Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper provides several surrogates for the Information Bottleneck (IB) and Deterministic Information Bottleneck (DIB) loss functions that are more friendly to optimization. For the decoder uncertainty part, the authors show that using Dropout and cross-entropy loss provides an unbiased estimator for the decoder cross-entropy which upperbounds the decoder uncertainty. For the regularization terms in IB/DIB, the authors inject noises to the latent features to lower-bound the conditional entropy of latent representations, and further proposes three types of surrogate objectives for the regularziation terms. Emprical results on CIFAR/ImageNette (a subset of ImageNet of 10 classes) show that the proposed surrogates yield similar behaviours in terms of adversarial robustness and information plane and the scalability of the proposed method.

Strengths of the paper:
- As this paper claims, this is the first work that proposes some surrogate of IB loss functions that can be easily optimized and thus be scaled to large models and datasets (CIFAR/ImageNette). Results on both datasets show similar behavior (adversarial robustness, two-phase information plane) to IB loss based optimization.
- The injection of random noises into the latent representation is interesting and able to enforce lower-bound on the conditional entropy of latent representations, which further induces some surrogates that are optimization-friendly.
- This paper is well-written, fully-prepared and contains a large amount of results that are of wide interests of researchers working in this topic.

I don't have specific criticisms for this paper.

review_title: A very good paper on Information Bottleneck

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper makes a theoretical contribution of three "surrogate objectives" for the information bottleneck principle, followed by empirical results on MNIST, CIFAR-10 and ImageNette (a subset of 10 easily classified classes from ImageNet). The objectives assume we add a single simple of zero-entropy noise to each sample of the output z of the stochastic encoder p(z|x), and then give estimators on an upper bound of the information bottleneck objective.

Evaluation:
Overall, this is a fine paper - the introduction is especially well-written and I appreciated the inclusion of all the information plane images of training trajectories. However, the contributions are not sufficiently novel for acceptance at ICLR; this paper is mostly a duplicate of prior work in this area.

I am not convinced that their theoretical contribution is novel - it seems to be a variant (or a specific case) of prior work on Conditional Entropy Bottleneck (CEB) given in Fischer 2020 (https://arxiv.org/pdf/2002.05379.pdf). Their initial insight (Proposition 1) that recognizes that the information bottleneck objective I(XZ) - beta * I(YZ) can be rewritten as H(Y|Z) + beta' * I(XZ|Y), is exactly the insight given in CEB. This paper bounds the I(XZ|Y) term by assuming Z has zero-mean Gaussian noise (which can be chosen such that it is also zero-entropy noise). In contrast, the CEB paper gives a variational bound on the rewritten objective, and when optimizing this bound you sample from the encoder and use the samples to parameterize a distribution (where a gaussian is the simplest choice of distribution). It seems like this paper is producing a special case of CEB for gaussian assumptions on that distribution family.

In addition, the empirical contributions are decidedly not novel. They claim to present "the first successful evaluation of IB objectives on CIFAR-10 and ImageNette", but prior work contains these evaluations: Fischer 2020 (linked above) contains CIFAR-10 results and Fischer and Alemi 2020 (https://arxiv.org/pdf/2002.05380.pdf) contains robustness results on CIFAR-10 and ImageNet, on larger ResNets than the experiments in this paper.
review_title: bounds on information bottleneck objectives

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview: The authors provide a detailed analysis of the information bottleneck principle to explain how neural networks train and generalise. Specifically, since multiple competing IB objectives exist, the authors develop universal surrogate objectives that are easier to optimise and apply these to several neural network architectures for imaging tasks.

Quality and Clarity: The paper is clearly  well written. I particularly like the use of colour to match corresponding terms in each of the objectives which makes it easy to pin-point which pieces correspond.

Significance: The IB principle is a very useful and relevant concept for specifically optimising models to retain only the relevant information wrt a particular context or prediction task. It has been applied in several contexts eg deep generative models for identifying novel molecule structures (Wieser et al 2020) or deducing sufficient adjustment sets for causal inference (Parbhoo et al 2020) as well as DNNs in general (Tisbhy and Zaslavksy, 2015). Since it relies entirely on information theoretic quantities, it is widely applicable across several domains. Analysing these information theoretic objectives in order to make sense of these models is very important.

Pros:  1) The work presents a very rigorous analysis and discussion of multiple competing IB objectives and discusses the implications of each of these.

2) The authors present tractable surrogate objectives that can make optimisation easier. Since these are defined entirely in terms of entropies as well, the work is applicable across various kinds of domains -- a key advantage of the classic IB too.

Cons:

1) I think the authors do a good job in theoretically motivating the particular surrogate objectives, but I would have liked to have seen some discussion as to why using a surrogate objective is sensible in the first place, versus say performing comparisons of deep IBs and VAEs. VAEs use maximum marginal likelihood as an objective. How does using the surrogate objective compare to maximum marginal likelihood? What are the implications of this for downstream application of the IB? Should IBs with surrogate objectives only be used for compression or also for prediction tasks like VAEs?


2) I would have also like to have seen empirically how the surrogate objectives can generalise across domains that are not images? Are these objectives robust in other applications such as EHR data or say the chemical/molecular domain?

3) I also think the generalisation plots are not the most intuitive to understand from first glance and require more parsing and explanation in the text.



review_title: Interesting paper but could use some clarifications

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of paper:
The authors review the information bottleneck (IB) in the context of deep learning. They discuss the obstacles to applying the IB (and a deterministic variant, the DIB) to modern datasets, review approaches to doing so, and introduce their own scalable approach. Their approach introduces practical surrogate objectives for the information regularizer term, and uses dropout as the source of stochasticity. They take advantage of the scalability of their method to train a ResNet with (D)IB on MNIST, CIFAR-10, and ImageNette and study adversarial robustness and evolution in the information plane.

Pros:
1) The surrogate objectives the authors introduce allow the application of IB without restricting the latent distribution to take on a form with analytic entropy (i.e. gaussian), as is the case the deep variational IB (VIB).
2) The authors are the first, to my knowledge, to scale the DIB from tabular settings (where it was developed) to modern function approximation settings using a clever zero-entropy noise trick (although this did come at the cost of diverging from the deterministic solutions that would be optimal).
3) The authors are the first, to my knowledge, to use dropout as the source of stochasticity that IB requires. This has the advantage of allowing the authors to use nearly arbitrary DNN architectures with (D)IB, as opposed to inserting an explicitly stochastic (gaussian) layer.
4) The paper functions as a good review, independent of the authors’ contributions. A common complaint when reading ML papers is that they don’t discuss related work enough, so this paper was refreshing.

Cons:
1) I struggled to disentangle the novel contributions of the authors from the work they were reviewing. The novel contributions to my knowledge are the first 3 pros above. On the other hand, optimizing decoder uncertainty is not, for example (e.g. it is done by VIB). The authors need to do a better job at highlighting their own contributions but also making clear what is not.
2) I don’t think the authors make a very compelling explicit case for what advantages their approach has over VIB (the main alternative). I do believe there are advantages (see pros 1-3 above), but they are scattered throughout the paper and not always made explicit. I think the authors need a dedicated subsection addressing this. This section should also highlight the disadvantages (looser bound maybe?).
3) Relatedly, why not include direct comparisons to VIB in the experiments? The authors seem to imply that VIB wouldn’t scale to the datasets they tackle, since the experiments in the VIB paper involve pretrained embeddings and smaller models. But a) the VIB paper was written 4 years ago and hardware/software has improved since then b) the model sizes are the same order of magnitude.

Other comments:
1) I thought the heavy use of color distracted more than it helped, though I appreciate the effort.
2) This paper (https://arxiv.org/abs/1712.09657) also attempted to scale DIB to non-tabular problems (although far from the scalability of DNNs). The authors also added noise, but in this case to the data rather than the latents. Different problem being solved, but possibly interesting connection for the authors.

UPDATE

Following the author's response and updated draft, I've raised my score from a 6 to a 7.

UPDATE 2

Following discussion among the reviewers and especially a summary of experimental results by Reviewer 3, I'm lowering score back to a 6.
review_title: Need to better highlight novel contributions and compare/contrast with VIB

Meta-review: We have a very well informed reviewer who strongly feels that this paper is insufficiently novel and significant further discussion on how the paper might be raised to a publishable level with more empirical results.  I will have to side with the more engaged reviewers who feel that the paper should be rejected.
Decision: Reject