Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Description:

The information bottleneck (IB) is an information theoretic principle for optimizing a mapping (encoding, e.g. clustering) of an input, to trade off two kinds of mutual information: minimize mutual information between the original input and the mapped version (to compress the input), and maximize mutual information between the mapped input and an output variable. It is related to a minimization of an (expected) Kullback-Leibler divergence betwen conditional distributions of an output variable.

In this paper, instead of the original IB, authors consider a previously presented dual problem of Felice and Ay, where the Kullback-Leibler divergence is minimized in the reverse direction: from the conditional distribution of output given encoding, p(y|hat x), to the conditional distribution given the original input, p(y|x).
The "dual problem" itself has a more complicated form than the original IB, authors claim this is "a good approximation" of the original bottleneck formulation, and aim to prove various "interesting properties" of it. 

- An iterative algorithm (Algorithm 1) similar to the original IB algorithm but with a few more steps is provided.

- A theorem about critical points where cardinality of the representation changes is given, similar to the IB critical points, and another theorem about difference of curves on an information plane between the IB and dual-IB solutions.

- Authors also show that if the true conditional distribution of outputs given inputs has an exponential-family form, the dual-IB decoder also has a form in the same family, which is said to reduce computational complexity of the algorithm.



Evaluation:

This is an entirely theory-based paper; although an algorithm is given, it is not instantiated for any concrete representation learning task, and no experiments at all are demonstrated.

Overall, I feel the motivation is not clear and strong enough. The abstract does not illustrate the importance of the mentioned "interesting properties" well enough for concrete tasks. Reading the paper, the clearest motivations seem to be improving computational complexity, and having a clearer connection to output prediction in cases where the predictor may be sub-optimal. However, authors do not quantify these well:

- The computational complexity improvement is not made clear (quantified) in a concrete IB optimization task: it seems it is only for exponential families, and even for them only affects one part of the algorithm, reducing its complexity from dim(X) to d; the impact of this is not tried out in any experiment.

- For output prediction, authors motivate that dual-IB could have a more direct connection e.g. "due to finite sample, in which it can be very different from the one obtained from the full distribution"). Authors further claim that the dual-IB formulation can "improve the generalization error when trained on small samples since the predicted label is the one used in practice". However, this is not tested at all: no prediction experiments, no quantification of generalization error, and no comparisons are done, thus the impact of the clearer connection to output prediction is not tested at all, and no clear theorems are given about it either.

The property that the algorithm "preserves exponential form of the original data distribution, if one exists" is interesting in principle, but it is unclear if any real data would anyway precisely have such a distribution; what happens if the data is not exactly in an exponential family?

In its current state the paper, although based on an interesting direction, in my opinion does not make a sufficient impact to be accepted to ICLR.

Other comments:

"Application to deep learning" mentioned in Section 1.5 is only a sincle sentence in the conclusions.

There have been some other suggested alternative IB formulations, for example the Deterministic IB of Strouse and Schwab (Neural Computation 2017) which also claim improved computational efficiency. How does the method compare to those?

Section 1.5 claims the algorithm "preserves the low dimensional sufficient statistics of the data": it is not clear what "preserves" means here, certainly it seems the decoder in Theorem 7 uses the same kinds of sufficient statistics as in the original data, but it is not clear that hat(x) would somehow preserve the same values of the sufficient statistics.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper introduces a variant of the Information Bottleneck (IB) framework, which consists in permuting the conditional probabilities of y given x and y given \hat{x} in a Kullback-Liebler divergence involved in the IB optimization criterion.

Interestingly, this change only results in changing an arithmetic mean into a geometric mean in the algorithmic resolution.

Good properties of the exponential families (existence of non-trivial minimal sufficient statistics) are preserved, and an analysis of the new critical points/information plane induced is carried out.

The paper is globally well written and clear, and the maths are rigorously introduced and treated. Two minor comments would concern the definition of the Mutual Information (MI), which could be recalled to help the unfamiliar reader and improve self-containedness, and the notation of the expected value (\langle \rangle_p), unusual in Machine Learning where \mathbb{E} is often preferred.

Another point that could be enhanced is the intuition behind the IB and dualIB criteria: a small comment on their meaning/relevance as well as the rationale/implication of the probabilities permutation would be valuable addition.

Exhibiting a link with variational autoencoders (VAEs), and expliciting the differences in a VAE framework between the two criteria could also represent an interesting parallel, especially for machine learning oriented readers.

One of the main drawback of the present paper is however the lack of convincing experiments. Graphs showing the good behavior of the introduced framework are ok, but the clear interest of using dualIB rather than IB could be emphasized more. In particular, it does not seem that the issues about IB raised in the abstract (curse of dimensionality, no closed form solution) are solved with dualIB. Furthermore, dualIB optimizes the MI of the predicted labels, which is claimed to be beneficial contrary to the MI on the actual labels. However, no empirical demonstration of this superiority is produced, which is a bit disappointing.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new "dual" variant of the Information Bottleneck framework. The IB framework has been the subject of many papers in past years, with a focus on understanding the inner workings of deep learning. The framework poses machine learning as optimizing an internal representation to tradeoff between retaining less information about the input features and retaining more information about the output label (prediction). The existing framework measures the retained information about the prediction via mutual information, which can be expressed as a KL divergence. The new dual framework reverses the arguments of this divergence.

The paper shows that this dual IB closely mirrors the original IB while having several additional nice mathematical properties. In particular,  it can be shown that for exponential families the dual IB representation retains the exponential form.

Overall, I think this paper adds a meaningful new perspective to the IB framework and the analysis appears to be thorough. As such, I support acceptance. 

The paper could be improved by giving more interpretation of the formal results and experiments - i.e. tying this framework back to the higher-level questions and explaining what the quantities mean.

Figures 1, 2 part a: The meaning of both axes is unclear. The horizontal axis beta is a lagrangian parameter with no meaning outside the framework. The vertical axis Pr[y=0|\hat x] has no semantics since the problem has not been defined. What is the reader meant to take away from these figures?

Equation 3, part i: The numerator should have a beta subscript. The meaning of the denominator is not clear (it is a normalizing constant).


review_title: Official Blind Review #3

Meta-review: Main content:

Blind review #1 summarizes it well:

This paper introduces a variant of the Information Bottleneck (IB) framework, which consists in permuting the conditional probabilities of y given x and y given \hat{x} in a Kullback-Liebler divergence involved in the IB optimization criterion.

Interestingly, this change only results in changing an arithmetic mean into a geometric mean in the algorithmic resolution.

Good properties of the exponential families (existence of non-trivial minimal sufficient statistics) are preserved, and an analysis of the new critical points/information plane induced is carried out.

--

Discussion:

The reviews generally agree on the elegant mathematical result, but are critical of the fact that the paper lacks any empirical component whatsoever.

--

Recommendation and justification:

The paper would be good for ICLR if it had any decent empirical component at all; it is a shame that none was presented as this does not seem very difficult.
Decision: Reject