Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper studies the phase transition problem in the information bottleneck (IB) objective and derives a formula for IB phase transitions. Based on the theory developed in the paper, an algorithm is developed to find phase transition points. The interesting observation is phase transition can correspond to learning new class and this paper conjectures in IB for classification, the number of phase transitions is at most C-1, C the number of classes.  This observation deserves to be further explored and may be a key to a deeper understanding of neural networks. 

The theory developed connects the IB objective, dataset and the representation, and thorough proofs are given.  The experiments matches the theoretical findings. 

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors studied the phase transition in the information bottleneck, which is defined as the point where the IB loss landscape changes. The authors give a theorem showing the practical condition for IB phase transitions which is related to Fisher information. Then an algorithm finding IB phase transitions are proposed and applied to MNIST and CIFAR10 dataset.

The paper is well-organized. The problem formulation, definition, theorem, algorithm and application form a whole story about finding IB phase transition. However, I found that the motivation of this paper is not very clear. Why do we want to find the IB phase transition accurately by running such a complicated algorithm (Algorithm 1). Usually, the scalar \beta is considered as a hyperparameter. By tuning \beta, one can easily produce Figure 1 and see some phase transition points. One can also tune \beta to learn different classed as shown in Figure 3(b). So designing a complicated coordinate-descent based algorithm to find a parameter seems overkilling the problem.

Moreover, the mathematical part of this paper can be made more accurate. For example, in definition 1, the authors write "\eta << \beta" and "|\epsilon| << 1", where the quantities \eta and \epsilon are used many times in the proof. However,  "<<" is not a rigorous mathematical symbol. The authors should consider rewrite their proof by using limit.

Therefore, I think the motivation and the mathematical quality of this paper can be further improved before getting accepted.  
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper contributes theoretically to the information bottleneck (IB) principle. In particular, the author(s) provided theoretical reasoning on the phase transition phenomenon: when the beta parameter of IB varies, the generalization performance changes in a stepwise manner rather than continuously. The core result is given by theorem 1: the phase transition betas necessarily satisfy an equation, where the LHS is expressed in terms of an optimal perturbation of the encoding function X->Z.

Overall, the reviewer believes that this work is a solid contribution to the IB principle, and should be accepted. Remarkably, this work gives the first formulation on the phase transition of IB, which was an empirical observation. Furthermore, the author(s) give an algorithm to find the transition betas, which agrees with empirical studies on CIFAR and MNIST datasets.

The reasons for the weak acceptance instead of a strong acceptance are explained by the following weaknesses.

As a theoretical contribution, it is important to have good formulations and clear statements. The quality of the mathematical statements is not satisfactory and can be largely improved.

Definition 1: introduce the concept of relative perturbation first. As a definition, it must be self-contained. Therefore introduce the second-order variation first (don't point to the appendix).

Lemma 0.1: inf_r(z|x) \mathcal{G} \ge \beta, is equivalent to \forall r(z|x), \mathcal{G}\ge\beta, the latter is more elegant. What is \mathcal{G}, anyway?

Lemma 0.2:  State the condition on the scale of the $\Delta\theta$. The reviewer suspects that if $\Delta\theta$ is large enough, the statement can be violated.  This lemma feels like a digress. Why the Fisher information is needed for explaining the phase transition?

Theorem 1: the statement is fine. It is better to have all conditions/assumptions listed as this is the core theorem. To be more complete, the authors can discuss the number of roots after the theorem and what happens if there is no root. In general, there have to be a few remarks after theorem 1.

Definition 2: \rho_r(X,Y;Z) is defined with respect to what? For example, state "Given a joined distribution p(X,Y), the representation maximum correlation....". Use a different equivalent symbol for definition, such as ":=".

Theorem 2: Again, \forall{f(x,z)} is with respect to what? In the "if" part of the "if-then" statement, "\forall{f(x,z}}, it can be decomposed as .." is a false statement, change to "If f{x,z} can be decomposed"

In the abstract, avoid citations, and explain X, Y, Z, p(z|x), for completeness.

As the paper is over the recommended length,  the reviewer is asked to be harsher in the assessment.
review_title: Official Blind Review #3

Meta-review: This submission presents a theoretical study of phase transitions in IB: adjusting the IB parameter leads to step-wise behaviour of the prediction.  Quoting R3: “The core result is given by theorem 1: the phase transition betas necessarily satisfy an equation, where the LHS is expressed in terms of an optimal perturbation of the encoding function X->Z.”
This paper received a borderline review and two votes for weak accept.  The main comment for the borderline review was about the rigor of a proof and the use of << symbols.  The authors have updated the proof using limits as requested, addressing this primary concern.  On the balance, the paper makes a strong contribution to understanding an important learning setting and a contribution to theoretical understanding of the behavior of information bottleneck predictors.  
Decision: Accept (Poster)