Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper points out that the IB principle based on the Markov chain assumption ($Y \rightarrow X \rightarrow T_1 \rightarrow \cdots \rightarrow \hat{Y}$) cannot fully characterize the information flow in DNNs, and hence several claims previously made on the assumption may not be accurate. For better quantification of the mutual information, this paper proposes a probabilistic framework of IB in deep neural networks (DNNs) by taking into account the back-propagation training. Under the probabilistic framework, the paper derives two Markov chains that characterize the information flow in DNNs and provides experimental evidence for supporting the validity of the Markov chain assumptions. The paper claims that the probabilistic approaches allow us to more accurately estimate the mutual information than existing non-parametric approaches and experimentally demonstrates how different activation functions and hidden layers achieve IB trade-offs. 

main_review: This paper provides a novel probabilistic framework for understanding IB principles and estimating the mutual information in DNN. The probabilistic view provided by the authors is very interesting and convincing. The claims are well supported by theoretical and experimental evidence. The only concern is on the clarity. This is a dense paper with a lot of mathematical proofs and experimental results, so improved clarity will help readers to understand the idea and findings of this paper as well as its limitations. See details below.  

Major
- L63-66: Please specify clearly which claims are theoretically or experimentally supported in the introduction. Are those claims "different hidden layers achieve different IB trade-offs", and "MLP satisfies IB principle no matter what the architecture of the MLP is") supported by theoretical or experimental evidence?
- L81-87: It is not clear why this paragraph (generalizing CNN to MLP) is needed to explain that "the weights of a neuron can be viewed as a global feature, ... ."  If this is a rationale for the probabilistic representation, please consider re-write it.
- L129-130: Please provide a detailed rationale for Definition 2. I'm guessing this is because of the infinite mutual information problem. How can we use this definition without loss of generality? 

Minor
- L45: Please also provide the subsection number. (i.e. Appendix G -> Appendix G.3)
- L63: What is \overline{X}?
- L103: Please provide the definition of $f_n$. 
- L104: $n$th -> $n$-th
- L186, L228: Be consistent in referring equations (i.e. Use only one style between "Equation (2)" and "Equation 2" ) 
limitations_and_societal_impact: The authors didn't address the limitations of their work in the conclusion section. The scope of claims should be specified clearly.  
- Please specify the assumptions made for the probabilistic framework.
- Please specify claims that are only supported by evidence.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work introduces a novel probabilistic formulation for Deep Neural Network layers to circumvent the problem of estimating mutual information between continuous variables. The authors characterize two different directions for the information flow during training and inference time, empirically analyzing and validating their theory on a simple task.
main_review: The paper addresses the relevant problem of analyzing and characterizing the information flows in deep neural networks, by considering the Boltzman distribution on the activation of each layer and factorizing the dependencies in the backward training parameter update step.
Several parts of the paper are not entirely clear and not all the statements are strongly supported or sound. 

Here I summarize my main concerns:

1) The discrete variable $T_i$ for each layer is introduced as a function of the activations $\bf t_i$, but clearly the activations of layer $i+1$ depend on $\bf t_i$ rather than the categorical variable denoting the "maximum feature correlation" $T_i$, which does not contain the entirety of the information regarding the activations. The corresponding graphical model follows the following chain of dependencies:
$$\begin{matrix} X & \to  & {\bf t}_1  & \to  & {\bf t}_2 & \to  & \hat{Y} \\\\  &  & \downarrow & & \downarrow \\\\ & & T_1 & & T_2\end{matrix}$$
Theorem 1 and equation 8 factorize $p(\hat{Y},T_2,T_1|X)$ as $p(\hat{Y}|T_2)p(T_2|T_1) p(T_1|X)$ without considering the dependencies $X\to T_2$ and $X\to\hat{Y}$. Why are these dependencies not considered?

2) Since $T_i$ is defined based on ${\bf t}_i$, as a consequence of the data processing inequality, any evaluation of $I(X; T_i)$ represents only a lower bound of the actual measure of information $I(X;{\bf t}_i)$ flowing through the layer of interest.

3) The paper claims that $I(X; T_i)$ is generally unbounded in the continuous case, but provides estimators based on the empirical distribution $P(X=x) = \frac{1}{|\mathcal{D}|}[x\in\mathcal{D}]$, which is discete by definition. In this case ,the empirical entropy is finite $H(X)=\log|\mathcal{D}|\le \infty$ and so is the mutual information in analysis $I(X; T_i)\le H(X) = \log|\mathcal{D}|$.

4) The main text does not contain a clear definition of $\bar{X}$ and $f_n$ (Line 103 and 192). The term "dataset" is used to refer to either $\mathcal{D}$ or $X$ in different sections.

5) The paper is generally quite notation-heavy, which makes it difficult to read in some sections.

6) The experimental section in the main text is not extensive enough to be significant or convincing

Minor comments:

- Figure 1 is difficult to interpret and does not add much to the main text of the paper.
- Line 106: $w_n$ used instead of $\omega_n$
limitations_and_societal_impact: The paper does not clearly state the limitations of the proposed method.

the paper has a prevailing theoretical component, therefore the societal impact is not directly discussed.
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper presents a new probabilistic model for studying information flows in DNNs. To that end two Markov chains connecting the network's input, hidden layers, and output are introduced: (i) the forward chain models the conditional distribution of each layer given its input as a Gibbs distribution over the layer's weight vectors (the Hamiltonian is given by a non-linearity applied of dot products between weights and inputs); (ii) the backward chain, which is discussed much less, tries to capture the correlations introduces by BP. The authors show how to compute mutual information terms under their probabilistic model and provide some synthetic experiments under the proposed framework.
main_review: The general idea of formulating a probabilistic model under which information flows in DNNs may be studied (as opposed to the statistical route based on estimation from samples) is solid. However, I am afraid that the execution presented in this paper is sub par. For such a newly proposed model to be adopted the formulation must be described in a crystal clear manner and be well motivated by theory and empirics. Unfortunately, the paper falls short on both aspects, as is elaborated below. As a result, I cannot recommend acceptance. 

My main concerns (along with some smaller ones) are as follows:

1. The technical writing is sub par. The authors' choice to emphasize on the probability spaces corresponding to each layer does not make sense to me. They should just describe the random variables of interest, their distributions, supports, correlations, etc. Sample spaces and $\sigma$-algebras can be left implicit for several reasons. First, one can always find a common probability space over which all random variables are defined (e.g., any random variable with values in a polish space can be generated from a uniform (0,1) variable, so one only has to assume that this underlying space is 'rich enough' to support such a uniform variable). More importantly, the probability spaces $(\Omega_{T_i},\mathcal{F},P_{T_i})$ does not encode the entire probabilistic model since the latter is defined through conditional probabilities that are absent here.

2. Continuing from the above, the validity of the proposed model and its relation to the original IB formulation is unclear to me. The IB paradigm views $T_i$ as the vector of hidden representations (neuron values) corresponding to layer $i$. Here the authors think of $T_i$ as the weight vector whose correlation with the layer's input is highest. I can't see why this is a reasonable definition, and more importantly, its relation to the IB formulation is questionable.  This deviation from the IB principle is problematic given that the proposed model is motivated by it. 

3. The Markov chain formulation is also unclear. First, if $X \to Y \to Z$ forms a Markov chain then so does $Z\to Y\to X$, so the discussion of the chain's directionality should take this into account. Namely, it is more a question of which conditional distributions are specified and not of whether there is a Markovity relation in a given direction. Since the authors discuss both directions of the chain, do we know that the conditional distributions correspond (i.e., give rise to the same joint measure)? Say we have $Y \to X \to T_1\to \hat{Y}$ with law $P_{X,Y,T,\hat{Y}}=P_{X,Y}P_{T|X}P_{\hat{Y}|T}$ and $Y\to \hat{Y} \to T$ with law $Q_{Y,T,\hat{Y}}$. Do we know that, e.g., $P_{Y,T,\hat{Y}}=Q_{Y,T,\hat{Y}}$? If the authors mean that there are two phases of updates (forward and backward), then they should define more variables to reflect that. The fact that this is still unclear to me after reading the paper testifies that writing (and possible the model itself) must be improved

4. The paper lacks in depth as far as the provided claims (Theorem 1, Proposition 1, etc.). These are trivial consequences of the proposed model. Some claims seem somewhat exaggerated, e.g., the proof in Appendix D that $H(Y)=I(X;Y)$. There is nothing to prove here as, given $X=\mathbf{x}^j$, the authors just take $Y=\ell \mathbf{1}_{\{y^j=\ell\}}$. This trend of overcomplicating simple observations repeats throughout the entire text.

6. The authors say their model alleviates the infinite mutual information issue. While this is true, it is achieved by trivializing the problem. The authors implicitly assume that the data distribution is supported only over the samples $\{\mathbf{x}^j\}$, in which case clearly $I(X;Y)\leq H(X)<\infty$. There is no probabilistic model for $P_X$ outside the sample set. Consequently there is no sense in which the proposed model can account for generalization, which is a key aspect to statistical learning theory.

7. The authors wait too long before they describe the proposed model. Vague descriptions are provided from the abstract through the introduction, but nowhere in the intro is the model actually described (even in words). There is a paragraph on page 2 that talks about the probability spaces attached to each layer, but as mentioned above this is quite redundant and doesn't clarify the probabilistic model. The model is described first only after Def 1 on page 3, which is too late. Mention of the Gibbs measure, its Hamiltonian, the induced Markov chains, etc. should be provided earlier in the text to improve readability. 

8. The paper is tedious to read. The technical descriptions are swamped in verbal and heuristic statements, making it hard to extract the message being conveyed. Because of that I had to reread many parts several times. The authors should clean up and streamline the text. This would make it easier to understand the proposed model and appreciate its potential validity. I am not providing specific examples since there are just too many---the entire text needs work.

9. In Equation (10), do the authors mean conditional KL divergence (i.e., given $X=x$) or averaged one over the samples? The dependence of the KL term on the weights should be made explicit. 

10. The proof outline for Theorem 2 is confusing. It uses undefined notation such as $\frac{\partial\ell_{\mathrm{CE}}}{\partial\omega}$ and does little to explain where the statement comes from. The authors should either shape it into a standalone argument (statements such as `it can be shown' should be shown or at least explain further) or omit it from the main text. 

11. I can't see how the experiments validate the applicability of the proposed theory. The authors compute the proposed objects but whether these in fact describe real world DNNs remains unclear. 
limitations_and_societal_impact: Not applicable.
review_title: null

Meta-review: The direction suggested by this paper is potentially interesting, but there was substantial agreement (not consensus) among the reviewers that this paper is not ready for publication. Overall, the reviewers felt (even ater substantial author-reviewer discussion) that the paper is hard to understand, the overall experiments are unclear, and the experiments are not compelling.
Decision: Reject