Information Bottleneck: Exact Analysis of (Quantized) Neural Networks
Authors: Stephan Sloth Lorenzen
Christian Igel
Mads Nielsen
Venue: ICLR 2022 Poster (2022)
Keywords: information bottleneck
quantization
neural network

Abstract: The information bottleneck (IB) principle has been suggested as a way to analyze deep neural networks. The learning dynamics are studied by inspecting the mutual information (MI) between the hidden layers and the input and output. Notably, separate fitting and compression phases  during training have been reported. This led to some controversy including claims that the observations are not reproducible and strongly dependent on the type of activation function used as well as on the way the MI is estimated. Our study confirms that  different ways of binning  when computing the MI lead to qualitatively different results, either supporting or refusing IB conjectures.
To resolve the controversy, we study the IB principle in settings where MI is non-trivial and can be computed exactly. We monitor the dynamics of quantized neural networks, that is, we discretize the whole deep learning system so that no approximation is required when computing the MI. This allows us to quantify the information flow without measurement errors. 
In this setting, we observed a fitting phase for all layers and a compression phase for the output layer in all experiments; the compression in the hidden layers was dependent on the type of activation function. Our study shows that the initial IB results were not artifacts of binning when computing the MI. However, the critical claim that the compression phase may not be observed for some networks also holds true.

Published as a conference paper at ICLR 2022
INFORMATION BOTTLENECK: EXACT ANALYSIS OF (QUANTIZED) NEURAL NETWORKS
Stephan S. Lorenzen, Christian Igel & Mads Nielsen
Department of Computer Science University of Copenhagen
(lorenzen, igel, madsn] ) @di. ku. dk
ABSTRACT
The information bottleneck (IB) principle has been suggested as a way to analyze deep neural networks. The learning dynamics are studied by inspecting the mutual information (MI) between the hidden layers and the input and output. Notably, separate fitting and compression phases during training have been reported. This led to some controversy including claims that the observations and strongly dependent on the type of activation function used as well as on the way the MI is estimated. Our study confirms that different ways of binning when computing the MI lead to qualitatively different results, either supporting or refusing IB conjectures. To resolve the controversy, we study the IB principle in settings where MI is non-trivial and can be computed exactly. We monitor the dynamics of quantized neural networks, that is, we discretize the whole deep learning system so that approximation is required when computing the MI. This allows us to quantify the information flow without mcasurement errors. In this setting, we observed a fitting phase for all layers and compression phase for the output layer in all experiments; the compression in the hidden layers was dependent the type of activation function. Our study shows that the initial IB results were not artifacts of binning when computing the MI. However, the critical claim that the compression phase may be observed for some networks also holds true.
not reproducible
INTRODUCTION
Improving our theoretical understanding of why over- wer-parameterized deep neural networks generalize well is arguably one of main problems in current machine learning research (Poggio al., 2020). Tishby & Zaslavsky 2015) suggested to analyze deep neural networks based on their Information Bottleneck (IB) concept, which is built on measurements of mutual information (MI) between the activations of hidden layers and the input and target (Tishby al., 1999), for overview see Geiger (2020). Shwartz-Zii Ziv & Tishby (2017) empirically studied the IB principle applied to neural networks and made several qualitative observations about the training process; especially, they observed compression conjectured to be a reason for good generalization performance and has widely been considered in the literature (Abrol & Tanner 2020 Balda et al. [2018) 2019 Chelombiev et al. [2019] Cheng et al. 2019 Darlow
fitring phase and a compression phase. The latter Eiformationnthhooree
Storkey, 2000 Elad al.] [2019; Fang et al. [2018] Gabrié et al. al.[2019) Goldfeld et al. 20199
Jónsson et al. 2020; Kirsch et al. [2020; Tang Nguyen & Choi 2019 Noshad et al.) 2019 Schiemer
& Ye 2020 Shwartz-Ziv & Alemil 2020; Wickstrøm al. 2019 Yu al. 2020). The work and
conclusions by Shwartz-Ziv & (2017) (2017) received a lot of critique, with the generality of their claims being doubted: especially (aaxe al. (2018) argued that the results by SSwwartz-Ziv Tishby do not generalize to networks using a different activation function. Their critique was again refuted by the original authors with counter-claims about incorrect estimation of the MI, highlighting an issue with the approximation of MI in both studies.
Our goal is to verify the claims by Shwartz- Ziv & Tishby and the critique by Saxe et al. in a setting where the MI can be computed exactly. These studies consider neural networks as theoretical entities working with infinite precision, which makes computation of the information theoretic quantities problematic (for a detailed discussion we refer to Geiger 2020 also Section 3) Assuming

---

Published as a conference paper at ICLR 2022
continuous input distributions, deterministic network using any of the standard activation functions (e.g., RELU, TANH) can be shown to have infinite MI (Amjad & Geiger 2019). If an empirical input distribution defined by data set D is considered (as it is the case many of the previous studies), then handomly-initialln likely result in trivial measurements of MI in the sense that the MI is finite but always maximal, that is, equal to log [D] (Goldfeld et al. 2019; Amjad & Geiger 2019). In order to obtain non-trivial measurements of MI, real-valued activations are usually discretized by binning the values, throwing away information in the process. The resulting estimated MI can be shown to be highly dependent on this binning, we refer to Geiger (2000) for a detailed discussion. Instead of approximating the MI in this fashion, we take advantage of the fact that modern computers and thus neural networks are discrete in the sense that floating point value can typically take at most 232 different values. Because 32-bit precision networks may still be too precise to observe compression (i.e., information loss), we apply quantization to the neural network system to an extent that we can compute informative quantities, that is, we amplify the effect of the information loss due the discrete computations in the neural network. One may argue that we just moved the place where the discretization is applied. This is true, but leads to fundamental difference: previous studies applying the discretization post-hoc rely on the in general false assumption that the binned MI approximates the continuous MI well - and thus introduce measurement errors, which may occlude certain phenomena and/or lead artifactual observations. In contrast, our computations reflect the true information flow in network during training. Our study confirms that estimation of MI by binning may lead strong artifacts in IB analyses and shows that:
deterministic neural networks with invertible activation functions will most
Both fitting and compression phases occur in the output SOFTMAX layer.
For the hidden layers, the fitting phase occurs for both TANII and RELU activations.
When using TANH in the hidden layers, compression is only observed in the last hidden layer.
When using RELU,, we did not observe compression in the hidden layers.
Even when applying low precision quantization, more complex networks with many neurons in each layer are observed to be too expressive to exhibit compression, as no information is lost.
Our setting excludes that the MI approximation the reason for these different IB dynamics.
The next section introduces the IB concept with a focus on its application to neural networks including the critique and controversy well as related work. Section 3 discusses issues relating to the estimation of MI, and the idea behind our contribution. Section presents our experiments, results and discussion before conclude in Section 5
2 THE INFORMATION BOTTLENECK
Preliminaries. Given continuous random variable (ΣR.) X with density function p(æ: and sup- port X, the continuous entropy H (X) of X is a measure of the uncertainty associated with X and is given by H (X) - Ix p(x) log p(z)(x. Given two r.V. X and Y with density functions p(xx and q(y) and supports X' and V the mutual information I(;;) of X and Y is mea- sure of the mutual "knowledge" between the two variables. The symmetric ((; is given by p(a.y)
(X;Y)) Sy fx p(x, v) log p(=)m()) drdy. In many cases it is impossible to compute the con- tinuous entropy and MI for continuous r.v.s exactly, due limited samples or computational limits, or because it may not be finite (Geiger) 2020). Instead, we often estimate the quanti- ties by their discrete counterparts. When X is discrete r.V., we consider the Shannon entropy H (X) = Σ P(x) log P(x). Correspondingly, the mutual information (X;Y) of two discrete r.V.S X,Y, is given by (X;Y) En.y P(x, v) log P())(())
P(a.)) We have the following useful identity
for both the continuous and discrete MI:
I(x;Y) = ((X)-(X))) - HX()))
(I)
where H (X is the conditional entropy of X given Y.
2

---

Published as a conference paper at ICLR 2022
IB Definition. The IB method was proposed by Tishby al. (1999). It is an information theoretic framework for extracting relevant components of an input r.v. X with respect to an outpur r.v. Y. These relevant components are found by "squeezing" the information from X through bottleneck, in the form of an r.v. T. In other words, T is a compression of X. The idea generalizes rate distortion theory, in which we wish to compress X, obtaining T, such that (X; 7) is maximized subject to constraint on the expected distortion d(z, t) wrt. the joint distribution p(æ, t) (Tishby et al. 1999). In the IB framework, the distortion measure is replaced by the negative loss in MI between T'and the output Y, (T) Both IB and rate distortion are lossy compression schemes!
The data processing inequality (DPI) (Y; X) 2 I IY; T) holds for the IB; that is, the bottleneck r.V. cannot contain more information about the label than the input.
One drawback of the information bottleneck method is the dependence on the joint distribution, p(z, y), which is generally not known. |Shamir et al. (2100) addressed this issue and showed that the MI, the main ingredient in the method, can be estimated reliably with fewer samples than required for estimating the true joint distribution. As common in the IB literature, whenever we discuss the MI computed on a finite data set D, we assume that p(r, y) corresponds to the empirical distribution defined by D. which is true for the experiments in Section 4.1 In practice, the assumption has to be relaxed to the data being drawn i.i.d. from p(z, y). However, any uncertainty resulting from the finite sample estimation in the latter case is not considered in our discussions.
IB In Deep Learning, Tishby & Zaslavsky 20155) applied the IB concept to neural networks. They view the layers of deep neural network (DNN) as consecutive compressions of the input. They consider the Markov chain
Y - T, -
where Tj denotes the i'th hidden layer of the I--layer network and T ÿ denotes the output of the network. Again, the bottleneck must satisfy the DPI:
IY::x) ((): 2 I(Y:))
(..)))..1111 21 ():r))
(2)
XX;:) 2 (I 2 (x: 2 ... ....1((:)))
(3)
Estimating the MI of continuous variables is difficult (Alemi et al.
2017). ) evident from the many
different methods proposed (Kraskov et al., 2004 Kolchinsky & Tracey 2017] Noshad et al. 2019).
In the discrete case, IX; and I(T:) can be computed as
(x; = ((T))) =
(4)
(5)
following from 1 and using in 4 the assumption that T is deterministic function of X. However,
for deterministic neural networks the continuous entropies may not be finite (Goldfeld et al. 2019
Saxe et al. 2018 Amjad & Geiger 2019).
Shwartz-Ziv Tishby (017) estimate the MI via (4) and (5) by discretizing T and then computing the discrete entropy. They trained a network (shown in Figure La) a balanced synthetic data set consisting of 12-bit binary inputs and binary labels. The network was trained for a fixed number of epochs while training and test error were observed. For every epoch and every layer T, The discretization is done by binning of T: Given upper and lower bounds by by, and m E N, we let B: R + [m]] denote the binning operation, that maps E [br, bu] to the index of the corresponding bin from the set of m uniformly distributed bins in [b,, bu]. Overloading the notation, we apply B directly
vector in Idd in order to obtain the resulting vector in [m] of bin indices. Using discretized T = BT)), I T"; and I (T'; Y) are then computed directly by and (5) using estimates of P(T"). P(Y) and PYT over all samples D of X.
Shwartz-Ziv & Tishby used the TANH activation function for the hidden layers, with by = - b, =1 (b, for the output SOFTMAX layer) and m 30 bins. The estimated (X; T) and I (T; Y) are plotted in the information plane, providing a visual representation of the information flow in
For IB, finding the optimal representation T can be formulated as the minimization of the Lagrangian ((; - 58 (T;Y) Y) subject the Markov chain Y + X + T and / E R (Tishby & 7aslavsky 20155
3

---

Published as a conference paper at ICLR 2022
Comp Compersioo
+
T
0.8
72
I(7.Y)
0.6
Epochss
T
Pitting
T3
Ts
74
Input
Fitting
Tb
Hidden
0.2
Output
8 To
0.0
((X:
(a)
(b)
Figure (a) The network used in the standard setting, with the number of neurons in each fully connected layer. The output layer activation function is SOFTMAX MAX, while either TANH or RELU is used for the hidden layers. (b) An example of an information plane for a layer network, with the observed phases marked. Only the last three layers are distinguishable.
the network during training (see example in Figure Tbp. Based the obtained results, (Shwartz: Ziv & Tishby| (2017) make several observations; one notable observation is the occurrence of two phases: an empirical risk minimization phase and compression phase. The first phase, also referred to as fitring phase, is characterized by increasing I (T; Y) related to decreasing loss. The subsequent compression phase characterized by decreasing I (X;T). and it has been argued that this compression leads to better generalization.
Critique and Controversy. The work by TTisbyy & Zas Zaslavsky (2015) and (Shwartz-Ziv & Tishby (017)) has jump-started an increasing interest in the IB method for deep learning, with several papers investigating or extending their contributions, see the review by [Giger (2220). However, as mentioned, their work also received criticism. In particular, the compression phase as general phenomenon has been called into question. Saxe al. (2018) published a paper refuting several of the claims made by Shwartz-Ziv & Tishby 2017). They based their criticism on a replication of the experiment done by "Shwartz-Ziv & Tishby (017)) where they replaced the bounded TANH activation function with the unbounded RELU activation function. When discretizing, they used the maximum activation observed across all epochs for the upper binning bound b,. The article claims that the two phases observed by (Shwartz-Ziv & Tishby occurred because the activations computed using TaNh saturate close to the boundaries -1 and The claim is supported by experiments using RELU activations and m 100 bins, in which the two phases are not observed.
The critique paper by Saxe et al.) was published at ICLR 2018, but started a discussion already during the review process the previous year, when Shwartz-Ziv & Tishby defended their paper in the online discussion forum OpenReview.ne? posting response titled "Data falsifying the claims of this ICLR submission all together" (Saxe et al., (0017). The response specifically states that "The authors don't 't know how to estimate mutual information correctly", referring to Saxe et al. (2018), and goes on provide an example with network using RELU activations, which does indeed exhibit the two phases. In response. [aaxe et al. performed further experiments using different estimators for MI: state-of-the-ar non-parametric KDE approach (Kolchinsky & Tracey 2017) and k-NN based estimator (Kraskov et al. 2004). The authors still did not observe the two phases claimed by Shwartz-Ziv Tishby
Following the discussion on OpenReview.net, several other papers have also commented on the controversy surrounding the information bottleneck. Noshad et al. (2019) presented a new MI estimator EDGE, based on dependency graphs, and tested it on the specific counter example using
https:://penneeriev.neee
4

---

Published as a conference paper at ICLR 2022
RELU activations suggested by Saxe al. (2018) and they observed the two phases. Table I in the
review by Geiger (020) provides a nice overview over empirical IB studies and if the compression
phase was observed (Darlow & Storkey 2020; Jónsson et al. [2200] Kirsch et al., 20200] Noshad et al.
2019
Raj et al.
& Tishby (017) or not (Abrol & Tanner 2020; Balda et al. 2018
2019
Tang Nguyen & 2009 / [Shwartz-Ziv & Alemi 2020 Yu et al. (2020) or the results were
mixed (Chelombiev et al. Cheng et al. 2019 Elad et al. 2019 Fang et al. 2018 Gabrié et al.,
2019
Goldfeld al. 2019] Saxe al. 2018 Schiemer Ye 2020 Wickstrom et al. 2019). In
conclusion, an important part of the controversy surrounding the IB hinges on the estimation of the
information- theooretic quantities this issue has to be solved before researching the information flow.
Related Work. The effect of estimating MI by binning has been investigated before, again refer to Geieie (020) for a good overview and discussion. [Shwartz-Ziv Ziv & Alemi (2020) consider infinite ensembles of infinitely-wide networks, which renders MI computation feasible, but do not observe compression phase. [helombiev et al.] (2019) applies adaptive binning, which, while less prone to issues caused by having the "wrong" number of bins, is still an estimation and thus also suffers from the same problems. Goldfeld et al. (2019) explores IB analysis by use of stochastic neural networks which allow them show that the compression phase these noisy networks occurs due to clustering of the hidden representations. While theoretically interesting, the stochastic neural networks are still qualitatively different from deterministic ones and thus not directly applicable in practice, [Raj et al.) (2020) conducted an IB analysis of binary networks, where both the activations and weights can only be =1, which allows for exact computation of MI. The binary networks are significantly different from the networks used in the original studies, whereas applying the IB analysis to quantized versions of networks from these studies allows for a more direct comparison. At first glance, the work by Raj et al. (2020) could be viewed as taking our approach to the extreme. However, the computations and training dynamics of the binary networks are qualitatively different from our study and the original IB work. For example, the binary networks require modifications for gradient estimation (e.g.. Raj approximate sign, and swish sign).
et al. consider
3. ESTIMATING ENTROPY AND MUTUAL INFORMATION IN NEURAL NETWORKS
The IB principle is defined for continuous and discrete input and output, however continuous MI in neural networks is in general infeasible to compute, difficult to estimate, and conceptually problematic.
3.1 BINNING AND Low COMPRESSION IN PRECISE NETWORKS
Consider for moment deterministic neural network on a computer with infinite precision. Assuming a constant number of training and test patterns, constant number of update steps, invertible activation functions in the hidden layers and pairwise different initial weights (c.g., randomly initialized). Given any two different input patterns, two neurons will have different activations with probability one (Goldfeld et al. 2019 Geiger 2020). Thus, on any input, a hidden layer T will be unique, meaning that, when analysing data set D. all observed values of T will be equally likely, and the entropy H (T) will be trivial in the sense that it is maximum, that is, log [D].
However, when applying the binning T' BT)), information is dropped, as B is surjective and several different continuous states may map to the same discrete state; for m bins and dr neurons in T the number of different states of T' is mor_ and thus more inputs from D will map to the same state. As consequence, the estimated entropy is decreased. Furthermore, the rate of decrease follows the number of bins; the smaller the number of bins m, the smaller the estimated H (T') (as more values will map to the same state). The entropy H (T'), and therefore I (X; T') in deterministic networks, is upper bounded by dr log m.g. Thus, changing the number of bins used in the estimation changes the results. We also refer to Geiger (2020) for a more detailed discussion of this phenomenorf]
Computations in digital computers are not infinitely precise, a 32-bit floating point variable, for instance, can take at most 232 different values. In theory, this means that we can actually consider the states of a neural network discrete and compute the exact MI directly. In practice, we expect the precision to be high enough, that when using invertible activation functions the entropy of the
Note that Geiger contrast our terminology, uses quantizer or Q to denote post-hoe hinning.
5

---

Published as a conference paper at ICLR 2022
layers will be largely uninformative, almost information loss/compression will occur. However, using low precision quantized neural networks, we can compute the exact MI in the network without H (T) being trivial what regard the ideal setting to study IB conjectures.
3.2 EXACT MUTUAL INFORMATION IN QUANTIZED NETWORKS
Quantization of neural networks was originally proposed as a means to create more efficient networks, space-win wise and in terms of inference running time, for instance to make them available on mobile devices (Jacob et al. 2018: Hubara et al., 2017). The idea is to use low precision k-bit weights (typically excluding biases) and activations, in order to reduce storage and running time. For our use case, we are mainly interested in quantized activations, as they contain the information that is moving through the network.
The naive approach is to quantize after training (post-training quantization). In terms of estimating the entropy/MI, this corresponds to applying a binning with 2 bins to the non-quantized activations as done in the previous studies (Shwartz-Ziv & Tishby 2017 Saxe et al. 2018), as information available to the network at the time of training is dropped, and thus we expect it to show similar artifacts. Instead, we investigate networks trained using quantization aware training (QAT), in which activations are quantized on the fly during training (Jacob et al. 2018). Specifically, forward passes quantized while backward passes remain non-quantized. However, the IB and all MI computations in the quantized networks only refer to quantized quantities (i.e., forward pass computations) and thus measure the true information in the network at the considered training step. The actual quantization happens after each layer (after application of the activation function). During training, the algorithm keeps track of the minimum ((²") and maximum a activations in each layer T across the epoch, and the activation of each neuron in T is mapped one of 2 uniformly distributed values between (T) and 0a before being passed to the next layer (Jacob et al. 2018). Thus for each epoch, the neurons in a layer T take at most 2* different values, meaning at most 2 different states for T (because of the SOFTMAX activation function the effective number of states is 2k(dy-1) 1) for the final layer Y).
EXPERIMENTS
We conducted the exact bottleneck analysis in two different setups:
The setting of Shwartz-Ziv & Tishby (017)) and [aaxe et al. (2018) using the network shown in Figure Ia with either TANH or RELU activations, fitted on the same synthetic data set consisting of [D] 21 12-bit binary input patterns with balanced binary output (Section 4.T..
Learning a two-dimensional representation for the MNIST handwritten digits, where a fully connected network similar to the one in the previous setting using a bottleneck architecture with RELU activations for all hidden layers is trained (Section 4.2).
For each setting, we applied k = 8 bit quantization aware training to the given network. As in previous studies (Shwartz-Ziv & Tishby 2017] Saxe et al.] 2018), we used an 80% training/test split to monitor fitting, while computing the MI based on the entire data set (training and test data). The networks were trained using mini batches of size 256 and the Adam optimizer with a learning rate of 10-4 Weights of layer T were initialized using truncated normal with mean 0 and standard deviation 1/vdr-
In addition, we also replicated the experiments of (Swwarzz-Ziv & Tishby (017) and Saxe al. (2018) using varying number of bins, and we investigated 32- and 4-bit quantization. We also considered IB analysis of few other architectures for the MNIST task, including architectures with wider bottlenecks, without a bottleneck, and using convolutions. These experiments are described in Supplementary Material A and
Our implementation is based on Tensorflow (Abadi et al. 2016). Experiments were run on an Intel Core 19-9900 CPU with 8 10GHZ cores and a NVIDIA Quadro P2200 GPU
Available at: ht http://////cccccoooooeeeeeeeeeea
IBAnal ysi n@NNS

---

Published as a conference paper at ICLR 2022
8,000
1.0
0.8
6,000
0.99
4,000
2,000
0.2
Y T
0.0
0.915
10
12
9
10
12
I(X:
I(X;T)
Figure 2: The quantized TANHI network, trained the standard setting with the discrete MI computed exactly. The full information plane (left) and the upper right area (right) are shown. Plotted MI values are the means over the 50 repetitions.
8,000
10
0.8
6,000
1(;Y)
0.6
1(::))
4,000
2,000
0.2
T3
T
74
Ta
0.0
0.8
T(X:T)
I(X:T)
Figure The quantized RELU network, trained in the standard setting with the discrete MI computed exactly. The full information plane (left) and the upper right area (right) are shown. Plotted MI values are the means over the 50 repetitions.
4.1 EXACT MUTUAL INFORMATION FOR SYNTHETIC DATA
The 8-bit quantized network from Figure Ta was fitted for 8000 epochs to the synthetic data considered in /Shwartz-Ziv & Tishby (201)) 7). using either T'NH or RELU for the hidden lavers. The experiment was repeated 50 times and the means of obtained measurements are reported
Figure 2 and Figure 3 show the information planes for the quantized TANH and RELU networks respectively; additional plots showing the variance of the repetitions are included in Supplementary Material B Both quantized networks obtained good fits with similar accuracies to non non-quantized networks; see Figure F.14 and Figure F.15 in the supplementary material.
For the quantized TANh network, the information plane is very similar to the plane obtained in the standard setting using 28 256 bins. We clearly observe fitting and compression phase in the output layer, while (X; ; 7') and I (T; Y)) in general are large for the input and hidden layers. Looking the hidden layer before the output layer, we observe a fitting and compression phase, but
'ee repeated any trial with an accuracy less than 0.55. This occurred in rare cases, where a RELU network got stuck with a 'dead" layer (all activations are 0 for all inputs) from which it could not recover; such network not of interest for the analysis. The risk of 'dead" layers would have been lower if we used different weight initialization scheme, however, did want change the setup from the original studies.
7

---

Published as a conference paper at ICLR 2022
3,000
Compression in 2D layir
3-
2,500
2.5
1(2.1)
1,500
1.5
1,000
500
0.5
[(X:2))
Figure 4: The information plane for the bottleneck structured network fitted on MNIST. Plotted MI values are the means over the 20 repetitions. The compression phase in the 2 neuron layer is highlighted.
with the fitting phase interrupted by small compression phase. For the remaining layers, the MI measurements with input and label remained almost constant throughout the training process.
Likewise, the information plane for the quantized RELU network looks very similar to the plane obtained with binning. The MI is slightly higher, but the most notably difference the occurrence of slight increase in (X; T) after approximately 1800 epochs, observed only for the hidden layer before the output layer. For the remaining hidden layers, only the fitting phase is observed, with no or very little compression; an observation mirrored in the binning experiments by |Saxe al. (0018)
4.2 EXACT MUTUAL INFORMATION FOR LEARNING AN MNIST REPRESENTATION
We considered the task of leaning two-dimensional representation of the MNIST handwritten digits (Bottou et al., 1994 Deng 2012). We fitted a network with four hidden layers with RELU activations and 16, 8, 4 and 2 neurons, respectively, where the last two neurons feed into 10 neuron SOFTMAX output layer. The network was trained for 3000 epochs, and the experiment was repeated 20 times.
The bottleneck architecture for learning a two-dimensional representation leads to a non-trivial IB analysis. Using more expressive networks (e.g., without or with a wider bottleneck) resulted trivial constant maximum MI measurements (see Supplementary Material E
Figure shows the information plane obtained for the network; additional plots showing the variance between repetitions included in Supplementary Material B The network exhibits sub-par perfor- mance due to the limiting bottleneck structure, obtaining a maximum accuracy of around 90% (see Figure in the supplementary material). However, the architecture does admit non-trivial analy- sis with a clear fitting phase the output layer and the three last hidden layers. Small compression phases are also observed in the RELU layers around 1500 epochs, similar to the last hidden layer of the network trained on synthetic data. The compression indicates that the learnt representations for digits from the same class become more similar.
4.3 DISCUSSION
Our experiments largely confirm the findings by Saxe et al. (2018): we observe no compression phase in the quantized REL.U network for the synthetic data. While the quantized networks do not directly correspond to the networks in the original studies (Shwartz-Ziv & Tishby 2017: Saxe et al. 2018), they amplify the effect of the networks being executed on a digital computer. Note that using quantization (as often done for mobile and edge computing well as when working on encrypted data) does not violate any assumption in the IB theory on the contrary, as argued before, it prevents the information theoretic measures from becoming trivial.
The MI generally being higher in our experiments confirms our discussion from Section 3 infor- mation well-preserved through the network. There is longer loss of information from the
8

---

Published as a conference paper at ICLR 2022
estimation method. All information loss and hence the dynamics observed - is result of the neural network learning. Performing the same experiments with the full 32-bit non-quantized networks, this is even pronounced, as almost no loss of information is observed between layers. This setting approaches the continuous case, in which we would get H()) = log [D] for invertible activation functions. We include information planes for 32-bit networks with exact MI computation in Supple- mentary Material [ Figure C.II Conversely, when training networks with even lower precision, we see higher information loss between layers. This can be expected from (T)) S 22dr and the small, decreasing number dy of neurons in layer T (see Figure lb). Using h = 4 bit precision, more layers are clearly distinguishable, with results similar to those obtained by estimating MI by binning; however, the 4-bit quantized networks also provide worse fit indicating that when estimating MI by binning, information vital to the network neglected in the analysis. We provide information planes for 4-bit quantized networks in Supplementary Material C Figure C.10
similar observation could be made for the network fitted on MNIST. Using the low complexity bottleneck architecture, obtained non-trivial information measurements, Having only two neurons in the smallest layer limits the network in terms of performance, yielding a sub-optimal aceuracy around 90%. Using more complex networks, for instance, with wider bottleneck or without bottle- neck, we obtained accuracies over 95% (see Figure F.16). but, even when using 8-bit quantization, the information planes were trivial, with only the last layer discernible (see Supplementary Material E.. We observed similar trivial information planes for convolutional architecture, further supporting that only very little information actually lost in the complex networks (even with quantization), making the information bottleneck, when computed exactly, a limited tool for analysing deep learning.
5 CONCLUSIONS AND LIMITATIONS
The experiments and discussion of SSwwrtz-ZZi & Tishby (2017) and Saxe et al. 2018), as well as several follow-up studies, indicate that estimating MI by binning may lead to strong artifacts. We empirically confirmed these issues (see Supplementary Material AA:: varying the number of bins in the discretization procedure, we observed large differences in the qualitative analysis of the same network, confirming that the results of previous studies highly depend on the estimation method.
To alleviate these issues, we computed the exact MI in quantized neural networks. In our experiments using 8-bit quantized networks, the fitting phase was observed for TANH and RELU network architectures. The compression phase was observed in the oulpui SOFTMAX layer in both cases. For the TANh network, the MI was only non-trivial for the last two layers, but not for the other hidden layers. We observed compression in the last hidden layer when using TANH activations; similar to the phase observed by Shwartz-Ziv & Tishby (2077). For the RELU network, the MI below its upper bound in all layers; however, no compression was observed in any of the hidden layers using RELU activations. The latter observation confirms the statements by Saxe al. (2018) about the IB in RELU networks.
Applying the analysis to network trained on MNIST, found that bottleneck architecture learning low-dimensional representation showed compression phase. The compression indicates that the learnt representations for digits from the same class become more similar, as intended for representation learning. When applied to networks without a bottleneck (c.g., a simple convolutional architecture), the IB analysis produced only trivial information planes, which shows the limitations of the IB analysis (even with exact computations) for deep learning in practice.
The main contribution of our work is that we eliminated measurements artifacts from experiments studying the IB in neural networks. By doing so, future research on the IB concept for deep learning can focus on discussing the utility of the principle for understanding learning dynamics instead of arguing about measurement errors.
The proposed quantization as such is no conceptual limitation, because for invertible activation functions studying (hypothetical) truly continuous system would lead to trivial results. The biggest limitation of our study is obviously that we could not (yet) resolve the controversy about the compression phase (which was, admittedly, our goal), because depends on the network architecture whether the phase is observed or not. However, the proposed framework now allows for accurate future studies of this and other IB phenomena.
experiments confirmed that it

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
Source code and instructions for reproducing our experiments are available online: https://
ACKNOWLEDGEMENTS
The authors would like to thank the anonymous reviewers for their constructive comments. SSL acknowledges funding by the Danish Ministry of Education and Science, Digital Pilot Hub and Skylab Digital. MN acknowledges support through grant NNF20SA0063 138 from Novo Nordisk Fonden. CI acknowledges support by the Villum Foundation through the project Deep Learning and Remote Sensing for Unlocking Global Ecosystem Resource Dynamics (DeReEco).. The authors acknowledge support by the Pioneer Centre for AI (The Danish National Research Foundation, grant Pl).
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et Tensorflow: system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI) 16, pp. 265-283, 2016.
Vinayak Abrol and Jared Tanner. national Conference Machine Learning (ICML): Workshop on Uncertainty and Robusiness in Deep Learning, 2020.
under mean field initialization. In Inter-
Alexander Amir Alemi, Ian S. Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv, 1612.00410, 2017.
Rana Ali Amjad and Bemhard C. Geiger. Learning representations for neural network- classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9))2225-2233) 2019.
Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar. An information theoretic view on learning of artificial neural networks. In International Conference Signal Processing and Communication Systems (ICSPSS), pp. 1-8. IEEE, 2018
Emilio Rafael Balda, Arash Behboodi, and Rudolf Mathar. On the trajectory of stochastic gradient http:://ppppneeeeeeeeeeeeeeeee/e
descent in the information plane, 2019. URL
SkMON20ctx
Lčon Bottou, Corinna Cortes, John Denker, Harris Drucker, Isabelle Guyon, L.D. Jackel, Yann LeCun, Urs A. Müller, Eduard Säckinger, Patrice Simard, and Vladimir Vapnik. Comparison of classifier methods: case study in handwritten digit recognition. In International Conference on Pattern Recognition (ICPR), pp. 77 72, 1994.
Ivan Chelombiev, Conor Houghton, and Cian O'Donnell. Adaptive estimators show information compression in deep neural networks. In International Conference on Learning Representations (ICLR), 2019.
Hao Cheng. Dongze Lian, Shenghua Gao, and Yanlin Geng. Utilizing information bottleneck to evaluate the capability of deep neural networks for image classification. Entropy, 21(5):456, 2019.
Luke Nicholas Darlow and Amos Storkey. What information does resnet compress? 2003.06254, 2020.
arXiv,
Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-142,, 2012.
Adar Elad, Doron Haviv, Yochai Blau, and Tomer Michaeli. Direct validation of the information bottleneck principle for deep nets. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 0-0, 2019.
10

---

Published as a conference paper at ICLR 2022
Hui Fang, Victoria Wang, and Motonori Yamaguchi. Dissecting deep learning networks detwors--visualiiing -isualiziig mutual information. Entropy, 20(11):823, 2018.
Marylou Gabrié, Andre Manoel, Clément Luneau, Jean Barbier, Nicolas Macris, Florent Krzakala, and Lenka Zdeborová. Entropy and mutual information in models of deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019.
Bernhard C. Geiger. On information plane analyses of neural network classifiers a review. arXiv, 2003.09671, 2020.
Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kings- bury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In International Conference on Machine Learning (ICML.), pp. 2299-2308, 2019.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, (166))666668888 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anleger-arithmetic-only inference. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Hlynur Jónsson, Giovanni Cherubini, and Evangelos Eleftheriou. Convergence behavior of dnns with mutual-informaaioo-biiile regularization. Entropy, 22(7):727, 2020.
Andreas Kirsch, Clare Lyle, and Yarin Gal. Scalable training with information bottleneck objec- tives. In International Conference on Machine Learning (ICML)): Workshop on Uncertainty and Robustness in Deep Learning, 2020.
Artemy Kolchinsky and Brendan Tracey. Estimating mixture entropy with pairwise distances. Entropy, 19, 2017. doi: 10.3390/e19000361
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 69:066138, 2004.
Morteza Noshad, Yu Zeng, and Alfred O Hero. Scalable mutual information estimation using dependence graphs. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2962-2966. IEEE, 2019.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceed- ings of the National Academy of Sciences, 117(48):30339-3044, 2020.
Vishnu Raj, Nancy Nayak, and Sheetal Kalyani. Understanding learning dynamics of binary neural networks via information bottleneck. arXiv, 2006.07522, 2020.
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, and Bren- dan D. Tracey. Discussion on openreview.net. htttss://operrevlew..ee ry WPG- A- 2017. Accessed January 2017.
Andrew M. Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, and Brendan D. Tracey. On the information bottleneck theory of deep learning. In International Conference on Learning Representations (ICLR), 2018.
Martin Schiemer and Juan Ye. Revisiting the information plane, 2020. URL https://
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, 41 (210)))666-2711, 2010.
Ravid Shwartz-Ziv and Alexander A Alemi. Information in infinite ensembles of infinitely-wide neural networks. In Symposium on Advances in Approximate Bayesian Inference, pp. 1-17, 2020.
Ravid Shwartz- Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv,, abs/1 703.00810 10, 2017.
11

---

Published as a conference paper at ICLR 2022
Thanh Tang Nguyen and Jaesik Choi. Markov information bottleneck to improve information flow in stochastic neural networks. Entropy, 21(1))977, 2019.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5, 2015.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. The 37'th Allerton Conference Commmication, Control and Computing, 1999.
Kristoffer Wickstrøm, Sigurd Løkse, Michael Kampffineyer Shujian Yu, Jose Principe, and Robert Jenssen. Information plane analysis of deep neural networks via matrix-bart based Renyi's entropy and tensor kernels. arXív, 1909. 1136,, 2019.
Shujian Yu, Kristoffer Wickstrøm, Robert Jenssen, and José C Príncipe. Understanding convolutional neural networks with information theory: An initial exploration. IEEE Transactions on Neural Networks and Learning Systems, 32:435-442, 2020.
12

---

Published as a conference paper at ICLR 2022
A ESTIMATING MUTUAL INFORMATION BY BINNING
This section presents a replication of the experiments from Shwartz-Ziiv Tishby (017)) and Saxe al. (2018), but varying the number of bins for estimating MI. Non-quantized TANHI and RELU networks were trained in the standard setting, and MI was estimated using 30, 100 and 256 bins. The number of bins were chosen to match the numbers used in the earlier works (30 used by Shwartzz & Tishby] 100 used by Saxe et al.) and to match the discretization in the quantized neural networks considered Section
30 bins
100
6,000
6,000
102:
0.4
0.2
0.10
((X;7))
Figure A.5: Information planes for the TANH network with various number of bins. 30 bins corresponds to replication of the results from Shwartz-Ziiv & Tishby (2017). The network is trained the standard setting, and the average runs is plotted.
30 bins
100 biss
256 blus
8,000
6,000
102223)
0.4
2,000
0.2
1(N:75)
T1X:T)
T[ 733
Figure A.6: Information planes for the RELUU network with various number of bins. 100 bins replicate the results from Saxe al. (2018). The network is trained in the standard setting, and the average of 50 runs is plotted.
The resulting information planes are shown for the TANH and RELU networks in Figure A.5 and Figure A.6 respectieely..
clearly see the results of Shwartz-Ziv & Tishby (2177) replicated using 30 bins for the TANH network; two phases are clearly visible in each layer. As expected, for every layer T' with binned 7" B(T), (X; and (I"; YY increase with the number of bins used with the phases still remaining visible in the distinguishable layers.
From Figure A.5
For the RELU network with 100 bins, the results of Saxe et al. (2018) were also replicated, that is, the compression phase was not observed. When using only 30 bins for the RELU network, the estimation broke down completely in the sense that the DPI (2) is violated: IT';Y) is non-monotone, a phenomenon occurring because the estimation, which has also been observed by Geiger (2020). Again, see for any layer 7' that (X; T") and (T'; Y') increase with m, although to lesser degree
13

---

Published as a conference paper at ICLR 2022
1,0
5,00
0,000
3(3:222
1(7,
0.6
4,000
6.4
044
68
7,000
1,000
6,000
2,000
4,00
6,000
0.0
10
Ejoes
Figure B.7: Mean and variance for ((;;) (left) and (T;Y) (center), and the information plane for the median deviating (L2-distance from the mean) repetition (right), for the TANH network in the standard setting.
3.0
12
0.8
6,000
105:70
1(2:3)
4,000
0.4
2,000
0.2
0.2
Figure B.8: Mean and variance for (X; (left) and I IT; (center), and the information plane for the median deviating (L2-distance from the mean) repetition (right), for the RELU network in the standard setting.
than for the TANH network, most likely due to the RELU activation functions actually dropping information (by zeroing negative activations).
In general. the estimation of MI in either network is highly dependable on the number of bins used, indicating that the interpretation the information plane not straight forward, when using binning for estimation.
B MI VARIANCE
In this work, we investigate the effect of MI approximation in previous IB studies. These original studies are based on a qualitative analysis of mean curves. However, the question arises of whether mean curves correctly represent individual learning processes. While we do investigate this question in detail, the following section provides plots illustrating the variance between experimental trials.
Figures B.7 B.8 and B.9 show the mean and variance of (X;7) (left) and (T; Y) (center), as well as the information plane a single trial (right), for the TANh and RELUU networks the standard setting, and the bottleneck network applied to MNIST, respectively. For the single trial, selected the median deviating repetition, ranked by the L2-distance from the mean.
From the plots, we see that when the mean MI is large, the variance is usually low. The variance is similar across the different networks (note the different scaling of the y-axes), with slightly larger variance for the RELU activations. Together with the median deviating information plane, the plots suggest that the mean information plane, which averages out less pronounced phases of MI increase and decrease, represents individual repetitions fairly well.
14

---

Published as a conference paper at ICLR 2022
3,00
15
2,000
1,000
7,000
3,000
1/000 2,000
[(K:T.)
Figure B.9: Mean and variance for () (left) and I (T:Y) (center), and the information plane for the median deviating (L2-distance from the mean) repetition (right), for the Bottleneck network applied to MNIST.
TANH
8,000
6,000
(0):))
0.6
1(T.Y)
0.6
4,000
0.4
0.1
0.2
0.2
0.0
0.0
(X::))
I(X:T))
Figure 1010: Information planes for 4-bit quantized TANH network (left) and RELU network (right).
The mean of 30 runs is plotted.
Additionally, the compression phase in the TANH network can clearly be seen in Figure B.7 (left), while it is absent in the RELU network (Figure B.8
C EFFECT OF THE QUANTIZATION PRECISION
This section presents information planes obtained using and 32-bit quantized neural networks trained in the standard setting, in order to investigate the effect of the precision used in the quantization.
Figure C.T0 1. depicts the resulting information planes for the 4-bit networks. As each neuron can take only 2 16 different values, the total number of possible states per layer decreases significantly in this setting, and as expected we see lower overall MI measurements. For the TANH network, several more layers become distinguishable. The observed information planes looks similar to those observed in the original experiments by (Swwartz-Ziv & Tishby (2017) and al. (2018). However, the network accuracy has now degraded compared the non-quantized networks (see Supplementary Material F which indicates that the binning used in the estimation of the MI previous experiments has discarded information vital the network.
Figure C.II shows the resulting information planes for the 32-bit networks. As expected, we see an overall increase in MI; the information drops only very slowly through the network. Each layer has many possible states and given the small data set we we get closer to the behavior of a continuous system.
15

---

Published as a conference paper at ICLR 2022
RELU
8,000
1(2)
1(T:
4,000
0.1
2,000
0.2
0.2
0.0
0.0
0
I(X:7)
I(X:T))
Figure C.11: Information planes for 32-bit quantized TANH network (left) and RELU network (right). The mean of 30 runs is plotted.
TANH
RELU
8,000
1.0
1.0
0.8
0.8
6,000
1(;
0.6
((;Y)
0.6
4,000
0.4
0.4
2,000
0.2
0.2
0.0
0.0
12
I(X:T)
I(XTT)
Figure D.12: The information planes for the TANH (left) and RELU (right) networks when prefitting on random labels for 1000 epochs. The means 20 repetitions are shown. In the plots, Y always refers to the true labels.
D QUANTIZED NETWORK WITH RANDOMIZED PREFITTING
This section presents results for the standard setting using 8-bit quantized networks, but with weights prefitted to random labels. First, the network is fitted for 1000 epochs to the synthetic data set with the labels shuffled. The network is then trained for 8000 epochs with correct labels (as done in Section 4.1 The experiment repeated 20 times. The resulting information planes for the TANH and the RELU network are presented Figure D.12 left and right respectively.
Inspecting the plots, random labels. This is not surprising as the input is not shuffled.
that, for all discernible layers, I TX; appears to increase when fitted to
As consequence of the increase in (X; T), the hidden layers (most notable in the RELU network) also see an increase in Y): As the ability to distinguish different inputs from a hidden layer T increases, does I (T;Y) Y).
As expected, the network accuracy does not increase when training on random labels, and accordingly there is no lateral movement in the information plane for the output layer.
16

---

Published as a conference paper at ICLR 2022
BOTTLENECK-1
HOURCLASS
3.5
3.5
3
2.5
2.5
1(7: Y)
1(7%:)
2
1.5
1.5
3,000
2,500
0.5
0.5
2,000
10
14
16
10
16
((X;T))
[[X; T.)
1,500
4x10
Conv
3.5
3.5
1,000
3
2.5
2.5
I(T:Y)
((EaY)
2
2
1.5
1.5
1
0.5
8
16
10
[(X;T)
I[X; (T)
Figure F,13: Information planes for the additional networks trained on MNIST. Note for HOURGLASS that the three last hidden layers almost coincide completely.
E MNIST WITH OTHER NETWORK ARCHITECTURES
This section presents the results of training number of 8-bit quantized networks with varying architectures on MNIST. We consider the same setup as Section 4.2] but with the following architectures (all using RELU activations in the hidden layers and a final 10 layer SOFTMAX output):
The BOTTLENECK-4 network with 4 hidden layers widths 16, 12, 8 and 4, i.e. the bottleneck has width 4.
The HOURGLASS network with 6 hidden layers of widths 16, 8, 4, 2, 4, 8. The bottleneck width still 2, but the network expands again after the bottleneck, creating an hourglass shape.
The 4x10 network with 4 hidden layers, each of width 10.
The CONV network, simple convolutional network with structure: CV-MP-CV-MP-FC. where CV denotes (3,3) 2-channel convolutional layer, MP denotes (2,2) max pooling and FC denotes fully connected RELU layer with 20 neurons.
Below, we denote the network used in Section 4.2 by BOTTLENECK-2 2. Each experiment is repeated 20 times and the resulting information planes are reported in Figure E.13 As expected, given the increased complexity, all networks exhibit better performance than the bottleneck structured network considered in Section 4.2 (the accuracy of all networks presented in Figure F.16 in Supplementary
17

---

Published as a conference paper at ICLR 2022
Material F), obtaining accuracies close or above 95% for all networks except the HOURGLASS network.
From Figure E.13 observe the information curves for the BOTTLENECK-4 network to be similar in shape to the BOTTLENECK-2 2, but with higher MI measurements in general. This not surprising, as the layers are wider and thus more expressive.
As expected, the HOURGLASS network also exhibits information curves similar to BOTTLENECK- 2 (as the architecture is similar for the first few layers). Not surprisingly, the two expanding layers of widths 4 and 8 almost coincide completely with the bottleneck layer; information cannot increase but also not decreased significantly.
The two more expressive networks without bottlenecks, 4x 10 and CONV, have almost completely trivial information curves. Limited fitting for the output layer in 4x and compression the output layers for both networks can be observed. The experiments indicate the limitations of exact IB analysis for complex, large scale networks.
F NETWORK ACCURACIES
Figure F.14 and Figure F.15 report the accuracies obtained by the TaNh and RELU networks when fitted on the synthetic data, respectively. As can be seen, the networks suffers only slightly when
bis
libs
32
Non-quattion
0.92
0.5
Toska
0,000
6,000
Ryoch
Epodh
Spock
Fpoch
Figure F.14: Accuracies of the quantized TANII networks (three on the left) and the non-quantized TANH network (right). The means and 95% confidence intervals over 50 repetitions are reported.
hix
Xon-gpostiod
0.95
0.5
Tiobs
3,000
Ploch
Fpoch
Spack
Tyech
Figure F.15: Accuracies of the quantized RELU networks (three on the left) and the non-quantized RELU network (right). The means and 95% confidence intervals over 50 repetitions are reported.
using only bits in the quantization.
Figure 116 reports the accuracy of the networks fitted to MNIST. Unsurprisingly, the more complex networks (BBTTLENECK-- 1x10, CONV) obtain better accuracy with lower variance, compared to the networks with a low-width bottleneck (BOTTLENECK-2 2, HOURGLASS).
18

---

Published as a conference paper at ICLR 2022
Dornossred
Fuet
0.3
1,000
2300
1.0g
2,000
3.000
2,00
1,000
2,00
1,00
3,09
Poach
Pawck
Paeck
Figure F.16: Accuracies of the quantized networks applied to MNIST. The means and 95% confidence intervals over 20 repetitions are reported. The decrease in training accuracy for the bottleneck structured network (left) is due to the optimization objective being cross-entropy.
19


Reviews:
[{"review_id": "vA97yk7ZVbD", "reviewer": "Reviewer_bFPZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I find the paper very well written and interesting. It is non-technical and purely empirical, but the numerical experiments are well-carried and insightful, on a difficult and timely subject. The authors DO mention that they did not provide an answer to the motivating question at the root of the work, namely, on whether compression phases happen in general in neural network training, because their experiments show that it depends on the setting and activation function used (it may also be that compression happens or not truly depending on the setting, and so there is not a definite answer). But it brings on the table a scheme to approach that question more systematically and cleanly. I believe that the paper is a very valuable contribution and will motivate other groups to take their approach. I do not see any obvious weakness other than there is no theory, but it is completely fine for a well-done empirical paper.", "summary_of_the_paper": "This paper considers the important problem of mutual information estimation in neural networks, a problem at the root of a debate on the usefulness of the information-bottleneck approach for the analysis of information flow in neural networks. There exist many approximation schemes to get estimates of the inter-layers mutual information, but the issue is that they may lead to different conclusions due to their sensitivity to discretization schemes. The authors propose instead to study discretized neural nets, trained with a simple learning procedure taking into account the discretization and that thus does not need post-training discretization. In this case mutual informations can be computed exactly and the data quantifies the true information flow along training.", "summary_of_the_review": "I recommend publication, as the paper is well-witten and clear, and brings about a new scheme and neural net model to compute exaclty one of the most fundamental quantity in their training: the mutual information.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "P33ylpjfFOr", "reviewer": "Reviewer_KkRc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality, Clarity, Soundness, Correctness**\n\nThe paper is very well written, and the necessary background including the main controversy and claims is well presented (perhaps a bit too detailed even regarding the history of the controversy). Methods and results are clear, and important ablations are included in the appendix. My main criticism is that the results qualitatively depend on the quantization bit-width chosen. I greatly appreciate that the authors do not “shove this under the carpet” but discuss it in the main paper and show it in the appendix. This will be valuable to anyone looking to implement the same main idea. However, in terms of take-aways from the paper the main claims and the controversy remain elusive. While it is undeniable that the paper analyzes training dynamics of activation-quantized networks exactly - it is unclear how these dynamics relate to the training dynamics of “unquantized” networks analyzed via binning. More importantly, there is no way of objectively choosing the right quantization width (which is perhaps similar to not being able to choose the right (non-uniform) binning). This could leave different people drawing different conclusions while using the same method (somewhat similar to the original controversy). Therefore, I would not call the results or method incorrect - but I also would not claim that this paper solves the controversy once and for all.\n\n\n**Verdict**\n\nI enjoyed reading this paper - it is well written, presents the background and history well and adds another piece to the continuing saga. The main idea is straightforward but good, and experiments and ablations are well executed. Unfortunately, the paper does not conclude the saga. My two main points of criticism are:\n1) The paper can only say something about training dynamics of activation-quantized networks - and while it is not far-fetched to assume that they relate well to analyzing training dynamics of “unquantized” networks analyzed via simple binning, this relationship is not well understood at all (in fact it is conceivable that the regularization effect of the strong quantization interacts with the compression effect of SGD).\n2) Perhaps the main worry is that the results qualitatively depend on the quantization bit-width chosen. This again leaves room for different interpretations of the findings which the method cannot objectively resolve.\nOverall I think there is merit to publishing this paper since it is well executed and might inform other people attempting to follow the same main idea. I am currently leaning slightly towards acceptance; my only reservation would be the somewhat limited significance of the paper (given the goals and the two main problems mentioned above). \n\nSideNote: Generally speaking I think it might be time to evaluate what information-plane analysis really tells us about training dynamics and how to use these findings. Perhaps the main merit is to provide inspiration for theoretical analysis of SGD dynamics, rather than empirically measuring the strength of a “compression phase”. Ideally these results would lead to a degree of understanding where we know which factors influence the fitting and compression phase, and how this influence plays out exactly - and perhaps most importantly how SGD can have a regularizing effect that helps with generalization. Some works have already set out towards this path. Another line of attack could be through a series of control experiments where information-quantities are known (and estimation methods can be calibrated/evaluated against), and where training-dynamics can be controlled (e.g. there should be no non-trivial compression phase when training with random labels; if we initialize with a solution that has “memorized” the entire training-set, do we see a greatly accelerated fitting phase? does compression have an immediate onset, how does it depend on hyper-parameters, network-capacity, activation functions, etc.?).\n\n**Improvements**\n\n1) Even though difficult, the paper would benefit strongly from an objective criterion on how to choose the quantization bit-width, or a “calibration” process to find such a value. I know this is a long shot - but to me this would be the most significant improvement of the current work.\n\n2) Though a uniform quantization is the most simple, is it well justified? Empirically activation distributions are far from being uniform between the min and max. How would a non-uniform quantization scheme change results?\n\n3) Control experiment (just a suggestion, I do not expect to see results in the authors’ response): train the network to fit random labels and use this as an initialization rather than random initialization of weights. What do the information dynamics look like when training (with correct labels) from this initialization? Do we see strongly accelerated fitting, do we get immediate compression? Do we see any compression in the networks/layers where we typically don’t see a compression phase?\n\n**Detailed comments**\n\n1) Would be nice to add one/two sentences on how gradients are backpropagated through the quantization functions (or are these simply bypassed for the backwards pass?).\n\n2) Fig 2, 3 - please indicate which layer is which in the plots.\n\n3) Maybe shorten/move the historical bit (openreview comments) to a footnote. I really like the very comprehensive list of references and their findings.\n\n4) Page 2 first para: maybe emphasize that IB/rate-distortion are lossy compression.\n\n5) Page 2 after first para: would be helpful for the reader to write down the IB objective.\n\n6) Page 2 second para: sounds a bit like the DPI needs to be carefully taken into account, but it cannot be violated. Maybe rephrase to: “The DPI of course holds in the IB, …”.\n\n 7) Page 6 second para: typo “quantized after training”.\n", "summary_of_the_paper": "**Update after authors' response:** The authors have managed to clarify some issues and make a couple of small improvements to the manuscript. I am tempted to raise my score to a 7, but to me personally the paper does not quite pass the threshold for an 8 (which is the next possible rating on the conference scale). I am in favor of accepting the paper and will argue so during the reviewers' discussion.\n\n**Summary**\n\nThe paper investigates the information-plane analysis of deep neural network training dynamics. The original claim was that neural networks go through distinct fitting and compression phases in SGD training that are well characterized by the information plane. These findings have later been disputed (particularly due to the way mutual information is estimated) which has lead to tens of papers investigating the phenomenon with mixed conclusions. The main idea of this paper is to eliminate the estimation problem by training neural networks with quantized activations, where the discrete mutual information can be computed exactly (in the limit of infinite samples). The original claims, controversy, and follow-up works are introduced and discussed in great detail. Original experiments are repeated with a very similar protocol to facilitate comparability of results. Additional experiments on MNIST are conducted, and ablations are performed. Results show the two distinct phases for some activation functions and some layers, but not for other activation functions.\n\n**Main contributions**\n\n1) Thorough discussion of the history of the Information-plane analysis, the main claims and previous findings and the follow-up papers it has spawned. Significance: The background and related work is well researched and well presented, which is very helpful for readers who have not followed this line of research closely. The only downside is that there is a fairly recent review (which is cited in the paper) which somewhat limits significance.\n\n2) Quantized-activation training to avoid estimation errors when computing mutual information terms. Significance: the idea is very sensible in principle. Unfortunately, as some of the ablations show, the qualitative results can depend strongly on the quantization bit-width chosen. This is unfortunate since the main goal was to eliminate the influence of binning in naive estimation of the mutual information. Nonetheless, the results are exact and reliable for training quantized-activation neural networks - what is not clear is how to choose the quantization.\n3) Reproduction of original experiments of Schwartz-Ziv & Tishby and Saxe et al. By sticking as closely as possible to the original training protocols, the new results are as comparable as possible (which does not mean that it is guaranteed that quantized-activation training dynamics are similar to “non-quantized” dynamics analyzed via binning - but empirically this seems to hold to a large degree). The paper also shows interesting ablations and additional experiments on MNIST. Significance: The most important experiments to run are included in the paper and important ablations are shown. The significance of the results could be improved by showing larger scale experiments on different kinds of networks (CNNs, ResNets, Transformers, …).\n", "summary_of_the_review": "The paper nicely summarizes some recent findings regarding information-plane analysis of neural network training, the dispute that followed the original results, and further follow-up work. To resolve some of the main issues (stemming from potential estimation errors of mutual information) the paper proposes to use quantized-activation networks instead, where the mutual information terms can be computed exactly. This idea is simple, yet compelling - unfortunately the problem of how to choose the correct quantization bit-width remains. Crucially, different bit-widths can lead to qualitatively quite different conclusions and there is currently no objective way of choosing the bit-width. The paper shows this via ablations which are discussed in the main text. Additionally, it is currently unclear how the training dynamics of (heavily) quantized networks relate to the training dynamics of “non-quantized” networks; after all; the activation-quantization could potentially have a regularizing effect that might interact with the compression induced by SGD alone. I think the paper is executed well, which is why I am currently in favor of sharing the results of this fairly straightforward idea with the wider community. I am not sure though whether this approach in general is a fruitful direction to target further research at in order to understand SGD training dynamics - which limits the potential impact of this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FfnsNnXhvKz", "reviewer": "Reviewer_vYX6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "UPDATE: Reading the revised manuscript, I am willing to improve my rating. The paper has merits and deserves to be published, even though I have still reservations regarding a few aspects. See reply to authors' response for details.\n\nThe paper is interesting, well written, and covers a relevant controversy (albeit in a niche). While previous works focused predominantly on deterministic networks with continuous activation functions (in which the estimation of information quantities is problematic), this work is among the first in which the information planes (IPs) can be interpreted from an information-theoretic perspective. Despite this merit, the technical contribution of this paper seems to be limited by the considerations below:\n\n- Methodologically, the paper is indeed similar to Raj et al. 2020. The authors claim that their approach is still different because Raj et al. 2020 use different training schemes (such as the straight-through estimator) and that thus the training trajectories will be fundamentally different. Does this not also affect the quantization-aware training that was used here? If so, what insights can be transferred to vanilla SGD?\n- The authors claim that the resulting mutual information values are \"true\", i.e., that the effects of mutual information estimation are entirely removed. This statements needs qualification. Indeed, if the empirical distribution of the dataset D is considered the true distribution P_XY, then this statement is correct. However, if we assume that there is a true distribution P_XY from which D is sampled iid (which is a more common assumption), then the computed mutual information values are indeed again only estimates, estimated from a finite dataset D. I suggest to at least make the former assumption explicit (and to defend it accordingly).\n- Furthermore, the authors present IPs by averaging over 50 training trajectories. This may be problematic if the phenomena are not epoch-aligned over these 50 runs. Indeed, averaging may explain certain \"dents\" in the IP that are observed in, e.g., Figs. 3, A.5., and C.8. For the sake of completeness, I suggest to show (maybe in the supplementary material) overlaid IPs of individual layers for all 50 runs to see if they fall on top of each other or not.\n- Regarding the generality of the paper's claims and connecting to the authors statement that the presence of a compression phase depends on the network architecture: The manuscript would be stronger if the authors add several more network architectures to investigate the effects, e.g., by varying the bottleneck size of the bottleneck architecture for MNIST, but also considering entirely different architectural choices. That the presence of compression depends on the architecture is not a problem -- the question how it depends on the architecture remains interesting and may deserve further research.\n- Finally, it is not fully clear what practical or theoretical implications the results of this paper have. The controversy about the existence of a compression phase is known. Several authors have hinted at a possible connection between such a compression in the IP and geometric effects in latent space (e.g., see Goldfeld et al. 2019 or Geiger 2020). Whether there is a connection between compression in the IP and generalization is unclear, but Saxe et al. 2017 and Goldfeld et al. 2019 suggest that there is not. How does the present manuscript complement these insights? \n- Also, there seems to be little connection between the learning curves in Appendix E and phenomena observed in the IPs. Specifically, while there seems to be compression for tanh activation functions but not for ReLU, the corresponding learning curves do not look qualitatively differently. What conclusions can we draw from that?", "summary_of_the_paper": "The paper discusses the information plane of quantized neural networks. The authors further investigate whether or not there are compression phases during training, a question over which there has been much controversy in the past.", "summary_of_the_review": "An interesting, although small, contribution to a recent controversy in understanding deep neural networks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, some reviewers have raised several points of criticism regarding certain aspects of the model whose novelty/significance was a bit unclear. After the rebuttal and the discussion phase, however, everyone agreed that most of these concerns could be addressed in a convincing way, and finally all reviewers were in favor of this paper. After carefully going over all the reviews, the rebuttal and the discussions, I fully agree with the reviewers and came to the conclusion that this paper indeed contains some interesting, novel and relevant contributions.
Decision: Accept (Poster)