Information Plane Analysis of Deep Neural Networks via Matrix--Based Renyi's Entropy and Tensor Kernels
Authors: Kristoffer Wickstrøm
Sigurd Løkse
Michael Kampffmeyer
Shujian Yu
Jose Principe
Robert Jenssen
Venue:  (2020)
Keywords: information plane
information theory
deep neural networks
entropy
mutual information
tensor kernels

Abstract: Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability. However, it is by no means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, to construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers. MI estimators should also be able to naturally handle convolutional layers, while at the same time being computationally tractable to scale to large networks. None of the existing IP methods to date have been able to study truly deep Convolutional Neural Networks (CNNs), such as the e.g.\ VGG-16. In this paper, we propose an IP analysis using the new matrix--based R\'enyi's entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimensionality of the data. The obtained results shed new light on the previous literature concerning small-scale DNNs, however using a completely new approach. Importantly, the new framework enables us to provide the first comprehensive IP analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large-scale neural networks.

Under review as a conference paper at ICLR 2020
INFORMATION PLANE ANALYSIS OF DEEP NEURAL NETWORKS VIA MATRIX-BASED RÉNYI'S ENTROPY AND TENSOR KERNELS
Anonymous authors Paper under double-blind review
ABSTRACT
Analyzing deep neural networks (DNNs) via information plane (IP) theory has gained tremendous attention recently as a tool to gain insight into, among others, their generalization ability. However, it is by means obvious how to estimate mutual information (MI) between each hidden layer and the input/desired output, construct the IP. For instance, hidden layers with many neurons require MI estimators with robustness towards the high dimensionality associated with such layers. MI estimators should also be able naturally handle convolutional layers. while at the same time being computationally tractable to scale to large networks. None of the existing IP methods date have been able study truly deep Con- volutional Neural Networks (CNNs), such as the e.g. VGG-16. In this paper, we propose an analysis using the new matrix-based Rényi's s entropy coupled with tensor kernels over convolutional layers, leveraging the power of kernel methods to represent properties of the probability distribution independently of the dimen- sionality of the data. The obtained results shed new light the previous literature concerning scall-scale DNNs, however using a completely new approach. Im- portantly, the analysis of contemporary large-scale DNNs and CNNs, investigating the different training phases and providing new insights into the training dynamics of large- scale neural networks.
framework enables
provide the first comprehensive IP
INTRODUCTION
Although Deep Neural Networks (DNNs) are at the core of most state the art systems com- puter vision, the theoretical understanding of such networks is still not at satisfactory level (Shwartz-Ziv & Tishby (017). In order to provide insight into the inner workings of DNNs, the prospect of utilizing the Mutual Information (MI), measure of dependency between two random variables, has recently garnered a significant amount of attention (Cheng et al.) 2018 Noshad et al.)
2019 Saxe et al. 2018, Shwartz-Ziv & Tishby| 2017: Yu et al. 2018 Yu & Principe 2019). Given the input variable and the desired output Y for a supervised learning task, a DNN is viewed as transformation of X into a representation that is favorable for obtaining good prediction of Y. By treating the output of each hidden layer as a random variable T', one can model the MI (X; between X and T. Likewise, the MI [(T; Y) between T' and Y can be modeled. The quanti- ties T' and I (T; Y) span what is referred to as the Information Plane (IP). Several works have demonstrated that one may unveil interesting properties of the training dynamics by analyzing DNNs in the form of the IP (Yu & Principe 2019; Goldfeld et al. 2019:) Noshad et al. 2019 Chelombiev al. 2019). Figure 0 produced using our proposed estimator, illustrates one such insight that is similar to the observations of Shwwatzz-22 Ziv & Tishby (2017), where training can be separated into two distinct phases, the fitting phase and the compression phase. This claim has been highly debated subsequent research has linked the compression phase saturation of neurons (Saxe al. 2018) or clustering of the hidden representations (Goldfeld et al., 2019).
Contributions We propose novel approach for estimating MI, wherein kernel tensor-based estimator of Rényi's entropy allows us to provide the first analysis of large-scale DNNs as com- monly found in otate-of-the-ar methods. We further highlight that the multivariate matrix-based

---

Under review as a conference paper at ICLR 2020
approach, proposed by Yu al. (2019), be viewed special of approach. How- ever, our proposed method alleviates numerical instabilities associated with the multivariate matrix- based approach, which enables estimation of entropy for high-dimensional multivariate data. Fur- ther, using the proposed estimator, we investigate the claim of Cheng et al. (2018) that the entropy MI-based analysis
(X)) T(T; X) and H(Y) R T(T:Y) Y) in high dimensions (in which would be meaningless) and illustrate that this does not hold for our estimator. Finally, our results indicate that the compression phase is apparent mostly for the training data, particularly for more challenging datasets. By utilizing a technique such as early-stopping, a common technique to avoid overfitting, training tends to stop before the compression phase occurs (see Figure This may indicate that the compression phase is linked to the overfitting phenomena.
Iterations
Layer 1
3.0
Layer 2
5000
Layer 3
Layer 4
2.5
Layer 5
4000
P
2.0
P
I(:)))
P2tience=12
3000
1.5
2000
1.0
0.5
1000
((X:T)
Figure 1: IP obtained using our proposed estimator for a small DNN averaged over training runs. The solid black line illustrates the fitting phase while the dotted black line illustrates the compres- sion phase. The iterations at which early stopping would be performed assuming a given patience parameter are highlighted. Here, patience denotes the number of iterations that need pass without progress on validation set before training is stopped to avoid overfitting. It can be observed that for low patience values, training will stop before the compression phase. For the benefit of the reader, the bottom right corner displays a magnified version of the first four layers.
2 RELATED WORK
Analyzing DNNs in the IP was first proposed by Tishby & Zaslavsky (0155) and later demonstrated by (Shwartz-Ziv & Tishby (2017). Among other results, the authors studied the evolution of the IP during the training process of DNNs and noted that the process was composed of two different phases. First, initial fitting phase where I(T; Y) increases, followed where T(X;T) decreases. These results were later questioned by Saxe et al. (2018), who argued that the compression phase is not a general property of the DNN training process, but rather effect of different activation functions. However, a recent study by Noshad et al. (211) seems to support the claim of compression phase, regardless of the activation function. The authors argue that the base estimator of MI utilized in [Saxe et al.] (2018) might not be accurate enough and demonstrate that a compression phase does occur, but the amount of compression can vary between different activation functions. Another recent study by (Chelombiev et al. 2019) also reported compression phase, but highlighted the importance of adaptive MI estimators. They also showed that when re-reuulaiizttion
phase of compression
included in the training compression function. Also, some recent studies have discussed the limitations of the IP framework for analysis and optimization for particular types of DNN (Kolchinsky al. (2019): Amjad & Geiger (2019))
observed, regardless of activation
2

---

Under review as a conference paper at ICLR 2020
On a different note, Cheng et al. (2018) proposed an evaluation framework for DNNs based on the IP and demonstrated that MI can be used to infer the capability of DNNs to recogmize objects for an image classification task. Furthermore, the authors argue that when the number of neurons in hidden layer grows large, I(T;X) and T(Y; T) barely change and are, using Cheng et al. (2018) terminology, approximately deterministic, i.e. T(T; X) (()) and [ Y) s H(Y) Therefore, they only model the MI between X and the last hidden layer, that is the output of the network, and the last hidden layer and Y.
Yu & Principe 2019) initially proposed to utilize empirical estimators for Rényi's MI for investigat- ing different data processing inequalities in stacked autoencoders (SAEs). They also claimed that the compression phase in the IP of SAEs is determined by the values of the SAE bottleneck layer size and the intrinsic dimensionality of the given data. Yu et al. (2019) also extended the empirical estimators for Rényi's MI to the multivariate scenario and applied the new estimator to simple but realistic Convolutional Neural Networks (CNNs) (Yu et al. 2018). However, the results so far suffer from high computational burden and are hard to generalize to deep and large-scale CNNs.
3 MATRIX-BASE MUTUAL INFORMATION
Here, we review Rényi's a-order entropy and its multivariate extension proposed by Yu et al. 2019)
3.1 MATRIX-BASHD RÉNYI'S ALPHIA-ORDER ENTROPY
Rényi's a-order entropy is generalization of Shannon's 's entropy (Shannon 1948 Renyi 1961). For random variable X with probability density function (PDF) f(x with support X, Rényi's a-order entropy is defined as:
1
Ha(f)
log far (x))xx
Equation || has been widely applied in machine learning (Principe 2010), and the particular case of 2. combined with Parzen window density estimation (Parzen 1962), form the basis for Information Theoretic Learning (Principe 2010). However, accurately estimating PDFs in high-
dimensional data, which is typically the case for DNNs, is challenging task.
To avoid the problem of high-dimensional PDF estimation, Giraldo et al. (2012) proposed a non- parametric framework for estimating entropy directly from data without resorting to kernel density estimation:
Definition 3.1. (Giraldo et al. 2012) Let x; E X, -
denote data points and let X' x X IR be an infinitely divisible positive definite kernel (Bhatia 2006).. Given the kernel
matrix K E N with elements (K)ij = x(x,,,,) and the matrix A, (A)is = N (K)a(
1 (K);;
the masrix-based Rényi's o-drder entropy is given by
N
1 loga (Ir(A))=
1
(A) 1
log2
Σ (A)A
Here, X, denotes the jth eigenvalue of the matrix A. The properties of this quantity was analysed in detail in Giraldo et al. (2012). It was shown that the kernel matrix A, obtained from the raw data, acts much density matrix in quantum information theory (Nielsen & Chuang 2011 T). It was further shown that the (Gram) matrix A is related to an empirical covariance operator on embed- dings of probability distributions in Reproducing Kernel Hilbert Space (RKHS). This is similar to the approach of maximum mean discrepancy and the kernel mean embedding (Gretton et al. 2012] Muandet et al. 2017). Moreover, Giraldo al. (2012) showed that under certain conditions Equa- converges to the trace of the underlying covariance operator, as shown in Proposition B. in tion 2
Appendix B Notice that the dimensionality of the data does not appear in Proposition B. This means that S. (A) captures properties of the distribution, with certain robustness with respect to high-dimensions data. This a beneficial property compared to KNN and KDE based information estimators used in previous works (Saxe al. 2018 Chelombiev et al. 2019), which have difficul-
ties handling siggh-dmensional data (Kwak& & Chong- Ho Choil 2002). Not all estimators entropy have the same property (Paninski 2003). Certain approaches developed for estimating the Shannon
3

---

Under review as a conference paper at ICLR 2020
entropy suffer from the curse of dimensionality (Kwak & Chong Ho Choi 2002) Also, there is no
need for any binning procedure utilized in previous works (Shwartz- Ziv Tishby 2017). which are known to struggle with the relu activation function commonly used in DNN (Saxe et al. 2018). In
Appendix A we have conducted experiments on synthetic data to illustrate the behaviours of these estimators for high-dimensional data.
Remark 1. In the limit when a - 1, Equation reduces to the matrix-based von Neumann entropy (Nielsen Chuang 201T) that resembles Shannon's definition over probability states, and can be expressed as
N
lim (A( - Σ a-1
(3)
For completeness, the proof of Equation 3 can be found in Appendix C
In addition to the definition of matrix based entropy, Giraldo et al. (2012) define the joint entropy between x E X' and y E y
A Ay tr(A, o Ay) Ay)
Sa (Ax, Ay) Sa
where x; and y, are two different representations the same object and denotes the Hadamard product. Finally, the MI is, similar to Shannon's formulation, defined
Ay(Ax:Ay) Sa(Ax) + Sa(Ay) - (Ax,
(5)
3.2 MULTIVARIATE MATRIX-BASED RÉNYT'S ALPHIA-ENRROP FUNCTIONALS
The matrix matrix-based Rényi's a-order entropy functional is not suitable for estimating the amount of information of the features produced by convolutional layer in DNN as the output consists of C' feature maps, each represented by their own matrix, that characterize different properties of the same sample. Yu et al. (2019) proposed multivariate extension of the matrix-based Rényi's 's o-order entropy. which computes the joint-entrops among C variables as
A.....oA.
SA
Ac) = Sa A o...
(6)
(Ac)ij = (x (0))) Yu et al. 2018) also demonstrated how Equation [ could utilized for analyzing synergy and redundancy of convolutional layers in DNN, but noted that this formulation
where (Ai)i K1 (x. (1
encounter difficulties when the number of feature maps increases, such as in more complex CNNs. Difficulties arise due to the Hadamard products in Equation 6 given that each element of Ac,
(1,2,.... C). takes on value between 0 and 1 and the product C' such elements thus tends towards as C grows. Yu et al. (2018) reported such challenges when attempting model the IP of the VGG16 (Simonyan & Zisserman 2015). We have illustrated how this numerical instability manifests itself in Appendix
4 TENSOR-BASER MUTUAL INFORMATION
To invoke information theoretic quantities of features produced by convolutional layers and address the limitations discussed above, we introduce our rensor-based approach for utilizing entropy and MI DNNs, and show that the multivariate approach in section 3.2 arises as a special case.
4.1 TENSOR KERNELS FOR MUTUAL INFORMATION ESTIMATION
The output of convolutional layer is represented as a tensor X, BC @ R & RW for data point As discussed above, the matrix- Rényi's a-entropy can not include tensor data without modifications. To handle the tensor based nature of convolutional layers we propose utilize tensor kernels (Signoretto et al. 2011)) to produce a kernel matrix for the output of a convolutional layer. tensor formulation of the radial basis function (RBF) kernel can be stated as
Kren (% X,))
- [|x 10,xx,,10 x,
4

---

Under review as a conference paper at ICLR 2020
where - denotes the Hilbert-Frobeninn norm Siigoretto et al. 2011) and is the kernel width parameter. In practice, the tensor kernel in Equation 7 can be computed by reshaping the tensor into vectorized representation while replacing the Hilbert-Frobenius norm with Euclidean norm. We compute the MI in Equation b by replacing the matrix A with
(Aten)is = N (Kien) di (K
(8)
N Kton ( Xy).
While Equation 7 provides the simplest and most intuitive approach for using kernels with tensor data, it does have its limitations. Namely, tensor kernel that simply vectorizes the tensor ignores the inter-component structures within and between the respective tensor (Signoretto et al., 2011). For simple tensor data such structures might not be present and tensor kernel as described above tensor
suffice, however, other tensor kernels do exist, such as for instance the matricization- kernels Signoretto et al. (20IT). In this work we have chosen the tensor kernel defined in Equation 7 for its simplicity and computational benefits, which come from the fact that the entropy and joint entropy computed batch-wise by finding the eigenvalues of kernel matrix, the eigenvalues of the Hadamard product of two kernel matrices, and utilizing Equation 2 Nevertheless, exploring structure preserving kernels can be an interesting research path in future works. In Appendix E we have included simple example towards this direction, where the tensor kernels described in this paper is compared to a matricization-bant based tensor kernel. Note that the multivariate approach described in Section 3.2 can be regarded special case of our proposed method given under certain assumptions and a proof provided Appendix D
4.2 CHOOSING THE KERNEL WIDTH
With methods involving RBF kernels, the choice of the kernel width parameter, dr, is always critical. For supervised learning problems, one might choose this parameter by cross-validation based on validation accuracy, while unsupervised problems one might use rule of thumb (Shi & Malik 2000 (hi et al. 22099; Silverman 1986). However, in the case of estimating MI in DNNs. the data often high dimensional, in which case unsupervised rules of thumb often fail (Shi et al. 2009).
In this work, we choose a based on an optimality criterion. Intuitively, one can make the following observation: good kernel matrix should reveal the class structures present in the data. This can be accomplished by maximizing the so called kernel alignment loss (Cristianini et al., 2002) between the kernel matrix of given layer, K,, and the label kernel matrix, Ky The kernel alignment loss is defined as
(Ka, KO)F
A(K 0: K,) K |r
KF
where ||-||e and ( denotes the Frobenius norm and inner product, respectively. Thus, we choose our optimal a as
o² arg max, A A(K,, Ky).
To stabilize the a values across mini batches, we employ an exponential moving average, such that in layer f at iteration have
T = 80 + +(1 3)0
where 3 E [0, and σ6, = of
5 EXPERIMENTAL RESULTS
We evaluate our approach by comparing it to previous results obtained on small networks by con- sidering the MNIST dataset and an Multi Layer Perceptron (MLP) architecture that was inspired by Saxe et al. (2018). We further compare to a small CNN architecture similar to that of Noshad et al. (001)) before considering large networks, namely VGG16, and a more challenging dataset, namely CNN to denote the small CNN architecture. De- tails about the MLP and the CNN utilized in these experiments can be found in Appendix F All MI
CIFAR- 10. Note, unless stated otherwise,
estimates computed using Equation 3 4 4ndd 5 and the tensor approach described in Section 4.
5

---

Under review as a conference paper at ICLR 2020
Iterations
5000
1
Layer 2
Layer 4
Layer
3000
I(T)))
2.0
2000
15
1000
10
-0
2.0
3.0
4.0
5.0
I(X:T)
Figure 2: IP of a CNN consisting three convolutional layers with 4, 8 and 12 filters and one fully connected layer with 256 neurons and ReLU activation function in in each hidden layer. MI was computed using the training data of the MNIST dataset and averaged over runs.
Layer
3.
Layer
Layer
Layer
Layer
2.0
Layer
Layer 8
Layer 10
Layer 11
12
Layer 14
Layer 15
80
(X:TT
Figure 3: IP of the VGG16 on the CIFAR-10 dataset. MI was estimated using the training data and averaged over 2 runs. Color saturation increases as training progresses. Both the fitting phase and the compression phase is clearly visible for several layers.
Since the matrix- Rényi MI is computed at batch level certain degree of noise is present. We employ moving average smoothing approach where each sample is averaged over nain-bbtcches. For the MLP and CNN experiments we use & 10 and for the VGG16 we use k 50. We use batch size of 100 samples, and determine the kernel width using the kernel alignment loss defined in Equation 9 For each hidden layer, chose the kernel width that maximizes the kernel alignment loss in the range 0.1 and times the mean distance between the samples in one mini- bati-b Initially, sample 75 equally spaced values for the kernel width in the given range for the ML.P and CNN and 300 values for the VGG16 network. During training, we dynamically reduce the number of samples 50 and 100 respectively fact that the kernel width remains relatively stable during the latter part of training (illustrated in Appendix M. We chose the range 0.1 and 10 times the mean distance between the samples in one mini-batch to avoid the kernel width becoming too small and to ensure that we cover wide enough range of possible values. For the input kernel width we empirically evaluated values in the range 2-16 and found consistent results for values in the range 4-12. All our experiments were conducted with input kernel width of 8. For the label kernel matrix, we want kernel width that is as small possible to approach an ideal kernel matrix, but also avoid numerical instabilities. For all our experiments we use a value of 0. for the kernel width the label kernel matrix.
reduce computational complexity and motivated by the
Comparison to Previous Approaches First, we study the IP of the MLP examined in previous works on DNN analysis using information theory (Noshad et al. 2019 Saxe et al. 2018). We utilize stochastic gradient descent with learning rate of 0.09, cross-entropy loss function, and repeat the experiment times. Figure displays the of the MLP with a ReLU activation function in each hidden layer. MI estimated using the training data of the MNIST dataset. A similar experiment

---

Under review as a conference paper at ICLR 2020
Layer
30
Layer 2
Layer
Layer
2.0
Layer
Layer
Layer 10
10-
Layer 11
Layer 12
13
Layer 14
Layer 15
16
(DCT
Figure 4: IP of the VGGI6 on the CIFAR-10 10 dataset. MI was estimated using the test data and averaged over 2 runs. Color saturation increases as training progresses. The fitting phase clearly visible while the compression phase can only be seen in the output layer.
was performed with the tanh activation function, obtaining similar results. The interested reader can find these results in Appendix G
From Figure one can clearly observe a fitting phase, where both XX; and I(Y;T) increases rapidly, followed by compression phase where (T;; X) decrease and (TY remains unchanged. Also note that ;(;; T) for the output layer (layer 5 in Figure I stabilizes at an approximate value of l10).10). which is be expected. This can be seen by noting that when the network achieves approximately 100% accuracy, [(Y; Ý) s SYY), where Y denotes the output of the network, since Y and will be approximately identical and the MI between a variable and itself is just the entropy of the variable. The entropy of Y estimated using Equation which requires the computation of the eigenvalues of the label kernel matrix N Ky. For the ideal case, where (K,, kis = if y: = y/ and zero otherwise, Ky rank K matrix, where K is the number of classes in the data. Thus,
Ky has K non-zero eigenvalues which are given by AR Ky) > is the number of datapoints in class k, k = 1,2...., have Ne, = Noz estimate
Noz, where No. N
.., K. Furthermore, if the dataset is balanced we Nex = Ne. Then, d (N ky) N K. which gives us the entropy
K
S (
A
1 N
log2
AR
(T
1
k=1
(10)
log2
log [].
K
Next examine the IP of CNN, similar to that studied by Noshad al. (2019), with similar experimental setup as for the MLP experiment. Figure 2 displays the IP of the CÑN with a ReLU activation function in all hidden layers. A similar experiment
conducted using the tanh activa- tion function and can be found in Appendix H While the output layer behaves similarly to that of the MLP. the preceding layers show much less movement. In particular, no fitting phase is observed, which might be a result of the convolutional layers being able to extract the necessary information in very few iterations. Note that the output layer is again settling the expected value log, (10), similar to the MLP, as it also achieves close to 100% accuracy.
Increasing DNN size Finally, we analyze the IP of the VGG16 network on the CIFAR-10 dataset, with the same experimental setup as in the previous experiments. To our knowledge, this is the first time that the full IP has been modeled for such large-scale network. Figure 3 and 4 show the IP when computing the MI for the training dataset and the test dataset respectively. For the training dataset, we can clearly observe the same trend as for the smaller networks, where layers experience a fitting phase during the early stages of training and compression phase in the later stage. Note, that the compression phase is less prominent for the testing dataset. Also note the difference between the final values of [(Y: T) for the output layer estimated using the training and test data, which is a result the different accuracy achieved on the training data (se 100%) and test data (+s 90%). Cheng et al. 2018) claim that T(T; X) H(X) and I(Y:T) s H(Y) for high dimensional data,
7

---

Under review as a conference paper at ICLR 2020
and highlight particular difficulties with computing the MI between convolutional layers and the input/output. However, this statement is dependent on their particular estimator for the MI, and the results presented in Figure 3 and 4 demonstrate that neither XX) nor I(Y;T) is deterministic
for our proposed quantity.
Effect of Early Stopping We also investigate the effect of using early stopping the IP described above. Early stopping is a regularization technique where the validation accuracy is monitored and training stopped if the validation accuracy does not increase for set number of iteration, often referred to as the patience hyperparameter. Figure [ displays the results of monitoring where the training would stop if the early stopping procedure was applied for different values of patience. For a patience of 5 iterations the network training would stop before the compression phase takes place for several of the layers. For larger patience values, the effects of the compression phase can be observed before training is stopped. Early stopping is procedure intended to prevent the network from overfitting, which may imply that the compression phase observed in the IP of DNNs can be related to overfitting. The interested reader can find further experiments the compression phase and early stopping in Appendix !
Data Processing Inequality A DNN consists of a chain of mappings from the input, through the hidden layers and to the output. One can interpret a DNN as a Markov chain (Shwartz-2h) Ziv & Tishby,
2017 Yu & Principe 2019) that defines information path (Shwartz-Zii & Tishby 2017). which
should satisfy the following Data Processing Inequality (Cover & Thomas 2006):
I(x 2 7(x 2 2 T(X;Tr),
(11)
where L is the number of layers in the network. An indication of a reasonable measure of MI is that it should uphold the DPI. Figure Tin in Appendix Kjillustrates the mean difference in MI between two subsequent layers in the MLP and VGG16 network. Positive numbers indicate that MI decreases, thus indicating compliance with the DPI. We observe that our quantity complies with the DPI for all layers in the MI. and all except one in the VGG16 network.
6 CONCLUSION
In this work, propose novel framework for analyzing DNNs from MI perspective using lensor-based estimate of the Rényi's a-order entropy. Our experiments illustrate that the proposed approach scales to large DNNs, which allows us to provide insights into the training dynamics. We observe that the compression phase in neural network training tends to be more prominent when MI estimated on the training set and that commonly used early-stopping criteria tend to stop training before or at the onset of the compression phase. This could imply that the compression phase is linked to overfitting. Furthermore, we showed that, for our tensor-based approach, the claim that HX)) se I(T; X) and H(Y) s [(T; Y) does not hold. We believe that our proposed approach can provide new insight and facilitate a more theoretical understanding of DNNs.
REFERENCES
R. A. Amjad and B. C. Geiger. Learning representations for neural network-based classification using the information bottleneck principle. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2019. ISSN 0162-8828. doi: 10. 10.1109/TAAM..2019999900
Rajendra Bhatia. Infinitely divisible matrices. 221-235, 2006. ISSN 00029890, 19300972. URL http: h/wp::///www.... jstor. wtt.tttoooog/ssallo
The American Mathematical Monthly, 113(3);
27641890
Ivan Chelombiev, Conor Houghton, and Cian O'Donnell. Adaptive estimators show information compression in deep neural networks. In International Conference on Learning Representations, 2019.
Hao Cheng, Dongze Lian, Shenghua Gao, and Yanlin Geng. Evaluating capability of deep neural networks for image classification via information plane. In The European Conference on Com- pufer Vision (ECCV), September 2018.
8

---

Under review as a conference paper at ICLR 2020
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecom- munications and Signal Processing). Wiley-Interscience, New York, NY, USA, 2006. ISBN 0471241954.
Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target align- ment. In Advances in neural information processing systems, pp. 367-373, 2002.
Luis Gonzalo Sanchez Giraldo, Murali Rao, and José Carlos Principe. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory, 61:535-548, 2012.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna- lional Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. mlr wwwpsppeesssvee httpesssvv/g/ooorroooo gww.otloootttoooooooo Ca html
URL
Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kings- bury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2309-2308. Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL httt:::/ppocceddiniiiin html
mlr.
pxess/v97/goldfe
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 13:723-773, March 2012. ISSN 1532-4435.
URL http::/t///ll. acm. org/ citation. wwt.:cc222.21111...22221
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In /CML, 2015.
Artemy Kolchinsky, Brendan Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. In International Conference on Learning Representations, 2019. URL
httttss://peenreeee..eeet htw.:eeet/oormmm22d=eeeee/HHaa
N. Kwak and Chong-Ho Choi. Input feature selection by mutual information based on parzen win- dow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(12):1667-1671, Dec 2002. ISSN 0162-8828. doi: 10. 10.1T/9//A 14861.
Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, and Bernhard Schikopf. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends in Machine Learning, 1011,141, 01 2017. doi: 10.1561 10.1561/2200000660.
Michael A. Nielsen and Isaac L. Chuang. Quantum Computation and Quantum Information: 10th Anniversary Edition. Cambridge University Press, New York, NY, USA, 10th edition, 2011. ISBN 1107002176, 9781107002173.
M. Noshad, Y. Zeng, and A. O. Hero. Scalable mutual information estimation using dependence graphs. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2962-2966. May 2019. doi: 10. 109/CAASSSSP22219.88833
Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 15(6):1911-255. June 2003. ISSN 0899-7667. doi: 10.1 00.11620089966603321180000
URL http::/dxx
org org/10.
780272
Emanuel Parzen. On estimation of probability density function and mode. Ann. Math. Statist., 33(3):1065-1072 09 1962. doi: 10. 12.12144 1//00///1117774472 URL sattps://don org/10. 121 /214/aomm/117770222

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W, 2017.
Jose C. Principe. Information Theoretic Learning: Renyi's Entropy and Kernel Perspec- rives. Springer Publishing Company, Incorporated, 1st edition, 2010. ISBN 1441915699,
9781441915696.
Alfred Renyi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probabilit, Volume 1: Contributions to the Theory of Statistics, pp. 547-561, Berkeley, Calif,, 1961. University of California Press. URL https: //pp:j/eeeecclc..ocrr//aaaaaaaaa
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Bren- dan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory deep learning. In International Conference Learning Representations, 2018. URL https:
//operevveww het/forummidee Serpnndddrryyy-a-A WPG-A-
C. E. Shannon. mathematical theory of communication. Journal, ((4))623-656. 1948.
Bell System Technical doi: 10. 15380721538-7755 19433000.999..00 1b00999
URL
inwllireelbbrr... 1948. 208881bb091..00
wwt.ey.comdo.//bb////////// / 7305.
com/ www.ddii/////eeeeeeeee
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 2000.
Tao Shi, Mikhail Belkin, Bin Yu, al. Data spectroscopy: Eigenspaces of convolution operators and clustering. The Annals of Statistics, 3768)) 2009.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. CoRR, abs/1703 1sss/7033008111 10, 2017. URL http: arwiv.iiivv.....m org/ abs//7a3..11///////// 00810
Marco Signoretto, Lieven De Lathauwer, and Johan AK Suykens. A kernel-based framework to tensorial data analysis. Neural networks, 24(8):661-877 2011.
Bernard W Silverman. Density Estimation for Statistics and Data Analysis, volume 26. CRC Press, 1986.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
N. Tishby and N. Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5, April 2015. doi: 10.1 10.1009/TW.22155/133666
S. Yu, L. G. Sanchez Giraldo, R. Jenssen, and J. C. Principe. Multivariate extension of matrix- based renyi's -order entropy functional. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2019, doi: 00.109/7PAM1.201922927777
Shujian and Jose C. Principe. Understanding autoencoders with information theoretic con- cepts. Neural Networks, 117:104 123, 2019. ISSN 0893-6080. doi: https://doi.org/00.0015
j.neuet.2019.055.00 003. URL http: //t:: www. wtw.:.cceenedricccc. com/ aclence/article/
pii 0111/0089368001930000
Shujian Yu, Kristoffer Wickstrom, Robert Jenssen, and José C. Príncipe. Understanding convo- lutional neural network training with information theory. CoRR, abs/1800 bbs/1804400637, 2018. URL
http:/ fttt://arxisn wttp/bs////o//oo 06537
10

---

Under review as a conference paper at ICLR 2020
A VALIDATION OF MATRIX-BASEE ESTIMATORS ON HIGH-DIMENSIONAL SYNTHETIC DATA
To examine the behaviour of the matrix-based estimators described in Section we have conducted simple experiment estimating entropy and mutual information in 100-dimensional data following a normal distribution. First, we generate 500 samples from six 100-dimensional normal distribution with 0 mean and variance {0.25, 0.20, 0.15, 0.1, 0.05, 0.01} and estimate the entropy of the result- ing distributions. The results of this experiment displayed in Figure 5 where the leftmost figure displays two dimensional representation of the data and the the rightmost figure shows the esti- mated entropy for different values of the variance. In the rightmost plot, entropy is estimated using all samples and in a batch-wise setting, with batches of size 100 (like in the main paper). The exper- iment shows how the estimated entropy decreases as the variance of the distribution decreases, as expected. Also, it shows how the batch-wise approximation produce similar results the estimates using the full dataset. Second, we generate 500 samples from a 100- -00ddimensionll normal distribu- tions with mean and variance 0.25, and 500 samples from six 100-dimensional normal distribution with mean and variance (0.25, 0.20, 0.15, 0.1, 0.05, 0.01) and estimate the MI between the distri- butions. In the rightmost plot, MI is estimated using all samples and in batch-wiss setting, with batches of size 100 (like in the main paper). The results of this experiment is depicted in Figure 6 where the leftmost figure displays two dimensional representation the data and rightmost figure shows the estimated MI variance of one of the distributions is decreased and less of the distribu- tions overlap. As the variance of the second distribution decreases and the distributions overlap less and less, the MI estimates also decrease, demonstrating that the estimators are able capture de- pendencies in a high-dimensional setting. Moreover, the batch-wise approximation produces similar estimates the full dataset approach.
Figure 5: Leftmost figure displays a 2-dimensional illustration of the data described in Appendix A and rightmost figure shows how the entropy decreases as the variance of the distribution decreases. Entropy is estimated using Equation on a 100-dimensional normal distribution, both with all sam- ples (displayed in blue) and with batches of size 100 (displayed in red).
Figure 6: Leftmost figure displays a 2-dimensional illustration of the data described in Appendix A and rightmost figure shows how mutual information decreases less of the distributions overlap. Mutual information is estimated using Equation two 100-dimensional normal distribution, using all samples (displayed in blue) and with batches size 100 (displayed in red).
11

---

Under review as a conference paper at ICLR 2020
B BOUND ON MATRIX-BASER ENTROPY WITH RESPECT TO COVARIANCE OPERATOR
The properties of Equation 2 was analysed in detail in Giraldo et al. (2012). It was shown that the kernel matrix A, obtained from the raw data, acts much density matrix similar to quantum information theory (Nielsen & Chuang 2011). It was further shown that the kernel matrix is related an empirical covariance operator on embeddings of probability distributions in RKHS. This is similar to the approach of maximum mean discrepancy and the kernel mean embedding (rrtton al., 2012) Muandet al. (0177) The connection with the data population can be shown the theory of covariance operators. The covariance operator G : H. - H. is defined through the bilinear form
- J (f, (xx) (1(x), di x (x) Ex (f(X)g(Y)
where Px is probability measure and fig H. Based on the empirical distribution Fv N DN 8x,(x), the empirical version & of obtained from a sample x; of size N is given by:
( =
1 -f (f. (x)()))))))))))))(()) ()())) (xx(x)=
N Σ (f (xx))) (xxx)))
By analyzing the spectrum of & and G, Giraldo et al. 2012) showed the the difference between tr(G) and tr(G) can be bounded, as stated in the following proposition:
Proposition B.1. Let = > 8x,(x (x) be the empirical distribution. Then, as a consequence {
of Proposition 6.1 in Giraldo et al. (2012), tr (G2) tr ". The difference between tr(G)
and (r) can be bounded under the conditions of Theorem 6.2 Giraldo al. (2012) and for >> 1, with probability 1-5
tr (G²) -r 1o-tr(r ((x)
2 2log ar
S aC
(12)
where C is compact self-udjoint operator.
C PROOF OF EQUATION
3 IN SECTION 3
Proof.
1
0
lim (A) = lim a-11
loga
Σ 2
a-+1
1
0
since Σ/ d = tr()) = 1. L'Hôpital's rule yields
@ Br
loga
1> i=1 AA)
lim (A)
lim am
3a -aa) a)
Σ (n)((A)
lim
(13)
2n2 a-
Σ
12

---

Under review as a conference paper at ICLR 2020
D TENSOR-BASED APPROACH CONTAINS MULTIVARIATE APPROACH AS SPECIAL CASE
Let (f) E RWWC denote the vector representation of data point i in layer and let x; (0) E RHw denote its representation produced by filter c. In the following, we omit the layer index for ease of notation, but assume fixed. Consider the case when Ke(:s-) is an RBF kernel with kernel width
parameter de. That is nx(xx (0)
(c))
2a [xx 10
In this case, A K. and
A O O Ac
1 K ..... N
Ke
N
A Ac)
tr( K .... 1 Kc)
o Kc
No Ne lr(K ....o Kc)
Ki Kc, N
since tr(K, .... .c) N. Thus, element (i,j) given by
A1 o.... o Ac Ac
- -x
K Jo
tr [ Ac) 11
N
If we let 0 = 01 = 02 = = oC, this expression is reduced to
# De_ [xx(*) - 10 N
1
d x x1,
N Knnn (%i, Xz).
(14)
Ac) implying that the tensor method is equivalent to the multivariate matrix-based joint entropy when the width parameter is equal within a given layer, assuming an RBF kernel is used. However, the tensor-based approach eliminates the effect of nu- merical instabilities encounters in layers with many filters, thereby enabling training of complex neural networks.
Accordingly, S. (Arn) =
E NUMERICAL INSTABILITY OF MULTIVARIATE APPROACH AND STRUCTURE PRESERVING TENSOR KERNELS
As explained in Section 3.2. the multivariate approach of Yu et al. (2019) (Equation 6 struggles when the number of channels in an image tensor becomes large, as a result of the Hadamard products in Equation To illustrate this instability we have conducted a simple example. A subset of 50 samples is extracted from the MNIST dataset. Then, each image is duplicated (plus some noise) C' times along the channel dimension of the same image, i.e. going from a grayscale image of size 11, 28, 28) to new image of size (1, C, 28, 28). Since the same image is added along the channel dimension the kernel matrix should not change dramatically. Figure 7 displays the results of the experiment just described. The first row if Figure 7 shows the kernel matrices based on the multivariate approach proposed by by et al. (2019). When the tensor data only has one channel (first column) the kernel obtained is identical to the results obtained using the tensor kernel described in this paper. However, as the number of channels increase, the off-diagonal quickly vanishes and the kernel matrix tends towards a diagonal matrix. This is result of vanishing Hadamard products, as described in Section 3.2 of the main paper. Theoretically, the multivariate approach should yield the same kernel as with the tensor kernel approach, explained Scction 4 but the off-diagonal elements decrease so quickly that they fall outside numerical precision. The second row of Figure depiccs the kernel matrices obtained using the tensor kernel approach described in Section The the number of channels increase, which is to be
kernel matrices in this row are almost unchanged expected. Since the same image added along the channel dimension, the similarity between the sample should not change drastically, which is what this row demonstrates. The third row of Figure displays the kernel matrices obtained using so-called matricization- based tensor kernels Signoretto al. (2011), which tensor kernels that preserve structure between the channels of the tensor. In this case, this approach produces similar results to the tensor kernel used this paper, which is to be expected. Since the same image is added along the channel dimension there little information
13

---

Under review as a conference paper at ICLR 2020
Number charnela.
Number dhanees:
Number chamals...
channels: 50
Figure 7: Different approaches for calculating kernels based on tensor data. First row shows the multivariate approach Yu et al. al. (2019), second row depicts the tensor kernel approach used this paper, and third row displays the kernel obtained using matricization-basse tensor kernels (Signoretoo jet al. (20IT) that preserve structure between channels. Bright colors indicate high values while dark values indicate low values in all the kernel matrices.
to extract between the channels. We hypothesize that for small images with centered objects, such as with MNIST and CIFARIO, the structured tensor kernel does not capture much more information than the tensor kernel described in Section However, for more complex tensor data, exploring the potential of such structure preserving tensor kernels is an interesting avenue for future studies.
F DETAILED DESCRIPTION OF NETWORKS FROM SECTION 5
We provide detailed description of the architectures utilized in Section of the main paper. Weights initialized according He al. (2015) when the ReLU activation function was applied and initialized according to Glorot & Bengio (2100) for the experiments conducted using the tanh acti- vation function. Biases initialized zeros for all networks. All networks were implemented
using the deep learning framework Pytorch (Paszke et al. 2017).
F.1 MULTILAYER PERCEPTRON UTILIZED IN SECTION
The MLP architecture used in our experiments is the same architecture utilized in previous studies on the IP of DNN (Noshad et al. 2019; Saxe al. 2018), but with Batch Normalization (loff & Szegedy 2015) included after the activation function of each hidden layer. Specifically, the MLP in Section 5 includes (from input to output) the following components:
1. Fully connected layer with 784 inputs and 1024 outputs.
2. Activation function.
3. Batch normalization layer
4. Fully connected layer with 1024 inputs and 20 outputs.
5. Activation function.
6. Batch normalization layer
7. Fully connected layer with 20 inputs and 20 outputs.
14

---

Under review as a conference paper at ICLR 2020
8. Activation function.
9. Batch normalization layer
Fully connected layer with 20 inputs and 20 outputs.
11. Activation function.
12. Batch normalization layer
13. Fully connected layer with 784 inputs and 10 outputs.
14. Softmax activation function.
F.2
CONVOLUTIONAL NEURAL NETWORK UTILIZED IN SECTION
The CNN architecture in our experiments a similar architecture as the one used by Noshad et al. 2019). Specifically, the CNN in Section 5 includes (from input output) the following components:
Convolutional layer with input channel and filters, filter size 3 x 3, stride of and no padding.
2. Activation function.
3. Batch normalization layer
4. Convolutional layer with 4 input channels and filters, filter size x 3, stride of 1 and no padding.
5. Activation function.
6. Batch normalization layer
7. Max pooling layer with filter size 2 x 2, stride of 2 and no padding.
8. Convolutional layer with input channels and 16 filters, filter size 3 x 3, stride of 1 and no padding.
9. Activation function.
10. Batch normalization layer
11. Max pooling layer with filter size 2 stride of 2 and padding.
12. Fully connected layer with 400 inputs and 256 outputs.
Activation function.
14. Batch normalization layer
15. Fully connected layer with 256 inputs and 10 outputs.
16. Softmax activation function.
G IP OF MLP WITH TANH ACTIVATION FUNCTION FROM SECTION 5
Figure g displays the IP of the MLP described above with a tanh activation function applied in each hidden layer. Similarly to the ReLU experiment in the main paper, fitting phase is observed. where both I(T;XX and I(Y; I') increases rapidly, followed by compression phase where I(T; decrease and [YY; T) remains unchanged. Also note that, similar the ReLU experiment, :(Y:T) stabilizes close to the theoretical maximum value of (1g).10).
H IP OF CNN WITH TANH ACTIVATION FUNCTION FROM SECTION 5
Figure 9 displays the IP of the CNN described above with a tanh activation function applied in each hidden layer. Just for the CNN experiment with ReLU activation function in the main paper, fitting phase is observed for the majority the layers, which might indicate that the convolutional layers can extract the essential information after only few iterations.
15

---

Under review as a conference paper at ICLR 2020
tterations
Layer
Layer
3.0
Layer
5000
Layer
2.5
Layer
4000
((T:)
2.0
3000
1.5
2000
1.0
1000
0.5
((:T)
Figure 8: IP of MLP consisting of four fully connected layers with 1024, 20, 20, and 20 neurons and tanh activation function in in each hidden layer. MI was estimated using the training data of the MNIST dataset and averaged over runs.
Iterations
325
5000
3.00
275
4000
2.0
3000
225
2.00
Layer
Layer
175
Layer 1000
150
-0
400
((X:T)
Figure 9: IP of a CNN consisting three convolutional layers with 4, 8 and 12 filters and one fully connected layer with 256 neurons and tanh activation function in in each hidden layer. MI was estimated using the training data the MNIST dataset and averaged over 5 runs.
I KERNEL WIDTH SIGMA
We further evaluate our dynamic approach of finding the kernel width a. Figure 10 shows the variation of a in each layer for the MLP, the small CNN and the VGG16 network. We observe that the optimal kernel width for each layer (based on the criterion Section 4.2). stabilizes reasonably quickly and remains relatively constant during training. This illustrates that decreasing the sampling range a meaningful approach to decrease computational complexity.
J INFORMATION PLANE VIDEO
To further investigate the role of the compression phase in the IP have created video that available at the following link: hattps://stream:: training process of simple 3 layer perceptron with ReLU neurons in each layer, which trained
com/ 7 The video shows the
16

---

Under review as a conference paper at ICLR 2020
7gm Seer
Skgmm ah
kgc gam
Eosition
teration
(a) MLP
(b) CNN
(c) VGG16
Figure 10: Evolution kernel width a function of iteration for the three networks that we consid- ered in this work. The plots demonstrate how the optimal kernel width quickly stabilizes and stay relatively stable throughout the training.
solve the XOR problem. We used such a simple network allows us visualize the decision surface of each layer and how the data is transformed throughout the training, as well as the infor- mation plane. All information theoretic quantities are calculated using the matrix based approach described in this paper.
Several interesting aspects could be highlighted, but we would like to focus on the compression phase. After approximately 1250 iteration the network approaches 100 percent accuracy. At the same time, the output layer starts to show compression. By inspecting the decision surface in the in- put space (plot shown in first row and first column of the video) it is clear that the network has found a solution that separates the two classes. Early stopping would stop the training at approximately 1300 iterations, depending on the patience parameter. At the end of the training, the decision surface in the input space has become much more narrow, thus leaving less opportunity to correctly classify new samples that lay slightly outside the space of the training samples. Such observations corrob- orate the suggestions from the main paper, that the compression phase is not necessarily associated with improved generalization.
K DATA PROCESSING INEQUALITY
Figure II illustrates the mean difference in MI between two subsequent layers in the MLP and VGG16 network. Positive numbers indicate that MI decreases, thus indicating compliance with the DPI. We observe that our estimator complies with the DPI for all layers in the MLP and for all except one in the VGG16 network.
Note, the difference in MI is considerably lower for the early layers in the network, which is further shown by the grouping of the early layers for our convolutional based architectures (Figure 2
17

---

Under review as a conference paper at ICLR 2020
150
125
1.00
1X:70 - XX:
0.75
0.6
-0.25
0.
10
15
Layer
Layer
(a) MLP
(b) VGG16
Figure 11: Mean difference in MI of subsequent layers and L + 1. Positive numbers indicate
compliance with the DPI. MI was estimated on the MNIST training set for the MLP and on the CIFAR-10 10 training set for the VGG16.
18


Reviews:
[{"review_id": "S1lNaTIRKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=== SUMMARY ===\n\nThe paper...\n- suggests an estimation method of mutual information that is less susceptible to numerical problems when many neurons are involved\n- suggests an estimation method tailored to convolutional neural networks. \n- analyses whether H(X) ≈ I(T;X) resp. H(Y) ≈ I(T;Y) is true in high dimensions for their suggested estimator.\n- analyses how common regularization techniques affect the learning dynamics. They show that early stopping prevents the learning process from entering the compression phase\n\n\n=== RELATED WORK ===\n\nPrior work is clearly presented and well-treated. The introduction and related work give a good overview of prior work on information plane theory, touching on the important contributions from Tishby & Zaslavsky [2015], Schwartz Ziv & Tishby [2017], Saxe et. al [2018], and Noshad et. al [2019], among others. In Section 3, Renyi’s \\alpha-order entropy is outlined, and the multi-variate extension proposed by Yu et al. [2019] is described.\n\n\n=== APPROACH === \nMultiple papers are put into perspective, and the findings of these papers serve as a foundation for the suggested method\n- Renyi’s Entropy is presented as a generalization of different entropy measures (among them Shannon Entropy). Computing Renyi’s Entropy however necessitates to get a high-dimensional density estimation over the data distribution\n- Giraldo et. al. suggested a kernel-based method to estimate Renyi’s Entropy solely on data without estimating the (intractable) density. An estimation for the mutual information can also be found in said paper\n- Yu et al. suggested a generalization of b) that can handle C variables instead of only two. As this computation includes taking the product over C values that lie within (0,1/N) the result tends to 0.\n\nAn RBF-based kernel is defined for a fixed conv. layer by computing the Frobenius norm w.r.t two 3D-output tensors of the conv. layer X_i, X_j. A derived matrix A is defined similarly to Giraldo et al.\n\nThis way, C individual kernels are defined, which are combined according to Yu et al.\n\n\n=== EXPERIMENTS ===\n\nComparison to Previous Approaches\n- Very detailed and clear structure of experiment. \n\nIncreasing DNN size\n- The experiment shows that there is both a fitting and a compression phase\n- H(X) ≈ I(T;X) resp. H(Y) ≈ I(T;Y) was disproven\n\nEffect of Early Stopping\n- Early Stopping prevents the learning process from entering the compression phase. The authors conjecture that the compression phase is related to overfitting -- it would be interesting to see some evidence to support this.\n\nData Processing Inequality\n- Enough evidence is provided to sustain that the data processing equality also holds for the suggested estimator\n\n\n=== OTHER COMMENTS ===\n\nThe paper would benefit from providing more rationale on why the suggested estimation method is numerically stable (as opposed to Yu et al.), maybe by supplying another proof.\n\nTechnical mistakes\nSection 4.1. Introduces X_i as a 3D tensor of dimensions C x H x W. Eq (7) mentions the Frobenius norm which is defined over matrices. We assume that X_i should be rather defined as a 2D tensor of dimensions H x W for a fixed layer index after reading Section 4.2.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hye7dGFTKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors try to resolve the problem of estimating mutual information between high-dimensional layers in neural network. Specifically, the authors try to tackle the convolutional neural networks, by propose a novel tensor-kernel based estimator. They use the proposed estimator to discover the similar trends on the information plane as the previous works.\n\nIn general, I think the introduction of the tensor kernels for mutual information estimation is the key contribution of this paper. However, I think this contribution is a little bit incremental, compared to the multivariate matrix-based version introduced by Yu et.al. In formula (7), the authors use the Frobenius distance of two tensors as the input of the kernal, which is equivalent to vectorize the tensor and use Euclidean distance. This approach does not capture the special structure of tensor and seems incremental. Also, the phenomenon of \"turning point\" on information plane for neural network has been challenged since its first publication. Despite its high citation, the deep meaning of this phenomenon has not been clearly studied. So an incremental change on computing such a questionable phenomenon make the contribution of this paper not very strong.\n\nSo I would like to give a weak reject. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkg2H7Kp9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper concerns the \"information plane\" view of visualizing and understanding neural net training. Kernel-based estimators of Renyi entropy-like quantities are applied to study moderate-sized DNNs. A straightforward extension to CNNs is presented.\n\nMajor comments:\n\nI found this paper extremely hard to follow. I think a lot of my difficulty was a lack of familiarity with the papers this work was building on, but I also felt the main line of reasoning was unnecessarily opaque. I've tried to indicate where some clarifying wording might help readers such as myself.\n\nIn general I am very skeptical that reasonable entropy-like quantities can be estimated reliably for high-dimensional data using anything along the lines of kernel density estimation, especially based on a single minibatch or small collection of minibatches! The authors provide no experimental evidence that these estimates are even close to being accurate (for example on synthetic datasets where the true entropy / mutual information is known). Clearly the estimated quantities evolve during training, and that may be interesting in itself, but calling the estimated quantities \"mutual information\" seems like a leap that's not justified in the paper.\n\nThroughout the paper, CNNs are referred to as having some special difficulty for entropy estimates because of the channel dimension. This is misleading. For DNNs the activations are vectors, and so a kernel defined over rank 1 tensors is needed. Even without the channel dimension, CNN activations would be rank 2 tensors, and so a kernel would need to be defined over rank 2 tensors. Going from rank 2 to rank 3 doesn't pose any special difficulties. Indeed the most obvious way of defining a kernel over higher rank tensors is to flatten them and use a kernel defined over vectors, which is exactly what the paper does in practice.\n\n\nMinor comments:\n\nIn the introduction, it would be helpful to include some relevant citations after the sentence \"Several works have demonstrated that one may unveil interesting properties... IP\".\n\nIn section 3, the multivariate extension proposed by Yu et al. (2019) seems like an interesting side note (since it was used in a previous attempt to estimate information plane for CNNs), but it doesn't seem central to the paper, and I personally found it unnecessarily confusing to have it presented in the main text. What about moving sections 3.2 and 4.2 to the appendix for clarity?\n\nIn section 3.1, \"over a finite set\" is probably incorrect (\"probability density function\" implies a continuous space for X, as does the integral in (1)).\n\nIn section 3.1 (and the appendix), \"The ones developed for Shannon\" seems imprecise. \"Certain approaches developed for estimating the Shannon entropy\"?\n\nIt's not clear what \"have the same functional form of the statistical quantity\" means. Which statistical quantity? What aspects of the functional form are similar? Please elaborate in the paper.\n\nI think \"marginal distribution\" is incorrect. It's representing a whole probability distribution, not just a marginal distribution. Which marginal would it be in any case?\n\nSection 3.2 states \"The matrix-based... entropy functional is not suitable for estimating... convolutional layer... as the output consists of C feature maps\", but as discussed above, there is no special difficulty caused by the channel dimension. Even if the channel dimension were not present the difficulties would be the same. (Also, it seems like defining a kernel over the rank-3 tensors is an extremely natural / unsurprising thing to try given the set-up so far.)\n\nIn section 4.1, \"can not include tensor data without modifications\" seems misleading for a similar reason. One of the great things about kernels is that they can be defined on lots of objects, including things like rank 3 tensors!\n\nNear (8), it would be very helpful to state explicitly what is done for the joint entropy term in (5). It sounds like this term, which involves the Hadamard product, in practice amounts to summing the Euclidean distances between x's and between y's, and it might be helpful to the new reader to point this out. (It also highlights that the method is easy to implement in practice).\n\nThe discussion in section 4.2 is only valid for the RBF kernel, but the first paragraph of that section makes it sound like it is true more generally.\n\nAt the bottom of section 4.2, if the proposed approach is equivalent to the multivariate approach, then how can one suffer from numerical instability while the other doesn't? Also, numerical stability makes it sound like an artifact of floating point arithmetic, whereas I think the point that's being made is more mathematical? Please clarify in the paper.\n\nIn \"enabling training of complex neural networks\", shouldn't \"training\" be \"analysis\"?\n\nIn section 5, under \"increasing DNN size\", I wasn't clear on the meaning of \"deterministic\" in \"neither .... is deterministic for our proposed estimator\". Random variables can be deterministic or not, but how can a mutual information be deterministic?\n\nUnder \"effect of early stopping\", isn't looking at the test set entropies (as is done elsewhere in the paper) much more relevant to overfitting than considering different \"patience\" values?\n\nIn the bibliography, two different papers are \"Yu et al. (2019)\".\n\nIn appendix A, \"the same functional form of the statistical quantity\", \"marginal\", etc don't seem quite correct, as mentioned above. Also the first equation should not have a comma between f(X) and g(Y) (which if I understand correctly are being multiplied).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the information plane analysis of DNNs. Estimating mutual information is required in such analysis which is difficult task for high dimensional problems. This paper proposes a new "matrix–based Renyi’s entropy coupled with ´tensor kernels over convolutional layers" to solve this problem. The methods seems to be related to an existing approach but derived using a different "starting point". Overall, the method is able to show improvements in high-dimensional case.

Both R1 and R3 have been critical of the approach. R3 is not convinced that the method would work for high-dimensional case and also that no simulation studies were provided. In the revised version the authors added a new experiment to show this. R3's another comment makes an interesting point regarding "the estimated quantities evolve during training, and that may be interesting in itself, but calling the estimated quantities mutual information seems like a leap that's not justified in the paper." I could not find an answer in the rebuttal regarding this.

R1 has also commented that the contribution is incremental in light of existing work. The authors mostly agree with this, but insist that the method is derived differently.

Overall, I think this is a reasonable paper with some minor issues. I think this can use another review cycle where the paper can be improved with additional results and to take care of some of the doubts that reviewers' had this time. 

For now, I recommend to reject this paper, but encourage the authors to resubmit at another venue after revision.
Decision: Reject