A Probabilistic Representation for Deep Learning: Delving into The Information Bottleneck Principle
Authors: XINJIE LAN
Kenneth Barner
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Deep Learning
Probabilistical Modeling
Information Bottleneck

Abstract: The Information Bottleneck (IB) principle has recently attracted great attention to explaining Deep Neural Networks (DNNs), and the key is to accurately estimate the mutual information between a hidden layer and dataset. However, some unsettled limitations weaken the validity of the IB explanation for DNNs. To address these limitations and fully explain deep learning in an information theoretic fashion, we propose a probabilistic representation for deep learning that allows the framework to estimate the mutual information, more accurately than existing non-parametric models, and also quantify how the components of a hidden layer affect the mutual information. Leveraging the probabilistic representation, we take into account the back-propagation training and derive two novel Markov chains to characterize the information flow in DNNs. We show that different hidden layers achieve different IB trade-offs depending on the architecture and the position of the layers in DNNs, whereas a DNN satisfies the IB principle no matter the architecture of the DNN. 

A Probabilistic Representation for Deep Learning: Delving into The Information Bottleneck Principle
Anonymous Author(s)
Affiliation
Address
email
Abstract
The Information Bottleneck (IB) principle has recently attracted great attention explaining Deep Neural Networks (DNNs), and the key is to accurately estimate the mutual information between a hidden layer and dataset. However, some unsettled limitations weaken the validity of the IB explanation for DNNs. To address these limitations and fully explain deep learning in an information theoretic fashion, we propose probabilistic representation for deep learning that allows the framework to estimate the mutual information, more accurately than existing non-parametric models, and also quantify how the components of hidden layer affect the mutual information. Leveraging the probabilistic representation, we take into account the back prockk-pppgttionn training and derive two novel Markov chains to characterize the information flow DNNs. We show that different hidden layers achieve different IB trade-offs depending the architecture and the position of the layers in DNNs, whereas a DNN satisfies the IB principle no matter the architecture the DNN.
3
5
7
11
13
Introduction
14
Deep learning [1] has already achieved great success in numerous applications. Deep Neural Networks (DNNs), however, are still commonly viewed as 'black boxes' [27]. Considerable efforts have been devoted to explaining the internal mechanism of DNNs from various perspectives, such as mathematics 12). statistics [4] 20 23), computer vision [37 21), etc. Recently, the Information
16
18
Bottleneck (IB) principle has attracted attention in opening the 'black boxes' of DNNs 30 33)
20 Given joint distribution P(X, Y), the IB principle posits a random variable T = f(X) obeying the Markov chain Y + X + T and optimizes 7 by the IB Lagrangian 32
min T(X:7) (TY;T)) P(T(X)
(1)
where f()) is an arbitrary function, I(::)) denotes mutual information, and the Lagrange multiplier 23 0 controls the IB trade-off between compressing the input and preserving the information of the label Y. In the seminal work [30], Tishby al. manifest the IB trade-off in every layer of
25 DNNs = {x; -.: :t;:9) } via studying [(X; T,) and I(Y;)) where T; is the random variable of
the ith hidden layer t;. Especially, the authors ascribe DNN generalization to the compression [29]
27 In the context of deterministic DNNs, recent works reveal some limitations of the principle for
28 explaining DNNs. Amjad et al. argue that the IB principle becomes an ill- ill-posed optimization problem
29 due to T(X; T,) = oo [1]. and Kolchinsky et demonstrate that not every layer of DNNs satisfies strict IB trade-off. i.e., different layers only differ in [[x; T,) but IYY;T)) keeps consistent in all
31 layers [5). addition, Saxe et al. experimentally show that the compression does not occur in
DNNs with non-saturating activation functions, e,g., the popular ReLU function [28), and Goldfeld
33 et al. doubt the causality between the generalization of DNNs and the compression [0. These
unsettled limitations greatly weakens the validity of the IB explanations for DNNs.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

35
The key to examining the IB principle in DNNs is the accurate estimation of the mutual information. However, regarding DNNs as deterministic models hinders us from specifying the random variable Ti and the distribution P.T.), thus it is difficult to accurately estimate TX;; T:)) and T;): More specifically, in the absence of a clear definition of 7; simply assuming the activations of f, as the i.i.d samples of T, induces T, being a continuous random variable and T(X; T.) oo in deterministic DNÑs (see Appendix C in (28). The complicated architecture of DNNs makes it challenging to
36
37
38
39
40
41
specify P(T.). Therefore, most previous works have to indirectly estimate P(T.) via non-parametric models 35 such as the empirical distribution [30], Kernel Density Estimation (KDE) [88], and
42
Gaussian convolution [1O]. However, we experimentally confirm that classical non- parametraiicc models derives poor mutual information estimation [24] [22 in DNNs, and one reason is because activations do not satisfy the i.i.d. prerequisite of non paraameettiic models (see Appendix G). In summary, the limitations mainly stem from the lack of an explicit probabilistic representation for deep learning.
44
45
46
47
The IB principle only formulates the information flow in DNNs = (ar, t1, ... tt, UJ after training, and the corresponding Markov chain (see Fig. in 30)
48
(2) 49 indicates that the information of Y transfers T. in the forward direction and T; receives the
Y TT.
50 information of Y only via X. However, training DNNs by the back-propagation (25) implies that the
51 information of Y transfers to T, in the backward direction during training and retains information
52 in T, after training. Notably, Zhang et al. show that DNN can fit labels well even using Gaussian
58 noise as input to train the DNN (38. which implies that Ti can directly receive the information of Y. Hence, the IB principle does not comprehensively characterize the information flow in DNNs.
54
55 To address the above limitations and comprehensively explain DNNs in an information theoretic fashion, we introduce the probability space (Rr,, Pr.) [6] for the ith hidden layer t; in DNNs.
56
57
Compared previous works, the probability space (ST, F, Pr. enables us to: (i) accurately estimate
58
I(x; T.) and Ti) via specifying Ti and PTT:), and (ii) quantify the effect of the architecture of and the back-propagation on T(X; T.) and ;(Y;T.) via explicitly modeling all the ingredients of 4, such as the activation function and the weights in probabilistic way. To the best of our knowledge,
59
80
61 this is the first time the probability space of a hidden layer in DNNs as defined.
82
Leveraging (9n,, F, Pr.), we derive information theoretic explanations for DNNs as follows:
63
* Two Markov chains characterize the information flow in DNNs = {2r, t,. ... ,t,,
X +T .... -, -
- + +
64
Different hidden layers manifest different IB trade-offs depending on the architecture and the position of hidden layers in DNNs.
65
66
* A DNN satisfies the IB principle no matter the architecture of the DNN.
Preliminaries. P(X.Y) = P( is an unknown joint distribution between X and Y. A 68 dataset D = {(x) 12 |x> E C .3² E 2) -m consists of J i...d. samples generated from P(X,Y)
67
69 with finite L, labels, i.e., y E (1...., T.). In the context of supervised learning, we focus on
70 feedfworad fully connected DNNs {2, t,, ........ ,tz, 0). i.e., Multi-Layer Perceptions (MLPs) g
71 for the image classification task. Without loss of generality, we use the MLP- {a, t1, t2, 9) with
72 the cross-entropy loss fcE for most theoretical derivations. In addition, H.) denotes entropy.
73 In the MLP, & and to have N and K neurons, respectively, and f, = {Inn = a () (1)
74 where (w))) _= = ΣM winn Im + bn the with dot-product given the weight ((l)) and the bias bin, and 01 (-) denotes an activation function, e.g., ReLU. Similarly, t2 (t2x (w tt)
(2)
1
78 where (w , (1) = >n wn2 tim + b2z. The output layer ý is softmax with L nodes
y = {vr =
Zy exp
1
(3) te)l
Zy (xpla(((((((2))))
(4)
77 where (w)) (3) (2) = Ekm Whi
t2x + byt and Zy = Ete expl (3) te)l is the partition function.
In which the virtual random variable X has all the information of X except Y. namely H(X) (x|Y).

---

convolution layer
3X.3 Kernel
3x3 Kernel 2
2x2 Feature map
Zx2 Feature map 2/
4x4 Input
1/
(4gg 4x4 Kernel
4x4 Kernel
(or(e)))) 1x1
Activation
Activation
t2
A fully connected layer
Figure 1: Given 4 x input fully connected layer is equivalent to convolution layer with 4 x convolution kernels. The definition of convolution (Chapter 9.1 in [TT) implies that the 4 4 weights Wi and W2 define two global features, and the two activations & 22 indicate the cross-correlation between W1, W2 and z, respectively. Priz (wi and Priz(w2)2 measure the probability w and w2 being recognized as the feature with the largest cross-correlation to z, respectively.
78
2
A probabilistic representation for deep learning
To accurately estimate II) and T.), in this section, we specify the probability space [6] for 80 a fully connected layer and derive the probabilistic explanations of the entire ML..
81 It is known that a convolution kernel (namely the weights of convolution) defines a local feature,
82 and a convolution operation derives a feature map to measure the cross-correlation between the local feature and input in receptive field (Chapter 1.1 in III.. Notably, a fully connected layer
84 equivalent to a convolution layer with the kernel size having the same dimension as input. Thus the weights of neuron can be viewed as global feature, and fully connected layer with multiple neurons derives activations to measure the cross cross-correlate correlation between the multiple global features and the input. The cross correlconnn explanation for fully connected layer is visualized in Figure []
86
88 Assuming that a fully connected layer t consists of N neurons {t, = oiw,, =>|]>>> where E RM
89 the input of t. (Wa, z) = Σm=1 Wmn 2m + b, is the dot-product between z and Wn, and (--) is an activation function. Based on the cross-correlation explanation, the behavior of t is measure 31 the cross-correlations between z and the N possible features defined by the the weights
90
92 In the context of pattern recognition 34 define virtual random process 'experiment" as t recognizing one of the patterns/fcatures with the largest cross-correlation to z from the N possible 94 features. The experiment characterizes the behavior of t (i.e., before recognizing the features with the largest cross-correlation, t must measure the cross-correlations between z and all the N possible features) while meets the requirement of the 'experiment" definition (i.e., only one outcome will occur on each trial of the experiment (6). The probability space (Srr, F, Pr)) is defined as follows:
93
95
96
97
98
Definition 1. (2,,F. Pr) consists of three components: the sample space Qy has N possible )
defined by the weights of the N neurons; the event outcomes (features) {w,, {wmm 1.M JM 1
99
100
space F is the o-algebra; and the probability measure Pr is a Gibbs distribution [19] to quantify the
101
probability of w, being recognized as the feature with the largest cors-correlatton to
102
Taking into account the randomness of z. the conditional distribution Py z is formulated as
1 exp(tt,) = exp (W 2))1
Priz(wnlz)
(5)
ZT
Zr
where 7 is the random variable of z and Zr = IN (f) is the partition function.
103
104
(Dr, F, Pr) clearly explains all the ingredients of in probabilistic fashion. The nth neuron defines a global feature by the weights w,, and the activation in = o ((w,, 2)) measures the cross- correlatio between wn and The Gibbs distribution Priz indicates that if wn has the higher activation, i.e., the larger cross-correlation to z, it has the larger probability being recognized as the feature with largest cross- cors-correlltion to For instance, if z R and t includes N neurons, then 2 = {Wi, (w1,w2} defines two possible outcomes (features). where w,, = [Wmn 166 F (0, (W1), (w2), (W1, w2)) means that neither, one, or both of the features are recognized by given x, respectively. Priz [W1] | and Priz(Wg|2z are the probability of w and Wa being recognized the feature with the largest cross-correlation z, respectively.
105
107
109
111
112
We do not take into account the scalar value bn for detining Pr. it not affects the feature detined wa.
3

---

118 (Qr. Pr) explains the representation ability of deep learning. Compared to Restricted Boltzmann Machines (RBMs) [26] simply using binary units to indicate features being recognized or not given input, the Gibbs distribution 3 Friz(wnl z) measures the probability of w,, being recognized with the largest cross-correlation z, i.e., it characterizes the relation between features and input more accurately. Moreover, Equation 5 shows that in o (ww 2)) is the negative energy function [9 of the Gibbs distribution, thus Priz( [wn can be derived as long as ( z)) are known because the energy function is the sufficient statistics [2] of the Gibbs distribution. That enables subsequent hidden layers to generate high-level features of input via directly processing the activations {ln Jn=1 thus deep learning can form a hierarchical structure to represent much complex features.
114
115
116
117
118
120
122
(2r, F.F..P Pr) answers a fundamental question: which component of hidden layer contains the information of the layer? Since w, defines Ω2, the weights contain all the information of a layer. In particular, since the activation i, a (w,, 2)) is a function of wn, the data processing inequality
123
124
125
indicates that the information of t is no more than the information of wn Simulations in Section demonstrate that if activations do not correctly characterize the cross-correlation between weights and input, activations contain less information than weights do.
4
126
128
Based on (Lr, F, Pr), we define the random variable T follows:
129
Definition& 2. Given the fully connected layer t, we define the random variable Ω2 + Er as
T(w,) n,
(6)
130
where the measurable space Er = (1,..,,
Since 22r is composed of finite N possible outcomes, T is discrete random variable. Notably, the one-to-one correspondence between wn and n indicates
132
Pr (wnlz) (r)z)
133
If not considering the pack-propagation training. the weights (namely Qr;;) of each layer are fixed. Thus Ti+1 entirely depends on T; and the MI.P {a;; 11; ÿ) forms Markov chain
134
X X T + T2 + Y.
(8)
Based on the corresponding joint distribution P(Ý: T2. TilX) = P(T) ( (2) and Definition 2, derive probabilistic explanation for the entire MLP, which summarized in Theorem 1. The detailed derivation is presented in Appendix
135
136
137
138
Theorem 1. The MLP {x; t1; t2: }} formulates conditional Gibbs distribution
(r)x(1xx)= = ΣΣ
p(ý = =1, =b R, = n|X = x)
7MLL(w)) 7 (xp(((((((((((( (9)
k=1 n=1
where ZMLP(x) EL IN PY. (1, k. n/x) is the partition function.
139
Since Prix (1|a:) exactly equals the output ÿI of the MLP, namely Equation 4 we conclude that the entire architecture of the MLP forms a family of Gibbs distribution Pyix (zx). In general, the
140
141
142
mack-propagation updates a weight w based on the gradient of fcE with respect to w,
w(s + 1) w(s) Hc. w(s) PYL Ow(s)
(10)
(w(s)
143 where s is the index of training iteration, the training rate, and KL[-II-] is the KL-divergence.
144
Figure 2 summarizes the probabilistic explanation for deep learning based on the MLP. In general, single learning iteration, an epoch, consists of two phases: training and inference (after training). During inference, the MLP bridges X and Y via multiple intermediate features D7 Ωr. and 22g defined by weights, and formulates the statistical relation between Ý and X as family of conditional Gibbs distribution XX). During training, the back proppaaaaannionn updates weights to learn optimal intermediate features for searching an optimal P XX) ) to accurately approximate (X)
147
148
149
Recent works about Gibbs explanations for hidden layer discussed in Appendix

---

QT2
Training
E
Trick
c
Inference
1
= ())
P(T2IT))
1())T2)
Figure 2: The visualization of the probabilistic explanation for deep learning based on the ML.P
The information theoretic explanations for deep learning
150
151
To address the limitations of existing IB explanations, this section proposes some novel information
152
theoretic explanations for DNNs based on the proposed probabilistic representation.
153
Proposition 1.
The mutual information between fully connected layer and dataset is finite.
T( < co.
(11)
154 Proof: Definition shows Er = (1, N). Thus T is a discrete random variable and (T) oc, thereby ((; T) & (T) < co.
155
Proposition circumvents the infinite mutual information problem. In the absence of a clear definition T: 2t + Er, most previous works [28] 3 | simply viewing the activation ln as the sample of T, namely In, ET R. implies T being continuous and gives rise the infinite mutual information problem in deterministic DNNs. However, (Dr, F, Pr) indicates that i, actually is a variable
157
158
159
180
measuring the correlation between wn, and z rather than the sample of T. namely tn $ Er.
Theorem 2.
The information of Y flows into the MLP in the backward direction during training
Ti + T2 8
(12)
Proof: First, since Qr is defined by ws in (ΩT, F, F. Pr) and Equation (10) shows that w(s + 1) is determined by all the previous gradients (w(s) ) 1==1. and (() randomly initialized and a is
162
163
constant, we can derive that 2r is determined by 6G. Second, based on the back pack-propagation, the
164
185
relation between gradients in two adjacent layers in the MLP = {æ;; b1; l2; 9) is formulated as
dec
[P - (1|2)] 124
Ow KI
0
L Oet
o ( b))
OPc Jwmn
K
ofl(wW ,x)) Fm.
(3)
lim
CE (2)
nk
(2)
1=1 Owl
kl
kl
f2k
k=
Ow
nk
[2)
tin
(13)
186
Equation 13 shows that awis)
8ice is a function of Prix (1|z) and 86c
autt) a function of Oics
awt++1>> where
(3) denotes the weight of ÿ. The two points above enable to derive that Q7, is function of 2r.
188
and fy is a function P(Y|X). Based on Definition 2, we can further derive that T; is a function of T and y is function of Y, ie.., Ti T2 - Y + Y. (See the detailed proof in Appendix C
170
Theorem 2 is consistent with the prevailing explanation for deep learning. LeCunn al. show that deep learning exploits the hicrarchical property of signals [18), i.e., the layers farther from output learn lower-level features, such as edges, whereas the layers closer output assemble lower-level features into the higher-level features corresponding to labels (see Figure 2 in [37]). Notably, since lower-level features commonly exist in signals with different labels (e.g., lower-level features, such
171
172
173
174
175
as the edges of the vehicle frame and the circular contour of wheels, exist in both the car and the truck classes in the CIFAR-10 10 dataset [16] Figure 2). lower-level features do not contain much information of labels. Therefore, the layers farther from output do not have much information of
176
labels, which is consistent with the Markov chain Ti T2 t- Y Y.
5

---

Since all the information of Y stems from X (i.., H(Y I(X;Y) proven in Appendix D) Theorem 2 implies that partial information of X flows into the MLP in the backward direction during training. Equation 2 shows the information of X flowing into the MIPP in the forward direction during inference. Overall, the information of X flows in the backward and forward directions during training and inference, respectively. As a result, the Markov chain, Equation 2 proposed by recent works could not fully characterize the information flow of X in the MLP in each epoch. In other words, (Xx; T is not necessarily greater than / (X;T+++ in the MLP in each epoch.
179
180
182
183
184
186
Equation 2 shows that T, receives the information of Y via X during inference. Theorem 2 shows that T; also directly receives information of Y during training, because the back- poopppppagttiion updates weights (i.e.. S2y,)) ) based on the label Y. Thus Equation 2 cannot fully characterize the information flow of Y in the MLP in each epoch, when we take into account the back propagation training.
187
188
189
190
To fully characterize the information flow in the MLP in each epoch, we introduce Corollary 1. Corollary 1. The information flow in the MIP can be characterized by two Markov chains as
191
X +I + -T +Y
(14)
T
192
The virtual random variable X contains all the information of X except Y, i.e., (X) H(X))).
193
Proof of the first Markov chain: Since X does not have any information of Y, it can only flow into the MLP in the forward direction during inference. Again since X does not have any information of
194
195
Y. the information flow of Y during training will not affect the information flow of X. Therefore,
X Ti - T2 + Y characterizes the information flow of X in both training and inference phases.
196
197
Proof of the second Markov chain: Since the weights are fixed after training, the sample space and the distribution of hidden layers are fixed after training. Therefore, the information of Y transferred into hidden layers during training will retain there after training (i.e., during inference). In addition, Definition indicates that a fully connected layer t = {t, = ((w,, +)) H measures the cross-
198
199
200
correlation between wh and x during inference, thus {w (1) In=1 can be viewed as representation of Z. As result, even though Z has all the information of Y. the information of Y that t can learn
201
202
from Z is determined by how much information of Y the representation {w> (2) J has. Overall, the
203
204
information flow of Y during inference will be the same that during training. Based on Theorem
we conclude that Ti + T2 Y + Y characterizes the information flow of Y in the MLP in both training and inference phases. Detailed derivations and explanations are presented in Appendix E
205
206
To quantify how much information of X and Y is learned by the MLP, we introduce Corollary 2. Corollary 2. The mutual information between dataset and the entire MLP can be expressed as
208
((xTTun T(X; ; + T(Y:Y)
(15)
; TMLP) = {)
209
where TMLP denotes a random variable corresponding to the entire architecture of the MLP.
210
Proof:
Since H(Y)- = I y( (Appendix D). H(X) = H(X) + I(X = H + H(Y). Hence, Corollary 2 can be derived by Corollary and the chain rule. The proof is in Appendix
211
4
Simulations
212
213
In this section, we propose a mutual information estimator based on (Ωr, F, F, Pr) and demonstrate the probabilistic representation and information theoretic explanations for deep learning on a synthetic dataset with known entropy. Additional experiments on benchmark datasets are in Appendix H
214
215
216
4.1
Setup
Mutual information estimator. Based on the definition of mutual information, we have
T(X: -(.,)))(T)))) - (T.
(16)
Previous works simply estimate T(X:T,) HTT)), because T: is assumed to be entirely dependent
219 on X in the Markov chain, Equation 2 thereby XT) O. However, Corollary shows that T;
depends on both X and Y if taking into account the training phase, thereby (T, + 0.

---

ImageD (label 11.
Imagel ilabel []
Image2 liabel [1.Q]
Image3 (aael [0
125
10
66
[C]
Figure 3: (A) the deterministic image á.. Image0 is generated by adding (u, o2) without rotation, Imagel is generated by rotating à along the secondary diagonal direction and adding N(u, a²), Image2 and Image generated by rotating à along the vertical and horizontal directions, respectively, and adding M(, )²)-
Table 1: The number of neurons(nodes) and the activation function in the layers of the MLPs
to
9
x(-)
MLPI
(32 x 32)
ReLU(2) max
MLP2
1024 (32 32)
Tanb(x) (c" e "M(e" +e"*)
MLP3
1024 (32 x 52)
6
ReLU
To accurately estimate (X;; Ti), we need to specify P(T and P(T.). Based on (F,,,,, 222 we formulate Pr,xx( (nl of the three fully connected layers in the MLP as
Prix(()/=)) Zel explo ((wh) x²))), P (N|x²) = 7F2 & (3)
(17)
Pry x (1/x/) = 1
Zry (xp tt (ar))))
223
To derive the marginal distribution P(T:), we sum the joint distribution P(T,, over a E X,
P(T: n) =
Emex m)=E>ee (x = EnSeD Px (x) (n|=') EwED (T(xx( (18)
224 where Px (x') is estimated by the empirical distribution 1/J given D. Finally, we can derive T,)
225 by Equation 16 17 and 18 Similarly, based on the definition of mutual information, we have
I(Y; HTT)) HT).Y).
(19)
228
To estimate H (T))), we reformulate P(T)Y) as
Pray (nl() EmEX Pr,xx (n x) (x+1) NUJ I
Pr.px (n|x)),
(20)
227
where Pxty (x>1) (1) is estimated by the empirical distribution 1/N(1) and (()) denotes the number of samples with the label in D. Finally, we can derive T.Y; by Equation 18
228
19 and 20
Synthetic dataset. The dataset consists of 512 gray-scale 32 x 32 images, which evenly generated by rotating a deterministic image à in four different orientations and adding Gaussian noise with expectation μ (()) and variance o² 1, namely
230
r()) + (N o²), where r(-) denotes the rotation method shown in Figure 3 The reason for adding Gaussian noise is to avoid DNNs directly memorizing the deterministic image. In addition, the binary labels [1,0] and [0,1] evenly divide the synthetic dataset into two classes. As result, the synthetic dataset has (approximately) 2 bits information and the labels have bit information. Compared to popular benchmark dataset
232
234
235
with unknown features and entropy, e.g., MNIST 17 and Fashion-MNIST (36), the features and the
236
237
entropy of the synthetic dataset are clear and known, which enables us examine the probabilistic representation and the mutual information estimator.
238
Neural Networks. We train three MLPs, namely MLP1, MLP2 and MLP3, on the synthetic dataset by a variant of Stochastic Gradient Descent (SGD) method, namely Adam [3] over 1000 epochs with the learning rate 0.03. Table I summarizes the architecture of the three MLPs.
240
4.2
Validating the probability space and the mutual information estimator
We demonstrate the sample space Ωor by visualizing the weight of the eight neurons in b,, iee. 1024 = {won 1 mm1. in 5 different epochs (i.e., 124, in Figure 4 (Left). As training continues, we observe that w( quickly learns all the spatial features of the synthetic dataset. For
243
244
245
246
instance W2 has low magnitude at top-left positions and high magnitude at bottom-right positions,
(1) which correctly characterizes the spatial feature of Image0. Similarly, w w (1), and wg))) correctly characterize the spatial feature of Imagel, Image2, and Image3 in Figure 3 respectively.
247
248
We only show the learned weights in MLP1 because we observe that the learned weights in MLP1 and MLP2 are very similar, though they different activation functions.
7

---

0.3
(1)
H(X1
1 M
MLP1
1.5
MLP2
1.0
MLP3
1000 128
-0.1
0.5
-0.2
-0.3
w
w
w
4
10"
10c
10c
102
Figure 4: (Left) The eight features (w
I==1 leamned by the weights of the eight neurons in 5 different epochs
fwom Im=1 are reshaped into 32 x 32 to show the spatial structure. (Right) The variation of (X;Ti) in the MLPI, MLP2, and MLP3 during 1000 epochs.
(i.e 0,1,4,18,, 1000), where w(y)
Table 2: The Gibbs probability Pp. (wn
[mmage0] in MI.1 and MI.P2 in the 1000 epoch
wf""
w/")
w
wg"
w6
wg")
(w> at)
-63.6
208.8
-181.6
45.1
-55.6
157.5
-210.0
-30.1
(xx)
0.0
208.8
0.0
45.1
0.0
157.5
0.0
0.0
exp Iin (ar)
1.0
4.79e+90
1.0
3.86c+19 19
1.0
2.51e+68
1.0
1.0
Piect
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.0
-1.0
1.0
-1.0
1.0
-1.0
1.0
-1.0
-1.0
exp exp [x)))(())))
0.36
2.71
0.36
2.71
0.36
2.71
0,36
0.36
25 0.037
0.272
0.037
0.272
0.037
0.272
0.037
0.037
f1 (1z) = a Tinh (1o))) (1)
21) and (a)) = ""ol"((("" .x)) are the activations given the same (w/) xx).
We demonstrate that P(T (X) correctly measures the probability of {wi [W[)) Jn=1 being recognized the
249
feature with the largest cross-correlation to x in Table 2 For instance, W correctly characterizes
250
the feature of Image0 and has the largest cross-correlation (wo ..) 190.8, thus it has the largest probability PR ( [Image0)- = 1.0 being recognized as the feature with largest cross-correlation to Image0. In contrast, since w/ incorrectly characterizes the feature of Image0 and has the lowest cross-correlation (w) (1) .) = 2200.0, so it has the lowest probability Ti|X (wo? [ImageO) = 0.0
251
(1)
252
(1)
254
255
being recognized as the feature with largest cross-correlation to Image0.
We observe that activation function (abbr. ACT) plays important role in the distribution. Specifically, ReI.U, a non-saturating (unbounded) ACT [9]. preserves the positive cross-correlatoons
257
while resets all the negative ones as zero. PReLU (w² (1) [Imaee0) = 1.0 shows that ReLU derives the
259
correct probability of w/ being recognized as the feature with largest cross- pross-correlation. In contrast, though w/"h) has stronger cross-correlation to Image0 than w i.e., (w)") xx) > (1 ,x),, Tanh, a saturating (bounded) ACT, derives S (x) = /Tan (x) = 1.0, and makes w1 to incorrectly have the same probability 0.272 to w being recognized as the feature with the largest corsecoorel to Image0, i.e., Tanh hinders & from correctly recognizing the features of input. The simulations for validating the probability space based on other synthetic images presented in Appendix G
260
261
263
285
To validate the mutual information estimator, we follow recent works (30 28
train the three MLPs with 50 different random initialization and study the average mutual information. Figure 4 quickly increases 1.81 and keeps stable in the MLP1, i.e., t1 learns most information of the dataset as (XX)= = 2.0. Notably, the result is consistent with the variation of the weights in Figure 4 (Left), which shows that the weights correctly characterize the features of the dataset and keeps stable after the fourth epoch. As a comparison, we observe that I(X; Ti) keeps stable at 0.44 in the MLP2, which confirms the statement that Tanh hinders i from correctly recognizing the features of input. In addition, Figure (Right) shows that T(X; ss 0.79 in MI.3 is smaller than Ti se 1.81 in MLP1, which is consistent with Definition 1, i.e., a layer with fewer neurons would represent fewer possible features, thus it contains less information.
286
267
(Right) shows that T [Xx; TT)
268
269
271
273
275 In summary, we demonstrate the probability space (S2, F, Pr) and show that if an ACT cannot preserve the cross-correlation between (eights(features) and input, it would distort the distribution of a layer, thereby affecting the mutual information between the layer and data/labels. In addition,
276
277
278
we show that the proposed mutual information estimator outperforms the existing non-parametric
279
models, e.g., empirical distribution (30 and KDE (28), based on the synthetic dataset. Especially,
activations do not satisfy the i..dd. prerequisite of non-parametric models is an important reason for
281
non-parametric models deriving inaccurate mutual information in DNNs. Due to limited space, the
experimental comparison and study of non- non-parametrie models are presented in Appendix G
8

---

MLP1(8 6-2 RoLU)
MLP2(8-6-2 Tanh))
MLP3(2-6 6-2 ReLU)
(0:TT)
2.0
15-
1.5
(X: Fa)
155
RX: Tal
105
(2(1)
NX. 9
T2: Tal
0.5
90:9)
0.5
0.5
0.2
0.3
0.0
107
10
102
10a
10f
10
10a
10.
10P
10'
10
10a
1.00
"he 4o
0.2
(()TT) T2)
0.4
NU
15 T2)
0.50
0.1
()
0.2
91
0.0
0.00
0.0
10a
10A
100
10a
10F
102
10a
10a
10P
150
107
10a
1.00
0.2
0.8
0.5
0.50
--- (P:TO
0.6
NY: 75)
RY Tc)
0.-
(9:9)
10:9
0.4
RY.9 9)
0.00
1.0
10a
10
30*
10F
102
107
1.0
1.0
10
107
10*
2.0
2.0
2.0
1.5
1.5
1.0
0.5
10C TaLr)
I0Y: Touel
0.5
(XC: Time
(CY: Tuce?
0.5
IU) TAL
/1Y: Tuas
0.0
0.0
100
10r 107
30
102 107
10
10P
101
107
10"
Figure 5: All the x-axis index training epochs. In each column, the first three figures show 1 [(x; T:), II);
and I() respectively. The forth figure shows [(x; Trmp) and ;YY; in MLP. The pink line denotes H(Y)= = 1.0 and the orange line denotes (XX) 2.0.
283
4.3
Validating the information theoretic explanations for DNNs
284
In Figure 5 we observe I(X; Ti) < <(X;Y) in MLP2 and MLP3, which confirms that the Markov chain proposed by previous works, Equation (2), cannot fully explain the information flow in MLPs, if taking into account the back-propagation training. As a comparison, the second and third row show I (i) >
205
12; 2 [(X; Y) and (Y; S. 72) 2 (Y; in all the three MLPs, which validates that Corollary I, i.e., Equation (14) characterizes the information flow in MLPs.
286
289
Figure 5 demonstrates that different hidden layers achieve different IB trade-offs depending on the architecture and the position of the layers in MLPs. In terms of architecture, I(Y > 0.8 and T;) > 0.75 in MLP1 indicate that t1. with ReLU, achieves a good prediction without much compression, whereas I(Y; Ti) 0.5 and I (;) < 0.1 in MLP2 show that t,, with Tanh, achieves a different IB trade-off. In addition, T(Y; Ti) Ri 0.45 and T(X; Ti) x 0.25 in MLP3 show the effect of neuron numbers on the IB trade-off. In terms of position, I(Y;Y) and ((X;)) 0 in MLP1 means that ý has a different IB trade-off to t1 in MLP1.
290
291
292
295
296
We demonstrate that a MLP satisfies the IB principle no matter what the architecture of the MLP is. Figure 5 visualizes T(X; TMP)) and T(Y:TMM.P) TMIP)) based on Corollary 2. It shows that all of three
297
MIPs satisfy the IB principle, namely (TIP)) < H(X) = 2 and T(Y; PTMP)) = HYY = 1,
299
though they have different architectures. Importantly, in contrast to previous work (28 claiming that
the compression not exists in DNNs with non-saturating ACT, such as ReLU, Figure 5 clearly shows
301
that the compression exists in all the MLPs, no matter the activation function of MLPs.
302
We further demonstrate the information theoretic explanations for DNNs on the benchmark MNIST and Mishion-MNIST datasets. The experiments are presented in Appendix H
303
Conclusion and future work
304
305
In this work, we (1) specify the probability space for a hidden layer for (2) accurately estimating the mutual information and (3) clearly explaining how the components of the layer affect the mutual
306
307
information. We take into account the pooe-propggttion training and derive two novel Markov chains
IB principle no matter the architecture of the DNN. In contrast, different hidden layers show different 308
to characterize the information flow in DNNs. Furthermore, we demonstrate that DNN satisfies the
310
IB trade-offs depending on the architecture and the position of the layers in DNNs. A potential direction is to study the generalization of DNNs based on the probabilistic representation.

---

References
312
[1] Rana Ali Amjad and Bernhard Claus Geiger. Learning representations for neural network-based classi- lication using the information bottleneck principle. IEEE transactions on pattern analysis and muchine intelligence, 2019.
314
315
316
[2] George Casella and Roger L. Berger. Statistical inference. Cengage Learning, 2021.
317
[3] Ivan Chelombiev, Conor Houghton, and Cian O' D'onnell. Adaptive estimators show information compres- sion in deep neural networks. In International Conference on Leurning Representations. 2019.
318
319
[4] Thomas Cover and Joy Thomas. Elements of Information Theory. Wiley-Interscience, Hoboken, New Jersy, 2006.
Balázs Csanád Csáji al. Approximation with artificial neural networks. Faculty of Sciences, Ervs Lordd University, Hungary, 24(48):7. 2001.
322
323
[6] Rick Durrett. Probability: theory and examples, volume 49, Cambridge university press, 2019.
324
[7] Marylou Gabrié, Andre Manoel, Clément I Luneau, Nicolas Macris, Florent Krzakala, Lenka Zdeborová, et al. Entropy and mutual information in models of deep neural networks. In Advances in Neural Information Processing Systems, pages 1821 1821-1831, 2018.
325
[8] Matt W Gardner and SR Dorling, Artificial neural networks (the multilayer perceptron) review of applications in the atmospheric sciences. Atmosphrric environment, 32(14-15),227-26336 1998.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256, 2010.
330
331
332
[10] Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in deep neural networks. In Proceedings of the 36th International Conference Machine Learning, volume 97, pages 2299-2308, 2019.
333
[11] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
338
337
[12] Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. Conference on Learning Theory, pages 2306-2327. PMLR, 2020.
338
339
[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv: 1412.6980, 2014.
[14] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pages 1885-1894. PMLR, 2017.
[15] Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. arXiv preprint arXiv: 1808.07593, 2018.
344
345
[16]
Alex Krizhevsky. Learning multiple layers features from tiny images. Technique Report, 2009.
346
[17] Y. LeCun, BE. Boser, and JS. Denker. Handwritten digit recognition with a back poocpprrpagttion network. In NeurIPS, pages 396-494, 1990.
347
348
[18] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
349
[19] Yann LeCun, Sumit Chopra, Raia Hadsell, Marc' Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy-based learning. MIT Press, 2006.
[00] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz. Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In ICLR, 2018.
352
353
[21] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them.
354
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1188-,,,,,
355
2015.
356
[22] David McAllester and Karl Stratos. Formal limitations the measurement of mutual information. In International Conference Artificial Intelligence and Statistics, pages 875-888 PMLR, 2020.
10

---

358
[23] Julian D Olden and Donald A Jackson. Illuminating the "black box": randomization approach for understanding variable contributions in artificial neural networks. Ecological modelling. 154(1-2):35-115. 2002.
360
[44]
Liam Paninski. Estimation of entropy and mutual information. Neural computation. 15(6):1191-1253 2003.
362
[25]
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- propagating errors. Nature, 323:533 536, October 1986.
384
[26] Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics. pages 445-455. PMLR, 2009.
386
[27] Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intelligence: Under- standing, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708. UrXiv:1708.08296, 2017.
388
[28] Andrew Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Tracey, and David Cox. the information bottleneck theory of deep learning. In International Conference on Representation Learning, 2018.
370
371
372
[29] Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with information bottleneck. Theoretical Computer Science, 11112993)2266--7711, 2010.
373
374
[30] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 1003,00010 00010, 2017.
375
378
[3]]
Noam Slonim. The information bottleneck: Theory and applications. PhD thesis, Citeseer, 2002.
[32]
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0000077, 2000.
378
379
[33] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pages 1-5. IEEE, 2015.
380
[34] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions neural networks, 10(5)-888-999, 1999.
382
383
[35] Larry Wasserman. All of nonparametric statistics. Springer Science Business Media, 2006.
[36] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
385
[37] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818 833, Springer, 2014.
387
[38] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 13.0000 2016
389

---

Checklist
390
391
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
385
394
(b) Did you describe the limitations your work? [Yes] see Section 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
397
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
389
(b) Did you include complete proofs of all theoretical results? [Yes]
400
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material as URL)? [Yes] the URL in Appendix [
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] see Section 441 Appendix and Appendix H
(c) Did you report multiple times)? [Yes] see Section
bars (c.g.. with respect to the random seed after running experiments
408
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] see Appendix
408
409
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
410
(a) your work uses existing assets, you cite the creators? [Yes]
(h) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes] Appendix(H]
413
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [N/A]
415
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
417
5. If you used crowdsoureing or conducted research with human subjects...
419
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
420
[N/A]
421
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
422
423
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
424
12


Reviews:
[{"review_id": "lcsnJEF4FhP", "reviewer": "Reviewer_3nVv", "summary": "The paper points out that the IB principle based on the Markov chain assumption ($Y \\rightarrow X \\rightarrow T_1 \\rightarrow \\cdots \\rightarrow \\hat{Y}$) cannot fully characterize the information flow in DNNs, and hence several claims previously made on the assumption may not be accurate. For better quantification of the mutual information, this paper proposes a probabilistic framework of IB in deep neural networks (DNNs) by taking into account the back-propagation training. Under the probabilistic framework, the paper derives two Markov chains that characterize the information flow in DNNs and provides experimental evidence for supporting the validity of the Markov chain assumptions. The paper claims that the probabilistic approaches allow us to more accurately estimate the mutual information than existing non-parametric approaches and experimentally demonstrates how different activation functions and hidden layers achieve IB trade-offs. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a novel probabilistic framework for understanding IB principles and estimating the mutual information in DNN. The probabilistic view provided by the authors is very interesting and convincing. The claims are well supported by theoretical and experimental evidence. The only concern is on the clarity. This is a dense paper with a lot of mathematical proofs and experimental results, so improved clarity will help readers to understand the idea and findings of this paper as well as its limitations. See details below.  \n\nMajor\n- L63-66: Please specify clearly which claims are theoretically or experimentally supported in the introduction. Are those claims \"different hidden layers achieve different IB trade-offs\", and \"MLP satisfies IB principle no matter what the architecture of the MLP is\") supported by theoretical or experimental evidence?\n- L81-87: It is not clear why this paragraph (generalizing CNN to MLP) is needed to explain that \"the weights of a neuron can be viewed as a global feature, ... .\"  If this is a rationale for the probabilistic representation, please consider re-write it.\n- L129-130: Please provide a detailed rationale for Definition 2. I'm guessing this is because of the infinite mutual information problem. How can we use this definition without loss of generality? \n\nMinor\n- L45: Please also provide the subsection number. (i.e. Appendix G -> Appendix G.3)\n- L63: What is \\overline{X}?\n- L103: Please provide the definition of $f_n$. \n- L104: $n$th -> $n$-th\n- L186, L228: Be consistent in referring equations (i.e. Use only one style between \"Equation (2)\" and \"Equation 2\" ) ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors didn't address the limitations of their work in the conclusion section. The scope of claims should be specified clearly.  \n- Please specify the assumptions made for the probabilistic framework.\n- Please specify claims that are only supported by evidence.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "ZMGrVEGbe8m", "reviewer": "Reviewer_e1nM", "summary": "This work introduces a novel probabilistic formulation for Deep Neural Network layers to circumvent the problem of estimating mutual information between continuous variables. The authors characterize two different directions for the information flow during training and inference time, empirically analyzing and validating their theory on a simple task.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses the relevant problem of analyzing and characterizing the information flows in deep neural networks, by considering the Boltzman distribution on the activation of each layer and factorizing the dependencies in the backward training parameter update step.\nSeveral parts of the paper are not entirely clear and not all the statements are strongly supported or sound. \n\nHere I summarize my main concerns:\n\n1) The discrete variable $T_i$ for each layer is introduced as a function of the activations $\\bf t_i$, but clearly the activations of layer $i+1$ depend on $\\bf t_i$ rather than the categorical variable denoting the \"maximum feature correlation\" $T_i$, which does not contain the entirety of the information regarding the activations. The corresponding graphical model follows the following chain of dependencies:\n$$\\begin{matrix} X & \\to  & {\\bf t}_1  & \\to  & {\\bf t}_2 & \\to  & \\hat{Y} \\\\\\\\  &  & \\downarrow & & \\downarrow \\\\\\\\ & & T_1 & & T_2\\end{matrix}$$\nTheorem 1 and equation 8 factorize $p(\\hat{Y},T_2,T_1|X)$ as $p(\\hat{Y}|T_2)p(T_2|T_1) p(T_1|X)$ without considering the dependencies $X\\to T_2$ and $X\\to\\hat{Y}$. Why are these dependencies not considered?\n\n2) Since $T_i$ is defined based on ${\\bf t}_i$, as a consequence of the data processing inequality, any evaluation of $I(X; T_i)$ represents only a lower bound of the actual measure of information $I(X;{\\bf t}_i)$ flowing through the layer of interest.\n\n3) The paper claims that $I(X; T_i)$ is generally unbounded in the continuous case, but provides estimators based on the empirical distribution $P(X=x) = \\frac{1}{|\\mathcal{D}|}[x\\in\\mathcal{D}]$, which is discete by definition. In this case ,the empirical entropy is finite $H(X)=\\log|\\mathcal{D}|\\le \\infty$ and so is the mutual information in analysis $I(X; T_i)\\le H(X) = \\log|\\mathcal{D}|$.\n\n4) The main text does not contain a clear definition of $\\bar{X}$ and $f_n$ (Line 103 and 192). The term \"dataset\" is used to refer to either $\\mathcal{D}$ or $X$ in different sections.\n\n5) The paper is generally quite notation-heavy, which makes it difficult to read in some sections.\n\n6) The experimental section in the main text is not extensive enough to be significant or convincing\n\nMinor comments:\n\n- Figure 1 is difficult to interpret and does not add much to the main text of the paper.\n- Line 106: $w_n$ used instead of $\\omega_n$", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not clearly state the limitations of the proposed method.\n\nthe paper has a prevailing theoretical component, therefore the societal impact is not directly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5 h"}, "extra_text": {}}, {"review_id": "Wo3xtUGuCk_", "reviewer": "Reviewer_hXwD", "summary": "The paper presents a new probabilistic model for studying information flows in DNNs. To that end two Markov chains connecting the network's input, hidden layers, and output are introduced: (i) the forward chain models the conditional distribution of each layer given its input as a Gibbs distribution over the layer's weight vectors (the Hamiltonian is given by a non-linearity applied of dot products between weights and inputs); (ii) the backward chain, which is discussed much less, tries to capture the correlations introduces by BP. The authors show how to compute mutual information terms under their probabilistic model and provide some synthetic experiments under the proposed framework.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The general idea of formulating a probabilistic model under which information flows in DNNs may be studied (as opposed to the statistical route based on estimation from samples) is solid. However, I am afraid that the execution presented in this paper is sub par. For such a newly proposed model to be adopted the formulation must be described in a crystal clear manner and be well motivated by theory and empirics. Unfortunately, the paper falls short on both aspects, as is elaborated below. As a result, I cannot recommend acceptance. \n\nMy main concerns (along with some smaller ones) are as follows:\n\n1. The technical writing is sub par. The authors' choice to emphasize on the probability spaces corresponding to each layer does not make sense to me. They should just describe the random variables of interest, their distributions, supports, correlations, etc. Sample spaces and $\\sigma$-algebras can be left implicit for several reasons. First, one can always find a common probability space over which all random variables are defined (e.g., any random variable with values in a polish space can be generated from a uniform (0,1) variable, so one only has to assume that this underlying space is 'rich enough' to support such a uniform variable). More importantly, the probability spaces $(\\Omega_{T_i},\\mathcal{F},P_{T_i})$ does not encode the entire probabilistic model since the latter is defined through conditional probabilities that are absent here.\n\n2. Continuing from the above, the validity of the proposed model and its relation to the original IB formulation is unclear to me. The IB paradigm views $T_i$ as the vector of hidden representations (neuron values) corresponding to layer $i$. Here the authors think of $T_i$ as the weight vector whose correlation with the layer's input is highest. I can't see why this is a reasonable definition, and more importantly, its relation to the IB formulation is questionable.  This deviation from the IB principle is problematic given that the proposed model is motivated by it. \n\n3. The Markov chain formulation is also unclear. First, if $X \\to Y \\to Z$ forms a Markov chain then so does $Z\\to Y\\to X$, so the discussion of the chain's directionality should take this into account. Namely, it is more a question of which conditional distributions are specified and not of whether there is a Markovity relation in a given direction. Since the authors discuss both directions of the chain, do we know that the conditional distributions correspond (i.e., give rise to the same joint measure)? Say we have $Y \\to X \\to T_1\\to \\hat{Y}$ with law $P_{X,Y,T,\\hat{Y}}=P_{X,Y}P_{T|X}P_{\\hat{Y}|T}$ and $Y\\to \\hat{Y} \\to T$ with law $Q_{Y,T,\\hat{Y}}$. Do we know that, e.g., $P_{Y,T,\\hat{Y}}=Q_{Y,T,\\hat{Y}}$? If the authors mean that there are two phases of updates (forward and backward), then they should define more variables to reflect that. The fact that this is still unclear to me after reading the paper testifies that writing (and possible the model itself) must be improved\n\n4. The paper lacks in depth as far as the provided claims (Theorem 1, Proposition 1, etc.). These are trivial consequences of the proposed model. Some claims seem somewhat exaggerated, e.g., the proof in Appendix D that $H(Y)=I(X;Y)$. There is nothing to prove here as, given $X=\\mathbf{x}^j$, the authors just take $Y=\\ell \\mathbf{1}_{\\{y^j=\\ell\\}}$. This trend of overcomplicating simple observations repeats throughout the entire text.\n\n6. The authors say their model alleviates the infinite mutual information issue. While this is true, it is achieved by trivializing the problem. The authors implicitly assume that the data distribution is supported only over the samples $\\{\\mathbf{x}^j\\}$, in which case clearly $I(X;Y)\\leq H(X)<\\infty$. There is no probabilistic model for $P_X$ outside the sample set. Consequently there is no sense in which the proposed model can account for generalization, which is a key aspect to statistical learning theory.\n\n7. The authors wait too long before they describe the proposed model. Vague descriptions are provided from the abstract through the introduction, but nowhere in the intro is the model actually described (even in words). There is a paragraph on page 2 that talks about the probability spaces attached to each layer, but as mentioned above this is quite redundant and doesn't clarify the probabilistic model. The model is described first only after Def 1 on page 3, which is too late. Mention of the Gibbs measure, its Hamiltonian, the induced Markov chains, etc. should be provided earlier in the text to improve readability. \n\n8. The paper is tedious to read. The technical descriptions are swamped in verbal and heuristic statements, making it hard to extract the message being conveyed. Because of that I had to reread many parts several times. The authors should clean up and streamline the text. This would make it easier to understand the proposed model and appreciate its potential validity. I am not providing specific examples since there are just too many---the entire text needs work.\n\n9. In Equation (10), do the authors mean conditional KL divergence (i.e., given $X=x$) or averaged one over the samples? The dependence of the KL term on the weights should be made explicit. \n\n10. The proof outline for Theorem 2 is confusing. It uses undefined notation such as $\\frac{\\partial\\ell_{\\mathrm{CE}}}{\\partial\\omega}$ and does little to explain where the statement comes from. The authors should either shape it into a standalone argument (statements such as `it can be shown' should be shown or at least explain further) or omit it from the main text. \n\n11. I can't see how the experiments validate the applicability of the proposed theory. The authors compute the proposed objects but whether these in fact describe real world DNNs remains unclear. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The direction suggested by this paper is potentially interesting, but there was substantial agreement (not consensus) among the reviewers that this paper is not ready for publication. Overall, the reviewers felt (even ater substantial author-reviewer discussion) that the paper is hard to understand, the overall experiments are unclear, and the experiments are not compelling.
Decision: Reject