Where is the Information in a Deep Network?
Authors: Alessandro Achille
Stefano Soatto
Venue:  (2020)
Keywords: Information
Learning Dynamics
PAC-Bayes
Deep Learning

Abstract: Whatever information a deep neural network has gleaned from past data is encoded in its weights.  How this information affects the response of the network to future data is largely an open question.  In fact, even how to define and measure information in a network entails some subtleties. We measure information in the weights of a deep neural network as the optimal trade-off between accuracy of the network and complexity of the weights relative to a prior.  Depending on the prior, the definition reduces to known information measures such as Shannon Mutual Information and Fisher Information, but in general it affords added flexibility that enables us to relate it to generalization, via the PAC-Bayes bound, and to invariance.  For the latter, we introduce a notion of effective information in the activations, which are deterministic functions of future inputs.  We relate this to the Information in the Weights, and use this result to show that models of low (information) complexity not only generalize better, but are bound to learn invariant representations of future inputs. These relations hinge not only on the architecture of the model, but also on how it is trained.

Under review as a conference paper at ICLR 2019
WHERE IS THE INFORMATION IN A DEEP NETWORK?
Anonymous authors
Paper under double-blind review
ABSTRACT
Whatever information a deep neural network has gleaned from past data en- coded in its weights. How this information affects the response of the network to future data is largely an open question. In fact, even how to define and mea- sure information the weights of deep neural network as the optimal trade-off between accuracy of the network and complexity of the weights relative to a prior. Depending on the prior, the definition reduces to known information measures such as Shannon Mutual Information and Fisher Information, but in general it affords added flexi- bility that enables to relate it to generalization, via the PAC-Bayes bound, and to invariance. For the latter, we introduce a notion of effective information in the activations, which are deterministic functions of future inputs. We relate this to the Information in the Weights, and use this result to show that models of low (in- formation) complexity not only generalize better, but are bound to learn invariant representations of future inputs. These relations hinge not only on the architecture of the model, but also how it is trained.
network entails some subtleties. We measure information
1 INTRODUCTION
At the end of training deep neural network, all that is left of past experience is set of values stored in its weights. So, studying what "information" they contain seems like a natural starting point to understand how deep networks learn.
But how is the information in a deep neural network even defined? The weights are not a ran- dom variable, and the network outputs deterministic function of its input, with degenerate (infi nite) Shannon Mutual Information between the two. This presents a challenge for theories of Deep Learning based on Shannon Information (Saxe et al., 2018). Several frameworks have been devel- oped to reason about information in fixed sets of values, for instance by Fisher and Kolmogorov, but they either do not relate directly to relevant concepts in Deep Learning, such as generalization and invariance, or cannot be estimated in practice for modern deep neural networks (DNNs).
Beyond how they define information, existing theories of Deep Learning are limited by whose infor- mation they address: Most approaches focus information of the activations of the network the output of layers - rather than their parameters, or weights, although recent information-thent -hhooeetcc approaches to study the weights are discussed in the next section. The weights are representation of past data (the training set of inputs and outputs), trained for predicting statistics the training set itself (e.g., the output), relative to prior knowledge. The activations are representation of (possibly unseen) future inputs (test set), ideally sufficient to predict future outputs, and invariant to nuisance variability in the data that should not affect the output. We have access to future data, and the Shannon Information their representation contains does not account for the finite training set, hence missing a link generalization.
But how are these properties of sufficiency and invariance achieved through the training process? Sufficiency alone is trivial any invertible function of the data is, in theory, sufficient but comes at the expense of complexity (or minimality) and invariance of the representation. Invariance alone is similarly trivial - any constant function invariant. A learning criterion therefore must trade off accuracy, complexity and invariance. The best achievable complexity trade-off is what we define
In this paper, we refer complexity as information complexity, to be distinguished from complexity of the hypothesis space, for instance measured by the VC Dimension.

---

Under review as a conference paper at ICLR 2019
as Information for the task. The challenge is that we wish to characterize sufficiency and invariance of representations of the test data, while we only have access to the training set.
So, throughout this paper, we discuss four distinct concepts: (1) Sufficiency of the weights, captured by a training loss (e.g., empirical cross-entropy); (2) complexity and minimality of the weights, captured by the information they contain; (3) sufficiency of the activations, captured by the test loss which we cannot compute, but can bound using the Information in the Weights; (4) invariance of the activations, property of the test data, which is not explicitly present in the formulation of the learning process when training a deep neural network. To do all that, we first need to formally define both information of the weights and of the activations.
1.1 SUMMARY OF CONTRIBUTIONS AND RELATED WORK
Our first contribution is to measure the Information in the Weights of a deep neural network as the trade-off between the amount of noise we could add to the weights (measured by its entropy relative prior), and the performance the network would achieve in the task at hand. Informally, given an encoding algorithm, this is the number of bits needed to encode the weights in order to solve the task some level of precision, as customary in Distoristt Theory. The optimal trade-off traces a curve that depends on the task and the architecture, and solutions along the curve can be found by optimizing an Information Lagrangian. The Information Lagrangian in the general form of an Information Bottleneck (IB) (Tishby et al., 1999). but is fundamentally different from the IB used in most prior work in deep learning (Tishby & Zaslavsky, 2015), which refers to the activations, rather than the weights. Our measure of information is practical even in large-scale networks, with millions of parameters, and retains the dependency on the number of samples in the training set.
Our second contribution is to derive relation between the two informations (Section 4), where we show that the Information Lagrangian of the weights of deep networks bounds the Information Bottleneck of the activations, but not vice-versa. This is important, as the IB of the activations is degenerate when computed on the training set, hence cannot be used at training time to enforce properties. On the other hand, the Information Lagrangian of the weights remains well defined, and through our bound it controls invariance at test time.
Our method requires specifying a parametrized noise distribution, as well as a prior, to measure information. While this may seem undesirable, we believe it is essential and key to the flexibility of the method, as it allows us to compute concrete quantities, tailored to DNNs, that relate gener- alization and invariance in novel ways. Of all possible choices of noise and prior to compute the Information in the Weights, there are few standard ones: An uninformative prior yields the Fisher Information of the weights. A prior obtained by averaging training over all relevant datasets yields the Shannon mutual information between the dataset (now a random variable) and the weights. A third important choice is the noise distribution induced by stochastic gradient descent (SGD) during the training process, which we discuss in the paper.
As it turns out, all three resulting notions of information are important to understand learning in deep networks: Shamnon's relates closely to generalization, via the PAC-Bayes Bound (Section 3.1). Fisher's relates closely invariance in the representation of test data (activations) as show in Section 4. The noise distribution of SGD is what connects the two, and establishes the link between invariance and generalization. Although it is possible to minimize Fisher or Shannon Information independently, we show that when the weights are learned using SGD, the two are related. This our third contribution, which is made possible by the flexibility of our framework (Section 3.3). Finally, in Section 5 we discuss open problems and further relations with prior work.
There is growing literature on information and generalization bounds for the weights of deep networks (Xu & Raginsky, 2017: Pensia et al., 2018). Given a data generating distribution D ~ ((x,)) y). a training algorithm w = A(D) is said be (c. (c.ju)-information stable if T(wr; D) < 6. The generalization gap of training algorithm can then be bounded in terms of its information stability. Indeed, the quantity I (w; D) is related to our definition of information in the weights (Section 3.2), but we emphasize that our general definition of amount of information in the weights extends to the case where both the dataset D and w are assumed to be given and fixed (as it is often common in Deep Learning,, and not resampled every time. First, some of main results are to prove that convergence to flat minima (low Fisher Information) and "path" stability of SGD (Hardt et al., 2015) imply "information" stability in the sense of Xu & Raginsky (2017) (Proposition 3.7). Unlike those
2

---

Under review as a conference paper at ICLR 2019
works, our bound does not depend solely the noise induced by the steps of SGD, but also the geometry of the loss lanscape, which allows to better capture some fundamental properties of Deep Learning. Second, while those work only bound the generalization performance the training task, we connect the information stability with amount of information the activations of a DNN and invariance to nuisances (Section 4). This this can be used to guarantee the quality of the learned representation in transfer learning setting.
Note that the fact that some weights can be perturbed at little loss has been known empirically for while (Hinton & Van Camp, 1993). We exploit this property to define information for particular set of weights, in a manner that is quite distinct from standard PAC-Bayes, using Fisher's information instead.
This paper takes Achille al. (2019) as the starting point of investigation, attempting to measure the quantities at play more accurately, which leads us beyond Shannon's 's formalism, to a more general setting that also includes Fisher's 's formalism and the relation between the two, mediated by the prop- erties of deep neural networks and SGD. All the (information) quantities we measure are specific to a particular weight vector, not its distribution.
PRELIMINARIES AND NOTATION
We denote with E X an input (e.g., an image), and with y Y "task variable," random variable which we are trying to infer, egg. a label Y = II....,C.. A dataset is finite collection of samples D (( v)) that specify the task. A DNN model trained with the cross-entropy loss encodes conditional distribution Pw (y(z), parametrized by the weights w. meant to approxi- mate the posterior of the task variable y given the input æ. The Kullbach-Liebler
KL-divergence, is the relative entropy between p(z) and q(x): KL [] ((x)) = Be--(t)) (((((((()/))) It is always non-negative, and zero if and only if p(x) = ((x). It measures the (asymmet- ric) similarity between two distributions. Given family of conditional distributions pie (y/x² parametrized by a vector w, we can ask how much perturbing the parameter w by a small amount ow will change the distribution, as measured by the KL-divergence. To second-order. this is given 5w Fdw + o(()) where F is the Fisher Information Matrix (or simply "Fisher"), defined by F = Pa.rr-(())p () (V log (w 3' log Pre (y|z)]
by Ez KL, Pio (1/2:) || Pwr+& (y/x))
pewp(s)p (yla)) [ V notice that the Fisher depends on the ground-truth data distribution p(r, y) only through the domain variable v, not the task variable y. since y ~ Pw (y)aa) is sampled from the model distribution when computing the Fisher. This property will be used later.
ur log (y]lx]. For its relevant properties see Martens (2014). It is important to
Given two random variables I and 2, their Shannon mutual information is defined as T(x; 2) Ezwp(e) [p([p())) p(2) ]] that is, the expected divergence between the distribution of z after an observation of and the prior distribution of z. It is positive, symmetric, zero if and only if the variables are independent (Cover & Thomas, 2012), and measured in Nats when using the natural logarithm.
In supervised classification one is usually interested in finding weights w that minimize the cross- entropy loss [.)ww) [( log usually minimized using stochastic gradient descent (SGD), which updates the weights w with an estimate of the gradient computed from a small number of samples (mini-batch). That is, Wk+1 = Wh V (w). where εN are the indices of a randomly sampled mini-batch and Lea (w) 1 E |- Pw Pw (yi(z;)]. Notice that Egg LVLg (w)] = VLD(w), (w), so we can think of the mini-batch gradient VLg. (w) as noisy version of the real gradient. Using this intuition we can write:
Pw Po [y]x:)] on the training set D. The loss Lo(w) is
We+1 = Wy V + /n Te (wne)
(I)
with the induced "noise" term Te, (w) v (VLç. (w) VL(W)). Written in this form, eq. (1) is a Langevin diffusion process, with (non-isotropic) noise Te, Li et al. (2017); Chaudhari & Soatto (2018).
3

---

Under review as a conference paper at ICLR 2019
3 INFORMATION IN THE WEIGHTS
One could define the Information in the Weights as their coding length after training. This, however, would not be meaningful, as only small subset of the weights matters: If perturbing given weight configuration w (w' < w + ow) were to yield no change in the cross-entropy loss (i.e. Lp(w') x LD(w)), one could argue that such weights contain "no information" about the task. Storing those weights with low precision, loss. On the other hand, if slightly perturbing a configuration of weights were to yield a large increase in the loss, one could argue that such weights are very "informative," and store them with high precision. But what perturbations should one consider (e.g., additive or multiplicative)? And how "small" should they be? What distribution should the perturbations be drawn from? To address these issues, we introduce the following definition:
pruning them, or randomizing them, would yield no performance
Definition 3.1 (Information in the Weight). The complexity of the task D at level B, using the post- distribution Q(w(D) (D) and the pre-distribution P(w). is
Cg(D; P. Q) = (LL-
+ 8 SKL ((wL)) II P(
(2)
Information in the Weights
is the reconstruction error under the distribution Q(w (D); KL Qu (w([D) P(w) ) measures the entropy of Q(w(D) relative to the P(w). If (w[D) minimizes eq. (2) for a given B, we call KL( Q* (D) || P(w)) the amount of Information in the Weights for the task 'D at level B (or "Information in the Weights," " when the meaning is clear from the context).
where (L
Note that the definition of information is based on the loss Lo on the training set, which depends on the number of samples in D, and does not require access to the underlying data distribution. We call (D) a "post-distribution" because it is picked after seeing the dataset D. We do not call it "posterior" to emphasize that it is an arbitrary distribution and does not have any Bayesian interpretation. Similarly, P(w) is an arbitrary "pre-distribution" distinct from Bayesian "prior," picked before the dataset seen. Having stressed the arbitrary nature of P(w) and Qw) from now on with simply call them prior and posterior for simplicity. We also refer to the variability implied by Q as "noise" even though the mechanism by which it acts could be deterministic.
Special cases of (2) include the case 3 = 1, when eq. (2) formally coincides with the evidence lower-bound (ELBO) used to train Bayesian Neural Networks. However, while the ELBO assumes the existence of a Bayesian posterior (w (D) of which Qw][D) is an approximation, we require no such assumption. The use of the ELBO for different values of 33 and the connection with rate- distortion theory has also been explored in Hu et al. (2018). Closer to our viewpoint is Hinton & Van Camp (1993), that shows that, for 3 = 1, eq. (2) is the cost to encode the labels in D together with the weights of the network. This justifies considering, for any choice of P and Q, the term KT. Q (w (w] P(w)) as the coding length of the weights using some algorithm, although this is true only if they are encoded together with the dataset. A drawback with these approaches is that they lead to non-trivial results only if KL (D || P(w)) is much smaller than the coding length of the labels in the dataset (i.e., N log Y nats, assuming a uniform label distribution). Unfortunately, this is far from being the case with typical deep neural networks.
Rather than the particular value of the coding length, we focus on how it changes as a function of for given noise model, tracing a Pareto-optimal curve which defines the Information in the Weights we have proposed above.
3.1 INFORMATION IN THE WEIGHTS CONTROLS GENERALIZATION
Equation (2) defines a notion of information that, while related the learning task, does not imme- diately relate to generalization error or invariance of the representation. Throughout the rest of this work, we build such connections leveraging on existing work. We start by using the well-known PAC-Bayes bound (McAllester, 2013) to connect the information that the weights retain about the training set to performance on the test data.
NNote that this definition is compatible with a deterministic training process, or with stochastic training process that yields a point estimate of the weights, or with stochastic training process that yields distribution of weights.
4

---

Under review as a conference paper at ICLR 2019
Theorem 3.2 (McAllester (2013), Theorems 2-4). Assume the dataset D {(z,, Wk) is sampled i.idd. from a distribution I), and assume that the per-sample loss used for training is bounded by Liax (we can reduce to this case by clipping and rescaling the loss). For any fixed 3 1/2. prior P(w), and weight distribution Q(w(D), with probability at least 1 - 6 over the sample of D, have:
1
QUIp + 3 KL( P) P)+ + B Blog 11 5
Luest ((ess(Q)= S
(3)
N(1 23)
where (Q) == =00)):reerrppee Ez..grppzzs) v)) (uwalPPllyaa is the expected per-sample test error that the model incurs using the weight distribution (w)(D). Moreover, given a distribution p(D) over the datasets, we have the following bound in expectation over all possible datasets:
Ep Luese (Q (w[D N(1 23)
[o + +3 BB ED|KL( ((wl((
(4)
Hence, minimizing the complexity Cs(D;P,Q) can be interpreted as minimizing an upper-bound on the test error, rather than merely minimizing the training error. In Dziugaite & Roy (2017), non-vacuous generalization bound computed for DNNs, using a (non-centered and non-isotropic) Gaussian prior and Gaussian posterior distributions.
3.2 SHANNON VS. FISHER INFORMATION IN THE WEIGHTS
Definition 3.1 depends on an arbitrary choice of the noise distribution and of the prior. While this may appear cumbersome, it captures the fact that to properly measure the information in deep network we need to adapt the choice of noise to the model. In this section, we show how different priors and posteriors result in known definitions of information, in particular Shannon's 's and Fisher's. This section is inspired by Achille et al. (2019), who derive these relations in the even more general setting of Kolmogonov Complexity.
In some cases, there may be an actual distribution T (D) over the possible training sets, so we may aim find the prior P(w) that minimizes the expected test error bound in eq. (4), which we call adapted prior. The following proposition shows that the information measure that minimizes the bound expectation is the Shannon Mutual Information between weights and dataset.
Proposition 3.3 (Shannon Information in the Weights). Assume the dataset 'D is sampled from a distribution π (D), and let the outcome of training on sampled dataset DD be described by distribution Q( (D). Then the prior P(w) minimizing the expected complexity Ed (D(Ca(D; P, Q)) is the marginal P(w) = ED (Q(w[[(()))) and the expected Information in the Weights is given by
[K[ (w)[D) P(w) ) = I(w; D)-
(5)
Here 1ww; D) is Shannon's 's mutual information between the weights and the dataset, where the weights are seen as a (stochastic) function of the dataset given by the training algorithm (SGD).
The above proposition is textbook material, but it is proven in the Appendix for completeness. Note that, in this case, the prior P(w)) is optimal given the choice of the training algorithm (i.e., the map A 'D - (w) and the distribution of training datasets x (D). Using this prior we have the following expression for the expectation over D of eq. (2):
(6) Notice that this is the general form of an Information Bottleneck (Tishby et al., 1999). However, the use of the IB in Deep Learning has focused on the activations (Shwartz-Ziv & Tishby, 2017). which are the bottleneck between the inputs x and the output y. Instead, the Information Lagrangian eq. (6) concerns the weights of the network, which are the bottleneck between the training dataset D and inference on the future test distribution. Hence, it directly relates to the training process, the finite nature of the dataset, and can yield bounds on future performance. The particular Information Bottleneck for the weights was first used by Achille & Soatto (2018), but derived in more limited setting that did not allow the flexibility needed to establish the bounds we describe in this paper using Fisher Information.
ED|Cs(D; Q)1 = En(Em--Q(I [[)(w)]] + 3I(w; D).
While the adapted prior of Proposition 3.3 allows us to compute an optimal generalization bound, it requires averaging with respect to all possible datasets, which requires knowledge of the task
5

---

Under review as a conference paper at ICLR 2019
distribution (D) and is, in general, unrealistic for deep learning. At the other extreme, can consider an uninformative prior, and obtain the (log-determinant of the) Fisher as a measure of information.
Proposition 3.4 (Fisher Information in the Weights), Assume an isotropic Gaussian prior P(w) ~ N (0, X²1) and Gaussian posterior Q(w (D) N(w", EE), where w² is any global minimizer of the cross cross-entropy loss. Then, for X + DC, as the prior becomes (improper and) uninformative, we have that:
1. For small B, the covariance E' that minimizes Cg(D; tends to = in accordance with the Cramér-Rao bound, where H = V [Dw) is the Hessian of the cruss-entropy loss, and F is the Fisher Information Matrix;
2. The Information in the Weights is given by
KT (T(((()))) || P(w))())))))))))))))))))))))PP log Fl + log X2 0(1).
Note that the constant 1/ke log X2 does not depend on Q(w (D) and hence can be ignored. Remark 3.5. The above proposition assumes that the configuration of the weights to which we are adding noise a global minimum, in which case the Hessian and the Fisher matrix coincide. In fact, we have the following decomposition of the Hessian (Martens, 2014, eq. 6 and Sect. 9.2):
& Σ Σ LV. =)l=-s H (FAN)EDJ=I
H=F+ N
(8)
where = = (x; is the output of the network for input Hi, L( =) = = -ン,- oy.d log( (2)) is the cross-entropy loss for the i-th sample, and Нin, is the Hessian of the j-th component of z. most training samples are predicted correctly, then V L.I( Ro O and H =s F. Otherwise, there is guarantee that H will be positive definite, making the second-order approximation used in Proposition 3.4 invalid, since it suggests that adding noise along the negative directions can decrease the loss unboundedly. Following Martens (2014), use more robust second-order approximation by ignoring the second part of eq. (8), hence using the Fisher as stable positive semi-definite approximation the curvature. this setting, eq. (7) remains valid at all points. The additive term diverging to infinity is expected when using an improper prior, and does not affect considerations about the gradients or the minimizers. Note that there is no assumption that the curvature of the loss be constant near convergence.
3.3 INFORMATION IN THE LEARNING DYNAMICS
In Section 3.1 we have seen that the Shannon Information of the weights controls generalization. In Section 4 we will see that the Fisher controls invariance of the activations. Can just pick one measure of information and use it to characterize both generalization and invariance?
In principle, the Fisher can also be used in Theorem 3.2 obtain generalization bounds; however, is likely to give a vacuous bound if used directly, as is usually much larger than the optimal Shannon Information. In this section, we argue that, for deep network trained with stochastic gradient descent on given domain, Fisher and Shannon go hand-in-hand. This hinges on the fact that: (i) The Fisher depends
the domain, but on the labels, hence all tasks sharing the same domain share the same Fisher, (ii) SGD implicitly minimizes the Fisher, hence, (iii SGD tends to concentrate the solutions a restricted area of low Fisher solutions, hence minimizing the Shannon Information.
While (i) follows directly from the definition of the Fisher Information Matrix (Section 2), (ii) not immediate, as SGD does not explicitly minimize the Fisher. The result hinges on the fact that, by adding noise to the optimization process, SGD will tend to escape sharp minima, and hence, since the Fisher is measure of the curvature of the loss function, it will evade solutions with high Fisher. We can formalize this reasoning using a slight reformulation of the Eyring- law (Berglund, 2011) 1) for stochastic processes in the form of eq. (1).
Proposition 3.6 (Berglund (2011), eq. 1.9). Let w" be a local minimizer of the loss function Top (w)) Consider the path Y joining w² with any other minimum which has the least increase

---

Under review as a conference paper at ICLR 2019
in the loss function. The point with the highest loss along the path is a saddle point w" (the rele- vant saddle) with a single negative eigenvalue X1 (1ww)). Then, in the limit of small step size t, and assuming isotropic gradient noise, the expected time before SGD escapes the minimum w" is given by
[[r]
121(w=)) e (F(ww))J(w))
2T
where we have defined the free energy F(w) LD(W) + log F(W)], where F(w) is the Fisher computed at w. and T x T// B, where B is the batch size. In particular, increasing T (the "temper- alure" of SGD) makes SGD more likely avoid minima with high Fisher Information.
We can informally summarize the above statement saying that SGD, rather than minimizing di- rectly the loss function, minimizes a free energy F(w) = LD(ww + 3 log | [(w)]. Hence, the Fisher Information in the Weights controls the dynamics by slowing down learning when more information needs to be stored, as made precise in the next proposition. Note that the improper nature of the prior plays no role in the free energy (constants do not matter); the slowing down corresponding to large Fisher information has also been observed empirically by Achille et al. (2018).
We can now finally prove (iii), connecting the Fisher Information with the Shannon, which are at face value unrelated. The proof leverages an approximation of the mutual information using the Fisher Information presented in Brunel & Nadal (1998).
Proposition 3.7. Assume the space of datasets 'D admits a differentiable parametrization. Assume that p(D w) is concentrated along a single dataset (i.e., the one used for training). Then, we have the approximation:
(2TT)*
Iww; = =H()) -ED log
V Fo((w)) (wr) NDDw
where H is the entropy and we assume pl DD) N(w (D), F(w (D))-1), w² (D) are the weights obtained at the end of training on dataset DD and we assume that F(F (D)) & VDw" (D). The term V is the Jacobian of the final point with respect changes of the training set.
Notice that the norm Vow'l| || of the Jacobian VDw" can be interpreted as a measure of the sta- bility of SGD, that is, how much the final solution changes if the dataset is perturbed (Hardt al., 2015). Hence, reducing the Fisher Fu (w= (D)) of the final weights found by SGD (i.e., the flatness of the minimum), or making SGD more stable. i.e., redueing V DDw* both reduce the mutual information (ww: D), and hence improve generalization per the PAC-Bayes bound. The Gaussian assumption for the weight distribution for convenience of computation.
THE ROLE OF INFORMATION IN THE INVARIANCE OF THE REPRESENTATION
Thus far we have seen that training& DNN using SGD recovers weights that are a sufficient (w minimizes the training loss) and minimal (they have low Information, either Shannon's 's or Fisher's) representation of the training dataset 'D. The PAC-Bayes Bound guarantees that, on average, suffi- ciency of weights- a representation of the training set implies sufficiency of the activations a representation of the input datum at test time. What we missing is a guarantee that, in addition to being sufficient, the representation of the test datum is also minimal, that is the information the activations is also minimized. Why do care that the representation of future data be minimal? Because it has been established by Achille & Soatto (2018) that this leads to invariance of the repre- sentation nuisance variability (a sufficient representation is invariant if and only if it is minimal). In this section, we derive this missing link. In this case, minimality should be expressed in terms of Shannon's Mutual Information, since we want the representation to be invariant on average over future data, to which we do not have access (unlike the training dataset, which was the subject of pre- vious sections). This entails some subtleties that have caused some confusion in the literature (Saxe
Foor example by parametrizing the labels by the weights of overfitting model, and sampling through differentiable sampling algorithm.
"That is, that the Fisher does not change much if we perturb the dataset slightly. This assumption is mainly keep the expression uncluttered, and a similar result can he derived without this additional hypothesis.
7

---

Under review as a conference paper at ICLR 2019
al., 2018; Chelombiev et al., 2019). After the training is complete, activations deterministic function of the input, so some infommation-theoree quantities are degenerate. Achille & Soatto (2018) argue that the weights of DNN should be considered stochastic. where the stochasticity is imputed by the amount of information they store, and prove that the the mutual information between activations and inputs is in fact upper-bounded by Information in the Weights, considered as a noisy communication channel. A similar point of view was taken later by Goldfeld et al. (2018), who estimate mutual information under the hypothesis of inputs with isotropic noise. Both Achille & Soatto (2018) 8) and Shwartz-Ziv & Tishby (2017) suggest connecting the noise in the weights and/or activations with the noise of SGD, although no formal connection has been established thus far.
Our main contribution, developed in this section, is to establish the connection between minimality of the weights and invariance of the activations, which resolves conflicting points of view. First, for a fixed deterministic DNNs, without stochasticity, we introduce the notion of effective information in the activations which, rather than measuring the information that an optimal decoder could extract from the activations, measures the information that the network effectively uses in order to classify. Using this definition, we show that the Fisher Information in the Weights bounds both the Fisher and Shannon Information in the activations. that we already related the Fisher Information to the noise of SGD in Proposition 3.6.
4.1 INDUCED STOCHASTICITY AND EFFECTIVE INFORMATION IN THE ACTIVATIONS
We denote with z f (x)) the activations of generic intermediate layer of DNN, a deterministic function of x. According to the definition of Information in the Weights, small perturbations of uninformative weights cause small perturbations the loss. Hence, information the activations that is not preserved by such perturbations is not used by the classifier. This suggests the following definition.
Definition 4.1. (Effective Information in the Activations) Let be the value of the weights, and let n ~ NO, Ei,, with Σ = 3FF-(ww) be the Gaussian noise minimizing eq. (2) at level 3 for an uninformative prior (Proposition 3.4). We call effective information (at noise level 3) the amount of information about æ that is not destroyed by the added noise:
Teg,a(x; 2) = [xr; zn))
where fw+n(x) are the activations computed by the perturbed weights w + N(w M.)).
Using this definition, we obtain the following characterization of the information the activations. Proposition 4.2. For small values of 3 we have:
(i) The Fisher Information F2/5
E. (2 log (0gg((22z)) of the activations w.t.. the inputs is:
F212 V J, Fw Valw.
where V (I)w(x) is the Jacobian of the representation given the input, and (x) is the Jacobian of the representation with respect the weights. particular, the Fisher of the activations goes to zero when the Fisher of the weights F. goes to zero.
(ii) Under the hypothesis that, for any representation z, the distribution p(z|zz) of inputs that could generate it concentrates around its maximum, we have:
(2Tee)*
Leg, H(x) E. log
(10)
hence, by the previous point, when the Fisher Information of the weights decreases, the effective mutual information between inputs and activations also decreases.
Hence, decreasing the Fisher Information that the weights have about the training set (which can be done by increasing the noise of SGD) decreases the apparently unrelated effective information between inputs and activations at test time. Moreover, making Vzfw(()) small, i.e., reducing the Lipschitz constant of the network, also reduces the effective information.
We say that a random variable n is nuisance for the task y if n affects the input x but is not informative of y. Lee. [(n, v) 0. We say that representation is maximally invariant to if
8

---

Under review as a conference paper at ICLR 2019
[22,n) n) is minimal among all sufficient representations, which are all the representations that capture all the information about the task contained in the input, y(2 = [(z. y). The following claim in Achille & Soatto (2018), connects invariance to compression (minimality):
Proposition 4.3 (Achille Soatto (2018), Proposition 3.1). A representation 2 is maximally in- variant to all nuisances at the same time if and only if [xx,2) 2) is minimal among the sufficient representations.
Together with Proposition 4.2, this shows that network which has minimal complexity (i.e., min- imal Information in the Weights) is forced to learn a representation that is effectively invariant to nuisances; that is, invariance emerge naturally during training by reducing the amount of information stored in the weights.
As side note, we may wonder what distribution of the inputs would maximize the effective mu- tual information Teir,a (a;; 2); that is, what distribution the network is maximally adapted to repre- sent (Brunel & Nadal, 1998). Maximizing ( 2) with respect to p(z) we obtain: p"(x) //Fate// F// Using this, we obtain the following bound on the mutual information [en for any input distribution:
2) S log (S F= dz).
Intuitively, this can be interpreted as the volume of the representation space; that is, how many well separated representations can be obtained mapping inputs x, taking into account that, because of a small Lipschitz constant of the network, or because of noise, multiple inputs may be mapped to similar representations.
5 DISCUSSION
Once trained, deep neural networks are deterministic functions of their input, and we are interested in understanding what "information" they retain, what they discard, and how they process unseen data. Ideally, we would like them to process future data by retaining all that matters for the task (sufficiency) and discarding all that does not (nuisance variability), leading to invariance. But do not have access to the test data, and the literature does not provide a rigorous or even formal connection between properties of the training set and invariance nuisance variability in the test data.
This paper extends and develops results of Achille & Soatto (2018) and is, to the best of our knowl- edge, the first to define the information in deep network, which is in the weights that represent the training set, in a way that connects it to generalization and invariance, which are properties of the activations of the test data. This Information in the Weights is neither Shannon's 's (used in Achille & Soatto (2018) nor Fisher's, but more general one that encompasses the two as special cases.
We leverage several existing results in the literature: the Information Lagrangian is introduced in Achille & Soatto (2018), but we extend it beyond Shannon Information, which presents some chal- lenges when the source of stochasticity is not explicit. We draw on Fisher's Information, that for- malizes a notion of sensitivity of a set of parameters, and is not tied to a particular assumption of generative model. We leverage the PAC-Bayes bound connect sufficiency of the weights suffi- ciency of the activations, and provide the critical missing link to connect minimality of the weights that arises from the inductive bias of SGD when training deep networks with minimality of the activations.
We put the emphasis on the distinction between Information in the Weights, done by Achille & Soatto (2018), and information in the activations, which several other information-theorrticc ap- proaches to Deep Learning have focused. One pertains representations of past data, which can measure. The other pertains to desirable properties of future data, that we cannot measure, but we can bound. We provide measurable bound, exploiting the Fisher Information, which enables reasoning about "effective stochasticity" even if network is a deterministic function.
Our results connect to generalization bounds through PAC-Bayes, and account for the finite nature of the training set, unlike several other eeformmtionnthhoreee approaches to Deep Learning that only provide results expectation.
Thhre are infinitely many, for instance the trivial function x, and any invertible function of it.

---

Under review as a conference paper at ICLR 2019
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. Journal of Machine Learning Research, 19()):14771190, 2018. URL http: / //arxiv.org//////ooo...........a abs 11006 0135.
Alessandro Achille, Glen Mbeng, and Stefano Soatto. The Dynamics of Differential Learning I: Information-Dynamme and Task Reachability. arXiv e-prints, art. arXiv:1810.02440, October 2018.
Alessandro Achille, Giovanni Paolini, Glen Mbeng, and Stefano Soatto. The Information Complex- ity of Learning Tasks, their Structure and their Distance. arXiv e-prints, art. arXiv:1944.0399, Apr 2019.
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep net- works. In International Conference Learning Representations, 2019. URL https:// openreview net
Nils Berglund. Kramers' law: Validity derivations and generalisations. arXiv preprint arXiv:1106.5799 2011.
Nicolas Brunel and Jean-Pierre Nadal. Mutual information, fisher information, and population cod- ing. Neural computation, 10(7):133-11777, 1998.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In International Conference of Learning and Repre- sentations, 2018.
Ivan Chelombiev, Conor Houghton, and Cian O'Donnell. Adaptive estimators show information compression in deep neural networks. arXiv preprint vrXiv:1902.000333 2019.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, 2017.
Ziv Goldfeld, Ewout van den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kings- bury, and Yury Polyanskiy. Estimating information flow in neural networks. arXiv preprint arXiv:1810.05728, 2018
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint 10%.025509001400 2015.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5-13. ACM, 1993.
Shell Xu Hu, France Champs-sur Marne, Pablo G Moreno, Neil Lawrence, and Andreas Dami- anou. 3-bnn: rate-distortion perspective bayesian neural networks. NewrIPS Bayesian Deep Learning Workshop, 2018.
Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. International Conference on Machine Learning, pp. 2101-2110, 2017.
James Martens. New insights and perspectives on the natural gradient method. arXiv preprint
arXiv:1412.1199, 2014.
David McAllester. pac-bayesian tutorial with a dropout bound. arXiv preprint arXiv: 137.22130..211. 2013.
Ankit Pensia, Varun Jog, and Po-Ling Loh. Generalization error bounds for noisy, iterative algo- rithms. In 2018 IEEE International Symposium on Information Theory (ISIT), pp. 546-550. IEEE, 2018.
10

---

Under review as a conference paper at ICLR 2019
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Bren- dan Damiel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. International Conference of Learning Representations, 2018.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprinl (tt:::77030000000 2017.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In Information Theory Workshop (ITW), 2015 IEEE, pp. 1-5. IEEE, 2015.
Naftali Tishby, Fernando C Pereira, and William Bialck. The information bottleneck method. The 37th annual Allerton Conference Communication, Control, and Computing, pp. 368-377, 1999.
Aolin Xu and Maxim Raginsky. ceemmation-theorete analysis of generalization capability of learn- ing algorithms. Advances in Neural Information Processing Systems, pp. 2524-2533, 2017.

---

Under review as a conference paper at ICLR 2019
A EMPIRICAL VALIDATION
Effeci of bench margins disrii a
1e4
Shomon Fisher information
Model porametiiation
5-0200
Fateriro
Shamoo Informatoon
8.008
3.5
12
8:009
4000
60
0.004
0.002
-0.
0.0
0.5
Figure 1: (Left) Plot of the marginal distribution of the parameter w at the end of training, marginal- ized over all possible training tasks D, as the batch size B of SGD changes. Notice that the batch size gets smaller SGD shifts farther away from areas with high Fisher Information (dotted line), supporting Proposition 3.6. (Center) Effect of the batch size on the Information in the Weights. Remarkably, while changing the batch size should only affect the Fisher Information, it also reduces the Shannon Information of the weights following the same qualitative dependence, in support of Proposition 3.7. (Right) Redundant parametrization ((0) used in the experiment to emulate some of the key properties of the loss landscape of deep networks.
Relation between Fisher and Shannon Information In this section we want empirically verify the link between Fisher Information and Shannon Information of in a machine learning problem trained with SGD, which we explored in Section 3.3. Our main objective verify that decreasing the Fisher Information (which can be done by changing the the batch size and learning rate (Proposition 3.6), does indeed decrease the Shannon Information (Figure 1, center). It is known in general that the Fisher Information can be used to upper- upper-bound bound the Shannon Information (recall from Section 3.2 that Shannon Information is the minimum attainable). However, we show empirically here that even for simple 1D example this bound is remarkably loose (by orders of magnitude). Rather the strong connection between Fisher and Shannon can better be explained in terms of Proposition 3.7 that we introduce.
of SGD, in particular
In general, computing the Shannon Information [D; w) between a dataset D and the parameters w of model is not tractable. However, here show an example of simple model that can be trained with SGD, replicates some of the aspects typical of the loss landscape of DNNs, and for which both Shannon and Fisher Information can be estimated easily. According Figure shows that (center) increasing the temperature of SGD, for example by reducing the hatch sizes, reduces both the Fisher and the Shannon Information of the weights (Proposition 3.7), and (left) this is due to the solution discovered by SGD concentrating in areas of low Fisher Information of the loss landscape when the temperature is increased (Proposition 3.6).
predictions in Section 3.3,
The dataset D = {xr with N = 100, is generated by sampling a mean j -nif[-1, 1] and sampling D; [H,, 1). The task to regress the mean of the dataset by minimizing the loss Lp = N E (z; - (()))² where @ are the model parameters (weights) and ø is some fixed parametrization. To simulate the weempaaramttiiztioonn and complex loss landscape of DNN, we pick (00) as in Figure (right). Notice in particular that multiple value of 0 will give the same (0)) This ensures that the loss function has many equivalent minima. However, these minima will have different sharpness, and hence Fisher Information, due to Φ(0) being more sharp near the origin. Proposition 3.6 suggests that SGD is more likely to converge to those minima with low Fisher Information, which is confirmed in Figure I (left), which shows the marginal end point over all datasets 'D and SGD trainings. Having found the marginal q(w) over all training and datasets, we can compute the the Shannon Information I(D; 0) Ep [KL q (w)l (D) || ((w)) that we take q(wlD)) = N(w", (W",F=1)) F-)), where w" is the minimum recovered by SGD at the end of training and F-1 is minimum variance of the estimation, which is given by the Cramér-Rao bound. The (log determmaennn of) loss function Lp. In Figure (center) we show how these quantities change the batch size B varies. Notice that when B = N = 100, the algorithm reduces to standard gradient descent, which maintains the largest information in the weights.
The toy model is implemented as follow:
)} Note
can instead easily be computed in close form given the
12

---

Under review as a conference paper at ICLR 2019
information
Information
Theee oinheer at lpoohh 30
2.000
60
40
20
Batch Size
dataset
Figure 2: (Left)) Trace of the Fisher Information of the weights of ResNet- 18 after 30 epochs of training on CIFAR-IO, for different values of the batch size. Smaller batch size have smaller Fisher Information, supporting Proposition 3.6. (Right) Fisher Information at the end of the training on subset of CIFAR-10 with only the first kt classes: After training on fewer classes, the network has less information in the weights, suggesting that the Information in the Weight has semantic role.
The value of the Fisher Information cannot be directly compared to the Shannon Information, since it defined modulo an additive constant due to the improper prior. However, using a proper Gaussian prior that leads to the lowest expected value, we obtain value of the "Gaussian" Information in the Weights between 4000-5000 nats, versus the nats of the Shannon Information: minimizing a much larger (Fisher) bound SGD can still implicitly minimize the optimal Shannon bound.
this section we show the trade-off between amount of in- formation in the weights, complexity of the task (Figure 2, right), and value of / (Figure 2, left) a more realistic problem. More precisely, we validate our predictions on a larger scale off off-the-shelf ResNet- 18 trained CIFAR-10 with SGD (with momentum 0.9, weight decay 0.0005, annealing the learning rate by 0.97 per epoch).
Fisher Information for CIFAR-10
First, we compute the Fisher Information (more precisely its trace) the end of the training training for different values of the batch size (and hence of the "temperature of SGD"). In accordance with Proposition 3.6, Figure 2 (left) shows that after 30 epochs of training the networks with low batch size have a much lower Fisher Information. Second, to check whether the Fisher Information cor- relates with the amount of information contained in the dataset, we train using only 2, 4, and so on classes of CIFAR-10. Intuitively, the dataset with only 2 classes should contain less information than the dataset with 10 classes, and correspondingly the Fisher Information in the Weights of the network should be smaller. We confirm this prediction in Figure 2 (right).
Fisher Information and dynamics of feature learning We now investigate how the amount of information in the weights of a deep neural network changes during training, in particular to see whether changes in the Fisher Information correspond to the network learning features of increasing complexity. In Figure we train a 3-layer fully connected network simple classification prob- lem of 2D points and plot both the Fisher and the classification boundaries during training. Since compute the Fisher Matrix exactly using the
the network is relatively small, in this experiment definition. As different features are learned, we observe corresponding "bumps" the Fisher infor- mation matrix. This is compatible with the hypothesis advanced by Achille et al. (2019), whereby feature learning may correspond to crossing of narrow bottlenecks (high curvature; high Fisher) in the loss landscape, which is followed by compression phase as the network moves away toward flatter area of the loss landscape.
B PROOFS
Q(wlD), we want to find the prior P*(w) that minimizes the expected complexity of the data:
Proof of Proposition 3.3. For a fixed training algorithm
Ps (w)= argmin (DD)
P(w)
argmin
Ep (B Eo ((U
P(w)
13

---

Under review as a conference paper at ICLR 2019
Learning ditee
Compresson
Mire tchee
100
No Nertow learred
Heering Suntun
Compressoon
Figure 3: Plot of the reg-determinan of the Fisher Information Matrix during training of 3-layers fully connected network on a simple 2D binary classification task. As the network learns an increas- ingly complex classification boundary the Fisher increases. Moreover, learning of feature correspond to small bumps in the Fisher plot, supporting the idea that feature learning may coincide with crossing of narrow bottlenecks the loss landscape.
Notice that only the second term depends P(w). Let Q(w) Eo[Q(w[D)] be the marginal distribution of w, averaged over all possible training datasets. We have
Ep ( Q(wlD) P(w) )} = ( Q(w) D) ([)(w)))))))))))))))))))))))[( (D/KL(Q(())ll [
Since the KL divergence always positive, the optimal "adapted" prior given by P (w) = ((w). i.e. the marginal distribution of over all datasets. Finally, by definition of Shannon's mutual information, we get
I(w; D) (L(Q(u
x (D)) ED> (KL Q(w[D) (KW(Q( Q(w) )}
Proof of Proposition 3.4. Since both P(w) and (w are Gaussian distributions, the KL diver- gence can be written
1 KL ((w(D) 1 |unl + X2 tr(E) + k X2 loo | k
+
where &s is the number of components of w.
Let wt be local minimum of the cross-entropy loss (p and let H be the Hessian of Lo (p (ylar) in w*. Set μ w". Assuming that quadratic approximation holds in sufficiently large neighborhood, we obtain
3
| 11
Cp(D;P,Q)= = Lp(Pw- (v|=)) + tr(H Σ) +
+
tr(2)) + log X2 log | -
2
X2
x2
The gradient with respect to
aC3 P.Q)
B
+
E-1
2X2
Setting it zero, obtain the minimizer Σ
2 (H H
8 n1
14

---

Under review as a conference paper at ICLR 2019
Recall that the Hessian of the cross-entropy loss coincides with the Fisher information matrix F at w", because w is a critical point (Martens, 2014). Since LD (P (y)x)), and hence H. is not normalized by the number of samples N, the exact relation is H . F. Taking the limit for + oc, we obtain the desired result.
IT
Proof of Proposition 3.7. It shown in Brunel & Nadal (1998) that, given two random variables and y, and assuming that p(æ[l)) is concentrated around its MAP, then the following approximation holds:
1
(2xe) Ffyl
(1;y)) Re H(x)
E.g
log
(11)
where Ful== By-pl// V [ogp())))] the Fisher Information that has about y, and k dim We want apply this approximation to [ww; D), using the distribution p(w[D) N(w"(D) F(w" (D))-1). Hence, we need to compute the Fisher Information F D that the dataset has about the weights. Recall that, for a normal distribution N(u(0), I())), the Fisher Information given by
Fm,n- = Gg,,, -
00 u +
2 tr (O,,,
E) 20, Σ))
Using this expression in our case, and noticing that by our assumptions we can ignore the second part, we obtain:
Fuip = VDW" Fu(w")VD''
which we can insert in eq. (11) to obtain:
1 IED
(2Tee
[w:D) N H(D
log
2
Fwip
(2TTe)*
= H()) - EE
log
D F Fw("w) (wt)V Voww
Proof of Proposition 4.2. (1) We need to compute the Fisher Information between 2n and æ, that is:
Ezje = Exwp(=l>)
v² log = (=)()))))
In the limit of small 3, and hence small n, we expand 2n = fw+n(x) to the first-order about w as follows:
Zn = fu-n(()) Jy o([[n]])
where Jj is the Jacobian of (x) seen as a function of w, with dim(Jy) = dim 2 x dim w. Hence, given that n NO,, Ex), obtain that given approximately follows the distribution p(zn (2)
We can now plug this into the expression for Falz and compute:
Fzt1 Exmp(=/x)
120gp(zz.1z)
E - 2
[= (<<))=) m(()) + log [()]]]
Vafu E. Vafa.
(2) We now proceed to estimate the Shannon mutual information / (22; between activations and inputs. general, this does not have closed form solution, rather we use again the approximation of Brunel & Nadal (1998) as done in the proof of Proposition 3.7. Doing so we obtain:
(2Re)* F2
lan 2) H(x) Eg
log
2
(2
H(x)
E,r 2
log
VV J Fw Valu
15


Reviews:
[{"review_id": "rJl_CN_DKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a theoretical account of information encoded within deep neural networks subject to information theoretic measures. In contrast to other efforts that examine information encoded in weights, this work emphasizes the effective information in the activations. This characterization is further related to information in the weights, and a theoretical justification is made for what this means with respect to properties of generalization and invariance in the network. \nThe notion of attaching the weights that represent the training set to the activations that accord with the test set in a theoretical framework is interesting. In practice, I would have liked to see a bit more attachment of the theoretical formalisms to the empirical justification that follows the references. This, however, is a matter of personal bias as I don't typically produce papers that are principally theoretical contributions in my own work. Overall, the content of the paper seems sound and the theoretical and empirical justifications seem well founded but I also can't claim to be an expert in this area.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Sye3oGRmcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThis is a theoretical paper that builds on top of Achille and Soatto (2018), Achille et al. (2019), McAllester (2013), and Berglund (2011),. The paper attempts to answer the relationship between the inductive bias of SGD, generalization of DNNs, and the invariance of learned representation from an information theoretical point of view. \nThe paper mentioned many interesting links. In my opinion, the contributions are the following:\n1. Invoking the theoretical result of Berglund (2011) to justify why Fisher information is relevant -- SGD tends to avoid local minima with high Fisher information. \n2. Relating the Fisher information and the stability of SGD to I(w; D).\n3. Introducing the definition of effective information in the activation, and show that which is closely related to the Fisher information. \n\nAbout the rating:\nThis is basically a good paper but I have a few concerns: \n1. A large fraction of this paper are taken from Achille and Soatto (2018), Achille et al. (2019).  \n2. In terms of impact, the paper is somehow incomplete -- it only demonstrates that the Fisher information is important, but the insights didn't lead to any substantial improvement over the current deep learning framework. \n\nDetailed comments:\n1. In my opinion, defining \"information in the weights for the task D\" by KL(Q||P) is inaccurate. \nThe weights themselves are information, which form a representation or a lossy compression of the data (which is also an information). \nAccording to the rate-distortion theory, what we care about is the amount of information the representation attains rather than \"where\" the information are. Therefore, we should talk about \"rate\" or \"amount of information\" or \"mutual information\" rather than \"information\" itself.\nA missing reference regarding this point: \nHu et al. \"β-BNN: A Rate-Distortion Perspective on Bayesian Neural Networks.\" 2018,\nwhich derives the information Lagrangian directly from rate-distortion theory. \n2. More discussions on Xu and Raginsky (2017) is expected, since it proposes to use I(w; D) as a generalization bound. It seems, in terms of generalization, minimizing I(w; D) is a sufficient condition while minimizing Fisher is a necessary condition.  \n3. There are in fact 4 key aspects: sufficiency, minimality, invariance and generalization. It would be great to have a theorem to summarize the relationships between them.  \n4. Could you elaborate on the footnote 3? ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hygat6E1qH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper deals with where the information is in a deep network and how information is propagated when new data points are observed. The authors measure information in the weights of a DNN as the trade-off between network accuracy and weight complexity. They bring out the relationships between Shannon MI and Fisher Information and the connections to PAC-Bayes bound and invariance. The main result is that models of low information generalize better and are invariance-tolerant. \n\nThe paper is very well written and concepts are theoretically-well documented.\n\nIn Definition 3.1 for the ‘Information in the Weights’, how does the complexity of the task vary with \\beta? Is the Pareto curve provided in the paper? ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is full of ideas. However, a logical argument is only as strong as its weakest link, and I believe the current paper has some weak links. For example, the attempt to tie the behavior of SGD to free energy minimization relies on unrealistic approximations. Second, the bounds based on limiting flat priors become trivial. The authors in-depth response to my own review was much appreciated, especially given its last minute appearance. Unfortunately, I was not convinced by the arguments. In part, the authors argue that the logical argument they are making is not sensitive to certain issues that I raised, but this only highlights for me that the argument being made is not very precise.  I can imagine a version of this work with sharper claims, built on clearly stated assumptions/conjectures about SGD's dynamics, RATHER THAN being framed as the consequences of clearly inaccurate approximations. The behavior of diffusions can be presented as evidence that the assumptions/conjectures (that cannot be proven at the moment, but which are needed to complete the logical argument) are reasonable. However, I am also not convinced that it is trivial to do this, and so the community must have a chance to review a major revision.
Decision: Reject