Usable Information and Evolution of Optimal Representations During Training
Authors: Michael Kleinman
Alessandro Achille
Daksh Idnani
Jonathan Kao
Venue: ICLR 2021 Poster (2021)
Keywords: Usable Information
Representation Learning
Learning Dynamics
Initialization
SGD

Abstract: We introduce a notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training. We show that the implicit regularization coming from training with Stochastic Gradient Descent with a high learning-rate and small batch size plays an important role in learning minimal sufficient representations for the task. In the process of arriving at a minimal sufficient representation, we find that the content of the representation changes dynamically during training. In particular, we find that semantically meaningful but ultimately irrelevant information is encoded in the early transient dynamics of training, before being later discarded. In addition, we evaluate how perturbing the initial part of training impacts the learning dynamics and the resulting representations. We show these effects on both perceptual decision-making tasks inspired by neuroscience literature, as well as on standard image classification tasks.

Published as a conference paper at ICLR 2021
USABLE INFORMATION AND EVOLUTION OF OPTIMAL REPRESENTATIONS DURING TRAINING
Michael Kleinman' Alessandro Achille Daksh Idnani' Jonathan C. Kao'
Uniivessity of California, Los Angeles "Caltech
{mi chael. kleinman, dak dakshiinannjeeucaa.ee aachille@caltech. edu kao@seas. www.eaasslla............... edu
ABSTRACT
We introduce notion of usable information contained in the representation learned by a deep network, and use it to study how optimal representations for the task emerge during training. We show that the implicit regularization com- ing from training with Stochastic Gradient Descent with a high learning-ran and small batch size plays important role in learning minimal sufficient representa- tions for the task. In the process of arriving at a minimal sufficient representation, we find that the content of the representation changes dynamically during train- ing. In particular, we find that semantically meaningful but ultimately irrelevant information is encoded in the early transient dynamics of training, before being later discarded. In addition, we evaluate how perturbing the initial part of train- ing impacts the learning dynamics and the resulting representations. We show these effects on both perceptual decision-making tasks inspired by neuroscience literature, as well as on standard image classification tasks.
INTRODUCTION
An important open question for the theory of deep learning is why highly over-parametrized neu- ral networks learn solutions that generalize well even though the model can in principle memorize the entire training set. Some have speculated that neural networks learn minimal but sufficient representations of the input through implicit regularization of Stochastic Gradient Descent (SGD) (Shwartz-Ziv & Tishby| 2017 Achille Soatto 2018), and that the minimality of the represental tions relates to generalizability. Follow-up work has disputed the validity of some of these claims when using deterministic deep networks (Saxe et al. 2018), leading to an ongoing debate on the notion of optimality of representations and how they are learned during training.
Part of the disagreement stems from the use of information-theome thhorettc quantities: most previous stud- ies in deep learning have analyzed the amount of information that the learned representation contains about the inputs using Shannon's 's mutual information. However, when the mapping from input to representation is deterministic, the mutual information between the representation and input is de- generate (Saxe et al. 2018 Goldfeld et al., 2018). Rather than study the mutual information in a neural network, here instead define and study the "usable information" in the network, which measures the amount of information that can be extracted from the representation by a learned de- coder, and is scalable high dimensional realistic tasks. We use this notion quantify how relevant and irrelevant information is represented across layers of the network throughout the training pro- cess, and how this is affected by the optimization algorithms and the network pretraining.
In particular, we propose to study simple task inspired by decision decision-making making tasks in neuroscience, where inputs and outputs are carefully designed to probe specific information processing phenom- ena. We then extend our findings to standard image classification tasks trained with state-of- of-te-of--h--rtt art models. Our neuroscience-inspiree task is the checkerboard (CB) task (Chandrasekaran et al. 2017 Kleinman et al. 2019). In the CB task, one discerns the dominant color of a checkerboard filled with red and green squares. The subject then makes a reach to a left or right target whose color matches the dominant color in the checkerboard (Fig [T). This task therefore involves making two binary choices: a color decision (i.e., reach to the red or green target) and a direction decision (i.e., reach to left right). Critically, the color of the targets (red left, green right; green left, red right) is

---

Published as a conference paper at ICLR 2021
random on every trial. The direction decision output is conditionally independent of the color de- cision, as detailed further in Fig Tp and Section [.6 even though the color information needs to be used to solve the task. This task allows us to evaluate how both of these components of information are represented through training and across layers.
We used this task and extensions to study the evolution of minimal representations during training. If a representation is sufficient and minimal, we refer to this representation as optimal (Achille & Soatto 2018). Our contributions are the following. (1) We introduce a notion of usable information for studying representations and training dynamics in deep networks (Section 3p. (2) We used this notion to characterize the transient training dynamics in deep networks by studying the amount of usable relevant and irrelevant information in deep network layers and across training epochs. We first use the CB task to gain intuition of the training dynamics in a simplified setting. We find that training with SGD is critical to bias the network toward learning minimal representations in intermediate layers (Section 4.1). This adds to the literature suggesting that SGD results in minimal representations of input information (Achille & Soatto 2018 Shwartz-Ziv & Tishby 2017) while avoiding some of the pitfalls. (3) We used the intuition gained from the simple task, evaluating our findings CIFAR-10 and CIFAR-100 task using modern architectures. Remarkably, we find that the networks increased usable information about an irrelevant component of information early in training and discarded it later on in training to arrive at minimal sufficient solution, consistent with a proposed (Shwartz-Ziv & Tishby 2017) though controversial theory (Saxe et al. 2018).
2 RELATED WORK
Some efforts to understand why neural networks generalize focus on representation learning, that is, how deep networks learn optimal (i..., minimal and sufficient) representations of inputs in der to solve a task. Typically, representation learning is focused on studying the properties of the asymptotic representations after training (Achille & Soatto| 2018). Recent work suggests that these asymptotic representations contain minimal but sufficient input information for performing a task (Achille& & Soatto| 2018 SSwwarzz-Ziv & Tishby 2017). Implicit regularization coming from SGD, and in particular from the use of large learning rates and small batch sizes, is believed to play an important role in forming these minimal sufficient representations.
How does the training process lead to these minimal but sufficient asymptotic representations? Shwartz-Ziv & Tishby (2017) propose that there are two distinct phases of training: an empirical risk minimization phase where the network minimizes the loss on the training set, and "com- pression" phase where the network discards information about the inputs that do not need to be represented to solve the task. Recently, (aax et al. (2018) challenged this view, arguing that the ob- served compression was dependent on the activation function and the mutual information estimator used in Shwartz-Ziv & Tishby (2017). These works highlight the challenges of estimating mutual information to study how representations emerge through training.
In general, estimating mutual information from samples is challenging for nigh-dimensional random variables (Paninski] 2003). The primary difficulty in estimating mutual information is estimating a nigh-dimensional probability distribution from the samples, since generally the number of samples required scales exponentially with the dimension. This is impractical for realistic deep learning tasks where the representations are high dimensional. To estimate the mutual information, (Shwartz- Ziv & Tishby (2017) used a binning approach, discretizing the activations into a finite number of bins. While this approximation is exact in the limit of infinitesimally small bins, in practice, the size of the bin affects the estimator (Saxe et al., 2018 Goldfeld et al. 2018). In contrast to binning, other approaches to estimate mutual information include entropic- based estimators (e.g., Goldfeld et al. )218)) and a nearest neighbours approach (Kraskov et al. 2004). Although mutual information is difficltt estimate, it is an appealing quantity to summarily characterize key aspects of the transient neural network training behavior because of its invariance to smooth and invertible transformations. In this work, rather than estimate the mutual information directly, we instead define and study the "usable information" in the network, which corresponds to variational approximation of the mutual information (Barber & Agakov] 2003, Poole et al. 2019) (see Sections 3 and A.I). Recently, such variational upproximations to mutual information have been viewed meaningful characterization of representations in deep networks, and the theoretical underpinnings of this approach are beginning
to be investigated (Xu et al. 2020 Dubois et al. 2020
2

---

Published as a conference paper at ICLR 2021
(a)
Trial 1:
Trial 2:
(b)
DNN
input
Z
output
Left target
Right target
Z
checkorbordd color
diresion choice
Color choice: green Direction choice: right
Color choice:
target
Z
Direction choice: right
(a) Checkerboard task. Given two binary target locations (left right) with randomly selected binary colors (red or green), one has to discern the dominant color in the checkerboard and reach the target of the dominant color. On every trial, there correct color and direction choice. However, the identities of the left and right targets are random every trial, decoupling the direction and color decision.
Figure 1:
(b) We trained a deep neural network to perform the task by specifying the proportion of green and red squares on the checkerboard, as well as two scalars denoting the colors of the left and right target. The network was trained to output the correct direction choice. As only the direction, but not the color choice, was reported, given a representation of the correct direction choice Za the network does not need to represent the color choice Ze in deeper layers. Ze is the representation of the target orientation.
Research into the training dynamics of deep networks, and how they represent relevant and irrelevant task information, is nascent. A related study by [Achille le et al. (2019) found that early periods of training were critical for determining the asymptotic network behavior. Additionally, it was found important for determining asymptotic performance (Golatkar
that the timing of regularization ct al. 2019), with regularization during this "critical period" having the most influential effect. Notably, both of these studies found initial increase in the amount of information that weights encode about the dataset (as measured by the Fisher information), that coincides with the critical period of learning. This phase followed later in training by "forgetting" phase where the network discards unnecessary information. This suggests that similar dynamic to the one we study can be observed in weight space instead of representation space.
3 USABLE INFORMATION IN REPRESENTATION
deep neural network consists of set of é layers. with each layer forming successive represen- tation of the input. A representation Ze may store information in variety of ways. It may be that a complex transformation is required to read out the information, or it may be that a simple linear decoder could read out the information. In both cases, from an information- thheoretcc perspective, the same information is contained in the representation, however, there is an important distinction regarding how "usable" this information is. Information is usable if later layers, which comprise affine transformations and element-wiss nonlinearities, can easily extract it to solve the task. Equiv- alently, usable information should be decodable by separate neural network also employing affine transformations and element-wiie wise nonlinearities.
Formally, we define the usable information that representation Z contains about quantity Y, which may refer to the output or a component of the input, as:
I (2; HYY LCE(p(y)))) (()l))
(I)
Here, H(Y) is the entropy, or uncertainty, of Y, and Leg is the cross-entropy loss on the test set of a discriminator network 4(ylz) trained to approximate the true distribution p(y]z). Our definition motivated in the following manner. The test set cross-entropy loss approximates how much uncer- tainty there is the output Y given Z and the discriminator. A low loss implies that there low uncertainty in Y given Z, or that the discriminator can extract lot of "information" about Y from Z. the logarithm in the cross-entropy loss is in base 2, is measured in bits. the value Y approximately the same for any Z, there would be little uncertainty in Y to begin with, important to know the amount of uncertainty in Y given with respect to the initial uncertainty in Y. What is most relevant is the amount of remaining uncertainty in Y given Z. Thus the
3

---

Published as a conference paper at ICLR 2021
difference in uncertainty HY) Leg the amount of "usable information" that Z contains about Y, as shown in our definition in Equation
This definition is appealing to study representations, in part, because it can be computed from sam- ples of Z and Y, and is a quantity that is comparable across network training. We estimate LCE using a small neural network that learns distribution g(y]=). To train the network, we sample acti- vations Z and the quantity Y and learn ((vlz) by minimizing the cross-entropy loss on a training set. We then evaluate the LCE on the test set (Equation T). We provide details about the neural network We also show in the Appendix that the usable information is a lower bound on the mutual information (Appendix (A.I). Importantl, usable information also is not constrained by the data processing inequality; that is, the informa-
and the training we used for decoding in Appendix [.3 and C.2
tion can be made more "usable" by transformation to later layers, consistent with the representation learning view that later layers are forming improved representations of the inputs (Xu et al. 2020).
EXPERIMENTS
Our goal was to characterize how optimal representations are formed through SGD training. We trained multiple network architectures on tasks and assessed the usable information in representa- tions across layers and training epochs. For given architecture and task, all hyper-parameters were kept constant throughout experiments, unless explicitly stated.
To develop intuition, we initially investigate how small fully connected networks represent the rel- evant and irrelevant information in the CB Task. We trained two different network architectures,
'Small FC: 5 layers, with 10 - 7 5 4-3 3 units in each layer, 'Medium FC': 100 20 20 20. Small FC was a network used in prior literature (Shwartz- Ziv & Tishby 2017; Saxe et al. 2018). Our networks were conneced and used ReLU activation. We trained the networks using SGD with a constant learning rate to perform the CB task, described in detail in Appendix B.4 The hyper-parameters used for the CB experiments are listed in Appendix B.5
In our CB task experiments, we quantified the usable color and direction information in the hidden representation, Zg- In the formation. We emphasize that, unless otherwise specified, the network was only trained to output the correct direction choice, so given representation of the direction, representing the color choice is irrelevant. Therefore, a minimal representation should not include information about the color choice, since not necessary to represent given a representation of the direction decision. To make the task more complex, we also generalized the CB task to have n. = 10 and n = 20 targets.
2 CB task, the color information represents half of the input in-
We then use this framework to examine how relevant and irrelevant information are represented in more realistic tasks and architectures, and how hyper-parameters affect the learning dynamics. We define a coarse labelling of task labels and study how the network represents the fine and coarse labelling through training, using ResNet-18 (He et al. (016) and All-CNN (Springenberg et al. 2015) on CIFAR-10 and CIFAR-100.
4.1 SGD WITH RANDOM INITIALIZATION RESULTS IN MINIMAL SUFFICIENT REPRESENTATIONS IN THE CB TASK
We first assessed the optimality of the network representations by training Small FC networks on the CB task using n = 2 colors (Fig[a) using a random initialization for the weights. In particular, the initial weights do not contain information about the dataset. We computed the usable color and direction information across layers of the neural network and epochs of training. In our plots, later layers are denoted by darker shades. deeper layers, there was a decrease usable color informa- tion, corresponding to more minimal representations. After training. the asymptotic representation in the last layer contained zero usable color information and 1 bit usable direction information. To visualize this minimal sufficient representation, we plotted the activations of the 3 units in the last layer of the Small FC network for different inputs. These visualizations are labeled by the correct color (red and green) and direction (cross circle). In the asymptotic representation, representation of the input color is overlapping (red and green), while the representation of the direction output is separable (crosses and circles), forming a minimal sufficient representation.
4

---

Published as a conference paper at ICLR 2021
(a)
(b)
50
12
50
Epact
Eponn
Eper
(d)
Epoch:
Epoch: 10
Epoch
Epoch:
0.026
2.5
0. 0.004
hid00e
0.000
0.0
20
0 Paieaaa
Ideea onl
0.010
Fidden 1
Wntmm wakk
Niste uni
hidoo
Iedben
3
Figure 2: SGD with random initialization leads minimal representations. (a) Small FC net- work trained the π 2 checkerboard task. Max usable direction and color information: bit. This network was trained without regularization for 100 epochs using SGD with a learning rate of 0.05 and batch size of 32. Blue (orange) lines correspond to usable information about the direction (color) decision in the representation. Darker shades of color correspond to deeper layers in the network. In the asymptotic representations, we observed that direction information high across layers, while color information decreased in the later layers, The usable color information was ap- proximately zero in the last layer of the Small FC network. (b) Medium FC network trained with = 10 checkerboard colors. Max usable direction and color information: 3.32 bits. In the last layer, there nearly zero usable color information. Across layers, there is decrease in usable color information, and an increase in usable direction information. (c) Medium FC network trained with 20 checkerboard colors, batch size of 128 and learning rate of 0.5. Max usable direction and color information: 4.32 bits. In the later layers (darker shades) there is small usable color infor- mation, but large usable direction information. (d) Visualization
the activations of the last layer of Small FC from (a) at epochs [0, 10, 20, 100], where the correct color choice is denoted by the marker color (red or green) and the correct direction choice is denoted by marker shape (crosses or dots). After training the crosses and dots are overlapping, corresponding nearly zero usable color information and nearly bit of direction information. This is a minimal and sufficient representation to solve the task.
To test if this observed minimality was result of our simple task, we extended the CB task to variant with input checkerboard colors, with corresponding output direction classes. We trained networks using larger architecture (Medium FC). We show results for n = 10 and n = 20 classes in Fig Pb. We observed similar phenomena to the n case: there was decreasing usable color information in deeper layers, and nearly zero color information in the last layer's 's representation. In contrast, there was significant usable direction information across all layers in the asymptotic rep- resentation, with usable information about the direction increasing for deeper layers. We validated our results using different random initializations (Figures 10
These results show that, for simple task with SGD and random initialization, minimal sufficient representations emerge through training. Asymptotic representations were sufficient to perform the task, but contained less usable color information in deeper layers, approaching zero color informa- tion in the last layer. In this simple task, we observed that it was possible for the network to solve the task with nearly zero usable color information in its last layer across training (Fig pb.c).
We also examined how changing the initialization by pretraining the network to output the color choice affected the resulting representations. We found that the resulting representations were not
5

---

Published as a conference paper at ICLR 2021
fine
3.0
Informi Informetion (btt))
2.0
80
1.5
Usable
1.0
-50
0.0
120 140
Epoch
Figure 3: Usable fine and coarse class information in a ResNet-18 on CIFAR-10. The fine classes (show in blue) correspond to the 10 CIFAR-I0 classes. The coarse classes correspond superclass consisting of all the even and odd classes. We trained the network to output the correct coarse class, which corresponds to 1 bit of information. Through training epochs, while the validation accuracy (green dashed line) is increasing, the information about the coarse class also increases towards bit. Early in training. the usable information about the fine label also increased, even though the network was not explicitly provided any information about the fine class. Around epoch 100, the network "forgets" this fine label information. The scale of the validation accuracy is shown the right hand side of the plot.
minimal for the = 2 checkerboard case (Fig (a), retaining some structure from the initialization (l). This result also held for the CB task with 10 and n 20 (Fig (Fig(p.c). Furthermore, found that pretraining on the color choice led to worse generalization performance (Fig 7
4.2 ACQUISITION AND FORGETTING OF USABLE INFORMATION IN MODERN DEEP NETWORKS
Using a similar approach as we did for the CB task to characterize relevant and irrelevant informa- tion, we next investigated how modern deep neural networks trained with SGD learned task repre- sentations. To study learning dynamics, we investigated (1) how networks learned and represented task information well as information about representative semantically meaningful variable, and (2) how this information was represented across training epochs. To this end, we defined coarse la- bels corresponding to groups of classes in the CIFAR-10 and CIFAR-100 datasets. The CIFAR-100 dataset defines fine labels corresponding to each of the 100 classes, as well as 20 coarse labels corre- sponding meaningful groupings of from the 100 classes. In the CIFAR- 10 case, defined two coarse labels arbitrarily, corresponding to even and odd class labels. Thus, when training the net- work to output the coarse label, we can investigate the network's representation of the semantically meaningful fine label description, which serves as a proxy for the computation and representations that the network is learning. We note that, when trained to output the coarse label, minimal repre- sentation should contain no additional information about the fine label.
We trained ResNet-18 (He et al. 2016 output the coarse label of CIFAR-10, using an initial learning rate of 0.1 with exponential annealing (0.97), momentum (0.9), and a batch size of 128. We investigated the usable information in the last layer of the ResNet-18, which has a dimension of 512. We found that while training the network to predict the coarse grained class, the network acquired information about the grained class, evidenced by an increase in usable information during training (orange curve) while validation accuracy (green dashed line; scale on the right hand side of plot) was increasing (Fig B Strikingly, while the validation accuracy and usable coarse-graines class information increased, the information about the fine labels first increased and then decreased (around epoch 100). then decreased to minimality, storing no additional usable information about the fine labels than contained in the coarse labels. These learning dynamics Shwartzz Ziv Tishby (2017), but due to controversies of their information estimation and experi- mental setup, have been widely debated (Saxe et al. 2018). We emphasize that even though did
proposed

---

Published as a conference paper at ICLR 2021
(a)
(c)
25--
Figure 4: Sensitivity to hyper-parameters. (a-c) The usable coarse and fine label information through training with a batch size 64, 256, and 512 (a batch size of 128 was used in Fig 3 learning dynamics only undergo compression at small batch sizes of 128 less. The validation accuracy is higher for smaller batch sizes as well. The plot of a batch size 1024 is in Fig g (d-f) Usable coarse and fine label information using initial learning rates of 0.075, 0.05 and 0.01 (a learning rate of 0.1 was used in Fig 3 With larger learning rates, the network observed an increase and decrease in fine label information. With smaller learning rate 0.01, the network exhibited an increase in fine label information, without subsequent decrease. The final validation accuracies (green dashed lines) are approximately comparable (96.5%, 96.8% and 95.8% respectively) though lowest with initial learning rate of 0.01 when the network did not form a minimal representation.
The
not ask the network to acquire information about the fine labels, SGD naturally led the network to learn information about the fine label, and then decreased this information later in training.
Together, these results show that SGD tends to result in minimal representations, which may be guided by interesting learning dynamics. To achieve this minimality, the network displays learning motif where it learns additional information early in training, then discards it later We next investigate how these findings depend on hyper-parameter choices, architecture, and task.
4.3 SENSITIVITY OF USABLE INFORMATION TRAINING DYNAMICS TO HYPER-PARR PARAMETERS, ARCHITECTURE, AND TASK
Using this framework, we evaluated how deep networks. We focus on the ResNet-18 trained on CIFAR-I0 in Figure We varied the batch sizes from 64 to 1024 and found that a small batch size led to dynamics similar to that of Fig 3 while a larger batch size did not lead to minimal representations (Fig 4-c). Results for a batch size of 1024 are shown in Fig 8 The learning rate also affected the learning dynamics. We found that all networks increased the information about the fine labels during training. However, we found that only for large initial learning rates did the network "forget" the superfluous information. Results for a learning rate of 0.001 are also shown in the appendix in Fig We found that small learning rates (0.001) or large batch sizes (512 larger) led to lower validation accuracy. Thus, the implicit regularization coming from the use of SGD with a small batch size and large learning rate, which common in practical settings, is crucial for learning minimal sufficient representations. Here have provided an underpinning for these choices by exposing their associated learning dynamics.
choices affected the learning dynamics in
Additionally, we investigated whether the phenomenon of acquiring "superfluous" task information was common across different architectures and tasks. We used an All-CNN (Springenberg et al.) 2015) trained CIFAR-10 output the binary coarse label, observing similar trend with an

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
KROW
CPAN-000
Fiee Couse
80
100
129
143
140
89
L23
149
160
Figure 5: Different Architecture, Task, and learning schedule (a) Using an All-CNN architecture Springenberg et al. (2015), we observe similar trend in the learning dynamics of usable informa- tion, with a increase and decrease in the fine label information during the CIFAR-10 task. This decrease does not lead to a completely minimal representation, though it does become close to min- imal. (b) We trained a ResNet-18 on the coarse labels the CIFAR-100 task, and tracked the information the network had about the fine and coarse label through training. We find that the net- work converges to an approximately minimal representation, though it did not undergo a noticeable increase and decrease in the fine label information, suggesting that this learning motif depends on the structure of the task. (c) Pretraining the network to output the fine labels before epoch 20 led to improved final performance (85.6% vs 83.5%) in (b). Note that the validation accuracy for the first 20 epochs was the validation accuracy on the 'fine' labels task, and was the validation accuracy on the 'coarse' task after epoch 20.
increase and decrease in the usable information about the fine label (Fig 5a). In this case, the infor- mation about the fine label did not decrease to minimality, but nonetheless, there was a significant reduction in the fine label information, suggesting that SGD naturally compresses additional input information. Finally, we evaluated how a ResNet- 18 represented task information using the CIFAR- 100 dataset. This dataset is accompanied with 100 fine labels and 20 coarse labels, corresponding to groupings of the 100 classes. We used the same to output the coarse labels, observing an increase to approximately 3.5 bits of usable information. The network achieved nearly minimal representation (p).
as in Fig We trained the network
It is important to note that for this setting of hyper-parametert in the CIFAR-100 task (the same as in the CIFAR-10 10 case). SGD did not show a visible increase followed by decrease in usable in- formation in the fine labels, result different than what we observed in CIFAR- 10. We conjectured this could be due to at least three potential reasons: (1) the hyper-parameter settings may be subop- timal, which we observed may result in learning dynamics that do not increase then decrease fine information (Fig D. f). (2) In CIFAR-100, coarse and fine labels are semantically similar, so there may not be not much more information to be naturally learned in the fine than the coarse labels, and further that it is possible that while the information about the fine labels remains approximately flat, the network is forgetting information about aspects of the fine labels while learning other parts of fine label information in the process of increasing coarse label information and arriving at nearly minimal representation. (3) CIFAR-100 has relatively few examples, 500 per fine label, impacting the learning of fine label information. Despite these limitations, our results from CIFAR-10 suggest that SGD learning dynamics that increase then decrease information about the fine label should re- sult in more optimal representations and higher validation accuracy. To test this, we performed an experiment where we pretrained the network to output fine label information until epoch 20, after which the network then was trained to output coarse information. This training process resulted in learning dynamics that resembled SGD learning in Fig We observed that these learning dynamics resulted in networks with a 2.1% increase in validation accuracy (compare Fig 5b and c). These results support that learning dynamics that increase, and then decrease, information about inputs, may result in more optimal representations that achieve higher validation accuracy.
5 DISCUSSION
We introduced a notion of the usable information in the representation, which reflects the amount of information that can be extracted by learned decoder, for understanding the training dynamics
8

---

Published as a conference paper at ICLR 2021
in deep networks. This definition is appealing, in part, due to its flexibility. For instance, if it is important to understand how accessible the information is to a linear decoder, it suffices to apply our formulation of usable information using linear decoder trained with cross-entropy loss. In contrast, the goal is to extract all information present in a representation, regardless of how accessible this information is, one can train high capacity nonlinear decoder. Since neural networks are powerful function approximators, as the function approximation improves, the decoder will approach the optimal decoder. In this case, the usable information approaches Shannon mutual information, as the lower bound becomes tight (Section A.II Future theoretical and empirical work should investigate the tightness of this bound and its dependence on training parameters.
In our case, we used a relatively small nonlinear neural network as the decoder, which provided insight into the evolution of optimal representations through training on simple tasks inspired by neuroscience literature and on image classification tasks. These tasks allowed us to show that the implicit regularization of SGD plays an important role in learning minimal sufficient representa- tions. In particular, in standard hyper-parameter settings, the network learns to encode semantically meaningful but ultimately irrelevant information early in training, before later discarding this information arrive at minimal sufficient representation.
observed learning dynamics where
Monkeys performing the checkerboard task, like our networks, also had minimal sufficient represen- tations in an output (motor) area (Chandrasckaran et al. 2017 Kleinman et al. 2019). Despite the obvious implementation differences of both information processing systems, we speculate that the general effects coming from noisy learning process, which led to minimal sufficient representations in our artificial networks, may be an important factor leading to minimal sufficient representations in biological networks.
is remarkable that in the CIFAR-10 task, SGD naturally exploited the semantically meaningful structure of the fine labels, in order solve the coarse labels task. In general, it is difficult to identify the features that are being learned during training, and whether they correspond to some- thing semantically meaningful. However by defining a coarse label, our task setup allowed us to study how semantically meaningful information was represented during training. During training, the network increased the information about the semantically important part of the input, even when only asked to output the coarse label. It then decreased the information later in training. We did not notice such a major increase in CIFAR-I00, perhaps due to the nature of the dataset or hyper- parameter configuration. However, by inducing the network to follow similar learning dynamics to Fig by pretraining the network to output the fine labels, we were able to improve the performance on the coarse labelling task. This suggests that a detailed understanding of the training dynamics and the features learned is important for learning optimal representations and successfully transferring representations between tasks.
Using usable information, we observed an increase and decrease in the information about an inrele- vant variable, which has been proposed (Shwartz-Ziv & Tishby| (017), but has been debated, largely due to controversies over the estimation of Shannon's 's mutual information (Saxe et al., 2018). Our observation in accordaace with the ideas of Shwwatzz Ziv & Tishby (2017). and importantly we have observed these dynamics on modern architectures and realistic tasks. Our results are also con- sistent with a complementary view of information in the weights, where it has been observed that the Fisher Information increased and decreased during training (Achille et al. 2019), corresponding to a critical period in neural network training.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their thoughtful feedback and suggestion of new experi- ments. MK was supported by the National Sciences and Engineering Research Council (NSERC). JCK was supported by an NSF CAREER Award 1943467, a UCLA Computational Medicine Ama- zon Web Services Award, and NVIDIA GPU Grant.
9

---

Published as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50):1-34, 2018. URL http:
h/jmlrr.org//aaa pttpars//vv/11 1011911--20 646. html
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep net- works. In International Conference on Learning Representations, 2019. URL https://
openreview met/ffruuu wlt.orummmddBBBBttcccc....
David Barber and Felix Agakov. The im algorithm: variational approach to information maxi- mization. In Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS'OS 03, pp. 201-208, Cambridge, MA, USA, 2003. MIT Press.
Chandramouli Chandrasekaran, Diogo Peixoto, William T. Newsome, and Krishna V. Shenoy. Lam- inar differences decision-related neural activity in dorsal premotor cortex. Nature Communica- tions, 8( 8(1):614, 2017.
Yann Dubois, Douwe Kiela, David J. Schwab, and Ramakrishna Vedantam. Learning optimal rep- resentations with the decodable information bottleneck, 2020.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems 32, pp. 10677-10687. Cur- ran Associates, Inc., 2019.
Ziv Goldfeld, Ewout van den Berg, Kristjan H. Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow in neural networks. CoRR, abs/1810.05528, 2018. URL http: /ttt::///axiii
wttp/xxivv..o// wwwp:vv...rraavoooooooooooooerr abs 118.1..000 05728
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.11091 N0.119/CVPR.2016
Michael Kleinman, Chandramouli Chandrasekaran, and Jonathan C. Kao. Recurrent neural net- work models of multi-area computation underlying decision-making. bioRxiv, 2019. doi:
10.1101/798553, URL htt.:p/ee/wwi.....oaa.r...oo///e
798553
Alexander Kraskov, Harald Stögbauer. and Peter Grassberger. Estimating mutual information. Phys- ical Review E, 69(6), Jun 2004, ISSN 1550-2376. doi: 10.1 1001103/pyyyrrvv.6..0066 18660666......6.66666.... URL
http: httt:://dx.do doi 100 03/ 69. 066138
Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 156):1191--253, doi: 10.1 (00116266/9766633227782222 URL :tttps://do:. .org/
June 2003. ISSN 0899-76677
10. 11 0111111222999900333333 780272
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5171-5180, Long Beach, California. USA, 09-15 Jun 2019. PMLR. httthttpp///ooo//
mir pttpppptp/ss.opseeeeeeeee
Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Bren- dan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference Learning Representations, 2018. URL https: //openreview.
derr-dyGyyyyyy-- WPG-A-
Ravid Shwartz-Ziv and Nafaai Tishby. Opening the black box of deep neural networks via informa- tion. CoRR, abs/1703.00$ 2017. http://t://///x: wwtp:/wwwwg/////////////oo 008100
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net, 2015.
10

---

Published as a conference paper at ICLR 2021
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In 8th International Conference on Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
htttss://ppeneviie ew. net forum?id=r BBeyFFDD

---

Published as a conference paper at ICLR 2021
A PROOFS
A.
USABLE INFORMATION LOWER BOUNDS THE MUTUAL INFORMATION
The entropy of distribution is defined
log 1 p(z)
H = En-p(a)
(2)
The mutual information, I(X;Y) can be written terms of an entropy term and a conditional entropy term:
I( - H(Y12).
(3)
We want show that:
I(;Y)) 2 (,(2Y) = = H(Y - LoE -Lr(p(ul))
(4)
It suffices to show that:
2Y S LCE
(5)
where LeB is the cross-entropy loss on the test set. For our study, HY) represented the known distribution of output classes, which in our case were equiprobable.
ZY = ((x.)))((2.0)
log p(ylz)
(6)
log q(ylz)
Eesp(() (KLp((l)))loo()
cross-entropy loss
< ((2.g))p(2..) log 4 =)
LCE
(8)
To approximate H(Y Z), we first trained a neural network with cross-entropy loss to predict the output, Y, given the hidden activations, Z, learning a distribution q(vl2). The KL denotes the Kullback-Liebler divergence. We multiplied (and divided) by an arbitrary variational distribution ((v|=) in the logarithm of equation 6 leading to equation?? 7 The first term in equation 7 is the cross- entropy loss commonly used for training neural networks. The second term is a KL divergence and is therefore non-negative. In our approximator, the distribution 4(y(z) is parametrized by neural network. When the distribution (ıy(2) = p(y[2), our variational approximation of H (Y)))), and
hence approximation of T(Z:Y) Y) is exact (Barber & Agakov 2003 Poole et al. 2019
B ADDITIONAL RESULTS AND DETAILS IN THE CHECKERBOARD TASK
B. SGD WITH NON-RANDOM INITIALIZATION MAY NOT FORM MINIMAL REPRESENTATIONS IN THE CB TASK
Implicit regularization in SGD hypothesized result in minimal representation through com- pression of irrelevant input information, also called a "forgetting" phase (Shwartz-Ziv & Tishby) 2017 Achille Soatto 2018 Achille et al. 2019). We tested this hypothesis by initializing net- works with significant color information, and subsequently performing SGD on the CB task. We then evaluated whether SGD resulted in networks with minimal color representations. We initialized the weights by pretraining the network to output the color decision for 20 epochs, which required the network represent color information. After epochs, we reverted to training on the CB task, where only the direction decision was reported. Since the learning rate kept constant, the pretrained weights can be viewed as a different initialization in parameter space for the modified task.
Strikingly, we found that the resulting representations were not minimal for the n = checkerboard case (Fig (4.) This result also held for the CB task with 10 and 20 (Fig 7p.c). While we observed some compression of usable color information through training, the asymptotic repre- sentations had significantly greater than zero color information. In Fig 7b, we observed all layers
12

---

Published as a conference paper at ICLR 2021
(a)
50
Epact
Eponn
(d)
Epoch:
Epoch:
20
0:15
2.0
20
0.10
10
13
0.3
holaan ont
Iddan use
J Widne
0.0g 0g.25 5209
20
0.5
Nottin
Nldden
Addua
Figure 6: Usable color and direction information in a network through training following pretraining the network output color, direction. Pretraining occurred for the first 20 epochs, indicated by the dashed red line. Subsequently. the network was trained to output direction, in Fig 2 (a) Usable information for Small FC trained on the N CB task. Usable color information increased in training, and decreased when the loss function changed. However, the asymptotic representation not minimal.
(b) Medium FC trained on N 10 task. Similarly, the network formed representation of color during pretraining, but the asymptotic representation is not minimal. (c)
Medium FC trained on N 20 checkerboard task. (d) Visualization the Small FC network in (a) showing that an optimal representation is not formed. The asymptotic representation in the last area has separate representations for red and green crosses. These should be overlapping in a minimal representation.
had more usable color information than the direction information in the first layer. The network therefore solved the task using alternative representation that was not minimal. We visualized the activations corresponding to the asymptotic non-minimal representations of Small FC in Fig In the early epochs the red and green points converge (both crosses and dots) cessful pretraining. However, when we trained the CB task starting at epoch 20, the representations changed. While the dot clusters for red and green checkerboards overlapping, the cross clusters are not. This representation is not minimal as color information can be decoded above chance.
result of suc-
These results show that the initialization affects the asymptotic representation of neural networks. SGD, under particular initializations, may not lead to minimal representations of the task inputs. This suggests there is trade-off between learning a minimal representation and simply reusing the existing representations present in the initial weights. Initial structure in the network representations from pretraining, such as the separation of the red and green crosses in the last layer representation, was maintained even when performing SGD to train a different task. Together, these results suggest that while SGD compresses representations towards minimality, it finds solution that is functionally related to the initial representation. This may correspond to optima in the neighborhood of the initialization.
B.2 RELATIONSHIP BETWEEN PRETRAINING, MINIMALITY, AND GENERALIZATION THE CB TASK
Our results show that the minimality of network representations, and therefore solutions, depends initialization. All trained networks (for larger than 2), however, achieved zero training
13

---

Published as a conference paper at ICLR 2021
1000
075
55088
550
Usite
Ustto abarat
22
-54.0
0.4
-09.00
02-
-50.6
05.
06.0
03-
13
13
(c)
(d)
6.-
dricton
coo
6500
225
-26
500
2.-
Usitoo stanalee
177
Ustteo
-80
-75
Eaoch
Epooc
Figure 7: (a) Final usable information and validation accuracy (green dashed line) function of pretraining epoch for the CB task (n = 2) averaged over 8 random initializations. (b) Final usable information and accuracy over 8 random initializations. (c) Final usable information and accuracy as a function of pretraining 20) averaged over 8 random initializations.
function of pretraining epoch for the CB task
10) averaged
epoch for the CB task and accuracy as function of pretraining epoch for the CB task (n = 25) averaged over 8 random initializations. Error bars show the S.E.M.
(d) Final usable information
ror. A natural question to ask how does the pretraining affect the resulting representation and generalization performance?
To answer this, we varied the number of epochs that we pretrained the CB tasks of n = 2, n 10, and 20 classes, and quantified the usable color and direction information, well as the trained network's test accuracy to understand how the network generalizes (Fig. 7 We found that networks trained with longer pretraining had less minimal representations and worse generalization performance. This was true regardless of the number of classes, but the effect was more pronounced (in terms of absolute difference in accuracy) when the network did not solve the task perfectly without pretraining. We note that regardless of how long the networks were pretrained for, the networks were subsequently trained for the same number of epochs (80), with the same learning rate throughout training. One interpretation is that when using existing structure to solve the task, network learned a suboptimal solution to solving the task, increasing the chance of overfitting. Another interpretation is that the pretraining changed the distribution of the weights, affecting the minimality and gencralization.
B.3 DETAILS NEURAL NETWORK FOR USABLE INFORMATION IN THE CB TASK
To estimate usable information, we computed the cross-entropy loss of a decoder d(vl2) that pre- diets from Z. The decoder was a three-layer neural network, with 128, 64, and 32 units per layer, with Leaky-ReLU activations (slope 0.2), batch-norm and dropout (p 0.7). At each epoch, 1250 training samples were generated and supplied to the decoder, along with either the corresponding correct direction or color choice. We evaluated the roos-entroppy loss 3750 test samples to min- imize overfitting. We trained the network for 100 epochs using a learning rate 0.5 for 'Medium FC' and 0.05 for 'Small FC.
14

---

Published as a conference paper at ICLR 2021
B.4 CHECKERBOARD TASK DESCRIPTION
Following the conventions of Kleinman al. 2019), we modeled the CB task (Fig To. (FigTo) inputting the checkerboard color and target configuration to a neural network that outputted the direction choice (Fig [T).. We minimized the cross-entropy loss of the network output and the ground truth output. We extended the checkerboard task to the n checkerboard task by increasing the number of checkerboards. Each target was 1 out of the n colors, with the targets forming correct direction corresponds to the direction of the target having the same color of the checkerboard. We specified the color of each target using one-hot encoding, and the color of the checkerboard as a one-hot encoding. Noise with mean 0 and standard deviation of 0.1 was added to the checkerboard inputs. The target and checkerboard color inputs were concatenated to form an input vector. The correct direction of the target was the output.
'n-polygon' The
B.5 DETAILS OF CB EXPERIMENTS
The following are the hyper-parameter used our experiments. We trained two different network architectures, 'Small FC": 5 layers, with 10 - 7 --- 4 - 3 units in each layer, 'Medium FC": 100 20 20 20. We trained networks using SGD with a constant learning rate throughout training.
FC Small, 2:
batch size: 32, learning rate: 0.05, number of data samples: 10000 (90% train, 10% vali- dation)
Medium FC, 10:
batch size: 64, learning rate: 0.5, number of data samples: 25000 (90% train, 10% valida- tion)
Medium FC, 20:
batch size: 128, learning rate: 0.5, number of data samples: 50000 (90% train, 10% vali- dation)
Medium FC, 25:
batch size: dation)
128, learning rate: 0.5, number of data samples: 75000 (90% train, 10% vali-
B.6 DEFINITION OF RELEVANT AND IRRELEVANT INFORMATION IN THE CB TASK
In the CB task, the color of the checkerboard and target configuration (inputs) are necessary to determine the correct direction reach (output). While both color and direction decision made, after the direction is determined, the color decision no longer needs to be represented: the network can generate the correct output with only the direction representation. Formally, the output is conditionally independent of the color representation, Z,, given the direction representation Z₂ (i.e., y (Ze, Za,, as illustrated by the graph in Fig (p). Hence, given representation of the direction choice, the color choice (and target configuration) no longer needs to be represented. We emphasize that, in general, the output not independent of the color representation and target configuration representation Z., i.e., y JL (Zc, Zz), hence information about the dominant color of the checkerboard is necessary to compute y. When this conditional independence holds, we call the conditionally independent variable "irrelevant." We therefore refer to the color choice "irrelevant" and the direction choice as "relevant." We study how these components evolve together throughout training.
15

---

Published as a conference paper at ICLR 2021
42
-0
Awy
Usabbe Information 16035
Usable intermatonn (brea)
Validaiinn DCC CCC//DS
34
1.5
2.5
2.0
-5
0.5
0.
-50
9.0
70
0.
4.
140
160
100
Epoch
Epoch
Figure 8: (Left)) Usable information for initial learning rate of 0.001 in CIFAR-10. The information about the fine labels does not decrease, and the validation accuracy only reaches 92%, in ntrast to Fig where the validation accuracy reached 96%. (Right) Usable information for batch size of 1024 in CIFAR-10.
C ADDITIONAL RESULTS AND DETAILS IN THE CIFAR-10 10 AND CIFAR-100 TASK
C.I CIFAR-10 AND CIFAR-100 TASK DESCRIPTION
We trained ResNet-18 and an All-CNN architecture to output superclass corresponding to the twenty coarse-grained classes in CIFAR-100 and, in CIFAR-IO, arbitrary superclass corre- sponding to the even and odd classes. Accordingly, a minimal representation should only encode the superclass.
C.2 DETAILS OF NEURAL NETWORK FOR USABLE INFORMATION
To estimate usable information, computed the cross-entropy loss of decoder ((yl2) that predicts from Z. We used a two-layer neural network, with 200 and 100 with Leaky-ReLU activations (slope 0.2), batch-norm and dropout (p 0.7). At each epoch, 7500 samples were supplied to the decoder, along with either the corresponding correct direction or color choice. We evaluated the cross-entropy loss 2500 test samples. We trained the network for 50 epochs using Adam with learning rate of 0.01 and weight decay of 0.001.
C.3 DETAILS OF NEURAL NETWORK TRAINING
In our experiments, unless otherwise stated, we trained a ResNet-18 (He al. 2016) with an ini- tial learning rate of 0.1 decaying smoothly with factor of 0.97 each epoch, batch size of 128, momentum of 0.9 and weight decay with coefficient 0.0005. For the All-CNN (Springenberg et al, 2015) we used batch size of 128, initial learning rate of 0.05 decaying smoothly by factor of 0.97 at each epoch, momentum 0.9, and weight decay with coefficient 0.001. We used standard data augmentation with random translations up 4 pixels and random horizontal flipping. These parameter configurations were taken directly from prior work (Achille et al. 2019).
D ADDITIONAL PLOTS
16

---

Published as a conference paper at ICLR 2021
Unita
Cyshh
Opoct
Bridion
09
800
Figure Evolution of usable information for eight random initializations for the 2 CB task.
Andoo
U10099
03
09
800
Giecion
Cooe
185
Figure 10: Evolution of usable information for eight random initializations for the 10 CB task.
17

---

Published as a conference paper at ICLR 2021
Sincion
Olation
11000 0001
Andson
Figure 11: Evolution of usable information for eight random initializations for the n 20 CB task.
USB
Undbl
Clacion
69
0.0
Cruch
Cooc7
Figure 12: Evolution of usable information for eight random initializations for the = 2 CB task with epochs of pretraining. If the the usable information was negative, indicating that the decoder overfit, we set the usable information to O. Note that this occurred for very small number of points.
18

---

Published as a conference paper at ICLR 2021
1.3
1200 P80 0000 000
80
Figure 13: Evolution of usable information for eight random initializations for the n 10 CB task with 20 epochs of pretraining.
10000 (381
3.00
80
Brution
Figure 14: Evolution of usable information for eight random initializations for the CB task with 20 epochs of pretraining.
19


Reviews:
[{"review_id": "lr9ubnQ1CjD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a notion of \"usable information\" in neural network representations (essentially, decodability of information by a neural network), and suggests that learned representations are \"minimal\" (discard task-irrelevant information) when training begins from random initialization, but not necessarily when beginning from other initializations. \n\nPros:\n\n-- the definition of usable information is reasonable and likely useful for future analyses (though see below)\n-- the questions addressed by the paper are interesting / important and are in need of thorough empirical study\n\nCons:\n\n-- the tasks used in this paper are very simple, with even/odd MNIST classification being the hardest task considered (and most analysis is conducted on an even less complex task, which is similar to a simple XOR).  It is very hard to know whether the paper's conclusions would generalize to tasks of interest to the machine learning community, or even to other simple tasks with different structure\n\n-- It is not clear from the pretraining experiments whether the negative results are due to pretraining or just the scale of the weights.  If networks were initialized randomly with mean / std taken from the pretrained network, would they also not learn minimal representations?\n\n-- It seems that the results of the paper must necessarily depend on several hyperparameters which were not explored.  For instance, if the learning rate in early layers is set sufficiently small, the network should learn these simple tasks without minimal representations.  \n\n-- The result about generalization correlating with minimality was not confirmed on MNIST.  It is not clear whether this is because the result does not hold on MNIST or simply because the authors did not test it.\n\n-- Transfer learning is known to be helpful in some practical settings.  Is this framework able to account for situations when transfer might be helpful, as well as harmful?  More discussion of this is needed\n\n\nOverall, given that this is an empirical paper (no new theory is provided), it is important for the experiments to be extensive and comprehensive.  The experiments in this paper, though they touch on interesting ideas, are not thorough enough to convince a reader of the authors' broader claims.  \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iB4JfblGLD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies how initialization and the implicit regularization of SGD affect the training dynamics of neural networks in terms of minimality and sufficiency of learned representations. The main findings are that 1) SGD with random initialization learns almost minimal and sufficient representations and 2) SGD with an initialization that contains information about irrelevant factors fails to converge to minimal representations, increasing the chance of overfitting. These findings are interesting, useful for understanding neural networks, relevant to the ICLR community, but lack evidence of generality.\n\n**Task choices.**\nMost of the experiments in this paper are done on the checkerboard task. In this task one is given a checkerboard, each cell of which is either red or green. One of these colors appears more (is dominant) and the task is to decide which color is dominant. The key aspect is that the input also contains 2 targets: left and right, with one being green and one being red. Instead of directly predicting the dominant color, the subject should pick the target whose color matches the dominant color. Importantly, the color arrangement of targets is picked at random. The authors also consider the task of predicting whether an MNIST digit is odd or even. These two tasks are too simple, which restricts the generality of conclusions. I suggest to consider harder tasks, for example classifying CIFAR-100 images, where the target is the superclass (1-20) and the irrelevant factor is the exact class (1-5). Additionally, it would be interesting to consider cases when the training data is such that there is a small mutual information between the irrelevant factor and the target. Will SGD with random initialization find a solution that has even smaller mutual information with irrelevant factors (i.e. sacrificing sufficiency for minimality)?\n\n**Network.**\nThroughout the paper only fully connected networks are considered. Additionally, the last hidden layer always has <= 20 units. For generality, it would be better to consider also larger and more modern networks, such as ResNets.\n\n**Activation function.**\nSaxe et al. [1] showed that the choice of activation function is crucial when judging about compression in late stages of training. The presented paper can be improved by considering other choices of activation functions.\n\n**The role of SGD.**\nThe implicit regularization of SGD arises from its stochasticity. The findings of this paper suggest that this stochasticity has a key role in finding minimal representations. This should be verified by comparing to standard gradient descent (i.e. batch size = number of examples).\n\n**Minor comments**  \n1. The description of the checkerboard task starting at the last paragraph of the first page can be improved.\n2. Did you consider using a more powerful decoder? In Fig. 2c for example, we see that later layers have more usable information about the direction. This means that there is a room for strengthening the decoder.\n\nP.S. I am willing to increase the score if the authors address the above concerns about generality.\n\n# Update\nThanks for the rebuttal, it addressed my main concerns. The new results on CIFAR-10 and CIFAR-100 with fine and course labels match the results on the checkerboard task. This increases the generality of the main claims. The new experiments also confirm that the level of noise in SGD has a key role in finding minimal representations. Furthermore, they show that when training with SGD with enough amount of noise, the usable information with fine labels increases initially and then decreases. This improves our understanding of the phenomenon introduced by [2], which was later debated by [1]. For these mentioned reasons, I updated the rating from \"5: Marginally below acceptance threshold\" to \"7: Good paper, accept\".\n\n**References**  \n[1] Saxe, Andrew M., et al. \"On the information bottleneck theory of deep learning.\" Journal of Statistical Mechanics: Theory and Experiment 2019.  \n[2] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR,  abs/1703.00810, 2017.", "extra_scores": {}, "extra_text": {}}, {"review_id": "YSrVSP67rBH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors contribute to the recent research on whether neural network training (in particular, SGD) favors minimal representations, in which irrelevant information is not represented by deeper layers. They do so by implementing a simple neuroscience-inspired task, in which the network is asked to make a decision by combining color and target information. Importantly, the network's output is conditionally independent of the color information, given the direction decision, so the color information is in some sense irrelevant at the later stages. Using this, the authors quantify the 'relevant' and 'irrelevant' information in different layers of the neural network during training. Interestingly, the authors show that minimal representation are uncovered only if the network is started with random initial weights. Information is quantified using a simple decoder network.\n\nThe article is clearly written and has a simple (in a good way) and interesting message. However, I also have some criticisms, especially regarding the conceptual underpinnings.\n\nWhen any neural network is predicting a deterministic function f : X -> Y, *all input features* are irrelevant to the output distribution when conditioned on the output itself. In other words, the minimal representation in a deterministic task is simply the output itself. (The situation is different when the task involves predicting a non-degenerate probability distribution P(Y|X), in which case the minimal representation -- i.e., the sufficient statistics -- can have an arbitrary amount of information.) In the information bottleneck community, this was mentioned in https://arxiv.org/pdf/1703.00810.pdf (section 2.4) and explored in https://arxiv.org/abs/1808.07593.\n\nIn motivating the paper, the authors appear to confuse two types of \"irrelevant features\": \n\t(1) when an input feature is useless for prediction, i.e., changing it does not change the predictions, and \n\t(2) when information about an input feature is independent of the output distribution, when conditioned on the output.\nFor a deterministic prediction task, all features type 2, but not all features are type 1. The authors have the following text:\n      \"We believe this task ... captures key structure from deep learning tasks. For example, in image classification, consider classifying an image as a car, which take on various colors. A representation in the last layer is typically conditionally independent of irrelevant input variations (i.e., the representation does not change based on differences in color).\" \nIf I understand the example, this is building off the intuition that \"color of car\" is irrelevant because it is a type 1 feature (not useful for prediction). In fact, it can be conditionally independent because it is type 2. Moreover, in the authors' task \"color of checkerboard\" is not type 1 (it is very relevant for the output -- changing it changes the output) but it can also be conditionally independent (since it is type 2).\n\nGiven the above arguments, the degree to which features are conditionally independent in middle layers does not necessarily reflect how useful they are for prediction.\n\nI have two other, more minor comments:\n1) The notion of \"direction information\" is somewhat confusing, as one can think about two kinds of direction information: (1) information about which targets (i.e., directions) correspond to which colors (which is provided as part of the input), and (2) information about the final reaching direction (i.e., the output). Given the points made above, if I understand correctly, information about which targets correspond to which colors is just as irrelevant as the color information, when conditioned on the output. I would suggest referring to the second kind of information (the one mainly discussed in the paper) as \"output information\".\n2) The authors should probably cite (and may be interested in) https://arxiv.org/abs/2009.12789 (NeuroIPS 2020), which also proposes to estimate mutual information using a practical family of decoders.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LTsahI-y2dK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Broadly, this work is an attempt to understand how neural networks can form generalizable representations while being severely overparameterized. This work proposes an information theoretic measure, called the \"usable information\", and use it to quantify the amount of relevant information in different layers of a neural network during training. The key idea is that, in order for the information represented in one layer to be \"usable\" by the next layer, it should be decodable by a simple transformation (affine + element-wise nonlinearity).\n\nPros:\n- (significance) The \"usable information\" is a variant of mutual information, which replaces the expensive conditional entropy term with a cross-entropy loss that is more readily computable from a neural network. A computationally efficient measure for information has a potential for being a generally useful tool in the broad community.\n- (quality) I like the approach of this study, which resembles how natural science tries to understand the function of a complex system (such as the brain, i.e., the real neural network) empirically. The choice of the task was also appropriate: it provides a good intuition about the relevant vs. irrelevant information, as well as a bridge to neuroscience studies that may lead to insightful discussion.\n- (clarity) The paper is clearly written and easy to follow in most places.\n\nCons:\n- (originality) One thing I expected to find in the paper was some review of other information theoretic measures that were proposed/used in the context of neural networks; proposing a cheaper alternative for the mutual information itself can't be a new idea (although if it is, that would be worth noting too). It would be fair to include a discussion along this line, and perhaps point out the properties of \"usable information\" that makes it particularly appealing.\n- (clarity) The presentation of Fig 4 is not clear to me. (i) Which plots belong to which axis, and what is the third curve \"val\" in black? (ii) \"a positive correlation with the minimality of the representation and generalization performances\": this sounds vague. Can you quantify?\n\nAdditional comments: \n- \"Does SGD always lead to equivalent representations, or does SGD trace a path through parameter space that leverages structure present in the initialization?\": I don't understand this sentence in the introduction. Also related, it would be nice to add a sentence or two to unpack the idea of \"implicit regularization through SGD\".\n- In Fig 2d, why do the four marker types appear somewhat separated (although not by a large margin), e.g., red x, green o, green x then red o?\n- The observation about non-random initialization is very interesting. In this example, keeping the old information does not seem to compromise the performance of the network in the current task. Do you think this is generally true for neural networks, or could there be a regime where retaining information about a previously relevant (but no longer relevant) information has an actual cost?\n\nOverall, I think this is an interesting paper that presents a promising approach toward the understanding of how informative representations are formed by training, one of the most fundamental questions in deep learning.\n\n**UPDATE:** Most of my questions/comments are addressed in the revised version of the paper and the author responses. I maintain my support for acceptance.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes that we can understand the evolution of representations in deep neural networks during training using the concept of "usable information". This is effectively an indirect measure of how much information the network maintains about a given categorical variable, Y, and the authors show that it is in fact a variational lower bound on the amount of mutual information that the network's representations have with Y. The authors show that in deep neural networks the usable information that is maintained for different variables during training depends on the task, such that task irrelevant variables (but not task relevant variables) eventually have their usable information reduced, leading to "minimal sufficient representations". 

The initial reviews were mixed. A common theme in the critiques was the lack of evidence of the generalization and scalability of these results. The authors addressed these concerns by including new experiments on different architectures and the CIFAR datasets, leading one reviewer to increase their score. The final scores stood at 3, 7 ,7, 7. Given the overall positive reviews, interesting subject matter, and relevance to understanding learned representations in deep networks, this paper seems appropriate for acceptance in the AC's opinion.
Decision: Accept (Poster)