Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors contribute to the recent research on whether neural network training (in particular, SGD) favors minimal representations, in which irrelevant information is not represented by deeper layers. They do so by implementing a simple neuroscience-inspired task, in which the network is asked to make a decision by combining color and target information. Importantly, the network's output is conditionally independent of the color information, given the direction decision, so the color information is in some sense irrelevant at the later stages. Using this, the authors quantify the 'relevant' and 'irrelevant' information in different layers of the neural network during training. Interestingly, the authors show that minimal representation are uncovered only if the network is started with random initial weights. Information is quantified using a simple decoder network.

The article is clearly written and has a simple (in a good way) and interesting message. However, I also have some criticisms, especially regarding the conceptual underpinnings.

When any neural network is predicting a deterministic function f : X -> Y, *all input features* are irrelevant to the output distribution when conditioned on the output itself. In other words, the minimal representation in a deterministic task is simply the output itself. (The situation is different when the task involves predicting a non-degenerate probability distribution P(Y|X), in which case the minimal representation -- i.e., the sufficient statistics -- can have an arbitrary amount of information.) In the information bottleneck community, this was mentioned in https://arxiv.org/pdf/1703.00810.pdf (section 2.4) and explored in https://arxiv.org/abs/1808.07593.

In motivating the paper, the authors appear to confuse two types of "irrelevant features": 
	(1) when an input feature is useless for prediction, i.e., changing it does not change the predictions, and 
	(2) when information about an input feature is independent of the output distribution, when conditioned on the output.
For a deterministic prediction task, all features type 2, but not all features are type 1. The authors have the following text:
      "We believe this task ... captures key structure from deep learning tasks. For example, in image classification, consider classifying an image as a car, which take on various colors. A representation in the last layer is typically conditionally independent of irrelevant input variations (i.e., the representation does not change based on differences in color)." 
If I understand the example, this is building off the intuition that "color of car" is irrelevant because it is a type 1 feature (not useful for prediction). In fact, it can be conditionally independent because it is type 2. Moreover, in the authors' task "color of checkerboard" is not type 1 (it is very relevant for the output -- changing it changes the output) but it can also be conditionally independent (since it is type 2).

Given the above arguments, the degree to which features are conditionally independent in middle layers does not necessarily reflect how useful they are for prediction.

I have two other, more minor comments:
1) The notion of "direction information" is somewhat confusing, as one can think about two kinds of direction information: (1) information about which targets (i.e., directions) correspond to which colors (which is provided as part of the input), and (2) information about the final reaching direction (i.e., the output). Given the points made above, if I understand correctly, information about which targets correspond to which colors is just as irrelevant as the color information, when conditioned on the output. I would suggest referring to the second kind of information (the one mainly discussed in the paper) as "output information".
2) The authors should probably cite (and may be interested in) https://arxiv.org/abs/2009.12789 (NeuroIPS 2020), which also proposes to estimate mutual information using a practical family of decoders.

review_title: Studies minimality of neural network representations using a simple neuroscience-motivated task

final_score: rating: 3: Clear rejection
review_content: This work introduces a notion of "usable information" in neural network representations (essentially, decodability of information by a neural network), and suggests that learned representations are "minimal" (discard task-irrelevant information) when training begins from random initialization, but not necessarily when beginning from other initializations. 

Pros:

-- the definition of usable information is reasonable and likely useful for future analyses (though see below)
-- the questions addressed by the paper are interesting / important and are in need of thorough empirical study

Cons:

-- the tasks used in this paper are very simple, with even/odd MNIST classification being the hardest task considered (and most analysis is conducted on an even less complex task, which is similar to a simple XOR).  It is very hard to know whether the paper's conclusions would generalize to tasks of interest to the machine learning community, or even to other simple tasks with different structure

-- It is not clear from the pretraining experiments whether the negative results are due to pretraining or just the scale of the weights.  If networks were initialized randomly with mean / std taken from the pretrained network, would they also not learn minimal representations?

-- It seems that the results of the paper must necessarily depend on several hyperparameters which were not explored.  For instance, if the learning rate in early layers is set sufficiently small, the network should learn these simple tasks without minimal representations.  

-- The result about generalization correlating with minimality was not confirmed on MNIST.  It is not clear whether this is because the result does not hold on MNIST or simply because the authors did not test it.

-- Transfer learning is known to be helpful in some practical settings.  Is this framework able to account for situations when transfer might be helpful, as well as harmful?  More discussion of this is needed


Overall, given that this is an empirical paper (no new theory is provided), it is important for the experiments to be extensive and comprehensive.  The experiments in this paper, though they touch on interesting ideas, are not thorough enough to convince a reader of the authors' broader claims.  


review_title: Interesting ideas but insufficient experimental evidence

final_score: rating: 7: Good paper, accept
review_content: Broadly, this work is an attempt to understand how neural networks can form generalizable representations while being severely overparameterized. This work proposes an information theoretic measure, called the "usable information", and use it to quantify the amount of relevant information in different layers of a neural network during training. The key idea is that, in order for the information represented in one layer to be "usable" by the next layer, it should be decodable by a simple transformation (affine + element-wise nonlinearity).

Pros:
- (significance) The "usable information" is a variant of mutual information, which replaces the expensive conditional entropy term with a cross-entropy loss that is more readily computable from a neural network. A computationally efficient measure for information has a potential for being a generally useful tool in the broad community.
- (quality) I like the approach of this study, which resembles how natural science tries to understand the function of a complex system (such as the brain, i.e., the real neural network) empirically. The choice of the task was also appropriate: it provides a good intuition about the relevant vs. irrelevant information, as well as a bridge to neuroscience studies that may lead to insightful discussion.
- (clarity) The paper is clearly written and easy to follow in most places.

Cons:
- (originality) One thing I expected to find in the paper was some review of other information theoretic measures that were proposed/used in the context of neural networks; proposing a cheaper alternative for the mutual information itself can't be a new idea (although if it is, that would be worth noting too). It would be fair to include a discussion along this line, and perhaps point out the properties of "usable information" that makes it particularly appealing.
- (clarity) The presentation of Fig 4 is not clear to me. (i) Which plots belong to which axis, and what is the third curve "val" in black? (ii) "a positive correlation with the minimality of the representation and generalization performances": this sounds vague. Can you quantify?

Additional comments: 
- "Does SGD always lead to equivalent representations, or does SGD trace a path through parameter space that leverages structure present in the initialization?": I don't understand this sentence in the introduction. Also related, it would be nice to add a sentence or two to unpack the idea of "implicit regularization through SGD".
- In Fig 2d, why do the four marker types appear somewhat separated (although not by a large margin), e.g., red x, green o, green x then red o?
- The observation about non-random initialization is very interesting. In this example, keeping the old information does not seem to compromise the performance of the network in the current task. Do you think this is generally true for neural networks, or could there be a regime where retaining information about a previously relevant (but no longer relevant) information has an actual cost?

Overall, I think this is an interesting paper that presents a promising approach toward the understanding of how informative representations are formed by training, one of the most fundamental questions in deep learning.

**UPDATE:** Most of my questions/comments are addressed in the revised version of the paper and the author responses. I maintain my support for acceptance.
review_title: Interesting approach to understanding how representations form

final_score: rating: 7: Good paper, accept
review_content: The paper studies how initialization and the implicit regularization of SGD affect the training dynamics of neural networks in terms of minimality and sufficiency of learned representations. The main findings are that 1) SGD with random initialization learns almost minimal and sufficient representations and 2) SGD with an initialization that contains information about irrelevant factors fails to converge to minimal representations, increasing the chance of overfitting. These findings are interesting, useful for understanding neural networks, relevant to the ICLR community, but lack evidence of generality.

**Task choices.**
Most of the experiments in this paper are done on the checkerboard task. In this task one is given a checkerboard, each cell of which is either red or green. One of these colors appears more (is dominant) and the task is to decide which color is dominant. The key aspect is that the input also contains 2 targets: left and right, with one being green and one being red. Instead of directly predicting the dominant color, the subject should pick the target whose color matches the dominant color. Importantly, the color arrangement of targets is picked at random. The authors also consider the task of predicting whether an MNIST digit is odd or even. These two tasks are too simple, which restricts the generality of conclusions. I suggest to consider harder tasks, for example classifying CIFAR-100 images, where the target is the superclass (1-20) and the irrelevant factor is the exact class (1-5). Additionally, it would be interesting to consider cases when the training data is such that there is a small mutual information between the irrelevant factor and the target. Will SGD with random initialization find a solution that has even smaller mutual information with irrelevant factors (i.e. sacrificing sufficiency for minimality)?

**Network.**
Throughout the paper only fully connected networks are considered. Additionally, the last hidden layer always has <= 20 units. For generality, it would be better to consider also larger and more modern networks, such as ResNets.

**Activation function.**
Saxe et al. [1] showed that the choice of activation function is crucial when judging about compression in late stages of training. The presented paper can be improved by considering other choices of activation functions.

**The role of SGD.**
The implicit regularization of SGD arises from its stochasticity. The findings of this paper suggest that this stochasticity has a key role in finding minimal representations. This should be verified by comparing to standard gradient descent (i.e. batch size = number of examples).

**Minor comments**  
1. The description of the checkerboard task starting at the last paragraph of the first page can be improved.
2. Did you consider using a more powerful decoder? In Fig. 2c for example, we see that later layers have more usable information about the direction. This means that there is a room for strengthening the decoder.

P.S. I am willing to increase the score if the authors address the above concerns about generality.

# Update
Thanks for the rebuttal, it addressed my main concerns. The new results on CIFAR-10 and CIFAR-100 with fine and course labels match the results on the checkerboard task. This increases the generality of the main claims. The new experiments also confirm that the level of noise in SGD has a key role in finding minimal representations. Furthermore, they show that when training with SGD with enough amount of noise, the usable information with fine labels increases initially and then decreases. This improves our understanding of the phenomenon introduced by [2], which was later debated by [1]. For these mentioned reasons, I updated the rating from "5: Marginally below acceptance threshold" to "7: Good paper, accept".

**References**  
[1] Saxe, Andrew M., et al. "On the information bottleneck theory of deep learning." Journal of Statistical Mechanics: Theory and Experiment 2019.  
[2] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR,  abs/1703.00810, 2017.
review_title: Interesting findings, but not enough evidence of generality

Meta-review: This paper proposes that we can understand the evolution of representations in deep neural networks during training using the concept of "usable information". This is effectively an indirect measure of how much information the network maintains about a given categorical variable, Y, and the authors show that it is in fact a variational lower bound on the amount of mutual information that the network's representations have with Y. The authors show that in deep neural networks the usable information that is maintained for different variables during training depends on the task, such that task irrelevant variables (but not task relevant variables) eventually have their usable information reduced, leading to "minimal sufficient representations". 

The initial reviews were mixed. A common theme in the critiques was the lack of evidence of the generalization and scalability of these results. The authors addressed these concerns by including new experiments on different architectures and the CIFAR datasets, leading one reviewer to increase their score. The final scores stood at 3, 7 ,7, 7. Given the overall positive reviews, interesting subject matter, and relevance to understanding learned representations in deep networks, this paper seems appropriate for acceptance in the AC's opinion.
Decision: Accept (Poster)