Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Pretrained deep representations from large datasets have a plethora of applications across vision and natural language. Understanding how information is encoded across the dimensions in the representation is important to design better algorithms for the respective downstream tasks. 

This paper discusses that the learned representations exhibit significant redundancy. That is, not all the dimensions are needed for getting a decent performance on the downstream tasks, and propose a notion of diffused redundancy. The paper explores the existence of diffused redundancy across the architectures, upstream and downstream tasks, and learning algorithms. Based on the experiments, the paper concludes that the entire layer is not necessary for the downstream task, however, providing caution on how smaller representations sizes may cause unfair decisions
strength_and_weaknesses: I will go sequentially with the strengths and weaknesses

Strengths:
1) The Paper is well-motivated, decently written, and easy to understand. 
2) Systematic experimentation is done to understand redundancy in the representation of 4 smaller-scale datasets using ImageNet pre-trained models
3) Experimental findings are consistent across the downstream datasets, pre-training data, architecture, and learning algorithms (normal training, MRL, adversarial). 
4) The trade-off between representation size and fairness is very important (and also a good extension to MRL -- Kusupati et al., NeurIPS 2022)
5) The choice of definition of diffused redundancy with the assumption of $\delta=0.9$ is a good starting point.


Weakness:
1) The existence of redundancy is not novel, in fact, MRL has also discussed it in their appendix implicitly (diffused redundancy experiments with particular feature size) - however, concretizing it is a good first step taken by this paper
2) Baseline with correlation regularizer should’ve been discussed - “Reducing Overfitting in Deep Networks by Decorrelating Representations”
3) The paper should cite and discuss papers with structured pruning which are very relevant to the potential redundancy in the hidden layer.
4) Figure 1 should also compare algorithms, that is CKA v/s fraction to better understand if some algorithms have less redundancy than others. 
5) Plain CKA numbers are a bit hard to understand. That is, for certain tasks, a CKA of 0.7 may be good, but for other, it may not be. Therefore, a scatter plot downstream task performance v/s CKA at different neuron fractions perhaps could be more informative. 
6) On Figure 4: When we are comparing the pre-training datasets, it is difficult to draw conclusions if the accuracy difference for full set of neurons differs quite a bit (which is the case in this figure)
7) The figures could be made more easily readable, it is hard to go through the densely packed plots. 
8) Lastly, while the experiments are a good first step, the datasets often are easy -- I would like to see two sets of ablations a) with the variation of $\delta$ from 0.8 to 0.99 and b) experiments on harder datasets like ImageNetV2 and Places365. This would make the paper extremely strong and would help any future papers built on top of it. 

I am very excited about this paper and want to have a conversation with the authors through rebuttal and revision. I want the authors to take the weakness in good spirit to help improve the paper to make it worth a strong publication

clarity,_quality,_novelty_and_reproducibility: See the above section. The paper is highly reproducible with code in the supplementary. 
summary_of_the_review: Good step towards quantifying the redundancy in the representations. Experiments on small scale datasets and some missing baselines. However, a strong effort towards something useful at scale.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: # Intro / Section 2
- This paper introduces the property of diffused redundancy: any randomly chosen subset of a layer can perform similarly to the full layer on downstream tasks.
- The degree of diffuse redundancy is investigated for the final layers of image classification models trained on imagenet.
- Two downstream tasks are used to measure diffused redundancy, 1) representational similarity (CKA) between the whole layer and random subsets of the layer, and 2) classification accuracy when using the whole or a random subset of the layer to train a linear classifier for a downstream dataset.
- Redundancy is when a high performance can be accomplished with a small fraction of neurons.
- Diffusivity is when the variability of performance is low for different random subsets of neurons.

# Section 2.1 / Figure 1 
- Representational similarity and downstream accuracy are plotted for different sizes and random subsets of the full layer. A comparison is done between four downstream datasets (cifar10/100, flowers, oxford-iit-pets), and two models (ResNet50 with and without adversarial training).
- Degree of diffusion and redundancy depends on the downstream dataset.
- The adversarially trained model demonstrates greater diffused redundancy.

# Section 2.2 / Figure 2
- Plots of representation similarity between randomly chosen subsets.

# Section 3 / Figure 3
- Downstream accuracy vs subset sizes is compared for a larger collection imagenet classification models.

# Section 3.1 / Figure 4
- Downstream accuracy vs subset size is compared for models trained on imagenet1k and imagenet21k.
- The imagenet21k models demonstrate greater downstream accuracy.

# Section 3.2 / Figure 5
- Downstream accuracy vs subset size is compared for models with varying final layer sizes.
- The size of the final layer is proportional to downstream accuracy.

# Section 3.3 / Figure 6
- Downstream accuracy vs subset size is compared for models that are trained with MRL (smaller parts of the representation have structure).
- The MRL-optimized representation performs well for very low numbers of neurons, but is outperformed for larger subsets.

# Section 4 / Figure 7
- The loss in downstream accuracy when reducing the size of the random subset is primarily in a few classes instead of being uniformly distributed.
-This suggests that exploiting diffused redundancy has implications for fairness.
strength_and_weaknesses: # Strengths
- Demonstrates an interesting property of image classifiers.
- Comparisons are done between many different models.
- Potential drawback of creating a class imbalance is highlighted, although this somewhat contradicts the diffused redundancy hypothesis.

# Weaknesses
- The application to model optimization is limited since this technique only reduces the size of the final layer. There could be some comparison in memory usage, floating point operations, or model size to motivate DR as an optimization technique.
- It is not clear if there is any use for the definition of diffused redundancy (DR) (equation 1), while the definition states that it will be used to rigorously test the DR hypothesis. Figure 1a (downstream accuracy) and 1b (diffused redundancy) appear to convey the same information. The rest of the paper does not use this definition.
- Section 2.2 claims to explain why any random subset will work for downstream tasks. Figure 2 shows that two randomly chosen subsets have high representational similarity to each other. This further confirms that any random subset works, but does it explain why? After seeing that a random subset has a similarity to the full layer in figure 1, it is not surprising that random subsets are also similar to each other.
- In figures 3, 4, and 6 I believe the full representations all have different sizes, but they are not displayed. I am wondering if it is more important to show the absolute number of neurons on the x-axis instead of the fraction.
clarity,_quality,_novelty_and_reproducibility: # Clarity / Quality
- In Figure 1, it is not stated whether diffused redundancy is using CKA or downstream accuracy as the task. The acronym for adversarial training (AT) is never explicitly defined.
- Needs some proofreading. For example: "hence right more point" was probably supposed to be "hence, the rightmost point".

# Novelty
- I am not familiar enough with the related works to evaluate the novelty.

# Reproducibility
- Code and implementation details are provided.
summary_of_the_review: Diffused redundancy is an interesting property of the final layer of image classifiers, but its application is not clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors of this paper examine the notion of representation redundancy in large networks, and more specifically how it is diffused across the neurons of the representation. The authors quantify the measure of representation redundancy they use and demonstrate that dropping a large amount of neurons from the last layer randomly only affects the downstream accuracy by a small amount. The authors evaluate their observation on pretrained models on ImageNet, and demonstrate the level of redundancy present in these pretrained representations.
strength_and_weaknesses: Strengths:

- The paper poses an interesting question overall, in that the authors consider whether the learned representations have any particular structure in how information is redundant in them, and demonstrate that the representations produced by the network are significantly redundant.

- The experimental portion of the paper demonstrates the claim that the authors make, regarding redundancy in the representations of the network. More specifically, it can be seen that in most cases, the downstream accuracy in a given task decays slowly, as the number of neurons in the representation is decreased. The authors also perform a large set of experiments to show what precisely influences diffused redundancy (varying upstream/downstream tasks, layer width and existence of adversarial training).

Weaknesses:

- While I agree with the core observation of the paper, I also don’t find it particularly surprising. It is expected that a large part of the representation produced by the network is redundant, and can be removed without significantly harming the performance of the network. The main point the authors make is that this can be done by removing random neurons of the representation, which shows that this redundancy is spread across the entire representation. However, one can reasonably expect that, since the representation is still in high dimensions, the samples will still be well separated, with only a few neurons kept. To improve this, I believe that the authors should further elaborate on the implications of their observation in the introductory parts of the paper. 

- In the experimental section, a simple baseline one can compare against is keeping only the principal directions of the data, via PCA. While this is more complex than simply dropping neurons (since it requires a linear combination of the weights) I believe it would be useful to include this comparison as well.

- When comparing downstream tasks, I think it would also be useful to include the evaluation that arises when the downstream task is the same as the upstream (for example, ImageNet-1K). I think this would be useful to demonstrate whether the observation still holds in a more complicated downstream task.

- The authors include a section in their paper that touches upon issues of fairness that may arise by minimizing the size of the representations of the model. However, I think this part of the paper should be further extended. As it is, it seems to be somewhat disjoint from the rest of the paper. This can easily be an important topic for future work, so I think it should be highlighted further.

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear. The point that the authors want to make (that it is possible to drop a large fraction of neurons from the network and still retain downstream performance) is clear, and the paper overall is easy to read. As a minor comment, I would suggest the authors rearrange the figures in page 6, so that they don't cut in the text.

Regarding originality, as stated above the authors make an interesting observation, which however is not that surprising in my opinion. I believe that further analysis on the implications of this observation is needed to fully support this work.

Finally, the authors provide in the Appendix the necessary hyperparameters to reproduce their results.
summary_of_the_review: Overall, I think this paper makes an interesting observation, which however requires further analysis on its implications. Currently, I think that there needs to be a better understanding on how the observation made by the authors can be used. Elaborating on the points I mentioned above would help improve this paper.

Update post rebuttal: I have raised my score after the authors' response.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper explores the representations learned by CNNs, and shows that subsets of the neurons in a particular layer can be used to summarize that layer's contributions to the final prediction.  Thus there is a lot of redundancy in the representations.

Reviewers agreed that the experiments were extensive, interesting and the paper was well laid out.  The reviewers raised some important points about the evaluation, specifically suggesting a new baseline (which would be a fair amount of work to implement) and other datasets (one of which has been incorporated).  There was some missing related work, but this seems to have been addressed. 

My take is that this is a strong paper, but needs a bit more work before it's ready for publication 
justification_for_why_not_higher_score: There are several outstanding issues that require additional experiments which the authors were unable to complete during the rebuttal (one requires a significant amount of coding).  This paper is close, but not quite ready for publication.
justification_for_why_not_lower_score: n/a
Decision: Reject