Diffused Redundancy in Pre-trained Representations
Authors: Vedant Nanda
Till Speicher
John P Dickerson
Krishna P. Gummadi
Soheil Feizi
Adrian Weller
Venue: NeurIPS 2023 poster (2023)
Keywords: representation learning
redundancy
transfer learning
fairness
TL;DR: We show that a randomly selected fraction of neurons from a pre-trained representation achieve similar performance as the full representation.

Abstract: Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded in such pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy, ie, any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able to perform similarly as the whole layer on a variety of downstream tasks. For example, a linear probe trained on $20\%$ of randomly picked neurons from the penultimate layer of a ResNet50 pre-trained on ImageNet1k achieves an accuracy within $5\%$ of a linear probe trained on the full layer of neurons for downstream CIFAR10 classification. We conduct experiments on different neural architectures (including CNNs and Transformers) pre-trained on both ImageNet1k and ImageNet21k and evaluate a variety of downstream tasks taken from the VTAB benchmark. We find that the loss \& dataset used during pre-training largely govern the degree of diffuse redundancy and the "critical mass" of neurons needed often depends on the downstream task, suggesting that there is a task-inherent redundancy-performance Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for exploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences. Our code is available at \url{https://github.com/nvedant07/diffused-redundancy}.

Diffused Redundancy in Pre-trained Representations
Vedant Nanda University of Maryland & MPI-SWS
Till Speicher MPI-SWS
John P. Dickerson University of Maryland
Krishna P. Gummadi MPI-SWS
Soheil Feizi University of Maryland
Adrian Weller
The Alan Turing Institute & University of Cambridge
Abstract
Representations learned by pre-training neural network on large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded such pre-trained rep- resentations. We find that learned representations in given layer exhibit a de- gree of diffuse redundancy, i.e., any randomly chosen subset of neurons in the layer that is larger than threshold size shares a large degree of similarity with the full layer and able to perform similarly as the whole layer on a variety of downstream tasks. For example, linear probe trained on 20% of randomly picked neurons from the penultimate layer of a ResNet50 pre-trained on Ima- geNerlk achieves an accuracy within 5% of linear probe trained on the full layer of neurons for downstream CIFARIO classification. We conduct experi- ments different neural architectures (including CNNs and Transformers) pre- trained on both ImageNetIk and ImageNet2 and evaluate a variety of down- stream tasks taken from the VTAB benchmark. We find that the loss & dataset used during pre-training largely govern the degree diffuse redundancy and the "critical mass" of neurons needed often depends the downstream task, sug- gesting that there is a task-inherent redundancy coroddancyy-effommncc Pareto frontier. Our findings shed light the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to per- form many downstream tasks. We investigate the potential for exploiting this re- dundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences. Our code available settuffuuudddddudddnnnyy
Introduction
that achieve compet- itive accuracies on many benchmarks [45]. A key reason for the success of these models is their ability to learn useful representations of data (8).. While these models continue to get better, under- standing the properties of underlying learned representations continues to be challenge.
Over the years, many architectures have been proposed (such as [50 16 23)
Prior works have attempted to understand representations learned by deep neural networks through the lens of mutual information between the representations, inputs, and outputs 449 and hypothesize that neural networks perform well because of a "compression" phase where mutual information between inputs and representations decreases. Additionally, recent works have found that many
neurons are polysemantic, i.e., one neuron can encode multiple "concepts" [10] 39 and that one can
"Correspondence to: vnanda@mpi- -wws. org
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Different model architectures with varying penultimate layer lengths trained on ImageNetlk, WRNS0- 2 stands for WiecResseet0-2.. Implementation of architectures taken from timm [59] Diffused redundancy here measures what fractions of neurons (randomly picked) can discarded achieve within 90% performance of the full layer.
Feature Length
ImageNetIk Top-1 Accuracy
Diffused Redundancy for d 0.9
Model
CIFARTO
CIFART00
Flowers
Oxford-IllT-Pets
VIT S-16
384
64.82%
0.70
0.50
0.50
0.80
VIT S-32
384
55.73%
0.70
0.50
0.50
0.70
ResNet18
512
69.23%
0.80
0.50
0.50
0,90
ResNet50
2048
80.07%
0,90
0.50
0.20
0,90
WRN50-2
2048
77.00%
0,95
0.80
0.50
0,95
VGG16
4096
73.36%
0.95
0.80
0.80
0.95
then train sparse linear models such concepts to do "explainable" classification [61] However, it is not well understood if or how extracted features are concentrated or spread across the entire representation.
While the length of the feature vectors extracted from state-of-the-art networks P can vary greatly their accuracy on downstream tasks are not correlated to the size of the representation (see Table but rather depend mostly on the inductive biases and training recipes [60, [53] In all cases, the size of extracted feature vector (i.e. number of neurons) is orders of magnitude less than the dimensions of the input and thus allows for efficient transfer to many downstream tasks [22]
[55). We
show that even using random subset of these extracted neurons is enough to achieve downstream transfer accuracy close to that achieved by the full layer, thus showing that learned representations exhibit degree of diffused redundancy (Table
Early works in perception suggest that there are many redundant neurons in the human visual cor- tex [3] and some later works argued that similar redundancy in artificial neural networks should help in faster convergence [9]. this paper, we revisit redundancy in the context of modern DNN architectures that are trained on large-scale datasets. In particular, propose the diffused redun- dancy hypothesis and systematically measure its prevalence across different pre-training datasets, losses, model architectures, and downstream tasks. We also show how this kind of redundancy can be exploited to obtain desirable properties such as generalization performance but at the same time draw caution certain drawbacks of such an approach in increasing disparity in inter-class performance. We highlight the following contributions:
We present the diffused redundancy hypothesis which states that learned representations exhibit redundancy that is diffused throughout the layer, i.e., many random subsets (of sufficient size) of neurons can perform as well as the entire layer. Our work aims to better understand the nature of representations learned by DNNs.
We present an initial analysis of why diffused redundancy exists in deep neural networks, we show that randomly chosen subset of neurons size & performs as well as the projection of the entire layer the first & principal components. Intuitively this means that in DNNs' representations, many random subsets of size roughly capture all the variation in data that is possible with dimensions (since PCA represents directions of maximum variance).
We propose measure of diffused redundancy and systematically test our hypothesis across vari- ous architectures, pre- pre-training datasets & losses, and downstream tasks.
We find that diffused redundancy is significantly impacted by pre-training datasets & loss and downstream datasets.
We find that models that are explicitly trained such that particular parts of the full representa- tion perform well as the full layer, i.e., these models have structured redundancy (e.g. (26). also exhibit a sigmificant amount of diffused redundancy. Further, we also evaluate models trained with regularization that decorrelates activation of neurons and again find that these regularizations surprisingly do not affect diffused redundancy. These results suggest that this phenomenon is perhaps inevitable when DNNs have wide enough final layer.
Extracced features for the purpose of this paper refers to the representation recorded on the penultimate
layer, but the larger concept applies any layer

---

We quantify the degree of diffused redundancy as function of the number of neurons in given layer. As we reduce the dimension of the extracted feature vector and re-train the model, the degree of diffused redundancy decreases significantly, implying that diffused re- dundancy only appears when the layer is wide enough to accommodate redundancy.
Finally we draw caution to some potential undesirable sidc-effects of exploiting diffused redun- dancy for efficient transfer learning that have implications for fairness.
1.1 Related Work
Closest to our work is that of Dalvi et al. 7 who also investigate neuron redundancy but in the context of pre-trained language models. They analyze two language models and find that they can achieve good downstream performance with significantly smaller subset of neurons. However, there are two key differences to our work. First, their analysis of neuron redundancy uses neurons from all layers (by concatenating each layer), whereas we show that such redundancy exists even at the level of a single (penultimate) layer. Second, and perhaps more importantly, they use feature selection to choose the subset of neurons, whereas we show that features are diffused throughout and that even random pick of neurons suffices. Our work also differs by analyzing vision models (instead of language models) and using a diverse set of 30 pre-trained models (as opposed to testing only two models) which allows us to better understand the causes of such redundancy.
Efficient Representation Learning These works aim to learn representations which are "slim", with the goal of efficient deployment on edge devices (65 64 5]. Recently proposed paradigm of Matryoshka Representation Learning (26) aims to learn nested representations where one can per- form downstream tasks with only small portion of the representation. The goal of such represen- tations is to allow quick, adaptive deployment without having to perform multiple, often expensive, forward passes. These works could be seen as inducing structured redundancy on the learned rep- resentations, where pre-specified parts of the representation are made to perform similar to the full representation. Our work, instead, aims to look at diffused redundancy that arises naturally in the training of DNNs. We carefully highlight the tradeoff's involved in exploiting this redundancy.
Pruning and Compression Many prior works focus on pruning weights (29 13 15
31
30
and how it can lead sparse neural networks with many weights turned off. Our focus, however, on understanding redundancy at the neuron level, without changing the weights. Work on structured pruning is more closely related to our work [33 17,, however, a key focus of these works is to prune channels/filters from convolution layers. Our work is more focused on understanding the nature of learned features and more broadly applicable to all kinds of layers and models. We additionally focus randomly pruning neurons, whereas structured pruning methods perform magnitude or fcature-selection baase pruning.
Many works aim to understand learned representations with the Two works in this space are especially related to our work: sparse linear layers 61 which show that one can train sparse linear layers on top of extracted features from DNNs; and concept bottleneck models [2I which explicitly introduce layer in which each neuron corresponds
Explainabbilly/nteerrreaall
goal of better explainability
[35
meaningful semantic concept. Both these works ex- plicitly optimize for small/sparse layers, whereas our work shows that similar "small" layers already exist in pre-trained networks, and in fact, can be found simply with random sampling.
Understanding Deep Learning A related concept that of instrinsic dimensionality of DNN land- scapes (32). Similar to our work, intrinsic dimensionality also requires dropping random parameters (weights) of the network. We, however, are concerned with dropping individual neurons. Other works on understanding deep learning 49 I have also looked at the learned features, however, none of these works analyze the redundancy at the neuron level. Another related phenomenon to diffused redundancy is that of neural collapse [42), which states that representations of the penulti- mate layer "collapse" K points (where implies that for perfectly collapsed representations we need to store just enough information in the final layer activations to be able to represent K different points. We interestingly show that this information is spread throughout the layer with a significant degree of redundancy.
number of classes in the pre-training dataset). This
3

---

S. 2
aeea Redde Momeeeannyy
Transder
Froction Meuroea (T091122440)
Traction Neurems (166)12260))
(a) Downstream Ace; Standard
(b) Diff. Redundancy: Standard
(c) Downstream Acc: AT
(d) Dift. Redundancy; AT
02
Praction Nenanens (141a
24
Praction Neurems (Rital-206))
(e) CKA: Stanfard
CKA: AT
[Testing For Diffusee Redundancy in ResNet50 Pre-trained ImageNetlk] Top: transfer accu- racies + Diffused Redundancy (DR) measure (Eq on different downstream datasets, dotted horizontal line shows accuracy obtained using the full layer. We see that accuracy obtained using parts of representation varies greatly with pre-training loss (much more diffused redundancy in Adversarially Trained (AT) ResNet), but also depends on the downstream dataset. Bottom: comparing CKA between randomly chosen fraction of neurons the whole layer. Here evaluate CKA on samples from different datasets and find that similarity of subset layer rapidly increases, reaching similarity of greater than 90% on the adversarially trained ResNet with only 10% randomly chosen neurons. Values are averaged over 5 random picks and error bars show std. dev.
Figure 1:
2 The Diffused Redundancy Phenomenon
suggest that the repre- sentations need not store lot of information about the input. These findings imply that all neurons in learned representation might not be necessary to capture all the information in a particular layer. Extending these observations, we propose the diffused redundancy hypothesis:
Prior observations about a compression phase 149 and neural collapse 42
Learned features are diffused throughout a given layer with redundancy such that there exist many randomly chosen subsets of neurons that can achieve similar performance to the whole layer for variety of downstream tasks.
Note that our hypothesis has two related but distinct parts to it: 1) redundancy in learned features, and 2) diffusion of this redundancy throughout the extracted feature vector. Redundancy refers to features being replicated in parts of the representation so that one can perform downstream tasks with parts of representation as well with the full representation. Diffassion refers this redundancy being spread all over the feature vector (as opposed to being structured), i.e., many random subsets (of sufficient size) of the feature vector perform equally well.
In order to evaluate the redundancy part of the diffused redundancy hypothesis we use two tasks: 1) representation similarity between randomly chosen subsets of representation with the whole representation, and 2) transfer accuracy on out-of-distribution datasets (using a linear probe) of randomly chosen subsets of the representation compared to the whole representation. To estimate diffusion, we run each check for redundancy over multiple random seeds and plot the standard deviation over these runs.
Representation Similarity of Part vs Whole Centered Kernel Alignment (CKA) a widely used representation similarity measure and takes in two representations of n data points Z E Raad, and E Xd and gives a similarity score between and (24). Intuitively, CKA (with lincar kernel, see Appendix for details about CKA) measures if the two representations rank the n points similarly (where similarity is based on cosine distances). For a given neural network and n samples drawn from given data distribution, i.c., X D, let g( ((X) be the (penultimate) layer representation. m is a boolean vector representing a subset of neurons in g (X), then we aim to measure CKA(m@ g(X), ) ((X) to estimate how much redundancy exists in the layer. If indeed CKA(m gXX), ((X) is high (i.e. close to 1) then it's a strong indication that the diffused redundancy hypothesis holds.
Downstream Transfer Performance of Part vs Whole A commonly used paradigm to measure the quality of learned representations is to measure their performance on variety of downstream tasks (68 22). Here, attach linear layer (h) on top of the extracted features of network (g) to

---

do classification. This layer is then trained using the training dataset of the particular task (keeping g frozen). If features were to be diffused redundantly then accuracy obtained using h g. i.e.ir layer attached to the entire feature vector, should be roughly the same as ht' o (m Ô g); where m is boolean vector representing a subset of neurons extracted by 9, and h & h are independently trained linear probes.
For both tasks, i.e. representation similarity and downstream transfer performance, we evaluate on CIFARIO/100 (25), Oxford-IIIT-Pets [43] and Flowers (38] datasets, from the VTAB bench- mark 688 Training and pre-processing details are included in Appendix
Measure of Diffused Redundancy In order to rigorously test our hypothesis, we define a measure of diffused redundancy (DR) for given model (g) with M being set of all possible boolean vectors of size (g). i.e. size of the representation extracted from Each vector m E . represents a
possible subset of neurons from the entire layer. This measure is defined particular task (T) as follows:
DR(g,T,0) = -1- min S s.t. [M I Emem, T(s) 2 8
T(mog)
(1)
191
My = {m E M. m = m E 13,1)
Here T() denotes the performance of the model inside () for the particular task and is a user- defined tolerance level. For the task of representation similarity T(m @ g) is CKA between subset of neurons denoted by m @ g and and T(g) is always 1, since it denotes CKA between and g- For downstream transfer performance, T(m @ g) is the test accuracy obtained by training linear probe on the portion of representation denoted by m B and T(g) is the test accuracy obtained using the full representation. For 8 = 1, this measure tells what fraction of neurons could be discarded to exactly match the performance of the entire set of neurons. A higher value of DR denotes that only a few random neurons were needed to match the task performance of the full set of neurons, and thus indicates higher redundancy. Since M contains exponential number of vectors (21ml), precisely estimating this quantity is hard. Thus, we first choose a few f (number neurons to be chosen) define subsets of M. Then for each My f we randomly select 5 samples.
2.1 Prevalence of Diffused Redundancy in Pre-Trained Models
Figure [ checks for diffused redundancy in the penultimate layer representation of two types of ResNet50 pre-trained on ImageNetlk: one using the standard cross-entropy loss and another trained using adversarial training (34) (with & threat model and
3) We check for diffused redundancy using both tasks of representation similarity and downstream transfer performance.
Redundancy This is indicated along the x-axis of Fig T i.e., redundancy is shown when some small subset of the full set of neurons achieve almost as good performance as the full set of neurons. When looking at downstream task performance (Figs TT in order to obtain performance within some 5% of the full layer accuracy (dotted lines), the fraction of neurons that are task-dependent, e.g. across both training types we see that flowers (102 classes) and CIFAR100 (100 classes) require more fraction of neurons than CIFAR10 (10 classes) and oxford-iitt-etss (37 classes), perhaps because both these tasks have more classes. Additionally, across all datasets, the model trained with adversarial training exhibits more diffused redundancy than the one trained with standard loss (Fig Id 1b respectively). meaning we can discard far more neurons for the adversarially trained model to reach close to the full layer accuracy. Interestingly when looking at CKA between part of the feature vector with the full extracted vector (Figs If), we do not see significant difference in trends when evaluating CKA on samples from different datasets. However, we still see that we can achieve a given level of CKA with far fewer fraction of neurons in the adversarially trained ResNet50 as compared to the usually trained ResNet50.
be discarded
Diffused Redundancy This is indicated by small error bars in Figs were instead very structured, then different random picks of neurons would have high variance,
redundancy
'e also report results for lacr 4/255 in Appendix D and generally find that the &a models exhibit higher diffused redundancy. We choose to study adversarially robust models since they 've been shown to transfer better (4).
5

---

however, the enror bars here are very low, showing that performance is very stable across different random picks, thus indicating that redundancy is diffused throughout the layer.
While both tasks of downstream transfer and CKA between part and whole indicate higher diffused redundancy for the adversarially trained model, see that downstream transfer performance can differ substantially based on the dataset (while CKA remains fairly stable across the same datasets), indicating that downstream performance turns out to be "harder" test for diffused redundancy. Thus, in the rest of the paper, we examine diffused redundancy through the lens of downstream transfer performance and include CKA results in Appendix A
2.2 Understanding Why Many Random Subsets Work
8
dett
64-
Praction Meurona (T61a122540)
Practio Neurena (1sta=2946)
2.
Neurone
03
24
Nauran
(a)
Standard Training: CKA be- (b) Adversarial Training: CKA be- (c) Standard Training: PCA top (d) Adversarial Training; PCA top- (dotted) and bottom (dashed)
tween two random subsets
tween tWD ramdom subsets
k (dotted) and bottom-k (dashed)
Similarity between any two randomly picked sets of 4% neurons becomes fairly high (for "critical mass" of 1%), thus showing that any random pick beyond this threshold is likely to perform similarly; [2ck 2db Performance of linear probe trained for downstream task on randomly chosen neurons (solid lines) closely follows that of a linear probe trained on a projection on top le principal components (dotted lines) for a sufficiently large value of k.
Figure 2:
[Why Many Random Subsets Work] 2ake
2b
Many prior works explicitly train models to have "small" representations (e.g. [26 65 64 5 with the goal of efficient downstream learning. These works show that, when explicitly optimized, networks can
1.0
Transter Accuracy
9.8
9.6
perform downstream classification with fewer neu- rons than typically used in statc-of-the-ar architec- tures. We show, however, that such subsets already exist in models that are not explicitly trained for this goal, and in fact, one doesn't 't even have to try hard to find this subset; it can be randomly chosen. Later in section 3.3 we compare some of these efficient representation learning methods to randomly chosen subsets and carefully analyze the tradeoffs involved. Here, however, we seek to better understand why there exist so many randomly selected subsets that work just as well as the whole layer.
0.4
0.2
chrlo cler107
Tower oelord Flobb
0.0
0.0
0.2
0.4
3.6
0.5
1.0
Fraction of Naurons
Transter Accuracy
9.8
9.6
0.4
0..
clel
Fraction of Neurons
High CKA between two random subsets of the same size. First, we use representation similarity (CKA) 244 get a sense of how two random picks of neurons (of the same size) relate to each other. Concretely, we calculate CKA between two random picks of 1% neurons in the penultimate layer (aver- aged over 10 such randomly picked pairs) on sam- ples taken from different datasets. Fig 2a show CKA results averaged over these different picks of pairs of subsets of the full set of neurons. We see that after picking certain threshold, i.e. for a large enough value of k. the similarity between any two randomly picked pairs of heads fairly high. For example, for the adversarially trained ResNet50 (Fig 2bb. we observe that any 10% of neurons picked from the penultimate layer are highly similar (CKA of about 0.8), with very low error bars. A similar value of CKA is obtained with 20% of neurons for the standard ResNet50 model. These results indicate that, given a sufficient size, picking any random subset of that size has very similar representations and thus provides an initial intuition for why almost any random subset, with high probability, could work equally well.
Figure 3: [Random Projection (dotted) vs Ran- domly Choosing Neurons (solid)) We find that diffused redundancy (randomly choosing neu- rons) performs significantly better than random projections. This indicates that the "constrained" projection offered by diffused redundancy is cru- cial in achieving good downstream performance.

---

Downstream performance on l randomly chosen neurons closely follows performance on top k principal components. While high representation similarity necessary condition for two representations to have similar downstream performance, it's not a sufficient one. As seen earlier in Fig | similar values of CKA can still show different downstream accuracy. Thus, to further understand why any random pick of neurons achieves similar accuracy, we compare randomly chosen subset of neurons with a projection on the same number of top principal components. Fig 2 2d shows that the performance of linear probe trained on random subset of neurons initially lags behind the top PCA dimensions for small values of k<< 20%). but for higher values (> 20%) closely follows the performance top principal components. This indicates that a sufficiently sized random subset of size (roughly) captures the maximum variance in data that can be captured in is dimensions (upper bound by top k principal components since they are designed to capture maximum variance). As a sanity check, we also show the results for the bottom principal components (directions of least variance) and that randomly chosen neurons perform significantly better.
probe trained on the projection the same number of
Random projections into lower dimensions do not perform well on downstream tasks. The presence of diffused redundancy in a particular layer indicates that a dimension lower than the layer's size suffices to perform many downstream tasks. However, randomly sampling neurons (as we do in our experiments on diffused redundancy) can be seen as one very particular way of reducing dimension that is restricted by what the network has learned. This raises natural question, would our observation extend to less restricted ways of reducing dimensions? To test this, we compare diffused redundancy i.e., lincar probe trained on random sample of k neurons to a linear probe trained on a random projection of the layer's activations. To project a d dimensional layer into lower dimension (k) multiply it by (normalized) randomly sampled matrix from normal distribution Rdxk We report our results over seeds in Fig 3 and find that diffused redundancy significantly outperforms random projection.
3 Factors Influencing The Degree of Diffused Redundancy
Trandder Accuraicy
Transfer Accuracy
Trandfer Accuracy E
Trandfer
Praction of neurons
Fraction of neurons
Fraction of neurons
Fraction of neurons
CIFARIO
CIFARI00
(c) Flowers
(d) Oxford Iels
vgpl6 on nonroo veple, tn-robaat/eps3
resnet88 monrob
resh et50 nonrob
M norrob
pasmmalpptccneee 224 nonroo
resine wtw.eebstttteegeee.....
wide resretso 2e0basileesd
vR. small patchaz 224-neese
Figure 4: [Comparisons Across Architectures For Downstream Task Accuracyl All models shown here pre-trained ImageNetlk. We that diffused redundancy exists across architectures, and the trend observed [a adversarially trained models also holds here as models' curves that are more "inside" are the ones trained with standard loss.
In order to better understand the phenomenon of diffused redundancy we analyze 30 different pre- trained models, with different architectures, pre-training datasets and losses. We then evaluate each model for transfer accuracy on 4 datasets mentioned in Section [ While all results in this section are on the penultimate layer. we also report results on intermediate layers in Appendix E and find similar trends of diffused redundancy. All details for reproducibility can be found in Appendix B
Architectures We consider VGG16 [50], ResNer18, ResNet50, WideResNet50-2 (66. VIT-S16 & VIT-S32 (3). Additionally, we consider ResNet50 with varying widths of the final layer (denoted by ResNet50 -fx where denotes the number of neurons in the final layer).
Upstream Datasets ImageNet- 1k & ImageNet-21k (45)
Upstream Losses Standard cross-entropy, adversarial training (f2 threat model, e = 3 and Loo threat model. e 4/255 34 MRL Loss 26). DeCov 6 and varying strengths of dropout [52].
Downstream Datasets CIFARIO/1000, -Ir--et, and Flowers, same as Section 2 Addi-
tionally, also report performance on harder datasets such as ImageNetv2 44] and Places365 69
Results for Lo robust models be found in Appendix D
7

---

Transfer Recurscc
Thansfer Accursiy
Trapeter
02
Praction U.
02
05 Fraction neurons
Praction neurons
Fraction
CIFARIO
CIFAR100
(c) Flowers
Oxford-IIT--Pets
- vit small patesie 224 244-magenat
sit small patchie 224imagenaz2k
Vi. small patchzz 224 iagenet
vit small patcha2 224 imageneet
Figure 5: [Comparison Across Upstream Datasets] We see that degree of diffused redundancy depends great deal on the upstream training dataset, in particular, models trained on ImageNe121k exhibit higher degree of diffused redundancy, although the differences in the degree of diffuused redundancy downstream task dependent
in Appendix F) For all analyses in this section, we also report corresponding approximations for DR (Eq(l II in Appendix D
3.1 Effects of Architecture, Upstream Loss, Upstream Datasets, and Downstream Datasets
Extending the analysis in Section 2 we evaluate the diffused redundancy hypothesis on other archi- tectures. Fig 4 shows transfer performance for different architectures. All architectures shown in Fig 4 are trained on ImageNetlk. We find that our takeaways from Section also extend to other architectures.
compares two instances each V and ViT-S32, one trained on a bigger upstream dataset (ImageNet21k) and another on smaller dataset (ImageNetlk)
Note that the nature of all curves in both Figs 4 highly depends downstream datasets. This is also consistent with the initial observation of Section about diffused redundancy being downstream dataset dependent. Additionally, also report results on ImageNetV2 and Places365 in Appendix showing that diffused redundancy also holds on "harder" datasets.
3.2 Diffused Redundancy as Function of Layer Width
We take the usual ResNet50 with penultimate layer consisting of 2048 neurons and compare it with variants that are pre-trane pre-trained with much smaller penultimate layer, these are denoted by ResNet50 ffx where x (< 2048) is the number of neurons the penultimate layer. Fig shows how diffused redundancy slowly fades away as we squeeze the layer to be smaller: In fact, for ResNet50 ff8, we see that across all datasets we need > 90% the full layer to achieve perfor- mance close to the full layer. This shows that diffused redundancy only appears in DNNs when the layer is sufficiently wide to encode redundancy.
Becurnacy
Tronsfer Accuracy
0.9
0.8
0.2
2A
05
1.0
0.
22
0.4
05
10
04
05 neurons
Praction neurons
neurons
(+) CO
(b) C00
(c) Flowers
(d) Oxford-IIITPPet
resnet50
resnet50 ff256
resnet50 ff32 f
resnet50 M1024
resnet50, f8
resnet50 ff16
resnet50 #512
resnet50
resnet50 ffs
Figure 6: [Diffused Redundancy as Function of Layer Width] As we make the length of the layer smaller, the degree redundancy becomes lesser. For ReaNet50 ff8, i.e. ResNet50 with only 8 neurons final layer, we see that we need almost 90% of neurons to achieve similar accuracy as the f'ull layer.
3.3 Comparison With Methods That Optimize For Lesser Neurons
Matryoshka Representation Learning (MRL) is recently proposed paradigm that learns nested representations such that the first k. 2k, 4k, ..., N (where Ñ size of the full layer) dimensions of the extracted feature vector all explicitly made to be good minimizing upstream loss, with

---

the intuition of learning coarse-to- tonsse- representations. This ensures that one can flexibly use these smaller parts of the representation for downstream tasks. MRL. thus, ensures that redundancy shows up in learned representations in structured way, i.e. we know the first k, 2k, ... neurons can be picked and used for downstream tasks and should perform reasonably.
Here we investigate two questions regarding Matryoshka representations: 1) do these representations also exhibit the phenomenon of diffused redundancy? i.e. if we were to ignore the structure imposed by MRL-type training and instead just pick random neurons from all over the layer, do we still get reasonable performance?, and 2) how do they compare representations learned by other kinds of losses?
Figure investigates these questions by comparing ResNet50 representations learned using MRL loss to other losses. resnet50 mrl nonrob first (red line) denotes a ResNer50 trained using MRL loss and evaluated on parts of the representation that were optimized have low upstream loss (i.e. first k 2k,.... neurons, here = 8 and N = 2048) and resnet50,ms hrrll__onoob__a_ae (green line) refers the same model with the same number of neurons chosen for evaluation, except they re chosen at random from the entire layer.
First, we interestingly see that even the ResNet50 trained with MRL loss exhibits diffused redun- dancy (denoted by green line spiking very quickly for most datasets in Fig 7 despite having been trained only have structured redundancy. Based this observation, we conjecture that diffused redundancy is a natural consequence of having a wide layer. Second, we see that ResNet50 trained on MRL indeed does better in the low neuron regime across datasets (red line the extreme left part of the plots in Fig 7) but other models quickly catch up as we pick more neurons, thus indicat- ing that major efficiency benefits of MRL-ttpe models are best realized when using an extremely low number of neurons, else one can obtain similar downstream performances by simply picking random samples from existing pre-trained models.
Trandte Accuracc
Transfer Agcurscy
Tronsfer Accuracy
0.8
Brenuter
03
0.2 3A
1.0
05
32
1.0
04 05
Praction neurons
Fraction of neurons
Fraction neurons
Fraction of neurons
(a) CIBAR10
(b) CIFARI00
(c) Flowers
(d) IIt-
reneet0-ronrrbbraae raadomm
resnet50 mwt.mmm-nnnn rnttrroooo--rnoom
resnet50 0 mriinonrob-firss
Figure 7: [Comparison of Diffused Redundancy in MRL vs other losses] Here we compare ResNet50 trained using multiple losses including MRL (26] nenrob indicates that the model
trained with the standard crossentropy loss while robust12eps3 indicates adversarial training with & threat model and e 3. Red line shows results for part the representation explicitly optimized in MRL, whereas the green line shows results for parts that are picked randomly from the same representation. Even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy.
3.4 Methods That Prevent Co-adaptation of Neurons Also Exhibit Diffused Redundancy
We consider two additional modifications to upstream pre-training: we add regularization to the usual cross-entropy loss that decorrelates neurons in the feature representation using DeCov [6] and Dropout [52] Dropout does this implicitly by randomly dropping neurons during training and DeCov does this explicitly by putting loss on the activations of given layer. We pre-train different ResNet50 on ImageNet by varying strengths dropout ranging from 0.1 all the way up to 0.8 and also by separately adding the DeCov regularizer (regularization strength 1e-4) to give 9 additional pre-trained models. Intuitively these methods should lead increased diffused redundancy since by design these methods force the model to disperse similar information in different parts of layer to perform the same task downstream task. Interestingly, we see that diffused redundancy these models is almost completely independent of the regularization strength. This suggests that diffused redundancy might be an inevitable property of DNNs when the layer has sufficient width. Due to space constraints, we include these results in Appendix G
4 Possible Fairness-Efficiens
Tradeoffs in Efficient Downstream Transfer
One natural use case for diffused redundancy is efficient transfer to downstream datasets. As defined in Eq I and as also seen in Sections & 3 dropping neurons comes at small cost (6 in Eq I in

---

Her ald
resetlyy seltana
01
W
53 21
34
24 Accuracy
CIFARIO
(b) CIFARTOO
(c) Flowers
(d) Oxford-IITT-ets
Figure 8: [Gini Coefficient of Biaas-Wiss Accuracies we Drop Neurons] Higher value of Gini coefficient indicates higher inequality [2]. We see that for all models gini coefficients become higher as the accuracy reduces (as a result of dropping neurons). Additionally, in some regions (highlighted in the plots), the model explicitly optimized for efficient transfer (resnet50_mm) mr1) can give rise to higher gini values, resulting in a more unequal spread of accuracy over classes.
performance as compared to the full set of neurons. Here we take a deeper look into this drop in overall performance and investigate how it is distributed across classes. If the drop affects only a few classes, then dropping neurons although efficient for downstream tasks could have implications for fairness, which not only of concern to ML researchers and practitioners [66 to lawyers (56) and policymakers [57].
but also
We compare the spread of accuracies across classes using inequality indices, which are commonly used in economics to study income incquality 8 47 and have also recently been adopted in the fair ML literature [51]. We use gini index [12] and coefficient of variation [27] to quantify the spread of performance across classes. For perfect spread, both gini and coefficient of variation are O, and higher values indicate higher inequality.
comppres the gini index for various models at varying levels of accuracy (note that accuracy monotonically increases with more neurons, hence the right most point for model represents the model with all neurons). We make two observations: across all datasets and all models we find that loss in accuracy (compared to the full layer) comes the cost of a few classes, opposed to being smeared throughout classes, as indicated by high gini values on the left of each plot. Additionally, we observe that the model trained using MRL loss tends to have slightly higher gini values in the regions where the drop in accuracy is slightly higher (highlighted on the plots). To ensure that this trend is not simply due lower accuracy, we investigate the error distributions across classes (Appendix (C and find that predictions become more homogencous as we drop more neurons. Similar trends are also observed with coeff. of variation as shown in Appendix C These results draw caution to the potential unintended side-effects of exploiting diffused redundancy and suggest that there could be a possible tainess-efficieng tradeoff involved.
Tradcoffs There are well-established tradeoffs between ro- bustness and fairness in both standard (36) and adversarial training [62 (88. One major difference between these works and our setup is that the task they train on is also the task they test on. We instead operate in the transfer learning setup where we train a linear probe on the representation of a pre-trained model. While it's not immediately clear whether the discrepancy between class accuracies observed on the pre-training task (e.g. in (22) also leads to an inter-class accuracy discrepancy on downstream tasks when dropping neurons, it would be very interesting future work to establish more formal connections between the robustness-fairt tradeoff and its effects on the fairness sirmess-efficiencyy tradeoff presented in this paper.
Connections to
5 Conclusion and Broader Impacts
We introduce the diffused redundancy hypothesis and analyze a wide range of models with differ- ent upstream training datasets, losses, and architectures. We carefully analyze the causes of such redundancy and find that upstream training (both loss and datasets) plays a crucial role and that this redundancy also depends on the downstream dataset. One direct practical consequence of our obser- vation is increased efficiency for downstream training times which can have many positive impacts in terms of reduced energy costs (54) which is crucial in moving towards "green" Al 48) We, how- ever, also draw caution to potential pitfalls of such efficiency gains, which might hurt the accuracy of certain classes more than others, thus having direct consequences for fairness.
10

---

Acknowledgements
VN and JPD were supported in part by NSF CAREER Award IIS- 18S-184,237, NSF D-ISN Award #2039862, NSF Award CCF-1852352. NIH ROI Award NLM013039-01, NIST MSE Award #20126334, DARPA GARD #HRO01 12020007, DoD WHS Award #HQ003420F0035, ARPA-E Award #4334192 and Google Faculty Research Award. VN, TS, and KPG were supported in part by an ERC Advanced Grant "Foundations for Fair Social Computing" (no. 789373). AW ac- knowledges support from Turing AI Fellowship under grant EP/V025279/1 and the Leverhulme Trust via CFI. SF was supported in part by grant from an NSF CAREER AWARD 1942230, ONR YTP award N00014-22-1-2271, ARO's Early Career Program Award 310902-00001, Meta grant 23010 HR001 12000132 (DARPA/RED), HR001 19S00226 (DARPA/GARD), Army Grant No. W91 INF2, the NSF award CCF2212458, Amazon Research Award, and an award from Capital One.
References
[1] Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.013300 350, 125:126-127: 127, 2017.
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.
[3] Fred Attneave. Some informational aspects of visual perception. 61(3):183, 1954.
Psychological review,
[4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
[5] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for all: Train one network and specialize it for efficient deployment. arXiv preprint 190.:09900999
[6] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Re- ducing overfitting in deep networks by decorrelating representations. arXiv preprint M5IG::/511.00006 2015.
[7] Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Meth- Online, November 2020.
ods in Natural Language Processing (EMNLP), pages Association for Computational Linguistics.
[8] Fernando G De Maio. Income inequality measures. Journal of Epidemiology & Community Health, 61(10):849-852 2007.
[9] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. Advances Neural Information Processing Systems, 30, 2017.
[10] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[11] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019.
[12] Corrado Gini. Measurement of inequality of incomes. The economic journal, (1(121): 126. 1921.
[13] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
11

---

[14] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems. 29, 2016.
[15] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 770-778, 2016.
[17] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[18] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wal- lach. Improving faimess in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1-16, 2019.
[19] Yoshio Izui and Alex Pentland. Analysis of neural networks with redundancy. Neural Compu- tation, 2(2):226 238, 1990.
[20] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677. PMLR, 2018.
[21] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
[22] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Euro- pean conference on computer vision. pages 491-507. Springer, 2020.
[23] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkor- eit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. 2021.
[24] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR, 2019.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[26] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard Swward-Snyder, Kaifeng Chen, Sham Kakade. Prateek Jain, et al. Ma- tryoshka representations for adaptive deployment. arXiv preprint arXiv:2205.13144 2022.
[27] Barbara S Lawrence. Perspective-the black box of organizational demography. Organization science, 8(1):1222, 1997.
(28) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436- 444, 2015.
[29] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural infor- mation processing systems, 2, 1989.
[30] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint ar1iv:1810.023400, 2018
[31] Asriel Levin, Todd Leen, and John Moody. Fast pruning using principal components. Advances in neural information processing systems, 6, 1993.
12

---

[32] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018.
[33] Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087 2016.
[34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
[35] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by in- verting them, 2014.
[36] Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John Dickerson. Fairness through robustness: Investigating robustness disparity in deep learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transpurency, pages 4664477, 2021.
[37] Vedant Nanda, Till Speicher, Camilla Kolling, John Dickerson, Krishna P. Gummadi, and Adrian Weller. Measuring representational robustness of neural networks through shared in- variances. In ICML, 2022.
[38] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
[39] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020.
[40] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill,
2017.
[41] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowl- edge and data engineering, 22(10): 1345-1133,-3359 2009.
[42] Vardan Papyan, XY Han, and David Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40)24452224466 2020.
[43] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Puttern Recognition, 2012.
[44] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019.
[45] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):11-225,, 2015.
[46] Hadi Salman, Andrew llyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Pro- cessing Systems, 33:3533-355 2020.
[47] Robert R Schutz. On the measurement of income inequality. The American Economic Review, 41(1):107-122, 1951.
[48] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54-63, 2020.
[49] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.000. 2017.
[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409415566660 1556, 2014.
13

---

[51] Till Speicher, Hoda Heidari, Nina Grgie- Plaic, Krishna P Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. A unified approach to quantifying algorithmic unfairness: Measuring individual &group unfairness via inequality indices. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2239- 2248, 2018.
[52] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
[53] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision trans- formers. arXiv preprint arXiv:2106. 10270, 2021.
[54] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David R. Traum, and Lluís Marquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics. ACL 2019, Florence, Italy, July 28- August 2, 2019. Volume 1: Long Papers, pages 3645-3650. Association for Computational Linguistics. 2019.
[55] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270-279. Springer, 2018.
[56] Songül Tolan, Marius Miron, Emilia Gómez, and Carlos Castillo. Why machine learning may lead to unfairness: Eyidence from risk assessment for juvenile justice in catalonia. In Proceedings of the Seventeenth International Conference Artificial Intelligence and Law, pages 83-92, 2019.
[57] Michael Veale, Max Van Kleek, and Reuhen Binns. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making making. In Proceedings of the 2018 chi conference on human factors in computing systems, pages 2018.
[58] Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adver- sarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8193-8201, 2023.
[59] Ross Wightman. Pytorch image models. 2019.
https: com wtw./wiiggtt wtt.:/wwwwwwwwaaa....o
pytorch-image
models www.ammmalellllammmmmmmmaaaeeeea
[60] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. abs/2110.00476 2021.
[61] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse lincar layers for debuggable deep networks. In International Conference on Machine Learning, pages 1 11205- 11216. PMLR, 2021.
[62] Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International conference on machine learning, pages 1 14992- 11501. PMLR, 2021.
[63] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization, 2015.
[64] Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training tech- niques. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1803-1811, 2019.
[65] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. abs/ 111111822.88,,... 2018.
[66] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pages 962-970. PMLR, 2017.
14

---

[67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, 2013.
[68] Xiaohua Zhai, Joan Puigcerver. Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 11Xhv:1910.04806 2011.
[69] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A
10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
15

---

A Measuring Diffused Redundancy
A.1 CKA Definition
In all our evaluations we use CKA with a linear kernel [24] which essentially amounts to the follow- ing steps:
1. Take representations Y Raxdi and Z Rax
2. Compute dot product similarity within these representation, i.e. i.e.compte K = =YT L ZZz
3. Normalize K and L to get K' = =HKH L' HLH where In 111
4. Return CKA(Y, Z)
HSIC(K,L)
where HSIC(K, II= (flatten(K))
V (HSICL)
(latten(L))
We use the publicly available implementation (37). which provides an implementation that can be calcuated over multiple mini-batches: httpp:://gghhuu...cc////aaaaa///
A.2
Additional CKA results
Fig 1 shows CKA comparison between randomly chosen parts of the layer and the full layer for dif- ferent kinds of ResNet50. We observe that even ResNet50 trained with MRL loss shows significant amount of diffused redundancy.
8
risnet5o
resnet5o
0.2
resnetto, neninnnnro
0.2
resnetio_ noiinoooo
resnet50 mwiinorroo mondom
resnetto, mrinorroo aanoo
0.0
0.0
Fraction of Neurons (Total=2048)
Fraction of Neurons (Total=2048)
(a) CIARROO
(b) CIFARI00
1.0
1.0
0.8
0.8
0.6
0.6
renett0-oowo-eendmm
menett00-owvo--mmee rantomrantom
resnetto rabus82op33 random
resnet50 ribust2opc3 random
rennet50 meknumeeb-fiia:
meset50 meinurreb-ins
resnet5o mrinerroo rando
resnetto mrinorroo aadom
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0,1
0.2
0.3
0.4
0.5
Fraction of Neurons (Total=2048)
Fraction of Neurons (Total=2048)
Flowers
Oxfor-IIIT-Pets
Figure 9: [Comparison of Diffused Redundancy MRL ys other losses, through the lens CKA] We see a similar trend as reported in in the main paper, where even the MRL model shows significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. The amount of diffused redundancy however much lesser than the resnets trained using the standard loss and adv. training as denoted by a much lower line across datasets.
B Training and Pre-Processing Details for Reproducibility
Here we list the sources of weights for the various pre-trained models used in our experiments:
ResNer18 trained on ImageNet1k using standard loss: taken from timm v0.6.1.
ResNet18 trained on ImageNetlk with ady training: taken from Salman et al. [46]:
16

---

ResNet50 trained on ImageNetlk using standard loss: taken from timm v0.6.1.
ResNet50 trained on ImageNetlk with adv training: taken from Salman al. [46 https:
//github. mode
ResNet50 trained on ImageNetlk using MRL and with different final layer widths (resne55_ffxx taken from released weights of by Kusupati et al. [26]:
https: ///
WideResNet50-22orr on ImageNetlk both standard and avd. training: taken from Salman et al. [46]: https://github.. tottaaottttoototoooooooe--------
VGG16 trained ImageNetlk with standard loss: taken from timm v0.6.1.
VGG16 trained on ImageNetlk with adv training: taken from Salman et al. [46] https:
//github. com htg.tcub..commmicccosttttttottt. moo.ccooootrooossttoooeesss
ViTS32 & VITS16 trained on ImageNet21k & ImageNetlk: taken from weights released by
Steiner et al.
5
All linear probes trained on the representations of these models are trained using SGD with learn- ing rate of 0.1, momentum of 0.9, batch size of 256, weight decay of le 4. The probe is trained for 50 epochs with a learning rate scheduler that decays the learning rate by 0.1 every 10 epochs. Scripts for training can also be found in the attached code.
For pre-processing,
re-size all inputs 224x224 (size used for pre-training) and apply the usual composition of RandomHorizootalFiip ColorJiterrbrrghhnes=-22222 contrast=0.25, satura- All inputs were mean normalized. For ima-
tion=0.25, hue=0.25). RandomRottaionndderreesss genetlk pre-trained models: mean = [0.485, 0,456, 0.406] and std = [0.229, 0.224, 0.225]. For imagenet2 k pre-trained models: mean [0.5,0.5,0.5]. std (0.5,0.5.0.5).
C Deeper Analysis of Fairness-Efficiency Tradeoff in Section
Analyzing Error Distributions To ensure that the higher gini coefficient shown in F we drop more neurons is not merely an artifact of lower overall accuracy, plot class-wise accuracies as we drop neurons (Figs II 12 & 13 We find that for the entire layer, accuracy starts at an almost drop neurons, the drop comes at
uniform distribution, and while overall accuracy deteriorates a larger cost for a few classes resulting in disparate inter-class accuracies.
Coeff of Variation for Measuring Inequality in Inter-Class Accuracy Fig 10 shows results for the same analysis shown in 8ig the main paper and we find similar takeaways even when using the coefficient of variation measure of inequality.
D Corresponding Diffused Redundancy Estimates For Analyses in Section 3 & Loo Robust Model Results
These are shown in Fig- respectively. This should allow for easy comparison of diffused redundancy (lines that are more outside have higher DR). For example, Figure 16)clerryy shows higher diffused redun- dancy in models trained larger upstream datasets (here ImageNet21k) since these curves lie more on the outside of the same model's curves for ImageNet
Corresponding diffused redundancy (DR) ablations for Figures 5
ures [5] 16
Additionally, we show numbers on x-axis for Figure in Figure 18 Figures 5 and compare models with same number of neurons in final layer and hence trends shown with fraction on the x-axis will be exactly the same with absolute numbers on the x-axis. However, Figure 18 allows direct comparison of the performance of the same absolute number of neurons across different models.
We report results for loo robust models (with 4/255) in Fig 14 and find that E model generally shows a greater degree of diffused redundancy.
E Results on Intermediate Layers
We additionally ran our experiment on other intermediate layers and report the results in Fig We present results for a ResNer50 pretrained on ImageNetlk using the standard CrossEntropy loss. The
17

---

1.0
resnet5o-nover obbut22ss33
mesnerddenomm rester mwwesSSrrbualleeee
Coefft. of Varlation
Coeff. of Variation
0.8
resnet50, snr nonrob 12t
0.6
0.4
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Accuracy
Accuracy
(a) CIFARIO
(b) CIFAR100
resset50 nonrab
Coeff. of Varlation
Coeff. of Variation
resnet50 nbustteeub
0.8
0.8
resnet50, mri nonros frrst
0.6
0.6
0.4
0.4
0.2
reshet50 Renc3
0.2
reanet50 mil-nenro--rsst
0.0
0.1
0.0
Accuracy
Accuracy
Flowers
(d) Oxfor-IIIT-Pet
Figure 10: [Coefficient of Variation As We Drop Neurons] We similar trend reported in Fig 8 the main paper where inequality increases drop neurons for all models on all datasets.
intermediate layers considered are characterized activations following each residual connection within distinct ResNet blocks. layerX. act3 means the Yth residual connection in the Xth ResNet block and act3 indicates that we're taking the value after the activation (ReLU) has been applied.
F Results on Harder Downstream Tasks: ImageNet and Places365
We report results harder downstream tasks such ImageNetlk, ImageNetV2, and Places365 in Figure 20) We find that when randomly dropping neurons, the model is still able to generalize to ImageNetlk and Places365 with very few neurons, i.e., the phenomena of diffused redundancy observed for smaller datasets, also holds for harder datasets. Interestingly we also observe that the accuracy gap between ImageNetlk and ImageNetV2 is maintained even drop neurons.
G Effects of Explicitly Preventing Co-adaptation of Neurons: Analysis of Dropout and DeCov
Regularizers such as dropout and DeCov, force different parts of the representations to be cor- related. Thus these regularizers can be seen as explicitly requiring different, compact parts of the representation be self-contained for the downstream task. Thus, intuitively, such methods should increase diffused redundancy. Here we investigate if our observation about diffused redundancy is influenced by such regularizers. We evaluate ResNet50 pre-trained on ImageNetlk with dropout in the penultimate layer ranging from a strength of 0.1 all the way to 0.8. We also train another ResNet50 model with the DeCov regularizer added to the usual crossentropy loss and put a weight of 0.0001 on the regularizer ensure that its numerical range is similar to that of the cross entropy loss term.
Results in Figures 21 & 22 suggest that such regularizers have almost no effect on diffused redun- dancy. Trends across datasets remain consistent regardless of the strength of dropout the weight given to DeCov regularizer. This observation further adds to the evidence that diffused redundancy likely be natural property of representations learned by DNNs.
18

---

1.0
1.0
Classwise Accuracles (mean)
classwise Accuracies (mean)
0.8
automobile
automobile
birg
bind
Eat
doer
frog
freg
horse
0.4
horse
ship
ship
taok
0.0005
0
0OD
0,005
00L
0.00
0.05
0.75
002
0.25
0.0005
0.01
0,002
000
0000
001
0.2
0.1
0.00
0.15
0.
0.15
0.
0.2
0.2
Fraction/number of neurons
Fraction/number neurons
(a) CIFARIO, Standard
(b) CIFARIO, Adversarial Training
aquarium A0/le
412l
1.0
1.9
bultanh
ch chimor
chipohie
crocel
logal
Classwise Accuracies (mean)
Classwise Accuracies (mean)
laen
lears
sapl:
na
mapls the
60X Deg
oak Pac
Pktup Dag
Tay
0.4
(0201
203
100%
2000
iweel pagner
Iefectiore
00
0.0
No
207
0.0
Fraction/number of
Fraction/number
(c) CIFARI0O, Standard
(d) CIFARIOD, Adversarial Training
Figure 11: [Error Distributions As A Function of Fraction of Neurons] We see that accuracy deteriorates we drop neurons, however, this drop comes larger cost for few classes and results in near homogenous predictions for the least number of neurons the left.
19

---

lurdicersg Rotze
Tord Fordioongg Potce
figm
Purs
puis
plant
pla wn
percaa
pircssn
peince
srinee d Sorric
rby
nuy Faad
great
Classwise Accuracies dies (mean)
tarbon
bo are
to
Classwits Accura cies
correen
commnh
ca Por
men
1040
Dougo
max
medsa
Fraction,number
Praction/mumber
(a) Flowers, Standard
Flowers, Adversarial Training
Figure 12: [Error Distributions As Function of Fraction of Neurons continued] We see that accuracy deteriorates as we drop ncurons, however, this drop comes at a larger cost for a few classes and results in near homogenous prodictions for the least number of neurons on the left.
Abyssinian
Abruc A00
Basset
Classwise Accuracies (mean)
Classwise Accuracies (mean)
0.8
British
0.8
English (i Baan
Engiish cot Ball
Germall Cirnncc
0.6
Germal
0.6
Jap
Lebnoor
Lest Dey
Minigture Fin Can
0.4
Ministure Ris
Rusel Tussioe
Sag
Rucci Bag
50 Bernard
Scoth
Scomi
Staffoddhhire Bull Citee
Staffordshire Bull SI
Fraction/number of neurons
Fraction/number of neurons
Standard
Adversarial Training
Figure 13: [Error Distributions Function Fraction Neurons continued] see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left.
20

---

Deffused (DR)
Offuredd ed Rodundancy (DR)
0.6
0.4
0.4
0.0
0.75
0.80
1.00
0.0
0.80
0.85
0.90
0.85
1.00
CIFARIO
CIFARIO
1.0
Diffused ed Redundanc anncy (DR)
1.0
0.0
0.6
Diffused
0.2
02
0.75
0.80
0.85
0.95
1.00
0.75
0.80
0.85
0.90
0.95
1.00
(c) Flowers
(d) Oxford IIfT
resnetSo-nonnab-raddom
resneSSDrobustiiees3raaddo
rennet5-robustlineess--raooo
Figure 14: [Results for loo robust model] We show results for ResNet50 trained with 3 different losses: CrossEntropy (nonrob), Adversarial Training with & threat model (robl2eps3), and with the Loc threat model (roblinfeps4). see that the & threat model shows most diffused redundancy. models are trained on ImageNetlk.
offfused Re Redundaanee dancyy (DOR)
0.8
0.8
0.4
6.4
Diffused
6.2
0.
(a) CIFARIO
(b) CIFARIOD
2.0
olffused Rodundann dancy [DR]
0.8
8.6-
0.6
Diffused
0.0
0.35
0.90
0:85
0.50
0.55
1.00
0.0
0.75
0:80
0.25
0.50
0.35
1.00
Flowers
(d) Oxford-IITT-Pet
www.
www.
Figure 15: (Comparisons Across Architectures For Downstream Task Accuracyl All models shown here are pre-trained on ImageNetlk. This Figure shows corresponding diffused redundancy values for Figure diffcrent 5 values. We see that diffused redundancy exists across architectures, and the trend observed in Figure Ti regarding adversarially trained models also holds here as models curves that more "inside" the trained with standard loss.
21

---

oiffused Redundanc (DR)
Diffused Redundanc
0.
0.6
1044
0.2
0.
0.80
0.35
(a) CIFARIO
(b) CIFAR
Diffused (DR)
Offfused ed Redunndancy (DR)
1.0
8.88
0.3
0.4
04
0.0
0.75
1.50
1.00
0.0
0.75
0.80
0.95
1.00
Oxlerd-IIT-Pes
vit_
224-imagener
vit vit_small pacplllppphh22__111 224-imagenet
vit Hit_small_patchen 124-imagenet21k
vit_small patncallpatch222__1 224-imagenet21k
[Comparison Across Upstream Datasets] We see that degree of diffused redundancy depends great deal on upstream training dataset, in particular models trained on ImageNet2 exhibit higher degree of diffused redundanacy, although the differences in the degree of diffused redundanacy are downstream task dependent
Figure 16:
Olffused Redundano (DR)
Diffused Redundanc
0.6
8.6
004
0.
02
0.80
0.35
(a) CIFARIO
(b) CIFARIOD
Diffused Redundancy (DR)
Offfused Redundancy (DR)
8.8
0.8
0.4
0.4
0.0
0.75
1.00
0.95
1.00
0.0
0.75
0.80
0.85
0.95
(c)
Oxford-I1TT-Pes
resnet50-nonrob-raddom
resnettOrobustieess-rndddd
resnet50 mri-nnorrob-frsst
Figure 17: [Comparison of Diffused Redundancy MRL vs other losses] Here we compare ResNet50 trained using multiple losses including MRL (26). Red line shows results for part of the representation ex- plicitly optimized in MRL, whereas green line shows results for parts that picked randomly from the same representation. Even the MRL model shows a significant amount of diffused redundancy despite being explic- itly trained to instead have structured redundancy. This figure shows diffused redundancy (DR) for plots in Figure 7
22

---

Transfer Accur Accuracy acy
0.8
0.0
0.4
0.
0.0
Number of neurons
Number
(a) CIFARIO
(b) CIFARIO0
0.8
Transfer or Accuracy
0.8
Transter:
0.4
6.4
0.-
Number of neurons
Number
(c) Flowers
(d) Oxford A.ford-IIIT-Pett
Naredo Fom randem
vid panede norrsb-undon ranedd broksada wwd.daaoeoaddatdoo
VLama Latchie 224-ramb
Figure 18: [Comparisons Across Architectures For Downstream Task Accuracy] This shows the same plots as Figure except showing absolute number of neurons on the x-axis
Tundy AOURO
Prectise nearees
Prattinn
Proction
Praitinn ceceees
(a) 1.aar1.0.act3
(b) layerl.
(c) Aayer1.2.cct
(d) layer2.0.ac33 0.act3
Teonter Acuncy
Fretiin nearora
Traction neorooa
Traction of neurera
Traction neurone
(c) layer2.1 lact3
1.yer2.2.ac3
(g) Iayer2.3.act
(h) 1ayer3.0.cct
Thasder AccumeE
Frectiin neaross
52
Fraetinn nearoes
Preetinn
Fraction
① Lact33.Lact3
1.ye13,2ac133
(k) 3.cc33
Po Recuracy
54
22
Waction
(m) aact
later4.0.act3
(o) layer4,
1ayer4,2.act3
cifar10
cifar100
flowers
oxford-iiit-pets
Figure 19: [Middle Layers; ResNet50, trained with CrossEntropy on ImageNettk] We see that as we deeper the network, accuracy progressively increases. We see even middle layers exhibit diffused redundancy, and accuracy plateaus very quickly for earlier layers. layerX. Y actt refers to the Yth residual connection in the Xth ResNet block and act3 indicates that we'n taking the value after the activation (ReLU) been applied.
23

---

1.0
0.5
Accuracy
9.4
Accuracy
0.8-
Imagenet ImagenetV2
0.4-
Places365
0.0
0.0
0.2
0.4
0.5
0.8
0.0
0.2
0.4
0.6
0.8
1.
Praction of Neurons
Praction Neurons
ResNer50 nonrohust
ResNet50 mbust Ege
Figure 20: [Performance ImageNettk,, ImageNetV2, and Places365] We check for the performance of randomly chosen subsets neurons on harder tasks like ImageNet1 k, ImageNetV2, Places365. We find that diffused redundancy holds for all these harder tasks as well. Additionally, we see that randomly dropping neurons still preserves the accuracy gap between ImageNetIk and ImageNetV2.
(a) Dropout 0.1
Dropout 0.2
(c) Diopour 0.3
(d) Dropout 0.4
d'ri0
dla/103
flovera
(e) Dropout 0.5
( Dropout = 0.6
(g) Dropout = 0.7
(h) Dropout 0.8
Trancfer Accurccy
dtar1a
6/2
0.
01 62
34
02
6.5
(i) DeCow Regularization (0.0001)
Figure 21: [Dropout and DeCov regularizer's effect on Diffused Redundancy]
24

---

(a) Dropout= 0.1
(b) Dropout 0.2
(c) Drupout = 0.3
(d) Dropout 0.4
diart
(c) Dropout 0.5
(f) Dropout = 0.6
(g) Dropout 0.7
(h) Dropout 0.8
Difrused Redundancy (08)
0.3
0.2
1.0
DeCov Rogularization (0.0001)
Figure 22: [Dropout and DeCov regularizer's effect on Diffused Redundancyl Same results as Figure 21 but showing DR estimates (Eq[]. Lines that are more towards the right (i.e on the "outside") mcan they exhibit more diffused redundancy.
25


Reviews:
[{"review_id": "dYNYsQLWeF", "reviewer": "Reviewer_MR5K", "summary": "Focusing on the (penultimate layer) representation learned by the pre-training model, this paper proposes a \"diffused redundancy hypothesis\" that includes two main concepts:\n\n- Redundancy: A small subnet can achieve comparable performance with the full model.\n- Diffused redundancy: The selection of the small subnet can be random.\n\nTo validate this hypothesis, this paper conducts experiments across various models, datasets, and training paradigms (e.g., standard training or adversarial training) to demonstrate the diffused redundancy phenomenon. Furthermore, this paper also investigates the possible factors that influence the degree of this phenomenon. Finally, this paper shows the trade-off between fairness (class-wise accuracy) and efficiency (pruning rate).", "questions": "1. Line 178: How about $\\ell_\\infty$ adversarial training and other perturbation bounds with $\\epsilon$?\n2. Lines 312-314: This paper claims that diffused redundancy can be used to reduce training and storage costs. Will model pruning or compression techniques serve a similar function? Will carefully pruning $k$% of the neurons result in better performance than randomly dropping $k$% of neurons? ($0\\le k \\le 100$)", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed diffused redundancy hypothesis is an unrealized phenomenon and may have a strong impact on the research community.\n2. This paper leverages several metrics, such as Centered Kernel Alignment (CKA), to interpret and understand this phenomenon, which is reasonable and enhances the soundness of the proposed hypothesis.\n3. This paper systematically investigates the underlying mechanism and influencing factors of the diffused redundancy hypothesis.", "weaknesses": "1. Given that model pruning and compression techniques can achieve comparable performance with only a few neurons, the diffuse redundancy phenomenon seems to be expected and less appealing. Intuitively, a wide network inevitably results in diffuse redundancy in the representations.\n2. This paper only investigates the representation on the penultimate layer and does not analyze this phenomenon in terms of middle layers.\n3. This paper only conducts empirical experiments to validate the proposed hypothesis but does not provide theoretical analysis of why this phenomenon exists.\n4. This paper does not provide an intuitive or theoretical understanding of the fairness-efficiency trade-off in Section 4.\n5.  Line 163, Equation 1. $DR(g, T, \\delta)$ is defined with the average $\\frac{T(m \\odot g)}{T(g)}$ for $m\\in M_f $.\nHowever, the diffuse redundancy hypothesis claims that any randomly chosen subset of neurons can achieve similar performance. Therefore, I think using the worst representation similarity instead of the average is more appropriate: $$DR(g,T,\\delta)=\\frac{1-\\min f, \\text{s.t.}\\min_{m\\in\\mathcal{M}_f} \\frac{T(m\\odot g)}{T(g)}\\ge \\delta}{|g|}$$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review! We address your concerns below:\n\n### [W#1 obvious in light of model compression] \nWe draw differences between our work and model compression literature in our paper in Section 1.1. To expand on that, model compression works aim to reduce the number of parameters in the neural network, without necessarily getting rid of entire neurons. In fact, pruning a ResNet50 by using the lottery ticket hypothesis [1] results in a penultimate layer of the same size as in the dense model. The main promise of such works is to be able to deploy “big” models in resource-constrained environments (eg: edge devices) since pruned models can speed up computation by making use of sparse matrix multiplications and require less storage (if most weights are zero then all these weights do not need to be stored). \nOn the other hand, our work investigates the nature of learned representations by dropping entire neurons from layers, without pruning away any parameters. Thus, we differ subtly, but quite significantly from traditional works on pruning. We would like to emphasize that our work is not obvious in light of work on pruning. The fact that even weight-pruned networks require all neurons to retain high performance should indicate that all neurons are necessary for downstream performance, but we show that this is not the case. Our work can be combined with other works on pruning, which will be an exciting avenue for future work.\nWe completely agree with the reviewers’ intuition about wide networks having more diffused redundancy, however, we believe such intuitions should be empirically tested and our work provides a principled framework to do that. We also find in our experiments that WideResNet50 exhibits more diffused redundancy than ResNet50 (Fig 10, Appendix).\n\n### [W#2 Intermediate Layers] \nOne reason for focusing on the penultimate layer in the paper is that that’s what’s typically used for downstream tasks. Nonetheless, we agree with the reviewer that this will be an interesting phenomenon to test at intermediate layers as well, and hence we’ve repeated our experiments for other layers – we find that similar observations hold even for intermediate layers. See Fig 1 in the attached response PDF for the plots.\n\n### [W#3&4 Lack of theory] \nWe would like to emphasize that the goal of our paper was not to provide a theoretical explanation of this phenomenon and hence we believe this is unjustified criticism. We make it very clear in the abstract that this is an empirical paper. Theoretical analysis of this phenomenon is certainly interesting for future research. Our goal, however, is to introduce the intriguing phenomenon of diffused redundancy and investigate it thoroughly by conducting empirical analyses. \nWe would also like to highlight that theoretical analysis is not trivial here, as is the case with many deep learning papers since any theoretical analysis needs to take into account many nuances of optimizing these models such as learning rate schedules, pre-training data distribution, downstream data alignment with pretraining data, architectural differences such as using self-attention vs convolutions, layernorm vs batchnorm, size of layers and so on. We do not yet have the basic tools to theoretically analyze all these phenomena, and thus turn to an empirical analysis of all these factors in our paper. Moreover, as pointed out in many debates at past ML conferences, theory usually follows after an empirical observation has been made [2]. This is also indicated in recent papers in deep learning that have introduced an interesting empirical phenomenon (Eg: The Lottery Ticket Hypothesis [1]) which have then been followed up with theoretical papers explaining the phenomenon (eg, see [3,4] with the theory behind the Lottery Ticket Hypothesis). \n\n### [W#5 Taking worst case in Eq 1] \nThis is a good point, thank you for the suggestion! We agree the worst-case accuracy is the ideal formulation, especially within our framing of “any subset works”. However, there are two issues for which we prefer the current formulation that considers the mean: (1) in order to compute worst-case accuracy empirically, we need to run this over many more random seeds, which becomes computationally expensive, and (2) worst case can be a very pessimistic estimate. For these reasons, we will re-word our claims about “any randomly chosen subset can achieve similar performance as the entire layer” to “there exist many randomly chosen subsets that can achieve similar performance as the entire layer”.\n\n### [Q#1 $\\ell_\\inf$ models] \nThanks for this suggestion! We have added results for $\\ell_\\inf$ robust models, please see Fig 3 in the attached response PDF. We see that $\\ell_2$ robust models have slightly higher diffused redundancy than $\\ell_\\inf$ models, which in turn have higher redundancy than nonrobust models.\n\n### [Q#2 Carefully selecting $k\\%$ neurons] \nWe have already included results for PCA, which carefully chooses dimensions to project the final layer onto. These are shown in Figures 2c and 2d and are discussed in Section 2.2. We have also added more results for different strategies for choosing $k\\%$ neurons in the response PDF (Fig 4) based on reviewer 1gmQ’s suggestions. All of these results show that randomly selected neurons have unexpectedly high predictive power.\n\n\n[1] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks by Frankle and Carbin https://arxiv.org/abs/1803.03635 \n\n[2] My take on Ali Rahimi's \"Test of Time\" award talk at NIPS by Yann LeCun https://www2.isye.gatech.edu/~tzhao80/Yann_Response.pdf \n\n[3] Proving the Lottery Ticket Hypothesis: Pruning is All You Need by Malach et al. https://arxiv.org/abs/2002.00585 \n\n[4] Optimal Lottery Tickets via Subset Sum: Logarithmic Over-Parameterization is Sufficient Pensia et al. https://arxiv.org/abs/2006.07990 "}, {"review_id": "lAxmUZmB8K", "reviewer": "Reviewer_qvdG", "summary": "The paper proposes investigating the emergence of redundant representations in pre-trained models. The setup is pretraining of various architectures (VIT, vgg, resnet) on Imagenet1k. The study pertains to how much of the representation is redundant and this is estimated by dropping features and finetuning on smaller datasets (cifar 10, 100, flowers and Pets). The paper proposes a measure of redundancy based on centred kernel alignment. The experiments show that this is true under this measure for random subsets of a certain size as long as the model has enough parameters. They also show that there is an apparent drop in fairness that would have to be addressed if features are to be pruned in the proposed way for some datasets like pets and flowers.", "questions": "* I am not sure how meaningful fig 7 is if the fairness is computed on the same labels as the quality. Do I understand correctly that the claim is Gini for say cifar 10 drops less than accuracy meaning it is dropped uniformly but pet's and flowers the Gini index goes up more quickly with accuracy thus is less fair.", "limitations": "none.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* It is an interesting phenomenon to discover and a good study.\n* The paper is generally well written.\n* The comparison across models and losses is interesting and may ultimately lead to improvements in model training and or design.", "weaknesses": "* Not sure how this is going to impact practical aspects of ML in the future. In particular not sure how meaningful the values of CKA are here. The measure (1) seems sensible though.\n* I think section 2.2 comes short of the state goal of explaining why any random subset works. It explains that this is happening but not why this is the result of the optimization.\n* I find the paragraph describing the \"correlation\" between the pruning and PCA confusing and misleading. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We address their concerns below:\n\n### [W#3 confusion in PCA vs Diffused Redundancy results] \nWe believe the reviewer is referring to L225-235, and associated figures 2c and 2d. The goal of this analysis is to compare diffused redundancy (our work) with PCA. In PCA the goal is to project the penultimate layer onto the first $k\\%$ principal components and compare how this performs to randomly choosing $k\\%$ of neurons in the final layer. We see that apart from very low values of $k$, performance of random neurons closely follows the performance of PCA. Please let us know if you’d like any further clarification or if you have any suggestions for us to improve the presentation of these results.\n \n### [W#2 sec 2.2 doesn’t explain why any subset works] \nWe believe we do offer some intuition for what could be going on by showing that randomly choosing $k\\%$ neurons capture the same amount of variance as top $k\\%$ principle components, for moderate values of $k$. We agree this does not answer the question of why gradient descent chooses solutions like this over other solutions that utilize the entire layer. However, we believe this warrants a separate investigation and is outside the scope of this paper.\n\n### [Q#1 meaningfulness of fig 7 when gini is calculated over class labels] \nYes, the reviewer’s interpretation is correct. We’ve done a deeper dive into this analysis to ensure its validity and we see that class accuracies are uniformly spread, and as we drop neurons, the accuracies become less uniform. We have added this in Fig 2 of the response PDF. However, we would like to clarify that these are toy datasets without many real-world fairness implications. Our goal was to show that such disparities can arise when using “compact” representations (such as choosing $k\\%$ of random neurons) and this “fairness” aspect must be taken into account when the underlying deep neural network is being used for high-stakes scenarios (eg: facial recognition).\n"}, {"review_id": "EUlYiMP74v", "reviewer": "Reviewer_USaP", "summary": "The authors introduce the concept of `diffused redundancy', which is the idea that a random sample of neurons reproduces the performance of the whole layer in many downstream tasks. The authors support the existence of this phenomenon with experiments on a variety of architectures pre-trained on ImageNet and explain it with the similarity between these random samples past a certain threshold size. The observed phenomenon is argued to be an 'inevitable' property of wide deep networks, as it seems to persist even when the training method should avoid it. Finally, the authors discuss the implications of diffused redundancy for fairness.", "questions": "1. In line 186, the authors mention that some tasks might require more neurons simply because they have more classes. Would it be possible to find a normalised measure that takes this effect into account?\n\n2. In Figure 2, panels (c) and (d), what is the cause of the discrepancy between dashed and solid curves for a low fraction of neurons?\n\n3. In section 3.2, is the dependency on the width simply due to the fact that there are not enough neurons? For instance, if the fraction of random neurons required to keep performance high with width 10^3 is 10%, then of course there cannot be diffused redundancy for width 10^2. If this is the case, then I would personally demote the discussion on the width to the appendix, either to make more space for the figures or to make space for the methods that decorrelate neurons, which might be more interesting for the general audience.\n\nFew additional comments/suggestions:\n\n4. I believe that the related works section should also discuss the neural collapse phenomenon.\n\n5. It would be helpful, for the sake of clarity, to have a definition of DR in words (i.e. $1$ minus the fraction of neurons required to have at least $\\delta$ accuracy in downstream tasks).", "limitations": "the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The work proposes a novel property of the hidden representations of deep neural networks which furthers our understanding of such representations. This new hypothesis is accurately framed within the existing literature and convincingly backed up by extensive numerical experiments. Overall, the manuscript is well-written and easy to follow.", "weaknesses": "I think this paper would benefit from a comparison of the diffused redundancy hypothesis with neural collapse (Papyan et al, 2020). In simple terms, if the neurons of the last layer are clustered into a number of groups much smaller than the width, then there will naturally be a threshold size $k$ such that if I pick $k$ neurons at random I will pick at least one from each cluster with large probability. If the number of clusters is taken as the number of classes, this would lead to a prediction for the behaviour of DR. This prediction would be the same, in principle, for different downstream tasks, but maybe this difference is only due to different downstream tasks having different classes. If true, this difference could be eliminated with an appropriate normalisation (see question 1 below).\n\nAnother criticism concerns the size of the figures. I would recommend the authors use the extra page for increasing the size of the figures.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and great suggestions! We address their concerns below:\n\n### [W#1 neural collapse] \nThank you for pointing out this interesting connection! We agree that diffused redundancy and neural collapse go hand-in-hand, and it would be especially interesting to see if diffused redundancy still holds before and around the terminal phase of training (ie before final layer activation has collapsed onto a much smaller subspace). If it’s indeed the case that neural collapse leads to diffused redundancy then we should likely not observe any diffused redundancy before the terminal phase. It will be very interesting future work to look into this! We will update the paper to include a discussion with neural collapse. \n\n### [Q#1 normalized measure] \nComputing the current measure, as it’s stated in our paper, does not depend on the number of classes in a downstream task. This is because we consider accuracy for downstream task performance. We can further consider balanced accuracy to eliminate any effects of classes. However, the point we wanted to make in our empirical experiments was that for some datasets we find we need more neurons from the penultimate layer to achieve same performance as the entire layer. Interestingly, these also happen to be datasets with more number of classes (CIFAR100 and Flowers in our experiments). \n\n### [Q#2 discrepancy between dashed and solid lines] \nDashed lines show the performance of PCA features whereas solid lines show. We see that for a lower number of neurons, PCA features perform significantly better than randomly choosing neurons and hence we see this discrepancy. Once hit around $k = 20\\%$ of neurons, we see that the solid line (diffused redundancy) catches up with the dotted line (PCA). \n\n### [Q#3 width] \nDiffused redundancy depends quite a bit on the architecture that was optimized during pre-training. We see that even though with a model like VGG16 (4096 neurons) one could discard 95% neurons (keeping ~200 neurons) without taking a hit on accuracy, one can still discard more than 70% neurons on a much smaller ViT model (retaining only about 115 neurons). While across architectures these differences could be happening due to a variety of other factors, in this section we wanted to carefully test this on the same architecture with the same pre-training data, by just varying the length of the penultimate layer. We can move this to the Appendix to make space for other suggestions!\n\n### [Clarity] \nWe will indeed use the extra page for increasing the figure size, thank you for the suggestion! We also agree that an intuitive definition of DR will aid readability and will add it to the paper.\n"}, {"review_id": "NLv2Q0gdHk", "reviewer": "Reviewer_yCaj", "summary": "The paper empirically investigates the various factors that affect how the features of a pretrained representations are encoded. They find that the features exhibit a degree of diffuse redundancy and randomly chosen subsets of the features in the penultimate layer have similar accuracy on downstream tasks and score high in common similarity metrics. They find that the loss and datasets used to pretrain the model affect the number of features required to achieve most of the performance on a given dataset. They show potential applications for efficient transfer learning and implications for unequal performance across classes when neurons are removed. ", "questions": "- For the adversarial training experiment, is the test set the same as standard training? (i.e. are the adversarial perturbations only done on the train split.)\n\n- The keys in the legend in figure 7 are unclear. For example, what does “nonrob” or “first” mean exactly?\n\n- For figure 5 it would be easier to draw conclusions if the x-axis was in length of the feature vector rather than fraction of used neurons so that models could be compared. For example, 20% of a model with a 256-length feature vector isn’t comparable to 20% of a model with a 64-length feature vector. If I wanted to know the best model to use for a given feature vector length, it would be difficult to see from the figure. \n \n", "limitations": "The authors adequately address the limitations of their work. They discuss how removing features from the feature vector could unequally affect the class accuracy. I think further experiments are needed to convincingly show that dropping features is the root cause of this phenomenon, but the authors bring up an interesting limitation of shortening the feature vector. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Using adversarial training to create more diffuse feature representations is an interesting observation and potentially useful for efficient retrieval or efficient inference when target datasets have many classes. In general, understanding how adversarial training affects the feature space of a pre-trained model is an interesting direction. \n- The experiments are relatively thorough with multiple architectures and datasets. \n- The paper is well written and organized. ", "weaknesses": "- The applications and implications of the diffused redundancy hypothesis given in the paper are not not well motivated, or lacking experimental evidence. \n\n  - One of the major stated contributions is decreased training time on downstream tasks. I think there are a few weaknesses in this argument. First, from my understanding, to determine the optimal number of neurons to drop for a given target dataset, you would need to perform multiple training runs with varying levels of neurons being dropped. Searching for this hyper-parameter would require additional FLOPS rather than reduce them. Second, I didn’t see analysis in the main paper or appendix about how many FLOPS are saved by using the reduced-length feature vector. Unless most of the FLOPS are in the final layer, the reduction in compute during training will be minor. Similar works, such as Slimmable Neural Networks, reduce parameters throughout the whole network for the purpose of efficiency, but at inference. Third, for most downstream datasets, fine-tuning is computationally cheap as the dataset and number of training epochs are small. Most of the work on efficiency is focused on pre-training or inference. \n\n  - The other stated implication of the diffused redundancy hypothesis is the accuracy inequality between classes when reducing length of the feature vector. The experiments in figure 7 don’t rule out the possibility that the gini-coefficient is merely correlated with the accuracy. It would be more convincing to see additional models on the scatter plot in figure 7 to show that this isn’t the case, and dropping neurons in the final layer are the main cause of the increase of the gini- coefficient\n\n- In the abstract it is stated that: “We find that learned representations in a given layer exhibit a degree of diffuse redundancy”, but from what I understand, the main experimental findings are only for the penultimate layer, so this claim is misleading. \n\n- CIFAR, OxfordPets, and Flowers are relatively small and less diverse than the other datasets in the appendix (ImageNetV2 and Places365). It would have been useful to see more analysis of how diversity and scale affect the conclusions. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments! We address their concerns below:\n\n### [W#1 decreased downstream training time] \nWe would like to emphasize that our goal is not model optimization. There has been long-standing research on structured pruning (He et al.[1]; Li et al.,[2]) that achieve optimizations (in terms of FLOPS, memory, inference times vs drop in performance) which are way better than randomly dropping features (our work). Our goal, however, is to point to an intriguing property of learned features in commonly used pre-trained DNNs. We tried to make this clear in the Introduction and hence we do not mention model optimization or reduction in training FLOPS as our contribution. We would also like to emphasize that one can still use diffused redundancy in conjunction with structured pruning methods (since these methods act on initial convolution layers) to get a model that is compact overall. One can also pre-compute the parameters that are best for downstream transfer and repeatedly use them, thus not having to run the computation of diffused redundancy repeatedly. However, such analyses are outside the scope of this paper and are very interesting directions for future work!\n\n\n### [W#2 gini being correlated to accuracy] \nThis is a good point, thank you for raising this! We’ve added gini plots for more models in the response PDF (Fig 2) and see that all models follow a similar trend of increasing gini as we drop neurons. Additionally, to further investigate your point, we’ve added the accuracy distributions over CIFAR10 classes (other datasets have many classes and thus could not fit into the response PDF, but we see similar trends) as a function of fraction of neurons (due to space constraints we could only include one model, but see similar trends throughout). For the low accuracy regime (left of the x-axis in Fig 1 in the response PDF, lower fractions), the heatmap shows that a high gini coefficient is clearly due to homogeneity in predictions – the model predominantly predicts a few classes and hence falters heavily on other classes. We see that this homogeneity only appears as we gradually drop neurons, and thus can conclude that the increase in gini coefficient is primarily due to dropping neurons.\n\n\n### [W#3 all results are only for penultimate layer] \nWe have added results for other layers in the response PDF (Fig 1) and see that a similar trend of diffused redundancy holds there as well. Regardless, we agree that the majority of our results are on the penultimate layer and will tone down this claim in the paper.\n\n\n### [W#4 larger and diverse datasets] \nWe have results on ImageNet1k, ImageNetV2, and Places365, all of which are not only more diverse but are also orders of magnitude larger in size. Additionally, ImageNetV2 represents an interesting case of a distribution shift. These results are in Appendix E. We see that we still observe diffused redundancy on these datasets, despite them being much “harder” than the other small datasets we consider. We can move the discussion of these results to the main paper.\n\n\n### [Q#1 accuracy for robust models] \nYes, all accuracies reported are “clean” test accuracies. We will make this clear in the paper.\n\n\n### [Q#2 legend in Fig 7] \n“First” means we take the first k neurons in the penultimate layer and “nonrob” refers to the non-robust model (i.e., model trained with the usual CrossEntropy loss). We explain this in L283-288 in the paper. We will also make it clear in the figure caption.\n\n\n### [Q#3 Fig 5 x-axis labels] \nThe goal of this analysis is not to know the best model for a given feature length but rather to show how diffused redundancy changes as we shrink the base model’s feature vector length. Since diffused redundancy is measured in terms of the fraction of neurons in a given layer, we showed plots with fractions on the x-axis. We already have the result for feature vector length in the appendix in Fig 13 for a different experiment, and see how this can sometimes be misleading since the nature of learned features varies quite a lot based on how many neurons we have in the layer. Regardless, we can certainly add the same figure with the number of neurons on the x-axis in the final version (we’re out of space in the response PDF), however, we’d kindly ask the reviewer to not hold our paper to a lower score due to this trivial change.\n\n\n[1] (Li et al., 2016) Pruning filters for efficient convnets\n\n[2] (He et al., 2017) Channel pruning for accelerating very deep neural networks\n"}, {"review_id": "zxuCZPtFrq", "reviewer": "Reviewer_1gmQ", "summary": "Authors argue that pre-trained representations exhibit a degree of \"diffused redundancy\", i.e. the accuracy on downstream tasks with linear probing is similar when we use the whole layer compared to a randomly subset of neurons. This claim is validated using representations pre-trained using ImageNet-1k and Imagent-21k and probe these representations on downstream tasks like CIFAR-10 and CIFAR100. The authors look at different factors that influence diffused redundancy like regularization or training loss--including variants like Matryoshka learning or adversarial training. Finally, they evaluate the potential impact on different class accuracies to understand the impact of exploiting diffused redundancy.\n", "questions": "The abstract (L17) claims that \"entire layers might not be necessary\". The authors could perhaps clarify that this is only true for the final layer, since their experiments are restricted to this layer (which doesn't discredit the importance of their findings).\n\nIn Figure 1(b) and Figure 1(d) the diffused redundancy plot seems to take only 7 different values which seems rather strange to me. Could the authors explain why this is the case? How many different $D_f$ sets are considered when estimating the diffused redundancy?\n\nCan these observations be extended to random projections of representations? Selecting a random subset of neurons can be viewed as looking at restricted class of random projections.\n", "limitations": "The authors have studied diffused redundancy under different settings and have adequately addresses limitations. Section 4 also addresses some potential limitations if future methods were to exploit diffused redundancy. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The authors identify a new phenomenon that seems to occur when we pre-train representations on large datasets.  The manuscript methodically studies diffused redundancy on a number of different downstream tasks, regularization mechanisms, layer width and loss functions and identify that it occurs across all these scenario but to different extents. They also observe that diffused redundancy increases with adversarial training which is interesting and find that the class accuracies are impacted unevenly if we use a subset of the neurons. Overall, I believe that this is an interesting finding with no complete explanation for when and why it happens. \n", "weaknesses": "**Does diffused redundancy occur for any random subset?**\n\nThe authors claim diffused redundancy holds for *any* subset of neurons (L5 and L53). However, this claim is only validated by averaging over 5 subsets of neurons (L174). Is it possible that the redundancy is true for only some subset of neurons? For example, if we consider the representation projected onto the last $k$ principal components, does the diffused redundancy observation still hold? Alternately, we could select $k$ neurons with the least variance and test if the linear probe accuracy is similar. I believe that the claim, that this holds for *any* subset of neurons, is not sufficiently validated.\n\nIf I understand the results correctly, there exists an alternative explanation that isn't completely ruled out. As an alternative, let us assume that most of the information is diffused among an *informative subset* of neurons. Since the experiments consider only 5 random subsets of neurons, it is possible that each random subset contains some neurons from this informative subset. Hence the information may not be diffused among *all* neurons in the final layer but only among some of them.\n\nOn a related theme, in Figure 2b, the PCA projected representations has an accuracy that is higher by as much as 20% compared to a random subset of neurons. However in L56, the authors claim that the representation performs as well as PCA. Am I misinterpreting these plots? This seems to indicate that most of the information exists in a smaller subset of neurons.\n\n**Is this specific to pre-training on ImageNet ?**\n\nI also wonder if these observations are specific to models pre-trained on ImageNet and it would be nice to investigate this further. How prevalent is this phenomenon and does it occur if we pre-train models on Places-365 or iMaterialist? I think another important experiment is to pre-train models on smaller datasets with very few classes (like CIFAR10) and see if diffused redundancy occurs in this scenario.\n\n**Definition of diffused redundancy**\n\nI also have some minor concerns regarding the definition of diffused redundancy. The definition (Eqn. 1), requires the accuracy averaged over different subsets of neurons ($M_f$) to be greater than some threshold $\\delta$. But this doesn't require *every* subset of neurons to have high accuracy. Maybe, a more appropriate definition is to require all subsets to have accuracy greater than some threshold if that is the central claim of the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and excellent suggestions! We answer their concerns below:\n\n### [W#1 comparison to last $k\\%$ principal components] \nThank you for this suggestion! We’ve added these results in the response PDF (Fig 4). We see that projections onto the lower PCA components perform significantly worse than both top PCA and randomly chosen neurons (our work). This suggests that randomly sampling neurons leads to capturing features that are more akin to projections on the top (high variance) principal components, rather than the projections on the lower (low variance) principal components. We agree with you that as stated, the claim that diffused redundancy holds for *any* subset of neurons is not entirely accurate and we will reword it to say that there are many subsets that can perform as well as the entire layer.\n\n### [W#2 what if we’re picking from an informative subset] \nThis is a great point! We believe this is compatible with diffused redundancy and it’s highly likely that a random subset picks neurons from an “informative” set of neurons. In fact, we see this in Figs 2c and 2d where the performance of diffused redundancy starts to catch up with PCA’s performance as we increase the fraction of neurons sampled. Finding this exact subset of informative neurons is not straightforward, but we show that simple random sampling gets us a long way.\n\n### [W#3 PCA vs Diffused Redundancy for lower value of $k\\%$] \nWe agree with the reviewer’s observation that PCA tends to be much better than diffused redundancy for very small values of $k$, but diffused redundancy mostly catches up by $k = 20\\%$. We will add this nuance to the paper.\n\n### [W#4 Other pre-training datasets] \nImageNet models have been widely used for a variety of tasks. In addition to ImageNet1k, we also added results for ImageNet21k, which is a significantly larger version of ImageNet with weakly labeled data. We see that these models exhibit even high diffused redundancy, thus, at first glance this seems to be a more general phenomenon, not restricted to ImageNet1k. \n\n### [W#5 worst case accuracy for Eq 1] \nThank you for this suggestion! We agree the worst-case accuracy is the ideal formulation, especially with the way we framed this as working for “any subset”. However, there are two issues for which we prefer the current formulation that considers the mean when empirically estimating diffused redundancy: (1) in order to compute worst-case accuracy empirically, we need to run this over many more random seeds, which becomes computationally expensive, and (2) worst case can be a very pessimistic estimate. For these reasons, we will re-word our claims about “any randomly chosen subset can achieve similar performance as the entire layer” to “there exist many randomly chosen subsets that can achieve similar performance as the entire layer”.\n\n### [W#6 results applicable only to penultimate layer] \nWe have added results for other intermediate layers in the response PDF (Fig 1) and we see that diffused redundancy also exists at these layers. However, we agree that all of our other analyses are on the penultimate layer, and we will clarify this in the paper.\n\n### [W#7 only 7 values for DR in Fig 1b and 1d] \nIn practice, we calculate diffused redundancy (DR) by attaching linear probes to different fractions of randomly chosen neurons from a given layer (here penultimate layer) and finetuning just this probe on the downstream dataset. In our experiments, we chose 10 different values of this fraction and hence (1b actually has 9 distinct values) we see discrete jumps in the plot. We added more fractions in our analysis and now in Fig 3 of the response PDF, we see that DR takes many more values. Generally, as we add more fractions, we should expect to see a smoother curve for DR plots.\n\n### [W#8 random projections vs random neurons] \nThis is a very interesting suggestion! We ran this experiment and report the results in Fig 4 of the attached response PDF. Interestingly we find that diffused redundancy does significantly better than random projections. This indicates that the “constrained” projection offered by diffused redundancy is crucial in achieving good downstream performance.\n"}]

Author rebuttal:
We thank the reviewers for their time, positive comments and constructive feedback. There were many excellent points raised by the reviewers. We ran the following experiments to address their concerns:

 - Fig 1: One common comment raised by reviewers was that all our experiments were only on the penultimate layer. We additionally ran our experiment on other intermediate layers and report the results here. We present results for a ResNet50 pretrained on ImageNet1k using the standard CrossEntropy loss – we see similar trends for other models as well, but have to omit the results due to space constraints. The intermediate layers considered are characterized as activations following each residual connection within distinct ResNet blocks. We show 4 such layers, each from a different block of the ResNet. Interestingly, the results indicate the existence of diffused redundancy even within intermediate layers.

 - Fig 2: This comment was raised by reviewer yCaj. With this analysys we wish to make sure that the reported drop in gini coefficient is actually due to dropping neurons and not merely an artifact of low accuracy. We show class-wise accuracies as a function of dropping neurons, along with gini plots of other models and find that while class-wise accuracies start at (almost) uniform values, as we drop neurons, performance on some classes deteriorates more than other classes, thus showing that dropping neurons directly results in disparate performance across classes.

 - Fig 3: We added diffused redundancy (DR) results for a $\ell_\inf$ ResNet50 robust model. We also added more fractions that are used to compute Diffused Redundancy, so the curve now takes "smoother" values than shown in the paper. We find that $\ell_2$ robust models show higher diffused redundancy than $\ell_\inf$ models.

 - Fig 4: Added comparisons of DR with random projections, projections on top PCA components (high variance), and projections on lower PCA components (low variance). We see that diffused redundancy works massively better than random projections and least PCA dimensions. 


Meta-review: All reviewers unequivocally agreed that this submission to be accepted.
Decision: Accept (poster)