Learned Neural Network Representations are Spread Diffusely with Redundancy
Authors: Vedant Nanda
Till Speicher
John P Dickerson
Soheil Feizi
Krishna Gummadi
Adrian Weller
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation learning
redundancy
transfer learning
fairness

Abstract: Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded in such pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy, ie, any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able to perform similarly as the whole layer on a variety of downstream tasks. For example, a linear probe trained on 20% of randomly picked neurons from a ResNet50 pre-trained on ImageNet1k achieves an accuracy within 5% of a linear probe trained on the full layer of neurons for downstream CIFAR10 classification. We conduct experiments on different neural architectures (including CNNs and Transformers) pre-trained on both ImageNet1k and ImageNet21k and evaluate a variety of downstream tasks taken from the VTAB benchmark. We find that the loss & dataset used during pre-training largely govern the degree of diffuse redundancy and the "critical mass" of neurons needed often depends on the downstream task, suggesting that there is a task-inherent sparsity-performance Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for exploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences.

Under review as a conference paper at ICLR 2023
LEARNED NEURAL NETWORK REPRESENTATIONS ARE SPREAD DIFFUSELY WITH REDUNDANCY
Anonymous authors Paper under double-blind review
ABSTRACT
Representations learned by pre-training neural network on large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, take a closer look at how features encoded in such pre-trained rep- resentations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy, í.e., any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able perform similarly as the whole layer stream tasks. For example, linear probe trained on 20% of randomly picked neurons from ResNet50 pre-trained on ImageNet k achieves an accuracy within 5% of a linear probe trained on the full layer of neurons for downstream CIFAR 10 classification. We conduct experiments on different neural architectures (includ- ing CNNs and Transformers) pre-trained on both ImageNet1k k and ImageNer21 k and evaluate variety of downstream tasks taken from the VTAB benchmark. We find that the loss & dataset used during pre-training largely govern the degree of diffuse redundancy and the "critical mass" neurons needed often depends on the downstream task, suggesting that there is task-inherent pparsity-erffrrmacce Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for ex- ploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences.
variety of down-
INTRODUCTION
Over the years, many architectures such VGG (Simonyan Zisserman 2014), ResNet (He al., 2016), and Vision Transformers (ViTs) (Kolesnikov et al. (021) have been proposed that achieve competitive accuracies on many benchmarks including the ImageNet (Russakovsky al.) 2015) challenge. A key reason for the success of these models is their ability to learn useful representations of data (LeCun et 2015).
Prior works have attempted to understand representations learned by deep neural networks through the lens of mutual information between the representations, inputs and outputs (Shwartz- Ziv & Tishby| (017)) 2017) and hypothesize that neural networks perform well because of a "compression" phase where mutual information between inputs and representations decreases. Moreover recent works on interpretability have found that many neurons in learned representations are polysemantic, iee. one neuron can encode multiple "concepts" " (Elhage et al. [002 Olah et al. 22220), and that one can then train sparse linear models such concepts do "explainable" classification (Wong et al.) 202T However, it is not well understood if or how extracted features are concentrated or spread across the full representation.
While the length of the feature vectors extracted from fatee-ofthhe-a networks can vary greatly, their accuracies downstream tasks are not correlated the size of the representation (see Table 1 but rather depend mostly on the inductive biases and training recipes (Wightman et al. 2021 Steiner al. 2021). In all cases, the size of extracted feature vector (i.e. number of neurons) orders of
'Extracted features for the purpose of this paper refers to the representation recorded on the penultimate layer, but the larger concept applies any layer

---

Under review as a conference paper at ICLR 2023
Table 1: Different model architectures with varying penultimate layer lengths trained on ImageNetIk. WRN50- stands WideResNet50-2. Implementation of architectures taken from timm [Wightman 2019). Dif- fused redundancy here measures what fractions of neurons (randomly picked) can be discarded to achieve within = 90% performance of the full layer.
Feature Length
ImageNetIk Top-1 Accuracy
Diffused Redundancy for 0.9
Model
CIFARIO
CIFARTOO
Flowers
Oxford-IIIT-Pets
ViT S-16
384
64.82%
0.70
0.50
0.50
0.80
VIT S-32
384
55.73%
0.70
0.50
0.50
0,70
R8sNet18
512
69.23%
0.80
0.50
0.50
0,90
ResNet50
2048
80.07%
0,90
0.50
0,20
0,90
WRN50-2
2048
77.00%
0.95
0.80
0.50
0,95
VGG16
4096
73.36%
0.95
0.80
0.80
0.95
magnitude less than the dimensionality of the input (e.g. ImageNet models, the inputs are 224 224 150528 dimensional) and thus allows efficient transfer to many downstream tasks (Kolesnikov et al. 2020; Bengio et al.) 2013) Pan & Yang 2009 TTan et al. 2018). We show that even when using random subset of these extracted neurons one can achieve downstream transfer accuracy close to that achieved by the full layer, thus showing that learned representations exhibit degree of redundancy (Table
Early works in perception suggest that there are many redundant neurons in the human visual cor- tex (Attneave) 1954) and some works argued that a similar redundancy in artificial neural networks should help in faster convergence (Izui & Pentland 1990).. In this paper revisit redundancy in the context of modern DNN architectures that have been trained on large-scale datasets. In particular, propose the diffused redundancy hypothesis and systematically measure its prevalence across different pre-training datasets, losses, model architectures and downstream tasks. We also show how this kind of redundancy can be exploited to obtaín desirable properties such as generalization performance and better parity in inter-class performance. We highlight the following contributions:
We present the diffused redundancy hypothesis which states that learned representations exhibit redundancy that is diffused throughout the layer. Our work aims to better understand the nature of representations learned by DNNs.
We propose a measure of diffused redundancy and systematically test our hypothesis across vari- ous architectures, pre-training datasets & losses and downstream tasks.
We find that diffused redundancy significantly impacted by pre-training datasets & loss and downstream datasets.
We find that models that explicitly trained such that particular parts of the full repre- sentation perform as well as the full layer, i.e., these models have structured redundancy (e.g. (Kusupati al., 2022)), also exhibit significant amount of diffused redundancy, show- ing that this phenomenon is perhaps inevitable when DNNs have wide enough final layer.
We quantify the degree of diffused redundancy as function of the number of neurons in given layer. As we reduce the dimension of the extracted feature vector and re-train the model, the degree of diffused redundancy decreases significantly, implying that diffused re- dundancy only appears when the layer is wide enough to accommodate redundancy.
Finally we draw caution to some potential undesirable side-effects of exploiting diffused redun- dancy for efficient transfer learning that have implications for fairness.
1.1 RELATED WORK
Closest to our work is that of Dalvi et (2020) who also investigate neuron redundancy but in the context of pre-trained language models. They analyze two language models and find that they can achieve good downstream performance with a significantly smaller subset of neurons. However, there are two key differences to our work. First, their analysis of neuron redundancy uses neurons from all layers (by concatenating each layer), whereas we show that such redundancy exists even at the level of single (penultimate) layer. Second, and perhaps more importantly, they feature selection to choose the subset of neurons, whereas we show that features are diffused throughout and that random pick of neurons suffices. Our work also differs by analyzing vision models
2

---

Under review as a conference paper at ICLR 2023
(instead of language models) and using diverse set of 21 pre- pre-trained models (as opposed to testing only two models) which allows us to better understand the causes of such redundancy.
Efficient Representation Learning These works aim to learn representations which are "slim", with the goal of efficient deployment on edge devices (Yu et al. 2018; Yu & Huang, 2019 Cai et al. 2019). Recently proposed paradigm of Matryoshka Representation Learning (Kusupati al.) 2022 aims to learn nested representations where one can perform downstream tasks with only a small portion of the representation. The goal of such representations is to allow quick, adaptive deployment without having to perform multiple, often expensive, forward passes. These works could be seen as inducing structured redundancy on the learned representations, where pre-specified parts of the representation are made to perform similar to the full representation. Our work, instead. aims to looks at diffused redundancy that arises naturally the training of DNNs. We carefully highlight the tradeoffs involved in exploiting this redundancy.
Pruning and Compression Many prior works focus on pruning weights (LeCun et al. 1989 Han
et al. 2015 Frankle & Carbin| 2019 Hassibi & Stork] 1992; Levin et al., 1993 Dong et al. 2017 Lee et al. 2018) and how can lead to sparse neural networks with many weights turned off. Our Tocus, however, is on understanding redundancy at the neuron level, without changing the weights. Work on structured pruning is more closely related to our work Li et al. (0166); He et al.) (017). however, key focus of these works is to prune channels/filters from convolution layers. Our work is more focused on understanding the nature of learned features and is more broadly applicable to all kinds of layers and models. We additionally focus randomly pruning neurons, whereas structured pruning methods perform magnitude or feature-selection based pruning.
Many works aim understand learned representations with the goal of better explainability (Mahendran & Vedaldi 2014, Yosinski et al.] 2015 Alain & Bengio
ExplaannabiiityyInteerrttaaaiiii
2018 Kim al., 2018, Olah et al.) 2017; 2020] Elhage et al., [022] Zeiler & Fergus 2013). How-
ever, two works are especially related to our work: sparse linear layers (Wong et al. 2021 which
show that one can train sparse linear layers on top of extracted features from DNNs; and concept bottleneck models (Koh et al., (0200) which explicitly introduce a layer in which each neuron cor- responds to a meaningful semantic concept. Both these works explicitly optimize for small/sparse layers, whereas our work shows that similar "small" layers already exist in pre-trained networks, and in fact, can be found simply with random sampling.
Understanding Deep Learning A related concept is that of instrinsic dimensionality of DNN land- scapes (Li et al. 2018). Similar to our work, intrinsic dimensionality also requires dropping random parameters (weights) of the network. We, however, are concerned with dropping individual neurons. Other works on understanding deep learning (Shwartz-Ziv & Tishby/ 2017] Achille & Soatto) 2017) have also looked at the learned features, however none of these works analyze the redundancy at neuron level.
2 THE DIFFUSED REDUNDANCY PHENOMENON
Prior observations about a compression phase (Shwartz-Ziv Ziv Tishby 2017) suggest that the rep- resentations need not store lot of information about the ippu;; and observations about there being polysemantic neurons (Olah et al. 2020) states that one neuron can store multiple concepts. Both findings allude to the possibility of not needing all neurons in the learned feature space. Extending these observations, we propose the diffused redundancy hypothesis:
Learned features are diffused throughout given layer with redundancy such that a randomly chosen subset of neurons can perform similar to the whole layer for variety of downstream tasks.
Note that our hypothesis has two related but distinct parts to it: 1) redundancy in learned features, and 2) diffuuson of this redundancy throughout the extracted feature vector. Redundancy refers to features being replicated in parts of the representation so that one can perform downstream tasks with parts of representation as well with the full representation. Diffusiin refers to this redundancy being spread all over the feature vector (as opposed to being structured), i.e., any random subset (of sufficient size) of the feature vector performs equally well.
In order to evaluate the redundancy part of the diffused redundancy hypothesis we use two tasks: 1) representation similarity between randomly chosen subsets of representation with the whole
3

---

Under review as a conference paper at ICLR 2023
Shatt
Fraciin Neurone (161)12266))
Fraction of Meurons (16ta)-
(a) Drenstream Ace; Standard
(b) Diff. Redundancy; Standard
(c) Downstream Ace; AT
(d) Diff. Redundancy; AT
0.3
8
Tovel
00 57
Prackion Neurona (Total-2048)
E. 2.4
Praction of Mourons (Totl=2048)
CKA; Standard
CKAA;
Figure 1: [Testing For Diffused Redundaney in ResNet50 Pre-trained on ImageNetlk] Top: transfer accu- racies + DR measure (Eq [ on different downstream datasets, dotted horizontal line shows accuracy obtained using the full layer. We see that accuracy obtained using parts of representation varies greatly with pre-training loss (much more diffused redundancy in adversarially trained ResNet). but also depends on the downstream dataset. Bottom: comparing CKA between randomly chosen fraction of neurons the whole layer. Here evaluate CKA on samples from different datasets and fnd that similarity a subset layer rapidly increases, reaching a similarity of greater than 90% on the adversarially trained ResNet50 with only 10% randomly cho- sen neurons. All values are averaged over 5 random picks and errors bars show standard deviation.
representation, and 2) transfer accuracy on out-of-distribution datasets (using linear probe) of randomly chosen subsets of the representation compared to the whole representation. To estimate diffusion, we run each check for redundancy over multiple random seeds and plot the standard deviation over these runs.
Representation Similarity of Part vs Whole Centered Kernel Alignment (CKA) a widely used representation similarity measure and takes in two representations of n data points Z E B= and Y E Raxd2 and gives similarity score between and (Korbblihh al. 2019). Intuitively, CKA (with linear kernel, see Appendix for details about CKA) measures if the two representations rank the n points similarly (where similarity is based on cosine distances). For a given neural network g and samples drawn from given data distribution, i.e., X ~ D, let g(X) be the (penultimate) layer representation. If m is a boolean vector representing subset of neurons in g(X), then we aim measure CKA(m g(X), (XX) to estimate how much redundancy exists in the layer. If indeed CKA(m O g(X), g (X)) is high (i.e. close to 1) then it's a strong indication that the diffused redundancy hypothesis holds.
Downstream Transfer Performance of Part vs Whole A commonly used paradigm to measure the quality of learned representations is measure their performance
variety of downstream tasks (Zhai et al. 2019 Kolesnikov et al. 2020). Here, we attach a linear layer (h) on top of the extracted features of network (g) do classification. This layer then trained using the training dataset of the particular task (keeping frozen). If features were to be diffused redundantly then accuracy obtained using h o g, i.e. linear layer attached the entire feature vector, should be roughly the same as h o (m @g) g): where m is boolean vector representing a subset of neurons extracted by g. and h & are independently trained linear probes.
For both tasks, i.e. representation similarity and downstream transfer performance, we evaluate on CIFAR10/10N (Krizhevsky al., 2009). Oxford--IIT-PPts (Parkhi et al., 2012) and Flowers (Nils-) back & Zisserman, 2008) datasets, from the VTAB benchmark (Zhai et al. 2019). Training and pre-processing details are included in Appendix B
Measure of Diffused Redundancy In order to rigorously test our hypothesis, we define a measure of diffused redundancy (DR) for a given model (g) with M being a set of all possible boolean vectors of size L.e. size of the representation extracted from g. Each vector m M represents possible subset of neurons from the entire layer. This measure is defined on a particular task (T) as follows:
4

---

Under review as a conference paper at ICLR 2023
min s.t. [ Emem,
1
Timsog) T(s) 2 8
DRig, T, =
(I)
[g]
M - {m E m== m E {0, 13111
Here T() denotes the performance of the model inside () for the particular task and is a user- defined tolerance level. For the task for representation similarity T(m g) is CKA between subset of neurons denoted by m @ ! and and ((g) is always I, since it denotes CKA between and g- For downstream transfer performance. (m C g) is the test accuracy obtained by training linear probe on the portion of representation denoted by m 3g and T(g) is the test accuracy obtained using the full representation. For 8 = 1, this measure tells what fraction of neurons could discarded to exactly match the performance of the entire set of neurons. A higher value of DR denotes that fewer random neurons could match the task performance of the full set of neurons, and thus indicates higher redundancy. Since M contains an exponential number of vectors (219)), precisely estimating this quantity is hard. Thus, we first choose a few f (number of neurons to be chosen) to define subsets of M. Then for each M we randomly select samples.
2.1 PREVALENCE OF DIFFUSED REDUNDANCY IN PRE- TRAINED MODELS
Figure T checks for diffused redundancy in the penultimate layer representation of two types of ResNet50 pre-trained on ImageNetIk: one using the standard cross-entropy loss and another trained using adversarial training (Madry et al. 2019) (with l threat model and e 3). We check for diffused redundancy using both tasks of representation similarity and downstream transfer perfor-
mance.
Redundancy This is indicated along the x-axis of Fig[1 i.e., redundancy is shown when some small subset of the full set of neurons can achieve almost as good performance as the full set of neurons. When looking at downstream task performance (Figs Tc.. in order to obtain performance within some 8% of the full layer accuracy (dotted lines), the fraction of neurons that can be discarded task desendeenn,,n,, e.g. across both training types (100 classes) require more fraction of neurons than CIFARIO (10 classes) and oxford-iit--ett (37 classes), perhaps because both these tasks have more classes. Additionally, across all datasets, the model trained with adversarial training exhibits more diffused redundancy (Fig [b] than the one trained with standard loss, meaning we can discard far more neurons for the adversarially trained model to reach close to the full layer accuracy. Interestingly when looking at CKA between part of the feature vector with the full extracted vector (Figs Te If.. we do not see significant difference in trends when evaluating CKA on samples from different datasets. However, we still see that we can achieve given level of CKA with far fewer fraction of neurons in the adversarially trained ResNet50 as compared to the usually trained ResNet50.
that flowers (102 classes) and CIFARI00
Diffusion This is indicated by small error bars in Figs Tak 1ck Iek
redundancy were instead very structured, then different random picks of neurons would have high variance, however the error bars here are very low, showing that performance is very stable across different random picks, thus indicating that redundancy is diffused throughout the layer.
While both tasks of downstream transfer and CKA between part and whole indicate higher diffused redundancy for the adversarially trained model, differ substantially based on the dataset (while CKA remains fairly stable across the same datasets), indicating that downstream performance turns out to be "harder" test for diffused redundancy. Thus, in the rest of the paper we examine diffused redundancy through the lens of downstream transfer performance and include CKA results in Appendix A
that downstream transfer performance can
2.2
UNDERSTANDING WHY Any RANDOM SUBSET WORKS
Many prior works explicitly train models to that have "small" representations (e.g. (Kusupati al., 2022 Yu et al. 2018 Yu & Huang] 2019 Cai et al. 2019) with the goal of efficient downstream learning These works show that, when explicitly optimized, networks can perform downstream classification with fewer neurons than typically used in state-of-the-art architectures. We show, however, that such subsets already exist in models that not explicitly trained for this goal and
5

---

Under review as a conference paper at ICLR 2023
in fact doesn't even have try hard to find this subset; it can be randomly chosen. Later in section 3.3 3 we compare some of these efficient representation learning methods to randomly chosen subsets and carefully analyze the tradeoffss involved. Here, however, seek to better understand why (almost) any random subset works.
We analyze this through the lens of representa- tion similarity (Komblith et al.) 2019). More specifically, we calculate CKA between two random picks of k% neurons in the penultimate layer (averaged over 10 such randomly picked pairs) on samples taken from different datasets. Fig 2 shows CKA results averaged over these different picks of pairs of subsets of the full set of neurons. We see that after a picking certain threshold, i.e. for large enough value of k, the similarity between any two randomly picked pairs of heads is fairly high. For ample. for the adversarially trained ResNet50 (Fig 2b), observe that any 10% of neurons picked from the penultimate layer are highly similar (CKA of about 0.8), with very low er- ror bars. A similar value of CKA is obtained with 20% of neurons for the standard ResNet50 model. These results indicate that, given suf- ficient size, picking any random subset of that size has very similar representations and this provides some intuition for why any random subset works equally well.
1.0
8
diica fowm
0.0
00
02
0.4
0.
(Total-2045)
(a) Standard Training
8
0.4
0.2
chi530
Fraction of Neurons (Total-20488)
(b) Adversarial Training
Figure 2: [Why Any Random Subset Works] Similar- ity between any two randomly picked sets of 4% neu- rons becomes fairly high (for a "critical mass" of &%), thus showing that any random pick beyond this thresh- old is likely to perform similarly.
3 FACTORS INFLUENCING THE DEGREE OF DIFFUSED REDUNDANCY
Tonnteer ALCurscy
(c) Flawers
(d) Oxford--IIT-Peal
vgpl6 on-nonres 20-mhhali2epp3
risnet18 nonroo
resine50 monToD
Wide resnet50 2norrob
Vt. small patchls 224 nonroo ve.small_ patch12 2222222eeee
mwwett0 2eobastDenc
Figure 3: (Comparisons Across Architectures For Downstream Task Accuracyl All models shown here pre-trained on ImageNetlk. We see that diffused redundancy exists across architectures, and the trend observed in Figure IC& regaarding adversarially trained models also holds here as models curves that are more "inside" are the ones trained with standard loss.
In order to better understand the phenomenon of diffused redundancy analyze 21 different pre- trained models, with different architectures, pre-training datasets and losses. We then evaluate each model for transfer accuracy
datasets mentioned in Section
Architectures We consider VGG16 (Simonyan & Zisserman 2014). ResNet18, ResNet50, WideResNNet5022 (He et al., 2016), VIT-S16 & VIT-S32 (Kolesnikov et al. 2021) Additionally we consider ResNet50 with varying widths of the final layer (denoted by ResNet ffx where x denotes the number of neurons in the final layer).
Upstream Datasets ImageNet- 1k & ImageNet-214 Russakovsky et al. 2015).
Upstream Losses Standard cross-entropy, adversarial training (&₂ threat model, 3) (Madry et al. 2019), MRL Loss (Kusupati et al. (022).
Downstream Datasets CIFAR 10/1000, Oxford-IIIT-Pets. and Flowers, as Section

---

Under review as a conference paper at ICLR 2023
Tunofy ACQUIIC
Fraction
Fraction
Fraction
94
Fraction
CIFARIO
CIFARIO0
(c) Flowers
Sxford-IITT-Pad Pets
o 226 imagenet
smmlll patchde, 224 224nngenet
vt_small_atth522 226- Imagenet
vTt matthho 224 Imagenet2lc
Figure 4: [Comparison Across Upstream Datasets] We see that degree of diffuusd redundancy depends a great deal on the upstream training dataset, in particular models trained on ImageNer2Ik exhibit higher degree of diffused redundanacy, although the differences in the degree of diffused redundanacy downstream task dependent
Pre-tree tre-trained weights for adversarially trained models were taken from (Salman et al. 2020). Weights for VITs (both ImageNer21 k and ImageNetIk) were taken from the code released by (Kolesnikov) and the ResNet50 trained with MRL loss jtt al. 2021, Steiner al., 2021). All ResNet 50 _0_
taken from code released by (Kusupati al. 2022). and all standard models were taken from timm and torchvision (Wightman/ 2019; Paszke et al. 2019).
3.1 EFFECTS OF ARCHITECTURE, UPSTREAM Loss, UPSTREAM DATASETS, AND DOWNSTREAM DATASETS
Extending the analysis in Section 2 we evaluate the diffused redundancy hypothesis on other archi- tectures. Fig 3 shows transfer performance for different architectures. All architectures shown in Fig 3 are trained on ImageNetlk. We find that our takeaways from Section P also extend to other architettrrs.
Fig compares two instances each of V and ViT-S32, one trained on a bigger upstream dataset (ImageNet21k) and another on smaller dataset (ImageNetlk)
Note that nature of all curves in both Figs 3 highly depends on downstream datasets. This is also consistent with the initial observation Section about diffused redundancy being downstream dataset dependent.
3.2 DIFFUSED REDUNDANCY A FUNCTION OF LAYER WIDTH
We take the usual ResNet50 with penultimate layer consisting of 2048 neurons and compare it with variants that pre-trained with much smaller penultimate layer, these are denoted by ResNet where x (< 2048) is the number neurons in the penultimate layer. Fig shows how diffused redundancy slows fades away ResNet 50_ff8, we see that across all datasets we need > 90% the full layer to achieve perfor- mance close the full layer. This shows that diffused redundancy only appears in DNNs when the layer is sufficiently wide to encode redundancy.
squeeze the layer be smaller. In fact, for
Sander Acurccy
30
Prectiie nearers
Fructise REGPOES
Preetien neurers
Praftion REGPORS
(a) CIFAR1O
(b) CIFAR100
(c) Flowers
(d) OxfordIIIT-Pets
resnet50
resnet50 0 f1256
resnet50_ f132
50 #11024
resnet50 f128
resnet50 ff16
resnet50 I1512
resnet50 f164
resnet50
Figure 5: [Diffused Redundancy as Function Layer Width] As we make length layer smaller, ResNet50 with only 8 neurons in final layer, degree of redundancy becomes lesser. For ResNet 50.EE8, we see that across all datascts we need almost 90% of neurons to achieve similar accuracy as the full layer.
8, ie

---

Under review as a conference paper at ICLR 2023
3.3 COMPARISON WITH METHODS THAT OPTIMIZE FOR LESSER NEURONS
Matryoshka Representation Learning (MRL) is recently proposed paradigm which learns nested representations such that first k, 2k, ...... .V (where N = size of the full layer) dimensions of the extracted feature vector all explicitly made be good minimizing upstream loss, with the intuition of learning coarse-to-fine representations. This ensures that one can flexibly use these smaller parts of the representation for downstream tasks. MRL, thus, ensures that redundancy shows up in learned representations in structured way, i.e., we know the first k, 2k, neurons can be picked and used for downstream tasks and should perform reasonably.
Here we investigate two questions regarding Matryoshka representations: 1) do these representations also exhibit the phenomenon diffused redundancy? i.e. we were to ignore the structure imposed by MRL-type training and instead just pick random neurons from all over the layer, do we still get reasonable performance?, and 2) how do they compare to representations learned by other kinds of losses?
Figure 6 investigates these questions by comparing ResNet50 representations learned using MRL loss to other losses. resnet50 mrl nonrob first (red line) denotes resnet50 learned using MRL. loss and evaluated on parts of representation that were optimized to have low upstream loss (i.e. first k N..N neurons, here k = 8 and N 2048) and resnet (green line) refers to the same model with same number of neurons chosen for evaulation, except they're chosen at random from the entire layer.
First, we interestingly see that even the ResNer50 trained with MRL loss exhibits diffused redun- dancy (denoted by green line spiking very quickly for most datasets in Fig 6). despite having been trained to only have structured redundancy. Based on this observation we conjecture that diffused redundancy is a natural consequence of having wide layer. Second, on MRL. indeed does better in the low neuron regime across datasets (red line on the extreme left part of the plots Fig 6). but other models quickly catch up as we pick more neurons, thus indi- cating that major efficiency benefits of MRL-type models are best realized when using extremely low number neurons, else one can obtain similar downstream performances by simply picking random samples from existing pre-trained models.
see that ResNet50 trained
"raction
Fraction neurona
Traction
Fraction
CIFARIO
CIFARI00
(c) Flowers
(d) Sxford-IITT-Pat
resnat50-norrch-ranne -andom-
resnet50, mod-nnnrob-raadoo
recnet50 mri-nanrob-frsst
Figure 6: [Comparison of Diffused Redundancy in MRL other losses] Here we compare ResNet50
trained using multiple losses including MRL (Kusupati el al. 2022). Red line shows results for part of representation explicitly optimized in MRL, whereas green line shows results for parts that are picked randomly from the same representation. Even the MRL model shows significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy.
4 POSSIBLE FAIRNESS- EFFICIECY TRADEOFFS EFFICIENT DOWNSTREAM TRANSFER
One natural use-case for diffused redundancy is efficient transfer downstream datasets, iee, use only a random subset instead of the entire feature vector. This would lead to faster training and lesser storage requirements since reduces number of additional parameters. As defined in Eq small
and also seen in prior works (e.g. Kusupati et al. (202))) dropping neurons comes cost in performance as compared to the full set of neurons. Here we take deeper look into this drop in overall performance and investigate how it is distributed across classes. If the drop affeets only a few classes, then dropping neurons although efficient for downstream tasks could have implications for fairness, which is not only of concern to MI. researchers and practitioners (Zafar
8

---

Under review as a conference paper at ICLR 2023
1.0
noret53 mereddt enbure2epe
0.3-
reavi53 minoavseftst
0.6
Ginl
6.2
0.2-
Accuracy
Accuracy
(a) CIFARIO
(b) CIFARIOD
0.8
0.8
Gin!
0.4-
6.4
0.0
0.0
0.1
0.2
0.3
0.4 Accuracy
0.5
8.6
0.7
0.0
0.0
0.2
0.4
0.6
08
Accuracy
(c) Flowers
(d) O1foOdd---TTTa
Figure 7: [Gini Coefficient of Class-Wiss Accuracies as we Drop Neurons] Higher value Gimi coefficient indicates higher incquality (Ginl 1921). We see that for all models gini coefficients become higher as the accuracy reduces (as a result of dropping neurons). Additionally in some regions (highlighted in the plots), the model explicitly optimized for efficient transfer (resmet mm can give rise to higher gini values, resulting unequal spread of accuracy over classes.
et al. 2017 Hardt et al. 2016 Holstein al. 2019). but also to lawyers Tolan al. (2119) and policymakers Veale et al. 2018).
We compare the spread of accuracies across classes using inequality indices, which are commonly used in economics to study income incquality (De Maio 2007| Schutz 195T) and have also recently been adopted in the fair ML literature (Speicher et al.) 2018). We use gini index (Gini 1921) and coefficient of variation (Lawrence) 1997) to quantify the spread of performance across classes. For perfect spread, both gini and coefficient of variation are 0, and higher values indicate higher inequality.
Figure 7 compares the gini index for various models varying levels of accuracy (note that accuracy monotonically increases with more neurons, hence right more point for a model represents the model with all neurons). We make two observations: across all datasets and all models we find that loss in accuracy (compared to the full layer) comes at the cost of few classes, as opposed being smeared throughout classes, as indicated by high gini values on the left of each plot. Additionally, we observe that the model trained using MRL loss tends to have slightly higher gini values in the regions where drop accuracy is slightly higher (highlighted on the plots). Similar trends are also observed with coeff. of variation shown in Appendix G These results draw caution to potential unintended side-effects exploiting diffused redundancy and suggest that there could be a possible fairess--efficien tradeoff involved.
5 CONCLUSION AND BROADER IMPACTS
We introduce the diffused redundancy hypothesis and analyze wide range of models with dif- ferent upstream training datasets & losses, architectures. We carefully analyze the causes of such redundancy and find that upstream training (both loss and datasets) plays crucial role and that this redundancy also depends on the downstream dataset. One direct practical consequence of our obser- vation is increased efficiency for downstream training times which have many positive impacts in terms of reduced energy costs (Strubell et al., (011)) which is crucial in moving towards "green" AI (Schwartz et al., 2020). We, however, also draw caution to potential pitfalls of such efficiency gains in Section 4 which might hurt the accuracy of certain classes more than others, thus having direct consequences for fairness. We see our work as contribution the scholarship on better understanding deep learning through an empirical lens, while also highlighting possible pitfalls.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We include all training and pre-pree proccsssssnng details in Appendix B used to generate the results in our paper. All our evaulation based on standard publicly available datasets which we have cited throughout the paper (and our code contains scripts to automatically download these datasets). Appendix B also includes links to pre-trained models (with proper citations) that exact numbers can be replicated.
We have also attached all the code
REFERENCES
Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arxiv: 125:126-127, 2017.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.
Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954.
Yoshua Bengio, Aaron Courville, and Paseal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798- 18288, 2013.
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint
2019.
Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in
pretrained transformer models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 4908 4926, Online, November 2020. Associ-
ation for Computational Linguistics. doi: 10.18653/v wttpp///222.c22 122000.emnlpmmai..... URL https:
Taallanthoooyy.oggg..
emplp main. 398
Fernando De Maio. Income inequality measures. Journal of Epidemiology & Community Health, 61(10):849-852, 2007.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodss, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, 2022. URL http:ptss//rrrasooooreereeieeeee
pub/2022/ 11022/toy_modde /etoodee//ndde..e html
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttttss///ppeeeiiiees
net/Eorum?id-ron b3RcF7
Corrado Gini. Measurement of inequality of incomes. The economic journal, 314121))124-126, 1921.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning, Advances in neural information processing systems, 29, 2016.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 1992.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference computer vision, pp. 1389-1397, 2017.
Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. Improving fairness machine learning systems: What do industry practitioners need? In Pro- ceedings of the 2019 CHI conference on human factors in computing systems, pp. 1-16, 2019.
Yoshio Izui and Alex Pentland. Analysis of neural networks with redundancy. Neural Computation, 2(2):226-238, 1990.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors PMLR, 2018.
(tcav). In International conference machine learning, pp.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference Machine Learning, pp- 5338-5348. PMLR, 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- ence on computer vision, pp. 491-507. Springer, 2020.
Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Un- terthiner, and Xiaohua Zhai. An image worth 16x16 words: Transformers for image recognition at scale. 2021.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR. 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of 2009.
features from tiny images.
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Noward-Snyded Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representations for adaptive deployment. arXiv preprint 20.:22255.1147.. 2022.
Barbara S Lawrence. Perspective- the black box of organizational demography. Organization science, 8(1):1-22, 1997.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):366-444 2015.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Tor. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint 1810.0230 2018.
Asriel Levin, Todd Leen, and John Moody. Fast pruning using principal components. Advances in neural information processing systems, 6, 1993.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Mcasuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 160vv1608.08777 2016.

---

Under review as a conference paper at ICLR 2023
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them, 2014.
Vedant Nanda, Till Speicher, Camilla Kolling, John P. Dickerson, Krishna P. Gummadi, and Adrian Weller. Measuring representational robustness of neural networks through shared invariances. In ICML, 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.239 00.23915ddisttll.00007. htpps:/diisilllpuu/////feeeeee httt://///aaauurisiuuuiiiaae
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi:
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-135) 2009.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, Н. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems 32, pp. http:/pppapprr....ippppppeeeeees ott.://wwooooaaaaaaaaaeeeeeaeaer
8024 8035. Curran Associates, Inc., 2019. URL URB/Lttee//ppeerr..eerr...p..psp
library.
pdf
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400.
PMLR, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1153))211-252, 2015.
Hadi Salman, Andrew Iyyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-3545, 2020.
Robert R Schutz. On the measurement of income inequality. The American Economic Review, (1):107-122, 1951.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54 63, 2020.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- tion. arXiv preprint 1rXiv:1703.008.01 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint 1rXiv:1409.1555 2014.
Till Speicher, Hoda Heidari, Nina Grgic-Hlae Hlaca, Krishna Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. A unified approach to quantifying algorithmic unfaimess: Measuring individual ggroup unfairness via inequality indices. In Proceedings of the 24th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2239-2248 2018
12

---

Under review as a conference paper at ICLR 2023
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2206.102277 2021.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David R. Traum, and Lluís Mårquez leds.), Proceedings of the 57th Conference of the Association for Computational Linguiitiess, ACL 2019. Florence, Italy, July 28- August 2, 2019. Volume 1: Long Papers, pp. 3645-36510 Association for Computational Linguisties, 2019. doi: 10. N0.18653/11/119-13555 URL, https: orgei.rrg//p ............../.11.... 18653/v1/ p19-1355
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018.
Songül Tolan, Marius Miron, Emilia Gómez, and Carlos Castillo. Why machine learning may lead to unfairness: Evidence from risk assessment for juvenile justice in catalonia. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, pp. 83-92. 2019.
Michael Veale, Max Van Kleek, and Reuben Binns. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making In Proceedings of the 2018 chi conference on human factors in computing systems, pp. 1-14, 2018.
Ross Wightman. Pytorch image models.
http::///etcuub..ommm.....
cetorrc-iiaqge models 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. abs/21 10.00476, 2021.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for de- buggable deep networks. In International Conference on Machine Learning, pp. 11205-11216. PMLR, 2021.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization, 2015.
Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEÉ/CVF international conference on computer vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang. Slimmable neural networks. abs/1812.00 0bs//811.00892,, 2018 URL httt:://axxiv http:/xxww..rrr/////12....0 abs//aa2.1111551/ 08928
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fair- ness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pp. 962-970. PMLR, 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, 2013.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint 19 2019.
13

---

Under review as a conference paper at ICLR 2023
A MEASURING DIFFUSED REDUNDANCY
A.1 CKA DEFINITION
In all our evaluations use CKA with a linear kernel (Korbblih et al. 2019) which essentially amounts to the following steps:
1. Take two representations Y E Raxdl and E Raxd2
2. Compute dot product similarity within these representation, ie Le.compute YYT, Zz"
3. Normalize K and L to get K' -HKH, L' HLH where H = In 111
H.IC(K,L) 4. Return CKA(Y, Z) K where HSIC = (n- (flatten(K) K*)
flatten(L'))
We use the publicly available implementation of Nanda et al. (2022), which provides an implemen- tation that can be calcuated over multiple mini-batches: ht htt /ttpt//gttttbbbbt cot/nvedantt//n7
STIR
A.2
ADDITIONAL CKA RESULTS
Fig g shows CKA comparison between randomly chosen parts of the layer and the full layer for dif- ferent kinds of ResNet50. We observe that even ResNet50 trained with MRL loss shows significant amount of diffused redundancy.
0.4
0.4
robupt2:0033
rebumt2:003
0.2
rusnet5o meinnerreb-frss
0.2
reshetso meieenreb-frat
resnot5o merinonroo random
resnet5o metinonrop random
0.0
0.0
Fraction Neurons (Total=2048)
Fraction Neurons (Total=2048)
(a) CIFARIO
(b) CIFAR100
1.0
1.0
0.8
0.8
resnet50 nahroa randem
resnet50 nahrob random
Sonnttti-nbsttieeess Srendom
www.tSt-nbbusiieeee Srandom
resnet5o, merinorroo frrst
resnet5o, nrinorroo
0.0
0.1
0.2
0.3
0.4
0.5
0.0
0.1
0.2 0.3
0.4
0.5
Fraction of Neurons (Total=2048)
Fraction of Neurons (Total=2048)
(c) Flowers
Oxfor-IIIT-Pus
Figure 8: [Compparion of Diffused Redundancy in MRL ys other losses, through the lens of CKAJ We see a similar trend as reported in Fig in the main paper, where even the MRL model shows significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. The amount of diffused redundancy however much lesser than the resnets trained using the standard loss and adv. training as denoted by a much lower line across datasets.
B TRAINING AND PROCESSSNSIN DETAILS FOR REPRODUCIBILITY
Here we list the sources weights for the various pre-trained models used in our experiments:
ResNet18 trained on ImageNet1k using standard loss: taken from timm v0.6.1.
14

---

Under review as a conference paper at ICLR 2023
ResNet18 trained on ImageNetlk with adv training: taken from Salman et al. (2020):
ResNet50 trained on ImageNetlk using standard loss: taken from t iimm v0.6.1.
ResNer50 trained on ImageNetIk with adv training: taken from Salman et al. 2020
bttps://qithub.. wwt.:/ocmmmccooooocco htt///rouustttttteooe www.srooooooods--tteeoro
ResNec50 trained on ImageNetlk using MRL and with different final layer widths (resnet:50.ffx): taken from released weights of by |Kusupati et al. (2022) https:
//githuubccom//Aa www.:/oo..AAAAAaA..a..
WideResNet50-22nn on ImageNetlk both standard and avd. training: taken from Salman al.
(2020): httpp://tttgtt http://gtttttttt/ com /mi crosoft wwt.tt/toooooooooooeoooooo models - transfer
VGG16 trained on ImageNetlk with standard loss: taken from timm v0.6.1.
VGG16 trained on ImageNetIk with adv training: taken from Salman et al. (0020)
https://githin
transferr
VITS32 & ViTS16 trained on ImageNet21k & ImageNerlk: taken from weights released by Steiner et 202T: https: com http://ppoggleeeeeeerrr/eeeeaeaa transforme
All linear probes trained on the representations of these models are trained using SGD with learn- ing rate 0.1, momentum 0.9, batch size of 256, weight decay of le - 4. The probe is trained for 50 epochs with learning rate scheduler that decays the learning rate by 0.1 every 10 epochs. Scripts for training can also be found in the attached code.
For pre-processing, re-size all inputs 224x224 (size used for pre-training) and apply the usual composition of BandoonHorioontlFFiip. Colorritterrrightne==033333 contrast=0.25,, satura- All inputs mean normalized. For ima-
tion=0.25, hue=0.25), LandoomRotationndegreeee genetlk pre-trained models: mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. For imagenet21k pre-trained models: mean [0.5,0.5,0.5], std [0.5,0.5.0.5].
C COFFF OF VARIATION FOR MEASURING INEQUALITY IN INTSR-CLASS ACCURACY
Fig shows results for the same analysis shown in Fig 7 of the main paper and we find similar takeaways even when using coefficient of variation as a measure inequality.
D ADDITIONAL REBUTTAL RESULTS
We ran the following additional experiments during the rebuttal phase:
Numbers on x axis for Figure 3 are shown in Figure 13 Figures and compare models with same number of neurons in the final layer and hence trends shown with fraction on the X-axis will be exactly the same with absolute numbers on the x-axis.
Corresponding diffused redundancy (DR) ablations for Figures 3 These are shown in Figures 10 11 respectively. This should allow easy interpretation of which mod- els/configurations have higher diffused redundancy (lines that are more outside have higher DR). For example Figure 11 clearly shows higher diffused redundancy in models trained on larger upstream datasets (here ImageNet21k) since these curves more on the outside of same model's curves for ImageNetlk.
Performance comparison between randomly chosen neurons and PCA based projection. This is shown Figure 14 We find that PCA closely follows performance of randomly chosen neurons for CIFARIO and OIf-rr-IIIT-Pets. However PCA outperforms randomly chosen neurons on CIFAR100 and Flowers, both of which have more number of classes.
Results on ImageNetlk and ImageNetV2. We report results on the harder task of Ima- geNetlk and an even harder task of generalizing to distribution shifts like ImageNetV2 in Figure [5] We find that when randomly dropping neurons, the model is still able to gen- eralize to ImageNet1k k with very few neurons, i.e., the phenomena of diffused redundancy
15

---

Under review as a conference paper at ICLR 2023
1.0
resnet50 pewettbehhualion
Coeff. of Varlation
Coaff, of Variation
0.8
resnotio, mrl nonnon
resnet50 pr nonroo frst
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Accuracy
Accuracy
(a) CIFRRI
(b) CIFARIO0
1.0
1.0
reshet5o nohreb
Coeff. of Varlation
Coeff: of Variation
0.8
0.8
resnett0 mi-nomma--rr
0.8
0.6
0.4
2.4
resret50 nonrab
0.2
mecnettierrbuullee Dapa)
0.2
resnet50, mn nonnot mow.:ommmmioonniiff. Airt
0.0
0.0
Accuracy
Accuracy
Flowers
Oxfor-IIIT-Pets
Figure 9: [Coefficient of Variation As We Drop Neurons] We similar trend reported in Fig of the main paper where inequality increases as we drop neurons models on datasets.
observed for smaller datasets, also holds for harder datasets. Interestingly we also observe that the accuracy gap between ImageNet Ik and ImageNet is maintained even as we drop neurons.
(DR)
1.0
Diffuced ed Redundancy (DR)
0.6
0.8
0.6
0.4
Difuueed
0.2
0.0
0.75
0.80
0.85
0.30
3.95
1.50
0.0
0.75
0.80
0.85
0.90
0.95
1.00
(a) CIFARIO
(b) CIFARIOD
Diffused Redundang 00000
Olffused Redundancy (DR)
0.8
10.6
0.2
022
0.80
0.55
(c) Flowers
(d) Oxford-IIT-Pets
viglt ereeora mindem *22l- www.stttieppiaaooooom madda
Sereell rcerio mindan vill rosuttoepc randam
vide poeeatt, ronesb-nndon
Aumma Lawcht 224. www-b rindan maida
Figure 10: (Comparisons Across Architectures For Downstream Task Accuracy] All models shown here pre-trained ImageNetlk. This Figure shows corresponding diffused redundancy values for Figure different 8 values. We see that diffused redundancy exists across architectures, and the trend observed Figure Tc regarding adversarially trained models also holds here as models curves that are more "inside" are the ones trained with standard loss.
16

---

Under review as a conference paper at ICLR 2023
Offfused Red Redundancy (DR)
Diffused Redundanc
0.8
0.
644
0.2
(a) CIFARIO
(b) CIFARIOD
1.0
Diffused Rodundancy (DR)
1.0
0.8
0.6
0.6
0.4
0.4
Diffused
0.0
0.75
0.80
0:85
0.50
0.55
1.00
0.0
0.75
0:80
0.85
0.50
0.35
1.00
Flowers
Oxford-IIT-Pets
vit mt_ssmallppacch1e 224-nonrob-random
vit vit_small patch32 224 non-nobobbbbadddm
vit_smail_patch16 http:lapatah12222nanooeea 224-nonrob-random
vit_small_patch33 22--22222222 nonnoobbr-baom
(Comparison Across Upstream Datascts] see that degree of diffused redundancy depends great deal on the upstream training dataset, in particular models trained on ImageNet2 k exhibit higher degree of diffused redundanacy, although the differences in the degree of diffused redundanacy are downstream task dependent
Figure 11:
Diftused Redundancy (DR)
1.0
offfused Redundancy (DR)
0.6
0.6
0.2
0.0
0.75
0.00
0.85
0.50
0.95
1.50
0.0
0.75
0.00
0.85
0.00
D.95
1.00
(a) CIFARIO
(b) CIFARIO0
olffused Redundancy (DR)
Diffused Redundanc
0.8
0.6
0.6
0.4
0.2
0.2
(c) Flowers
(d) Oxford Pets
resnet5O-nonro--raddom resnettOrrbbusiiegss-rndd
Soonett0.mm-nonnoo-rrrddo Besnetto_mri-nonro--ffeo
Figure 12: [Comparison of Diffused Redundancy in MRL vs other losses] Here compare ResNet50 trained using multiple losses including MRL (Kusupati representation explicitly optimized MRL, whereas green line shows results parts that are picked randomly from the same representation. Even the MRL model shows significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. This figure shows diffused redundancy (DR) for all plots Figure 6
2022). Red line shows results for part of the
17

---

Under review as a conference paper at ICLR 2023
1.0
1.0
Transtor Accuracy
0.6
0.8
Transter
0.4
1000
2000
3000
4030
1200
2000
2020
4000
Number neurons
Number
(a) CIFARIO
(b) CIFARIOD
1.0
1.0
Transter Accuracy
0.6
1000
2000
3000
4030
1000
2000
3020
4000
Number neurons
Number neurons
Flowers
(d) Oxford III Pets
Figure 13: [Comparisons Across Architectures For Downstream Task Accuracy] This shows the same plots
Figure except showing absolute number of neurons on the x-axis
Transfer Acuurayy
0.8
0.8
0.44
Fraction of Newrons
Fraction Neurons
(a) CIFARIO
(b) CIFARIOD
0.8
Transee Accuracy
0.8
Transter 0.4
0.4
0.2
0.5
Praction Neurons
Fraction Neurons
(c) Flowers
(d) Ixforrd-IIT-Petss
resnet50 nonrob pca
resnet50 nanzob random
resnet50 robusti2eps3 pca
resmet50-robustieenn random
Figure 14: [PCA Diffused Redundancy Comparison] We compare PCA-based dimensionality reduction randomly choosing subsets neurons on ImageNet pretrained ResNet50 (both nonrob and one trained with adversarial training). We find that for some downstream tasks PCA performance closely follows performance obtained via random ncurons. For tasks with large number of classes (CIFAR 100 and Dxford-IIIT-Petts PCA outperforms randomly chosen neurons. We also interestingly see that the gap between PCA and random significantly lesser for the adversarially trained model.
18

---

Under review as a conference paper at ICLR 2023
1.0
0.8
Accuracy
0.6
Accuraci
0.8-
0.2
0.2
0.0
1.
imagenet imagenetv2
Fraction Naurons
Fraction
(n) ResNer50 nonrohust
(b) ResNer50 mbust Fge
Figure 15: [Performance on ImageNetlk and ImageNetV2] We check for performance of randomly chosen subsets of neurons on harder tasks like ImageNetIk and also check for performance differences distribu- tion shift, iee. ImageNetV2 Recht al. (2019). We find that diffused redundancy holds for both these tasks. Additionally we see that randomly dropping neurons still preserves the accuracy gap between ImageNe Ik and ImageNetV2.
19


Reviews:
[{"review_id": "tBbOCKV8_5", "reviewer": "Reviewer_JaiD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "# Intro / Section 2\n- This paper introduces the property of diffused redundancy: any randomly chosen subset of a layer can perform similarly to the full layer on downstream tasks.\n- The degree of diffuse redundancy is investigated for the final layers of image classification models trained on imagenet.\n- Two downstream tasks are used to measure diffused redundancy, 1) representational similarity (CKA) between the whole layer and random subsets of the layer, and 2) classification accuracy when using the whole or a random subset of the layer to train a linear classifier for a downstream dataset.\n- Redundancy is when a high performance can be accomplished with a small fraction of neurons.\n- Diffusivity is when the variability of performance is low for different random subsets of neurons.\n\n# Section 2.1 / Figure 1 \n- Representational similarity and downstream accuracy are plotted for different sizes and random subsets of the full layer. A comparison is done between four downstream datasets (cifar10/100, flowers, oxford-iit-pets), and two models (ResNet50 with and without adversarial training).\n- Degree of diffusion and redundancy depends on the downstream dataset.\n- The adversarially trained model demonstrates greater diffused redundancy.\n\n# Section 2.2 / Figure 2\n- Plots of representation similarity between randomly chosen subsets.\n\n# Section 3 / Figure 3\n- Downstream accuracy vs subset sizes is compared for a larger collection imagenet classification models.\n\n# Section 3.1 / Figure 4\n- Downstream accuracy vs subset size is compared for models trained on imagenet1k and imagenet21k.\n- The imagenet21k models demonstrate greater downstream accuracy.\n\n# Section 3.2 / Figure 5\n- Downstream accuracy vs subset size is compared for models with varying final layer sizes.\n- The size of the final layer is proportional to downstream accuracy.\n\n# Section 3.3 / Figure 6\n- Downstream accuracy vs subset size is compared for models that are trained with MRL (smaller parts of the representation have structure).\n- The MRL-optimized representation performs well for very low numbers of neurons, but is outperformed for larger subsets.\n\n# Section 4 / Figure 7\n- The loss in downstream accuracy when reducing the size of the random subset is primarily in a few classes instead of being uniformly distributed.\n-This suggests that exploiting diffused redundancy has implications for fairness.", "summary_of_the_review": "Diffused redundancy is an interesting property of the final layer of image classifiers, but its application is not clear.", "strength_and_weaknesses": "# Strengths\n- Demonstrates an interesting property of image classifiers.\n- Comparisons are done between many different models.\n- Potential drawback of creating a class imbalance is highlighted, although this somewhat contradicts the diffused redundancy hypothesis.\n\n# Weaknesses\n- The application to model optimization is limited since this technique only reduces the size of the final layer. There could be some comparison in memory usage, floating point operations, or model size to motivate DR as an optimization technique.\n- It is not clear if there is any use for the definition of diffused redundancy (DR) (equation 1), while the definition states that it will be used to rigorously test the DR hypothesis. Figure 1a (downstream accuracy) and 1b (diffused redundancy) appear to convey the same information. The rest of the paper does not use this definition.\n- Section 2.2 claims to explain why any random subset will work for downstream tasks. Figure 2 shows that two randomly chosen subsets have high representational similarity to each other. This further confirms that any random subset works, but does it explain why? After seeing that a random subset has a similarity to the full layer in figure 1, it is not surprising that random subsets are also similar to each other.\n- In figures 3, 4, and 6 I believe the full representations all have different sizes, but they are not displayed. I am wondering if it is more important to show the absolute number of neurons on the x-axis instead of the fraction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oGCIt0916r", "reviewer": "Reviewer_9A8r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Pretrained deep representations from large datasets have a plethora of applications across vision and natural language. Understanding how information is encoded across the dimensions in the representation is important to design better algorithms for the respective downstream tasks. \n\nThis paper discusses that the learned representations exhibit significant redundancy. That is, not all the dimensions are needed for getting a decent performance on the downstream tasks, and propose a notion of diffused redundancy. The paper explores the existence of diffused redundancy across the architectures, upstream and downstream tasks, and learning algorithms. Based on the experiments, the paper concludes that the entire layer is not necessary for the downstream task, however, providing caution on how smaller representations sizes may cause unfair decisions", "summary_of_the_review": "Good step towards quantifying the redundancy in the representations. Experiments on small scale datasets and some missing baselines. However, a strong effort towards something useful at scale. ", "strength_and_weaknesses": "I will go sequentially with the strengths and weaknesses\n\nStrengths:\n1) The Paper is well-motivated, decently written, and easy to understand. \n2) Systematic experimentation is done to understand redundancy in the representation of 4 smaller-scale datasets using ImageNet pre-trained models\n3) Experimental findings are consistent across the downstream datasets, pre-training data, architecture, and learning algorithms (normal training, MRL, adversarial). \n4) The trade-off between representation size and fairness is very important (and also a good extension to MRL -- Kusupati et al., NeurIPS 2022)\n5) The choice of definition of diffused redundancy with the assumption of $\\delta=0.9$ is a good starting point.\n\n\nWeakness:\n1) The existence of redundancy is not novel, in fact, MRL has also discussed it in their appendix implicitly (diffused redundancy experiments with particular feature size) - however, concretizing it is a good first step taken by this paper\n2) Baseline with correlation regularizer should’ve been discussed - “Reducing Overfitting in Deep Networks by Decorrelating Representations”\n3) The paper should cite and discuss papers with structured pruning which are very relevant to the potential redundancy in the hidden layer.\n4) Figure 1 should also compare algorithms, that is CKA v/s fraction to better understand if some algorithms have less redundancy than others. \n5) Plain CKA numbers are a bit hard to understand. That is, for certain tasks, a CKA of 0.7 may be good, but for other, it may not be. Therefore, a scatter plot downstream task performance v/s CKA at different neuron fractions perhaps could be more informative. \n6) On Figure 4: When we are comparing the pre-training datasets, it is difficult to draw conclusions if the accuracy difference for full set of neurons differs quite a bit (which is the case in this figure)\n7) The figures could be made more easily readable, it is hard to go through the densely packed plots. \n8) Lastly, while the experiments are a good first step, the datasets often are easy -- I would like to see two sets of ablations a) with the variation of $\\delta$ from 0.8 to 0.99 and b) experiments on harder datasets like ImageNetV2 and Places365. This would make the paper extremely strong and would help any future papers built on top of it. \n\nI am very excited about this paper and want to have a conversation with the authors through rebuttal and revision. I want the authors to take the weakness in good spirit to help improve the paper to make it worth a strong publication\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ROritB7N89N", "reviewer": "Reviewer_tTBB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper examine the notion of representation redundancy in large networks, and more specifically how it is diffused across the neurons of the representation. The authors quantify the measure of representation redundancy they use and demonstrate that dropping a large amount of neurons from the last layer randomly only affects the downstream accuracy by a small amount. The authors evaluate their observation on pretrained models on ImageNet, and demonstrate the level of redundancy present in these pretrained representations.", "summary_of_the_review": "Overall, I think this paper makes an interesting observation, which however requires further analysis on its implications. Currently, I think that there needs to be a better understanding on how the observation made by the authors can be used. Elaborating on the points I mentioned above would help improve this paper.\n\nUpdate post rebuttal: I have raised my score after the authors' response.", "strength_and_weaknesses": "Strengths:\n\n- The paper poses an interesting question overall, in that the authors consider whether the learned representations have any particular structure in how information is redundant in them, and demonstrate that the representations produced by the network are significantly redundant.\n\n- The experimental portion of the paper demonstrates the claim that the authors make, regarding redundancy in the representations of the network. More specifically, it can be seen that in most cases, the downstream accuracy in a given task decays slowly, as the number of neurons in the representation is decreased. The authors also perform a large set of experiments to show what precisely influences diffused redundancy (varying upstream/downstream tasks, layer width and existence of adversarial training).\n\nWeaknesses:\n\n- While I agree with the core observation of the paper, I also don’t find it particularly surprising. It is expected that a large part of the representation produced by the network is redundant, and can be removed without significantly harming the performance of the network. The main point the authors make is that this can be done by removing random neurons of the representation, which shows that this redundancy is spread across the entire representation. However, one can reasonably expect that, since the representation is still in high dimensions, the samples will still be well separated, with only a few neurons kept. To improve this, I believe that the authors should further elaborate on the implications of their observation in the introductory parts of the paper. \n\n- In the experimental section, a simple baseline one can compare against is keeping only the principal directions of the data, via PCA. While this is more complex than simply dropping neurons (since it requires a linear combination of the weights) I believe it would be useful to include this comparison as well.\n\n- When comparing downstream tasks, I think it would also be useful to include the evaluation that arises when the downstream task is the same as the upstream (for example, ImageNet-1K). I think this would be useful to demonstrate whether the observation still holds in a more complicated downstream task.\n\n- The authors include a section in their paper that touches upon issues of fairness that may arise by minimizing the size of the representations of the model. However, I think this part of the paper should be further extended. As it is, it seems to be somewhat disjoint from the rest of the paper. This can easily be an important topic for future work, so I think it should be highlighted further.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject