SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: [Random Projection (dotted) vs Randomly Choosing Neurons (solid)] We find that diffused redundancy (randomly choosing neurons) performs significantly better than random projections. This indicates that the "constrained" projection offered by diffused redundancy is crucial in achieving good downstream performance.
Downstream performance on  $k$  randomly chosen neurons closely follows performance on top  $k$  principal components. While high representation similarity is a necessary condition for two representations to have similar downstream performance, it's not a sufficient one. As seen earlier in Fig 1, similar values of CKA can still show different downstream accuracy. Thus, to further understand why any random pick of neurons achieves similar accuracy, we compare a randomly chosen subset of neurons with a projection on the same number of top  $k$  principal components. Fig 2c& 2d shows that the performance of a linear probe trained on a random subset of neurons initially lags behind the top PCA dimensions for small values of  $k(< 20\%)$ , but for higher values ( $>20\%$ ) closely follows the performance on a probe trained on the projection on the same number of top principal components. This indicates that a sufficiently sized random subset of size  $k$  (roughly) captures the maximum variance in data that can be captured in  $k$  dimensions (upper bound by top  $k$  principal components since they are designed to capture maximum variance). As a sanity check, we also show the results for the bottom  $k$  principal components (directions of least variance) and see that randomly chosen neurons perform significantly better.
Random projections into lower dimensions do not perform well on downstream tasks. The presence of diffused redundancy in a particular layer indicates that a dimension lower than the layer's size suffices to perform many downstream tasks. However, randomly sampling neurons (as we do in our experiments on diffused redundancy) can be seen as one very particular way of reducing dimension that is restricted by what the network has learned. This raises a natural question, would our observation extend to less restricted ways of reducing dimensions? To test this, we compare diffused redundancy i.e., a linear probe trained on a random sample of  $k$  neurons to a linear probe trained on a random projection of the layer's activations. To project a  $d$  dimensional layer into a lower dimension  $(k)$  we multiply it by a (normalized) randomly sampled matrix from a normal distribution  $\mathbb{R}^{d \times k}$ . We report our results over 5 seeds in Fig 3 and find that diffused redundancy significantly outperforms random projection.
3 Factors Influencing The Degree of Diffused Redundancy
(a) CIFAR10
(b) CIFAR100
(c) Flowers
Figure 4: [Comparisons Across Architectures For Downstream Task Accuracy] All models shown here are pre-trained on ImageNet1k. We see that diffused redundancy exists across architectures, and the trend observed in Figure 1c&1a regarding adversarially trained models also holds here as models' curves that are more "inside" are the ones trained with standard loss.
(d) Oxford-IIIT-Pets
In order to better understand the phenomenon of diffused redundancy we analyze 30 different pretrained models, with different architectures, pre-training datasets and losses. We then evaluate each model for transfer accuracy on 4 datasets mentioned in Section 2. While all results in this section are on the penultimate layer, we also report results on intermediate layers in Appendix E and find similar trends of diffused redundancy. All details for reproducibility can be found in Appendix B.
Architectures We consider VGG16 [50], ResNet18, ResNet50, WideResNet50-2 [16], ViT-S16 & ViT-S32 [23]. Additionally, we consider ResNet50 with varying widths of the final layer (denoted by ResNet50_ffx where x denotes the number of neurons in the final layer).
Upstream Datasets ImageNet-1k & ImageNet-21k [45].
Upstream Losses Standard cross-entropy, adversarial training ( $\ell_2$  threat model,  $\epsilon = 3$  and  $\ell_{\infty}$  threat model,  $\epsilon = 4 / 255^4$ ) [34], MRL Loss [26], DeCov [6], and varying strengths of dropout [52].
Downstream Datasets CIFAR10/1000, Oxford-IIIT-Pets, and Flowers, same as Section 2. Additionally, we also report performance on harder datasets such as ImageNetv2 [44] and Places365 [69]
Results for  $\ell_{\infty}$  robust models can be found in Appendix D
(b) CIFAR100
Figure 5: [Comparison Across Upstream Datasets] We see that degree of diffused redundancy depends a great deal on the upstream training dataset, in particular, models trained on ImageNet21k exhibit a higher degree of diffused redundancy, although the differences in the degree of diffused redundancy are downstream task dependent
(c) Flowers
in Appendix F. For all analyses in this section, we also report corresponding approximations for  $DR$  (Eq 1) in Appendix D.
3.1 Effects of Architecture, Upstream Loss, Upstream Datasets, and Downstream Datasets
Extending the analysis in Section 2, we evaluate the diffused redundancy hypothesis on other architectures. Fig 4 shows transfer performance for different architectures. All architectures shown in Fig 4 are trained on ImageNet1k. We find that our takeaways from Section 2 also extend to other architectures.
Fig 5 compares two instances each of ViT-S16 and ViT-S32, one trained on a bigger upstream dataset (ImageNet21k) and another on a smaller dataset (ImageNet1k)
Note that the nature of all curves in both Figs 4&5 highly depends on downstream datasets. This is also consistent with the initial observation of Section 2 about diffused redundancy being downstream dataset dependent. Additionally, we also report results on ImageNetV2 and Places365 in Appendix F showing that diffused redundancy also holds on "harder" datasets.
3.2 Diffused Redundancy as a Function of Layer Width
We take the usual ResNet50 with a penultimate layer consisting of 2048 neurons and compare it with variants that are pre-trained with a much smaller penultimate layer, these are denoted by ResNet50_ffx where x (< 2048) is the number of neurons in the penultimate layer. Fig 6 shows how diffused redundancy slowly fades away as we squeeze the layer to be smaller. In fact, for ResNet50_ff8, we see that across all datasets we need  $>90\%$  of the full layer to achieve performance close to the full layer. This shows that diffused redundancy only appears in DNNs when the layer is sufficiently wide to encode redundancy.
(b) CIFAR100
(c) Flowers
Figure 6: [Diffused Redundancy as Function of Layer Width] As we make the length of the layer smaller, the degree of redundancy becomes lesser. For ResNet50_ff8, i.e.ResNet50 with only 8 neurons in the final layer, we see that we need almost  $90\%$  of neurons to achieve similar accuracy as the full layer.
3.3 Comparison With Methods That Optimize For Lesser Neurons
Matryoshka Representation Learning (MRL) is a recently proposed paradigm that learns nested representations such that the first  $k, 2k, 4k, \ldots, N$  (where  $N =$  size of the full layer) dimensions of the extracted feature vector are all explicitly made to be good at minimizing upstream loss, with
the intuition of learning coarse-to-fine representations. This ensures that one can flexibly use these smaller parts of the representation for downstream tasks. MRL, thus, ensures that redundancy shows up in learned representations in a structured way, i.e., we know the first  $k$ ,  $2k$ , ... neurons can be picked and used for downstream tasks and should perform reasonably.
Here we investigate two questions regarding Matryoshka representations: 1) do these representations also exhibit the phenomenon of diffused redundancy? i.e. if we were to ignore the structure imposed by MRL-type training and instead just pick random neurons from all over the layer, do we still get reasonable performance?, and 2) how do they compare to representations learned by other kinds of losses?
Figure 7 investigates these questions by comparing ResNet50 representations learned using MRL loss to other losses. resnet50_mrl_nonrob_first (red line) denotes a ResNet50 trained using MRL loss and evaluated on parts of the representation that were optimized to have low upstream loss (i.e. first  $k, 2k, \ldots N$  neurons, here  $k = 8$  and  $N = 2048$ ) and resnet50_mrl_nonrob_random (green line) refers to the same model with the same number of neurons chosen for evaluation, except they're chosen at random from the entire layer.
First, we interestingly see that even the ResNet50 trained with MRL loss exhibits diffused redundancy (denoted by green line spiking very quickly for most datasets in Fig 7), despite having been trained to only have structured redundancy. Based on this observation, we conjecture that diffused redundancy is a natural consequence of having a wide layer. Second, we see that ResNet50 trained on MRL indeed does better in the low neuron regime across datasets (red line on the extreme left part of the plots in Fig 7), but other models quickly catch up as we pick more neurons, thus indicating that major efficiency benefits of MRL-type models are best realized when using an extremely low number of neurons, else one can obtain similar downstream performances by simply picking random samples from existing pre-trained models.
(b) CIFAR100
Figure 7: [Comparison of Diffused Redundancy in MRL vs other losses] Here we compare ResNet50 trained using multiple losses including MRL [26]. nonrob indicates that the model was trained with the standard crossentropy loss while robust12eps3 indicates adversarial training with  $\ell_2$  threat model and  $\epsilon = 3$ . Red line shows results for part of the representation explicitly optimized in MRL, whereas the green line shows results for parts that are picked randomly from the same representation. Even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy.
3.4 Methods That Prevent Co-adaptation of Neurons Also Exhibit Diffused Redundancy
We consider two additional modifications to upstream pre-training: we add regularization to the usual cross-entropy loss that decorrelates neurons in the feature representation using DeCov [6] and Dropout [52]. Dropout does this implicitly by randomly dropping neurons during training and DeCov does this explicitly by putting a loss on the activations of a given layer. We pre-train different ResNet50 on ImageNet1k by varying strengths of dropout ranging from 0.1 all the way up to 0.8 and also by separately adding the DeCov regularizer (regularization strength  $= 1e - 4$ ) to give 9 additional pre-trained models. Intuitively these methods should lead to increased diffused redundancy since by design these methods force the model to disperse similar information in different parts of a layer to perform the same task downstream task. Interestingly, we see that diffused redundancy in these models is almost completely independent of the regularization strength. This suggests that diffused redundancy might be an inevitable property of DNNs when the layer has sufficient width. Due to space constraints, we include these results in Appendix G.
4 Possible Fairness-Efficiency Tradeoffs in Efficient Downstream Transfer
One natural use case for diffused redundancy is efficient transfer to downstream datasets. As defined in Eq 1 and as also seen in Sections 2& 3, dropping neurons comes at a small cost ( $\delta$  in Eq 1) in
(b) CIFAR100
Figure 8: [Gini Coefficient of Class-Wise Accuracies as we Drop Neurons] Higher value of Gini coefficient indicates higher inequality [12]. We see that for all models gini coefficients become higher as the accuracy reduces (as a result of dropping neurons). Additionally, in some regions (highlighted in the plots), the model explicitly optimized for efficient transfer (resnet50_mrl) can give rise to higher gini values, resulting in a more unequal spread of accuracy over classes.
performance as compared to the full set of neurons. Here we take a deeper look into this drop in overall performance and investigate how it is distributed across classes. If the drop affects only a few classes, then dropping neurons – although efficient for downstream tasks – could have implications for fairness, which is not only of concern to ML researchers and practitioners [66, 14, 18], but also to lawyers [56] and policymakers [57].
We compare the spread of accuracies across classes using inequality indices, which are commonly used in economics to study income inequality [8, 47] and have also recently been adopted in the fair ML literature [51]. We use gini index [12] and coefficient of variation [27] to quantify the spread of performance across classes. For a perfect spread, both gini and coefficient of variation are 0, and higher values indicate higher inequality.
Figure 8 compares the gini index for various models at varying levels of accuracy (note that accuracy monotonically increases with more neurons, hence the right most point for a model represents the model with all neurons). We make two observations: across all datasets and all models we find that a loss in accuracy (compared to the full layer) comes at the cost of a few classes, as opposed to being smeared throughout classes, as indicated by high gini values on the left of each plot. Additionally, we observe that the model trained using MRL loss tends to have slightly higher gini values in the regions where the drop in accuracy is slightly higher (highlighted on the plots). To ensure that this trend is not simply due to lower accuracy, we investigate the error distributions across classes (Appendix C) and find that predictions become more homogeneous as we drop more neurons. Similar trends are also observed with coeff. of variation as shown in Appendix C. These results draw caution to the potential unintended side-effects of exploiting diffused redundancy and suggest that there could be a possible fairness-efficiency tradeoff involved.
Connections to Robustness-Fairness Tradeoffs There are well-established tradeoffs between robustness and fairness in both standard [36] and adversarial training [62, 58]. One major difference between these works and our setup is that the task they train on is also the task they test on. We instead operate in the transfer learning setup where we train a linear probe on the representation of a pre-trained model. While it's not immediately clear whether the discrepancy between class accuracies observed on the pre-training task (e.g. as in [62]) also leads to an inter-class accuracy discrepancy on downstream tasks when dropping neurons, it would be very interesting future work to establish more formal connections between the robustness-fairness tradeoff and its effects on the fairness-efficiency tradeoff presented in this paper.
5 Conclusion and Broader Impacts
We introduce the diffused redundancy hypothesis and analyze a wide range of models with different upstream training datasets, losses, and architectures. We carefully analyze the causes of such redundancy and find that upstream training (both loss and datasets) plays a crucial role and that this redundancy also depends on the downstream dataset. One direct practical consequence of our observation is increased efficiency for downstream training times which can have many positive impacts in terms of reduced energy costs [54] which is crucial in moving towards "green" AI [48]. We, however, also draw caution to potential pitfalls of such efficiency gains, which might hurt the accuracy of certain classes more than others, thus having direct consequences for fairness.
VN and JPD were supported in part by NSF CAREER Award IIS-1846237, NSF D-ISN Award #2039862, NSF Award CCF-1852352, NIH R01 Award NLM013039-01, NIST MSE Award #20126334, DARPA GARD #HR00112020007, DoD WHS Award #HQ003420F0035, ARPA-E Award #4334192 and a Google Faculty Research Award. VN, TS, and KPG were supported in part by an ERC Advanced Grant "Foundations for Fair Social Computing" (no. 789373). AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1 and the Leverhulme Trust via CFI. SF was supported in part by a grant from an NSF CAREER AWARD 1942230, ONR YIP award N00014-22-1-2271, ARO's Early Career Program Award 310902-00001, Meta grant 23010098, HR00112090132 (DARPA/RED), HR001119S0026 (DARPA/GARD), Army Grant No. W911NF2120076, the NSF award CCF2212458, an Amazon Research Award, and an award from Capital One.
[1] Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350, 125:126-127, 2017.
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.
[3] Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954.
[4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
[5] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791, 2019.
[6] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. arXiv preprint arXiv:1511.06068, 2015.
[7] Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4908-4926, Online, November 2020. Association for Computational Linguistics.
[8] Fernando G De Maio. Income inequality measures. Journal of Epidemiology & Community Health, 61(10):849-852, 2007.
[9] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layerwise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017.
[10] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[11] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[12] Corrado Gini. Measurement of inequality of incomes. The economic journal, 31(121):124-126, 1921.
[13] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, 2013.
[68] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867, 2019.
[69] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
A Measuring Diffused Redundancy
A.1 CKA Definition
In all our evaluations we use CKA with a linear kernel [24] which essentially amounts to the following steps:
1. Take two representations  $Y \in \mathbb{R}^{n \times d1}$  and  $Z \in \mathbb{R}^{n \times d2}$ .
2. Compute dot product similarity within these representation, i.e. compute  $K = YY^{T}$ ,  $L = ZZ^{T}$
3.Normalize  $K$  and  $L$  to get  $K^{\prime} = HKH$ $L^{\prime} = HLH$  where  $H = I_{n} - \frac{1}{n}\mathbf{11}^{T}$
4. Return  $\mathrm{CKA}(Y,Z) = \frac{\mathrm{HSIC}(K,L)}{\sqrt{\mathrm{HSIC}(K,K)\mathrm{HSIC}(L,L)}}$ , where  $\mathrm{HSIC}(K,L) = \frac{1}{(n - 1)^2} (\mathrm{flatten}(K') \cdot \mathrm{flatten}(L'))$
We use the publicly available implementation of [37], which provides an implementation that can be calculated over multiple mini-batches: https://github.com/nvedant07/STIR
A.2 Additional CKA results
Fig 9 shows CKA comparison between randomly chosen parts of the layer and the full layer for different kinds of ResNet50. We observe that even ResNet50 trained with MRL loss shows a significant amount of diffused redundancy.
(a) CIFAR10
(c) Flowers
(b) CIFAR100
(d) Oxford-IIIT-Pets
Figure 9: [Comparison of Diffused Redundancy in MRL vs other losses, through the lens of CKA] We see a similar trend as reported in Fig 7 in the main paper, where even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. The amount of diffused redundancy however is much lesser than the resnets trained using the standard loss and adv. training as denoted by a much lower red line across all datasets.
B Training and Pre-Processing Details for Reproducibility
Here we list the sources of weights for the various pre-trained models used in our experiments:
- ResNet18 trained on ImageNet1k using standard loss: taken from timm v0.6.1.
- ResNet18 trained on ImageNet1k with adv training: taken from Salman et al. [46]:
- ResNet50 trained on ImageNet1k using standard loss: taken from timm v0.6.1.
- ResNet50 trained on ImageNet1k with adv training: taken from Salman et al. [46]: https://github.com/microsoft/robust-models-transfer.
- ResNet50 trained on ImageNet1k using MRL and with different final layer widths (resnet50_ffx): taken from released weights of by Kusupati et al. [26]: https://github.com/RAIVNLab/MRL.
- WideResNet50-2 on ImageNet1k both standard and avd. training: taken from Salman et al. [46]: https://github.com/microsoft/robust-models-transfer.
- VGG16 trained on ImageNet1k with standard loss: taken from timm v0.6.1.
- VGG16 trained on ImageNet1k with adv training: taken from Salman et al. [46]: https://github.com/microsoft/robust-models-transfer.
- ViTS32 & ViTS16 trained on ImageNet21k & ImageNet1k: taken from weights released by Steiner et al. [53]: https://github.com/google-research/vision_transformer.
All linear probes trained on the representations of these models are trained using SGD with a learning rate of 0.1, momentum of 0.9, batch size of 256, weight decay of  $1e - 4$ . The probe is trained for 50 epochs with a learning rate scheduler that decays the learning rate by 0.1 every 10 epochs. Scripts for training can also be found in the attached code.
For pre-processing, we re-size all inputs to  $224 \times 224$  (size used for pre-training) and apply the usual composition of RandomHorizontalFlip, ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25), RandomRotation(degrees=2). All inputs were mean normalized. For imagenet1k pre-trained models: mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. For imagenet21k pre-trained models: mean = [0.5,0.5,0.5], std = [0.5,0.5,0.5].
C Deeper Analysis of Fairness-Efficiency Tradeoff in Section 4
Analyzing Error Distributions To ensure that the higher gini coefficient shown in Fig 8 as we drop more neurons is not merely an artifact of lower overall accuracy, we plot class-wise accuracies as we drop neurons (Figs 11, 12 & 13). We find that for the entire layer, accuracy starts at an almost uniform distribution, and while overall accuracy deteriorates as we drop neurons, the drop comes at a larger cost for a few classes resulting in disparate inter-class accuracies.
Coeff of Variation for Measuring Inequality in Inter-Class Accuracy Fig 10 shows results for the same analysis shown in Fig 8 of the main paper and we find similar takeaways even when using the coefficient of variation as a measure of inequality.
D Corresponding Diffused Redundancy Estimates For Analyses in Section 3 &  $\ell_{\infty}$  Robust Model Results