SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vedant Nanda *
University of Maryland & MPI-SWS
Till Speicher
MPI-SWS
John P. Dickerson
Krishna P. Gummadi
Soheil Feizi
& University of Cambridge
Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded in such pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy, i.e., any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able to perform similarly as the whole layer on a variety of downstream tasks. For example, a linear probe trained on  $20\%$  of randomly picked neurons from the penultimate layer of a ResNet50 pre-trained on ImageNet1k achieves an accuracy within  $5\%$  of a linear probe trained on the full layer of neurons for downstream CIFAR10 classification. We conduct experiments on different neural architectures (including CNNs and Transformers) pretrained on both ImageNet1k and ImageNet21k and evaluate a variety of downstream tasks taken from the VTAB benchmark. We find that the loss & dataset used during pre-training largely govern the degree of diffuse redundancy and the "critical mass" of neurons needed often depends on the downstream task, suggesting that there is a task-inherent redundancy-performance Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for exploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences. Our code is available at https://github.com/nvedant07/diffused-redundancy.
Over the years, many architectures have been proposed (such as [50, 16, 23]) that achieve competitive accuracies on many benchmarks [45]. A key reason for the success of these models is their ability to learn useful representations of data [28]. While these models continue to get better, understanding the properties of underlying learned representations continues to be a challenge.
Prior works have attempted to understand representations learned by deep neural networks through the lens of mutual information between the representations, inputs, and outputs [49] and hypothesize that neural networks perform well because of a "compression" phase where mutual information between inputs and representations decreases. Additionally, recent works have found that many neurons are polysemantic, i.e., one neuron can encode multiple "concepts" [10, 39], and that one can
*Correspondence to: vnanda@mpi-sws.org
Table 1: Different model architectures with varying penultimate layer lengths trained on ImageNet1k. WRN50-2 stands for WideResNet50-2. Implementation of architectures is taken from timm [59]. Diffused redundancy here measures what fractions of neurons (randomly picked) can be discarded to achieve within  $\delta = 90\%$  performance of the full layer.
Model	Feature Length	ImageNet1k Top-1 Accuracy	Diffused Redundancy for δ = 0.9
CIFAR10	CIFAR100	Flowers	Oxford-IIIT-Pets
ViT S-16	384	64.82%	0.70	0.50	0.50	0.80
ViT S-32	384	55.73%	0.70	0.50	0.50	0.70
ResNet18	512	69.23%	0.80	0.50	0.50	0.90
ResNet50	2048	80.07%	0.90	0.50	0.20	0.90
WRN50-2	2048	77.00%	0.95	0.80	0.50	0.95
VGG16	4096	73.36%	0.95	0.80	0.80	0.95
then train sparse linear models on such concepts to do "explainable" classification [61]. However, it is not well understood if or how extracted features are concentrated or spread across the entire representation.
While the length of the feature vectors extracted from state-of-the-art networks  ${}^{2}$  can vary greatly, their accuracy on downstream tasks are not correlated to the size of the representation (see Table 1), but rather depend mostly on the inductive biases and training recipes [60, 53]. In all cases, the size of extracted feature vector (i.e. number of neurons) is orders of magnitude less than the dimensions of the input and thus allows for efficient transfer to many downstream tasks [22, 4, 41, 55]. We show that even using a random subset of these extracted neurons is enough to achieve downstream transfer accuracy close to that achieved by the full layer, thus showing that learned representations exhibit a degree of diffused redundancy (Table 1).
Early works in perception suggest that there are many redundant neurons in the human visual cortex [3] and some later works argued that a similar redundancy in artificial neural networks should help in faster convergence [19]. In this paper, we revisit redundancy in the context of modern DNN architectures that are trained on large-scale datasets. In particular, we propose the diffused redundancy hypothesis and systematically measure its prevalence across different pre-training datasets, losses, model architectures, and downstream tasks. We also show how this kind of redundancy can be exploited to obtain desirable properties such as generalization performance but at the same time draw caution to certain drawbacks of such an approach in increasing disparity in inter-class performance. We highlight the following contributions:
- We present the diffused redundancy hypothesis which states that learned representations exhibit redundancy that is diffused throughout the layer, i.e., many random subsets (of sufficient size) of neurons can perform as well as the entire layer. Our work aims to better understand the nature of representations learned by DNNs.
- We present an initial analysis of why diffused redundancy exists in deep neural networks, we show that a randomly chosen subset of neurons of size  $k$  performs as well as the projection of the entire layer on the first  $k$  principal components. Intuitively this means that in DNNs' representations, many random subsets of size  $k$  roughly capture all the variation in data that is possible with  $k$  dimensions (since PCA represents directions of maximum variance).
- We propose a measure of diffused redundancy and systematically test our hypothesis across various architectures, pre-training datasets & losses, and downstream tasks.
- We find that diffused redundancy is significantly impacted by pre-training datasets & loss and downstream datasets.
- We find that models that are explicitly trained such that particular parts of the full representation perform as well as the full layer, i.e., these models have structured redundancy (e.g. [26]), also exhibit a significant amount of diffused redundancy. Further, we also evaluate models trained with regularization that decorrelates activation of neurons and again find that these regularizations surprisingly do not affect diffused redundancy. These results suggest that this phenomenon is perhaps inevitable when DNNs have a wide enough final layer.
${}^{2}$  Extracted features for the purpose of this paper refers to the representation recorded on the penultimate layer, but the larger concept applies to any layer
- We quantify the degree of diffused redundancy as a function of the number of neurons in a given layer. As we reduce the dimension of the extracted feature vector and re-train the model, the degree of diffused redundancy decreases significantly, implying that diffused redundancy only appears when the layer is wide enough to accommodate redundancy.
- Finally we draw caution to some potential undesirable side-effects of exploiting diffused redundancy for efficient transfer learning that have implications for fairness.
Closest to our work is that of Dalvi et al. [7] who also investigate neuron redundancy but in the context of pre-trained language models. They analyze two language models and find that they can achieve good downstream performance with a significantly smaller subset of neurons. However, there are two key differences to our work. First, their analysis of neuron redundancy uses neurons from all layers (by concatenating each layer), whereas we show that such redundancy exists even at the level of a single (penultimate) layer. Second, and perhaps more importantly, they use feature selection to choose the subset of neurons, whereas we show that features are diffused throughout and that even a random pick of neurons suffices. Our work also differs by analyzing vision models (instead of language models) and using a diverse set of 30 pre-trained models (as opposed to testing only two models) which allows us to better understand the causes of such redundancy.
Efficient Representation Learning These works aim to learn representations which are "slim", with the goal of efficient deployment on edge devices [65, 64, 5]. Recently proposed paradigm of Matryoshka Representation Learning [26] aims to learn nested representations where one can perform downstream tasks with only a small portion of the representation. The goal of such representations is to allow quick, adaptive deployment without having to perform multiple, often expensive, forward passes. These works could be seen as inducing structured redundancy on the learned representations, where pre-specified parts of the representation are made to perform similar to the full representation. Our work, instead, aims to look at diffused redundancy that arises naturally in the training of DNNs. We carefully highlight the tradeoffs involved in exploiting this redundancy.
Pruning and Compression Many prior works focus on pruning weights [29, 13, 11, 15, 31, 9, 30] and how it can lead to sparse neural networks with many weights turned off. Our focus, however, is on understanding redundancy at the neuron level, without changing the weights. Work on structured pruning is more closely related to our work [33, 17], however, a key focus of these works is to prune channels/filters from convolution layers. Our work is more focused on understanding the nature of learned features and is more broadly applicable to all kinds of layers and models. We additionally focus on randomly pruning neurons, whereas structured pruning methods perform magnitude or feature-selection-based pruning.
Explainability/Interpretability Many works aim to understand learned representations with the goal of better explainability [35, 63, 2, 20, 40, 39, 10, 67]. Two works in this space are especially related to our work: sparse linear layers [61] which show that one can train sparse linear layers on top of extracted features from DNNs; and concept bottleneck models [21] which explicitly introduce a layer in which each neuron corresponds to a meaningful semantic concept. Both these works explicitly optimize for small/sparse layers, whereas our work shows that similar "small" layers already exist in pre-trained networks, and in fact, can be found simply with random sampling.
Understanding Deep Learning A related concept is that of intrinsic dimensionality of DNN landscapes [32]. Similar to our work, intrinsic dimensionality also requires dropping random parameters (weights) of the network. We, however, are concerned with dropping individual neurons. Other works on understanding deep learning [49, 1] have also looked at the learned features, however, none of these works analyze the redundancy at the neuron level. Another related phenomenon to diffused redundancy is that of neural collapse [42], which states that representations of the penultimate layer "collapse" to  $K$  points (where  $K =$  number of classes in the pre-training dataset). This implies that for perfectly collapsed representations we need to store just enough information in the final layer activations to be able to represent  $K$  different points. We interestingly show that this information is spread throughout the layer with a significant degree of redundancy.
(a) Downstream Acc; Standard
(b) Diff. Redundancy; Standard
(e) CKA; Standard
(c) Downstream Acc; AT
(f) CKA; AT
Figure 1: [Testing For Diffused Redundancy in ResNet50 Pre-trained on ImageNet1k] Top: transfer accuracies + Diffused Redundancy (DR) measure (Eq 1) on different downstream datasets, dotted horizontal line shows accuracy obtained using the full layer. We see that accuracy obtained using parts of representation varies greatly with pre-training loss (much more diffused redundancy in Adversarially Trained (AT) ResNet), but also depends on the downstream dataset. Bottom: comparing CKA between a randomly chosen fraction of neurons to the whole layer. Here we evaluate CKA on samples from different datasets and find that similarity of a subset of layer rapidly increases, reaching a similarity of greater than  $90\%$  on the adversially trained ResNet with only  $10\%$  randomly chosen neurons. Values are averaged over 5 random picks and error bars show std. dev.
(d) Diff. Redundancy; AT
2 The Diffused Redundancy Phenomenon
Prior observations about a compression phase [49] and neural collapse [42] suggest that the representations need not store a lot of information about the input. These findings imply that all neurons in a learned representation might not be necessary to capture all the information in a particular layer. Extending these observations, we propose the diffused redundancy hypothesis:
Learned features are diffused throughout a given layer with redundancy such that there exist many randomly chosen subsets of neurons that can achieve similar performance to the whole layer for a variety of downstream tasks.
Note that our hypothesis has two related but distinct parts to it: 1) redundancy in learned features, and 2) diffusion of this redundancy throughout the extracted feature vector. Redundancy refers to features being replicated in parts of the representation so that one can perform downstream tasks with parts of representation as well as with the full representation. Diffusion refers to this redundancy being spread all over the feature vector (as opposed to being structured), i.e., many random subsets (of sufficient size) of the feature vector perform equally well.
In order to evaluate the redundancy part of the diffused redundancy hypothesis we use two tasks: 1) representation similarity between randomly chosen subsets of a representation with the whole representation, and 2) transfer accuracy on out-of-distribution datasets (using a linear probe) of randomly chosen subsets of the representation compared to the whole representation. To estimate diffusion, we run each check for redundancy over multiple random seeds and plot the standard deviation over these runs.
Representation Similarity of Part vs Whole Centered Kernel Alignment (CKA) is a widely used representation similarity measure and takes in two representations of  $n$  data points  $Z \in \mathbb{R}^{n \times d_1}$  and  $Y \in \mathbb{R}^{n \times d_2}$  and gives a similarity score between 0 and 1 [24]. Intuitively, CKA (with linear kernel, see Appendix A for details about CKA) measures if the two representations rank the  $n$  points similarly (where similarity is based on cosine distances). For a given neural network  $g$  and  $n$  samples drawn from a given data distribution, i.e.,  $X \sim \mathcal{D}$ , let  $g(X)$  be the (penultimate) layer representation. If  $m$  is a boolean vector representing a subset of neurons in  $g(X)$ , then we aim to measure  $\mathrm{CKA}(m \odot g(X), g(X))$  to estimate how much redundancy exists in the layer. If indeed  $\mathrm{CKA}(m \odot g(X), g(X))$  is high (i.e. close to 1) then it's a strong indication that the diffused redundancy hypothesis holds.
Downstream Transfer Performance of Part vs Whole A commonly used paradigm to measure the quality of learned representations is to measure their performance on a variety of downstream tasks [68, 22]. Here, we attach a linear layer  $(h)$  on top of the extracted features of a network  $(g)$  to
do classification. This layer is then trained using the training dataset of the particular task (keeping  $g$  frozen). If features were to be diffused redundantly then accuracy obtained using  $h \circ g$ , i.e. linear layer attached to the entire feature vector, should be roughly the same as  $h' \circ (m \odot g)$ ; where  $m$  is a boolean vector representing a subset of neurons extracted by  $g$ , and  $h \& h'$  are independently trained linear probes.
For both tasks, i.e. representation similarity and downstream transfer performance, we evaluate on CIFAR10/100 [25], Oxford-IIIT-Pets [43] and Flowers [38] datasets, from the VTAB benchmark [68]. Training and pre-processing details are included in Appendix B.
Measure of Diffused Redundancy In order to rigorously test our hypothesis, we define a measure of diffused redundancy  $(DR)$  for a given model  $(g)$  with  $\mathcal{M}$  being a set of all possible boolean vectors of size  $|g|$ , i.e. size of the representation extracted from  $g$ . Each vector  $m \in \mathcal{M}$  represents a possible subset of neurons from the entire layer. This measure is defined on a particular task  $(T)$  as follows:
$$
D R (g, T, \delta) = 1 - \frac {\min  f , \text {s . t .} \frac {1}{| \mathcal {M} _ {f} |} \sum_ {m \in \mathcal {M} _ {f}} \frac {T (m \odot g)}{T (g)} \geq \delta}{| g |}, \tag {1}
$$
$$
\mathcal {M} _ {f} = \left\{m \in \mathcal {M} | \sum_ {i} m _ {i} = f; m \in \{0, 1 \} ^ {| g |} \right\}
$$
Here  $T(.)$  denotes the performance of the model inside () for the particular task and  $\delta$  is a user-defined tolerance level. For the task of representation similarity  $T(m\odot g)$  is CKA between a subset of neurons denoted by  $m\odot g$  and  $g$ , and  $T(g)$  is always 1, since it denotes CKA between  $g$  and  $g$ . For downstream transfer performance,  $T(m\odot g)$  is the test accuracy obtained by training a linear probe on the portion of representation denoted by  $m\odot g$  and  $T(g)$  is the test accuracy obtained using the full representation. For  $\delta = 1$ , this measure tells what fraction of neurons could be discarded to exactly match the performance of the entire set of neurons. A higher value of  $DR$  denotes that only a few random neurons were needed to match the task performance of the full set of neurons, and thus indicates higher redundancy. Since  $\mathcal{M}$  contains an exponential number of vectors  $(2^{|g|})$ , precisely estimating this quantity is hard. Thus, we first choose a few  $f$  (number of neurons to be chosen) to define subsets of  $\mathcal{M}$ . Then for each  $\mathcal{M}_f$  we randomly select 5 samples.
2.1 Prevalence of Diffused Redundancy in Pre-Trained Models
Figure 1 checks for diffused redundancy in the penultimate layer representation of two types of ResNet50 pre-trained on ImageNet1k: one using the standard cross-entropy loss and another trained using adversarial training [34] (with  $\ell_2$  threat model and  $\epsilon = 3$ )<sup>3</sup>. We check for diffused redundancy using both tasks of representation similarity and downstream transfer performance.
Redundancy This is indicated along the x-axis of Fig 1, i.e., redundancy is shown when some small subset of the full set of neurons can achieve almost as good performance as the full set of neurons. When looking at downstream task performance (Figs 1a&1c), in order to obtain performance within some  $\delta \%$  of the full layer accuracy (dotted lines), the fraction of neurons that can be discarded are task-dependent, e.g. across both training types we see that flowers (102 classes) and CIFAR100 (100 classes) require more fraction of neurons than CIFAR10 (10 classes) and oxford-iiit-pets (37 classes), perhaps because both these tasks have more classes. Additionally, across all datasets, the model trained with adversarial training exhibits more diffused redundancy than the one trained with standard loss (Fig 1d& 1b respectively), meaning we can discard far more neurons for the adversarially trained model to reach close to the full layer accuracy. Interestingly when looking at CKA between part of the feature vector with the full extracted vector (Figs 1e&1f), we do not see a significant difference in trends when evaluating CKA on samples from different datasets. However, we still see that we can achieve a given level of CKA with far fewer fraction of neurons in the adversarially trained ResNet50 as compared to the usually trained ResNet50.
Diffused Redundancy This is indicated by small error bars in Figs 1a&1c&1e&1f. If redundancy were instead very structured, then different random picks of neurons would have high variance,
3We also report results for  $\ell_{\infty},\epsilon = 4 / 255$  in Appendix D and generally find that the  $\ell_2$  models exhibit higher diffused redundancy. We choose to study adversarially robust models since they've been shown to transfer better [46].
however, the error bars here are very low, showing that performance is very stable across different random picks, thus indicating that redundancy is diffused throughout the layer.
While both tasks of downstream transfer and CKA between part and whole indicate higher diffused redundancy for the adversarially trained model, we see that downstream transfer performance can differ substantially based on the dataset (while CKA remains fairly stable across the same datasets), indicating that downstream performance turns out to be a "harder" test for diffused redundancy. Thus, in the rest of the paper, we examine diffused redundancy through the lens of downstream transfer performance and include CKA results in Appendix A.
2.2 Understanding Why Many Random Subsets Work
(a) Standard Training; CKA between two random subsets
(b) Adversarial Training; CKA between two random subsets
(c) Standard Training; PCA (dotted) and bottom (dashed)
(d) Adversarial Training; PCA top-k (dotted) and bottom-k (dashed)
Many prior works explicitly train models to have "small" representations (e.g. [26, 65, 64, 5] with the goal of efficient downstream learning. These works show that, when explicitly optimized, networks can perform downstream classification with fewer neurons than typically used in state-of-the-art architectures. We show, however, that such subsets already exist in models that are not explicitly trained for this goal, and in fact, one doesn't even have to try hard to find this subset; it can be randomly chosen. Later in section 3.3 we compare some of these efficient representation learning methods to randomly chosen subsets and carefully analyze the tradeoffs involved. Here, however, we seek to better understand why there exist so many randomly selected subsets that work just as well as the whole layer.
High CKA between two random subsets of the same size. First, we use representation similarity (CKA) [24] to get a sense of how two random picks of neurons (of the same size) relate to each other. Concretely, we calculate CKA between two random picks of  $k\%$  neurons in the penultimate layer (averaged over 10 such randomly picked pairs) on samples taken from different datasets. Fig 2a& 2b show
CKA results averaged over these different picks of pairs of subsets of the full set of neurons. We see that after picking a certain threshold, i.e. for a large enough value of  $k$ , the similarity between any two randomly picked pairs of heads is fairly high. For example, for the adversarially trained ResNet50 (Fig 2b), we observe that any  $10\%$  of neurons picked from the penultimate layer are highly similar (CKA of about 0.8), with very low error bars. A similar value of CKA is obtained with  $20\%$  of neurons for the standard ResNet50 model. These results indicate that, given a sufficient size, picking any random subset of that size has very similar representations and thus provides an initial intuition for why almost any random subset, with high probability, could work equally well.
Figure 2: [Why Many Random Subsets Work] (2a& 2b) Similarity between any two randomly picked sets of  $k\%$  neurons becomes fairly high (for a "critical mass" of  $k\%$ ), thus showing that any random pick beyond this threshold is likely to perform similarly; (2c& 2d) Performance of a linear probe trained for a downstream task on randomly chosen neurons (solid lines) closely follows that of a linear probe trained on a projection on top  $k$  principal components (dotted lines) for a sufficiently large value of  $k$ .