Title: Diffused Redundancy in Pre-trained Representations
Venue: NeurIPS 2023
Accepted: yes
Keywords: representation learning

redundancy

transfer learning

fairness
Abstract: Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded in such pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy, ie, any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able to perform similarly as the whole layer on a variety of downstream tasks. For example, a linear probe trained on $20\%$ of randomly picked neurons from the penultimate layer of a ResNet50 pre-trained on ImageNet1k achieves an accuracy within $5\%$ of a linear probe trained on the full layer of neurons for downstream CIFAR10 classification. We conduct experiments on different neural architectures (including CNNs and Transformers) pre-trained on both ImageNet1k and ImageNet21k and evaluate a variety of downstream tasks taken from the VTAB benchmark. We find that the loss \& dataset used during pre-training largely govern the degree of diffuse redundancy and the "critical mass" of neurons needed often depends on the downstream task, suggesting that there is a task-inherent redundancy-performance Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for exploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences. Our code is available at \url{https://github.com/nvedant07/diffused-redundancy}.

Diffused Redundancy in Pre-trained Representations

Abstract

Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded in such pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy , i.e. , any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able to perform similarly as the whole layer on a variety of downstream tasks. For example, a linear probe trained on 20% of randomly picked neurons from the penultimate layer of a ResNet50 pre-trained on ImageNet1k achieves an accuracy within 5% of a linear probe trained on the full layer of neurons for downstream CIFAR10 classification. We conduct experiments on different neural architectures (including CNNs and Transformers) pretrained on both ImageNet1k and ImageNet21k and evaluate a variety of downstream tasks taken from the VTAB benchmark. We find that the loss & dataset used during pre-training largely govern the degree of diffuse redundancy and the 'critical mass' of neurons needed often depends on the downstream task, suggesting that there is a task-inherent redundancy-performance Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for exploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences. Our code is available at https://github.com/nvedant07/diffused-redundancy .

1 Introduction

Over the years, many architectures have been proposed (such as [50, 16, 23]) that achieve competitive accuracies on many benchmarks [45]. A key reason for the success of these models is their ability to learn useful representations of data [28]. While these models continue to get better, understanding the properties of underlying learned representations continues to be a challenge.

Prior works have attempted to understand representations learned by deep neural networks through the lens of mutual information between the representations, inputs, and outputs [49] and hypothesize that neural networks perform well because of a 'compression' phase where mutual information between inputs and representations decreases. Additionally, recent works have found that many neurons are polysemantic , i.e. , one neuron can encode multiple 'concepts' [10, 39], and that one can

then train sparse linear models on such concepts to do 'explainable' classification [61]. However, it is not well understood if or how extracted features are concentrated or spread across the entire representation.

While the length of the feature vectors extracted from state-of-the-art networks 2 can vary greatly, their accuracy on downstream tasks are not correlated to the size of the representation (see Table 1), but rather depend mostly on the inductive biases and training recipes [60, 53]. In all cases, the size of extracted feature vector ( i.e. number of neurons) is orders of magnitude less than the dimensions of the input and thus allows for efficient transfer to many downstream tasks [22, 4, 41, 55]. We show that even using a random subset of these extracted neurons is enough to achieve downstream transfer accuracy close to that achieved by the full layer, thus showing that learned representations exhibit a degree of diffused redundancy (Table 1).

Early works in perception suggest that there are many redundant neurons in the human visual cortex [3] and some later works argued that a similar redundancy in artificial neural networks should help in faster convergence [19]. In this paper, we revisit redundancy in the context of modern DNN architectures that are trained on large-scale datasets. In particular, we propose the diffused redundancy hypothesis and systematically measure its prevalence across different pre-training datasets, losses, model architectures, and downstream tasks. We also show how this kind of redundancy can be exploited to obtain desirable properties such as generalization performance but at the same time draw caution to certain drawbacks of such an approach in increasing disparity in inter-class performance. We highlight the following contributions:

We present the diffused redundancy hypothesis which states that learned representations exhibit redundancy that is diffused throughout the layer, i.e. , many random subsets (of sufficient size) of neurons can perform as well as the entire layer. Our work aims to better understand the nature of representations learned by DNNs.

Wepresent an initial analysis of why diffused redundancy exists in deep neural networks, we show that a randomly chosen subset of neurons of size k performs as well as the projection of the entire layer on the first k principal components. Intuitively this means that in DNNs' representations, many random subsets of size k roughly capture all the variation in data that is possible with k dimensions (since PCA represents directions of maximum variance).

We propose a measure of diffused redundancy and systematically test our hypothesis across various architectures, pre-training datasets & losses, and downstream tasks.

-We find that diffused redundancy is significantly impacted by pre-training datasets & loss and downstream datasets.

-We find that models that are explicitly trained such that particular parts of the full representation perform as well as the full layer, i.e. , these models have structured redundancy ( e.g. [26]), also exhibit a significant amount of diffused redundancy. Further, we also evaluate models trained with regularization that decorrelates activation of neurons and again find that these regularizations surprisingly do not affect diffused redundancy. These results suggest that this phenomenon is perhaps inevitable when DNNs have a wide enough final layer.

-We quantify the degree of diffused redundancy as a function of the number of neurons in a given layer. As we reduce the dimension of the extracted feature vector and re-train the model, the degree of diffused redundancy decreases significantly, implying that diffused redundancy only appears when the layer is wide enough to accommodate redundancy.

Finally we draw caution to some potential undesirable side-effects of exploiting diffused redundancy for efficient transfer learning that have implications for fairness.

1.1 Related Work

Closest to our work is that of Dalvi et al. [7] who also investigate neuron redundancy but in the context of pre-trained language models. They analyze two language models and find that they can achieve good downstream performance with a significantly smaller subset of neurons. However, there are two key differences to our work. First, their analysis of neuron redundancy uses neurons from all layers (by concatenating each layer), whereas we show that such redundancy exists even at the level of a single (penultimate) layer. Second, and perhaps more importantly, they use feature selection to choose the subset of neurons, whereas we show that features are diffused throughout and that even a random pick of neurons suffices. Our work also differs by analyzing vision models (instead of language models) and using a diverse set of 30 pre-trained models (as opposed to testing only two models) which allows us to better understand the causes of such redundancy.

Efficient Representation Learning These works aim to learn representations which are 'slim', with the goal of efficient deployment on edge devices [65, 64, 5]. Recently proposed paradigm of Matryoshka Representation Learning [26] aims to learn nested representations where one can perform downstream tasks with only a small portion of the representation. The goal of such representations is to allow quick, adaptive deployment without having to perform multiple, often expensive, forward passes. These works could be seen as inducing structured redundancy on the learned representations, where pre-specified parts of the representation are made to perform similar to the full representation. Our work, instead, aims to look at diffused redundancy that arises naturally in the training of DNNs. We carefully highlight the tradeoffs involved in exploiting this redundancy.

Pruning and Compression Many prior works focus on pruning weights [29, 13, 11, 15, 31, 9, 30] and how it can lead to sparse neural networks with many weights turned off. Our focus, however, is on understanding redundancy at the neuron level, without changing the weights. Work on structured pruning is more closely related to our work [33, 17], however, a key focus of these works is to prune channels/filters from convolution layers. Our work is more focused on understanding the nature of learned features and is more broadly applicable to all kinds of layers and models. We additionally focus on randomly pruning neurons, whereas structured pruning methods perform magnitude or feature-selection-based pruning.

Explainability/Interpretability Many works aim to understand learned representations with the goal of better explainability [35, 63, 2, 20, 40, 39, 10, 67]. Two works in this space are especially related to our work: sparse linear layers [61] which show that one can train sparse linear layers on top of extracted features from DNNs; and concept bottleneck models [21] which explicitly introduce a layer in which each neuron corresponds to a meaningful semantic concept. Both these works explicitly optimize for small/sparse layers, whereas our work shows that similar 'small' layers already exist in pre-trained networks, and in fact, can be found simply with random sampling.

Understanding Deep Learning Arelated concept is that of instrinsic dimensionality of DNN landscapes [32]. Similar to our work, intrinsic dimensionality also requires dropping random parameters (weights) of the network. We, however, are concerned with dropping individual neurons. Other works on understanding deep learning [49, 1] have also looked at the learned features, however, none of these works analyze the redundancy at the neuron level. Another related phenomenon to diffused redundancy is that of neural collapse [42], which states that representations of the penultimate layer 'collapse' to K points (where K = number of classes in the pre-training dataset). This implies that for perfectly collapsed representations we need to store just enough information in the final layer activations to be able to represent K different points. We interestingly show that this information is spread throughout the layer with a significant degree of redundancy.

0.0

0.2

0.4

0.6

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

cifar10

cifar100

flowers

oxford-iiit-pets

(a) Downstream Acc; Standard

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

cifar10

cifar100

flowers

oxford-iiit-pets

(b) Diff. Redundancy; Standard

0.0

0.2

0.4

0.6

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

cifar10

cifar100

flowers

oxford-iiit-pets

(c) Downstream Acc; AT

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

cifar10

cifar100

flowers

oxford-iiit-pets

(d) Diff. Redundancy; AT

0.0

0.2

0.4

0.6

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

cifar10

cifar100

flowers

oxford-iiit-pets

(e) CKA; Standard

0.0

0.2

0.4

0.6

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

cifar10

cifar100

flowers

oxford-iiit-pets

(f) CKA; AT

2 The Diffused Redundancy Phenomenon

Prior observations about a compression phase [49] and neural collapse [42] suggest that the representations need not store a lot of information about the input. These findings imply that all neurons in a learned representation might not be necessary to capture all the information in a particular layer. Extending these observations, we propose the diffused redundancy hypothesis:

Learned features are diffused throughout a given layer with redundancy such that there exist many randomly chosen subsets of neurons that can achieve similar performance to the whole layer for a variety of downstream tasks.

Note that our hypothesis has two related but distinct parts to it: 1) redundancy in learned features, and 2) diffusion of this redundancy throughout the extracted feature vector. Redundancy refers to features being replicated in parts of the representation so that one can perform downstream tasks with parts of representation as well as with the full representation. Diffusion refers to this redundancy being spread all over the feature vector (as opposed to being structured), i.e. , many random subsets (of sufficient size) of the feature vector perform equally well.

In order to evaluate the redundancy part of the diffused redundancy hypothesis we use two tasks: 1) representation similarity between randomly chosen subsets of a representation with the whole representation, and 2) transfer accuracy on out-of-distribution datasets (using a linear probe) of randomly chosen subsets of the representation compared to the whole representation. To estimate diffusion , we run each check for redundancy over multiple random seeds and plot the standard deviation over these runs.

Representation Similarity of Part vs Whole Centered Kernel Alignment (CKA) is a widely used representation similarity measure and takes in two representations of n data points Z ∈ R n × d 1 and Y ∈ R n × d 2 and gives a similarity score between 0 and 1 [24]. Intuitively, CKA (with linear kernel, see Appendix A for details about CKA) measures if the two representations rank the n points similarly (where similarity is based on cosine distances). For a given neural network g and n samples drawn from a given data distribution, i.e. , X ∼ D , let g ( X ) be the (penultimate) layer representation. If m is a boolean vector representing a subset of neurons in g ( X ) , then we aim to measure CKA( m ⊙ g ( X ) , g ( X ) ) to estimate how much redundancy exists in the layer. If indeed CKA( m ⊙ g ( X ) , g ( X ) ) is high ( i.e. close to 1) then it's a strong indication that the diffused redundancy hypothesis holds.

Downstream Transfer Performance of Part vs Whole A commonly used paradigm to measure the quality of learned representations is to measure their performance on a variety of downstream tasks [68, 22]. Here, we attach a linear layer ( h ) on top of the extracted features of a network ( g ) to

do classification. This layer is then trained using the training dataset of the particular task (keeping g frozen). If features were to be diffused redundantly then accuracy obtained using h ◦ g , i.e. linear layer attached to the entire feature vector, should be roughly the same as h ′ ◦ ( m ⊙ g ) ; where m is a boolean vector representing a subset of neurons extracted by g , and h & h ′ are independently trained linear probes.

For both tasks, i.e. representation similarity and downstream transfer performance, we evaluate on CIFAR10/100 [25], Oxford-IIIT-Pets [43] and Flowers [38] datasets, from the VTAB benchmark [68]. Training and pre-processing details are included in Appendix B.

Measure of Diffused Redundancy In order to rigorously test our hypothesis, we define a measure of diffused redundancy ( DR ) for a given model ( g ) with M being a set of all possible boolean vectors of size | g | , i.e. size of the representation extracted from g . Each vector m ∈ M represents a possible subset of neurons from the entire layer. This measure is defined on a particular task ( T ) as follows:

Here T ( . ) denotes the performance of the model inside () for the particular task and δ is a userdefined tolerance level. For the task of representation similarity T ( m ⊙ g ) is CKA between a subset of neurons denoted by m ⊙ g and g , and T ( g ) is always 1, since it denotes CKA between g and g . For downstream transfer performance, T ( m ⊙ g ) is the test accuracy obtained by training a linear probe on the portion of representation denoted by m ⊙ g and T ( g ) is the test accuracy obtained using the full representation. For δ = 1 , this measure tells what fraction of neurons could be discarded to exactly match the performance of the entire set of neurons. A higher value of DR denotes that only a few random neurons were needed to match the task performance of the full set of neurons, and thus indicates higher redundancy. Since M contains an exponential number of vectors ( 2 | g | ), precisely estimating this quantity is hard. Thus, we first choose a few f (number of neurons to be chosen) to define subsets of M . Then for each M f we randomly select 5 samples.

2.1 Prevalence of Diffused Redundancy in Pre-Trained Models

Figure 1 checks for diffused redundancy in the penultimate layer representation of two types of ResNet50 pre-trained on ImageNet1k: one using the standard cross-entropy loss and another trained using adversarial training [34] (with ℓ 2 threat model and ϵ = 3 ) 3 . Wecheck for diffused redundancy using both tasks of representation similarity and downstream transfer performance.

Redundancy This is indicated along the x-axis of Fig 1, i.e. , redundancy is shown when some small subset of the full set of neurons can achieve almost as good performance as the full set of neurons. When looking at downstream task performance (Figs 1a&1c), in order to obtain performance within some δ % of the full layer accuracy (dotted lines), the fraction of neurons that can be discarded are task-dependent, e.g. across both training types we see that flowers (102 classes) and CIFAR100 (100 classes) require more fraction of neurons than CIFAR10 (10 classes) and oxford-iiit-pets (37 classes), perhaps because both these tasks have more classes. Additionally, across all datasets, the model trained with adversarial training exhibits more diffused redundancy than the one trained with standard loss (Fig 1d& 1b respectively), meaning we can discard far more neurons for the adversarially trained model to reach close to the full layer accuracy. Interestingly when looking at CKA between part of the feature vector with the full extracted vector (Figs 1e&1f), we do not see a significant difference in trends when evaluating CKA on samples from different datasets. However, we still see that we can achieve a given level of CKA with far fewer fraction of neurons in the adversarially trained ResNet50 as compared to the usually trained ResNet50.

Diffused Redundancy This is indicated by small error bars in Figs 1a&1c&1e&1f. If redundancy were instead very structured, then different random picks of neurons would have high variance,

however, the error bars here are very low, showing that performance is very stable across different random picks, thus indicating that redundancy is diffused throughout the layer.

While both tasks of downstream transfer and CKA between part and whole indicate higher diffused redundancy for the adversarially trained model, we see that downstream transfer performance can differ substantially based on the dataset (while CKA remains fairly stable across the same datasets), indicating that downstream performance turns out to be a 'harder' test for diffused redundancy. Thus, in the rest of the paper, we examine diffused redundancy through the lens of downstream transfer performance and include CKA results in Appendix A.

2.2 Understanding Why Many Random Subsets Work

0.0

0.2

0.4

0.6

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of Neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of Neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

cifar10

cifar100

flowers

oxford-iiit-pets

(a)

Standard

Training;

CKA be- tween two random subsets

(c)

Standard

Training;

PCA top

(dotted) and bottom (dashed)

(d) Adversarial Training; PCA top- k (dotted) and bottom-k (dashed)

Many prior works explicitly train models to have 'small' representations ( e.g. [26, 65, 64, 5] with the goal of efficient downstream learning. These works show that, when explicitly optimized, networks can perform downstream classification with fewer neurons than typically used in state-of-the-art architectures. We show, however, that such subsets already exist in models that are not explicitly trained for this goal, and in fact, one doesn't even have to try hard to find this subset; it can be randomly chosen. Later in section 3.3 we compare some of these efficient representation learning methods to randomly chosen subsets and carefully analyze the tradeoffs involved. Here, however, we seek to better understand why there exist so many randomly selected subsets that work just as well as the whole layer.

High CKA between two random subsets of the same size. First, we use representation similarity (CKA) [24] to get a sense of how two random picks of neurons (of the same size) relate to each other. Concretely, we calculate CKA between two random picks of k % neurons in the penultimate layer (averaged over 10 such randomly picked pairs) on samples taken from different datasets. Fig 2a& 2b show

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of Neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of Neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

cifar10

cifar100

flowers

oxford-iiit-pets

CKA results averaged over these different picks of pairs of subsets of the full set of neurons. We see that after picking a certain threshold, i.e. for a large enough value of k , the similarity between any two randomly picked pairs of heads is fairly high. For example, for the adversarially trained ResNet50 (Fig 2b), we observe that any 10% of neurons picked from the penultimate layer are highly similar (CKA of about 0 . 8 ), with very low error bars. A similar value of CKA is obtained with 20% of neurons for the standard ResNet50 model. These results indicate that, given a sufficient size, picking any random subset of that size has very similar representations and thus provides an initial intuition for why almost any random subset, with high probability, could work equally well.

Downstream performance on k randomly chosen neurons closely follows performance on top k principal components. While high representation similarity is a necessary condition for two representations to have similar downstream performance, it's not a sufficient one. As seen earlier in Fig 1, similar values of CKA can still show different downstream accuracy. Thus, to further understand why any random pick of neurons achieves similar accuracy, we compare a randomly chosen subset of neurons with a projection on the same number of top k principal components. Fig 2c& 2d shows that the performance of a linear probe trained on a random subset of neurons initially lags behind the top PCA dimensions for small values of k ( < 20%) , but for higher values ( > 20% ) closely follows the performance on a probe trained on the projection on the same number of top principal components. This indicates that a sufficiently sized random subset of size k (roughly) captures the maximum variance in data that can be captured in k dimensions (upper bound by top k principal components since they are designed to capture maximum variance). As a sanity check, we also show the results for the bottom k principal components (directions of least variance) and see that randomly chosen neurons perform significantly better.

Random projections into lower dimensions do not perform well on downstream tasks. The presence of diffused redundancy in a particular layer indicates that a dimension lower than the layer's size suffices to perform many downstream tasks. However, randomly sampling neurons (as we do in our experiments on diffused redundancy) can be seen as one very particular way of reducing dimension that is restricted by what the network has learned. This raises a natural question, would our observation extend to less restricted ways of reducing dimensions? To test this, we compare diffused redundancy i.e. , a linear probe trained on a random sample of k neurons to a linear probe trained on a random projection of the layer's activations. To project a d dimensional layer into a lower dimension ( k ) we multiply it by a (normalized) randomly sampled matrix from a normal distribution R d × k . We report our results over 5 seeds in Fig 3 and find that diffused redundancy significantly outperforms random projection.

3 Factors Influencing The Degree of Diffused Redundancy

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) CIFAR10

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) CIFAR100

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(c) Flowers

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) Oxford-IIIT-Pets

vgg16_bn-nonrob

vgg16_bn-robustl2eps3

resnet18-nonrob

resnet18-robustl2eps3

resnet50-nonrob

resnet50-robustl2eps3

wide_resnet50_2-nonrob

wide_resnet50_2-robustl2eps3

vit_small_patch16_224-nonrob

vit_small_patch32_224-nonrob

In order to better understand the phenomenon of diffused redundancy we analyze 30 different pretrained models, with different architectures, pre-training datasets and losses. We then evaluate each model for transfer accuracy on 4 datasets mentioned in Section 2. While all results in this section are on the penultimate layer, we also report results on intermediate layers in Appendix E and find similar trends of diffused redundancy. All details for reproducibility can be found in Appendix B.

Architectures We consider VGG16 [50], ResNet18, ResNet50, WideResNet50-2 [16], ViT-S16 & ViT-S32 [23]. Additionally, we consider ResNet50 with varying widths of the final layer (denoted by ResNet50 ffx where x denotes the number of neurons in the final layer).

Upstream Datasets ImageNet-1k & ImageNet-21k [45].

Upstream Losses Standard cross-entropy, adversarial training ( ℓ 2 threat model, ϵ = 3 and ℓ ∞ threat model, ϵ = 4 / 255 4 ) [34], MRL Loss [26], DeCov [6], and varying strengths of dropout [52].

Downstream Datasets CIFAR10/1000, Oxford-IIIT-Pets, and Flowers, same as Section 2. Additionally, we also report performance on harder datasets such as ImageNetv2 [44] and Places365 [69]

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) CIFAR10

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) CIFAR100

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(c) Flowers

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) Oxford-IIIT-Pets

vit_small_patch16_224-imagenet

vit_small_patch16_224-imagenet21k

vit_small_patch32_224-imagenet

vit_small_patch32_224-imagenet21k

in Appendix F. For all analyses in this section, we also report corresponding approximations for DR (Eq 1) in Appendix D.

3.1 Effects of Architecture, Upstream Loss, Upstream Datasets, and Downstream Datasets

Extending the analysis in Section 2, we evaluate the diffused redundancy hypothesis on other architectures. Fig 4 shows transfer performance for different architectures. All architectures shown in Fig 4 are trained on ImageNet1k. We find that our takeaways from Section 2 also extend to other architectures.

Fig 5 compares two instances each of ViT-S16 and ViT-S32, one trained on a bigger upstream dataset (ImageNet21k) and another on a smaller dataset (ImageNet1k)

Note that the nature of all curves in both Figs 4&5 highly depends on downstream datasets. This is also consistent with the initial observation of Section 2 about diffused redundancy being downstream dataset dependent. Additionally, we also report results on ImageNetV2 and Places365 in Appendix F showing that diffused redundancy also holds on 'harder' datasets.

3.2 Diffused Redundancy as a Function of Layer Width

We take the usual ResNet50 with a penultimate layer consisting of 2048 neurons and compare it with variants that are pre-trained with a much smaller penultimate layer, these are denoted by ResNet50 ffx where x ( < 2048 ) is the number of neurons in the penultimate layer. Fig 6 shows how diffused redundancy slowly fades away as we squeeze the layer to be smaller. In fact, for ResNet50 ff8 , we see that across all datasets we need > 90% of the full layer to achieve performance close to the full layer. This shows that diffused redundancy only appears in DNNs when the layer is sufficiently wide to encode redundancy.

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) CIFAR10

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) CIFAR100

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(c) Flowers

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) Oxford-IIIT-Pets

resnet50

resnet50_ff1024

resnet50_ff512

resnet50_ff256

resnet50_ff128

resnet50_ff64

resnet50_ff32

resnet50_ff16

resnet50_ff8

3.3 Comparison With Methods That Optimize For Lesser Neurons

Matryoshka Representation Learning (MRL) is a recently proposed paradigm that learns nested representations such that the first k, 2 k, 4 k, ..., N (where N = size of the full layer) dimensions of the extracted feature vector are all explicitly made to be good at minimizing upstream loss, with

the intuition of learning coarse-to-fine representations. This ensures that one can flexibly use these smaller parts of the representation for downstream tasks. MRL, thus, ensures that redundancy shows up in learned representations in a structured way, i.e. , we know the first k, 2 k, ... neurons can be picked and used for downstream tasks and should perform reasonably.

Here we investigate two questions regarding Matryoshka representations: 1) do these representations also exhibit the phenomenon of diffused redundancy? i.e. if we were to ignore the structure imposed by MRL-type training and instead just pick random neurons from all over the layer, do we still get reasonable performance?, and 2) how do they compare to representations learned by other kinds of losses?

Figure 7 investigates these questions by comparing ResNet50 representations learned using MRL loss to other losses. resnet50 mrl nonrob first (red line) denotes a ResNet50 trained using MRL loss and evaluated on parts of the representation that were optimized to have low upstream loss ( i.e. first k, 2 k, ...N neurons, here k = 8 and N = 2048 ) and resnet50 mrl nonrob random (green line) refers to the same model with the same number of neurons chosen for evaluation, except they're chosen at random from the entire layer.

First, we interestingly see that even the ResNet50 trained with MRL loss exhibits diffused redundancy (denoted by green line spiking very quickly for most datasets in Fig 7), despite having been trained to only have structured redundancy. Based on this observation, we conjecture that diffused redundancy is a natural consequence of having a wide layer. Second, we see that ResNet50 trained on MRL indeed does better in the low neuron regime across datasets (red line on the extreme left part of the plots in Fig 7), but other models quickly catch up as we pick more neurons, thus indicating that major efficiency benefits of MRL-type models are best realized when using an extremely low number of neurons, else one can obtain similar downstream performances by simply picking random samples from existing pre-trained models.

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) CIFAR10

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) CIFAR100

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(c) Flowers

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) Oxford-IIIT-Pets

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50_mrl-nonrob-random

resnet50_mrl-nonrob-first

3.4 Methods That Prevent Co-adaptation of Neurons Also Exhibit Diffused Redundancy

We consider two additional modifications to upstream pre-training: we add regularization to the usual cross-entropy loss that decorrelates neurons in the feature representation using DeCov [6] and Dropout [52]. Dropout does this implicitly by randomly dropping neurons during training and DeCov does this explicitly by putting a loss on the activations of a given layer. We pre-train different ResNet50 on ImageNet1k by varying strengths of dropout ranging from 0.1 all the way up to 0.8 and also by separately adding the DeCov regularizer (regularization strength = 1 e -4 ) to give 9 additional pre-trained models. Intuitively these methods should lead to increased diffused redundancy since by design these methods force the model to disperse similar information in different parts of a layer to perform the same task downstream task. Interestingly, we see that diffused redundancy in these models is almost completely independent of the regularization strength. This suggests that diffused redundancy might be an inevitable property of DNNs when the layer has sufficient width. Due to space constraints, we include these results in Appendix G.

4 Possible Fairness-Efficiency Tradeoffs in Efficient Downstream Transfer

One natural use case for diffused redundancy is efficient transfer to downstream datasets. As defined in Eq 1 and as also seen in Sections 2& 3, dropping neurons comes at a small cost ( δ in Eq 1) in

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.0

0.2

0.4

0.6

0.8

1.0

Gini

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(a) CIFAR10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.2

0.4

0.6

0.8

1.0

Gini

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(b) CIFAR100

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.2

0.4

0.6

0.8

1.0

Gini

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(c) Flowers

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

Gini

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(d) Oxford-IIIT-Pets

performance as compared to the full set of neurons. Here we take a deeper look into this drop in overall performance and investigate how it is distributed across classes. If the drop affects only a few classes, then dropping neurons - although efficient for downstream tasks - could have implications for fairness, which is not only of concern to ML researchers and practitioners [66, 14, 18], but also to lawyers [56] and policymakers [57].

We compare the spread of accuracies across classes using inequality indices, which are commonly used in economics to study income inequality [8, 47] and have also recently been adopted in the fair ML literature [51]. We use gini index [12] and coefficient of variation [27] to quantify the spread of performance across classes. For a perfect spread, both gini and coefficient of variation are 0, and higher values indicate higher inequality.

Figure 8 compares the gini index for various models at varying levels of accuracy (note that accuracy monotonically increases with more neurons, hence the right most point for a model represents the model with all neurons). We make two observations: across all datasets and all models we find that a loss in accuracy (compared to the full layer) comes at the cost of a few classes, as opposed to being smeared throughout classes, as indicated by high gini values on the left of each plot. Additionally, we observe that the model trained using MRL loss tends to have slightly higher gini values in the regions where the drop in accuracy is slightly higher (highlighted on the plots). To ensure that this trend is not simply due to lower accuracy, we investigate the error distributions across classes (Appendix C) and find that predictions become more homogeneous as we drop more neurons. Similar trends are also observed with coeff. of variation as shown in Appendix C. These results draw caution to the potential unintended side-effects of exploiting diffused redundancy and suggest that there could be a possible fairness-efficiency tradeoff involved.

Connections to Robustness-Fairness Tradeoffs There are well-established tradeoffs between robustness and fairness in both standard [36] and adversarial training [62, 58]. One major difference between these works and our setup is that the task they train on is also the task they test on. We instead operate in the transfer learning setup where we train a linear probe on the representation of a pre-trained model. While it's not immediately clear whether the discrepancy between class accuracies observed on the pre-training task ( e.g. as in [62]) also leads to an inter-class accuracy discrepancy on downstream tasks when dropping neurons, it would be very interesting future work to establish more formal connections between the robustness-fairness tradeoff and its effects on the fairness-efficiency tradeoff presented in this paper.

5 Conclusion and Broader Impacts

We introduce the diffused redundancy hypothesis and analyze a wide range of models with different upstream training datasets, losses, and architectures. We carefully analyze the causes of such redundancy and find that upstream training (both loss and datasets) plays a crucial role and that this redundancy also depends on the downstream dataset. One direct practical consequence of our observation is increased efficiency for downstream training times which can have many positive impacts in terms of reduced energy costs [54] which is crucial in moving towards 'green' AI [48]. We, however, also draw caution to potential pitfalls of such efficiency gains , which might hurt the accuracy of certain classes more than others, thus having direct consequences for fairness.

References

Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. arXiv preprint arXiv:1706.01350 , 125:126-127, 2017.

Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018.

Fred Attneave. Some informational aspects of visual perception. Psychological review , 61(3):183, 1954.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791 , 2019.

Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. arXiv preprint arXiv:1511.06068 , 2015.

Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 4908-4926, Online, November 2020. Association for Computational Linguistics.

Fernando G De Maio. Income inequality measures. Journal of Epidemiology & Community Health , 61(10):849-852, 2007.

Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layerwise optimal brain surgeon. Advances in Neural Information Processing Systems , 30, 2017.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread , 2022.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Corrado Gini. Measurement of inequality of incomes. The economic journal , 31(121):124126, 1921.

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems , 28, 2015.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016.

Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems , 5, 1992.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision , pages 1389-1397, 2017.

Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum´ e III, Miro Dudik, and Hanna Wallach. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems , pages 1-16, 2019.

Yoshio Izui and Alex Pentland. Analysis of neural networks with redundancy. Neural Computation , 2(2):226-238, 1990.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning , pages 5338-5348. PMLR, 2020.

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision , pages 491-507. Springer, 2020.

Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. 2021.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning , pages 3519-3529. PMLR, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representations for adaptive deployment. arXiv preprint arXiv:2205.13147 , 2022.

Barbara S Lawrence. Perspective-the black box of organizational demography. Organization science , 8(1):1-22, 1997.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature , 521(7553):436444, 2015.

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems , 2, 1989.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 , 2018.

Asriel Levin, Todd Leen, and John Moody. Fast pruning using principal components. Advances in neural information processing systems , 6, 1993.

Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations , 2018.

Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 , 2016.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.

Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them, 2014.

Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. Fairness through robustness: Investigating robustness disparity in deep learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 466-477, 2021.

Vedant Nanda, Till Speicher, Camilla Kolling, John P. Dickerson, Krishna P. Gummadi, and Adrian Weller. Measuring representational robustness of neural networks through shared invariances. In ICML , 2022.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , Dec 2008.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill , 2020. https://distill.pub/2020/circuits/zoom-in.

Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill , 2017. https://distill.pub/2017/feature-visualization.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering , 22(10):1345-1359, 2009.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40):24652-24663, 2020.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning , pages 5389-5400. PMLR, 2019.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems , 33:3533-3545, 2020.

Robert R Schutz. On the measurement of income inequality. The American Economic Review , 41(1):107-122, 1951.

Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM , 63(12):54-63, 2020.

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. A unified approach to quantifying algorithmic unfairness: Measuring individual &group unfairness via inequality indices. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 22392248, 2018.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.

Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 , 2021.

Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David R. Traum, and Llu´ ıs M` arquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 3645-3650. Association for Computational Linguistics, 2019.

Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks , pages 270-279. Springer, 2018.

Song¨ ul Tolan, Marius Miron, Emilia G´ omez, and Carlos Castillo. Why machine learning may lead to unfairness: Evidence from risk assessment for juvenile justice in catalonia. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law , pages 83-92, 2019.

Michael Veale, Max Van Kleek, and Reuben Binns. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In Proceedings of the 2018 chi conference on human factors in computing systems , pages 1-14, 2018.

Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8193-8201, 2023.

Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models , 2019.

Ross Wightman, Hugo Touvron, and Herv´ e J´ egou. Resnet strikes back: An improved training procedure in timm. abs/2110.00476, 2021.

Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning , pages 1120511216. PMLR, 2021.

Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International conference on machine learning , pages 1149211501. PMLR, 2021.

Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization, 2015.

Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1803-1811, 2019.

Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang. Slimmable neural networks. abs/1812.08928, 2018.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics , pages 962-970. PMLR, 2017.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, 2013.

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04867 , 2019.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2017.

A Measuring Diffused Redundancy

A.1 CKA Definition

In all our evaluations we use CKA with a linear kernel [24] which essentially amounts to the following steps:

Take two representations Y ∈ R n × d 1 and Z ∈ R n × d 2

Compute dot product similarity within these representation, i.e. compute K = Y Y T , L = ZZ T

Normalize K and L to get K ′ = HKH , L ′ = HLH where H = I n -1 n 11 T

Return CKA ( Y, Z ) = HSIC ( K,L ) √ HSIC ( K,K ) HSIC ( L,L ) , where HSIC ( K,L ) = 1 ( n -1) 2 ( flatten ( K ′ ) · flatten ( L ′ ))

We use the publicly available implementation of [37], which provides an implementation that can be calcuated over multiple mini-batches: https://github.com/nvedant07/STIR

A.2 Additional CKA results

Fig 9 shows CKA comparison between randomly chosen parts of the layer and the full layer for different kinds of ResNet50. We observe that even ResNet50 trained with MRL loss shows a significant amount of diffused redundancy.

0.0

0.1

0.2

0.3

0.4

0.5

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50_mrl-nonrob-first

resnet50_mrl-nonrob-random

(a) CIFAR10

0.0

0.1

0.2

0.3

0.4

0.5

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50_mrl-nonrob-first

resnet50_mrl-nonrob-random

(b) CIFAR100

0.0

0.1

0.2

0.3

0.4

0.5

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50_mrl-nonrob-first

resnet50_mrl-nonrob-random

(c) Flowers

0.0

0.1

0.2

0.3

0.4

0.5

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

CKA

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50_mrl-nonrob-first

resnet50_mrl-nonrob-random

(d) Oxford-IIIT-Pets

B Training and Pre-Processing Details for Reproducibility

Here we list the sources of weights for the various pre-trained models used in our experiments:

ResNet18 trained on ImageNet1k using standard loss: taken from timm v0.6.1.

ResNet18 trained on ImageNet1k with adv training: taken from Salman et al. [46]:

ResNet50 trained on ImageNet1k using standard loss: taken from timm v0.6.1.

ResNet50 trained on ImageNet1k with adv training: taken from Salman et al. [46]: https: //github.com/microsoft/robust-models-transfer .

ResNet50 trained on ImageNet1k using MRL and with different final layer widths ( resnet50 ffx ): taken from released weights of by Kusupati et al. [26]: https:// github.com/RAIVNLab/MRL .

WideResNet50-2 on ImageNet1k both standard and avd. training: taken from Salman et al. [46]: https://github.com/microsoft/robust-models-transfer .

VGG16 trained on ImageNet1k with standard loss: taken from timm v0.6.1.

VGG16 trained on ImageNet1k with adv training: taken from Salman et al. [46]: https: //github.com/microsoft/robust-models-transfer .

ViTS32 & ViTS16 trained on ImageNet21k & ImageNet1k: taken from weights released by Steiner et al. [53]: https://github.com/google-research/vision_transformer .

All linear probes trained on the representations of these models are trained using SGD with a learning rate of 0 . 1 , momentum of 0 . 9 , batch size of 256 , weight decay of 1 e -4 . The probe is trained for 50 epochs with a learning rate scheduler that decays the learning rate by 0 . 1 every 10 epochs. Scripts for training can also be found in the attached code.

For pre-processing, we re-size all inputs to 224x224 (size used for pre-training) and apply the usual composition of RandomHorizontalFlip, ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25), RandomRotation(degrees=2). All inputs were mean normalized. For imagenet1k pre-trained models: mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. For imagenet21k pre-trained models: mean = [0.5,0.5,0.5], std = [0.5,0.5,0.5].

C Deeper Analysis of Fairness-Efficiency Tradeoff in Section 4

Analyzing Error Distributions To ensure that the higher gini coefficient shown in Fig 8 as we drop more neurons is not merely an artifact of lower overall accuracy, we plot class-wise accuracies as we drop neurons (Figs 11, 12 & 13). We find that for the entire layer, accuracy starts at an almost uniform distribution, and while overall accuracy deteriorates as we drop neurons, the drop comes at a larger cost for a few classes resulting in disparate inter-class accuracies.

Coeff of Variation for Measuring Inequality in Inter-Class Accuracy Fig 10 shows results for the same analysis shown in Fig 8 of the main paper and we find similar takeaways even when using the coefficient of variation as a measure of inequality.

D Corresponding Diffused Redundancy Estimates For Analyses in Section 3 & ℓ ∞ Robust Model Results

Corresponding diffused redundancy (DR) ablations for Figures 4,5,7. These are shown in Figures 15,16,17 respectively. This should allow for easy comparison of diffused redundancy (lines that are more outside have higher DR). For example, Figure 16 clearly shows higher diffused redundancy in models trained on larger upstream datasets (here ImageNet21k) since these curves lie more on the outside of the same model's curves for ImageNet1k.

Additionally, we show numbers on x-axis for Figure 4 in Figure 18. Figures 5 and 7 compare models with same number of neurons in the final layer and hence trends shown with fraction on the x-axis will be exactly the same with absolute numbers on the x-axis. However, Figure 18 allows a direct comparison of the performance of the same absolute number of neurons across different models.

We report results for ℓ ∞ robust models (with ϵ = 4 / 255 ) in Fig 14 and find that ℓ 2 model generally shows a greater degree of diffused redundancy.

E Results on Intermediate Layers

We additionally ran our experiment on other intermediate layers and report the results in Fig 19. We present results for a ResNet50 pretrained on ImageNet1k using the standard CrossEntropy loss. The

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0.0

0.2

0.4

0.6

0.8

1.0

Coeff. of Variation

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(a) CIFAR10

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.2

0.4

0.6

0.8

1.0

Coeff. of Variation

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(b) CIFAR100

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.2

0.4

0.6

0.8

1.0

Coeff. of Variation

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(c) Flowers

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

0.8

1.0

Coeff. of Variation

resnet50-nonrob

resnet50-robustl2eps3

resnet50_mrl-nonrob-first

(d) Oxford-IIIT-Pets

intermediate layers considered are characterized as activations following each residual connection within distinct ResNet blocks. layerX.Y.act3 means the Y th residual connection in the Xth ResNet block and act3 indicates that we're taking the value after the activation (ReLU) has been applied.

F Results on Harder Downstream Tasks: ImageNetV2 and Places365

We report results on harder downstream tasks such as ImageNet1k, ImageNetV2, and Places365 in Figure 20. We find that when randomly dropping neurons, the model is still able to generalize to ImageNet1k and Places365 with very few neurons, i.e. , the phenomena of diffused redundancy observed for smaller datasets, also holds for harder datasets. Interestingly we also observe that the accuracy gap between ImageNet1k and ImageNetV2 is maintained even as we drop neurons.

G Effects of Explicitly Preventing Co-adaptation of Neurons: Analysis of Dropout and DeCov

Regularizers such as dropout and DeCov, force different parts of the representations to not be correlated. Thus these regularizers can be seen as explicitly requiring different, compact parts of the representation to be self-contained for the downstream task. Thus, intuitively, such methods should increase diffused redundancy. Here we investigate if our observation about diffused redundancy is influenced by such regularizers. We evaluate ResNet50 pre-trained on ImageNet1k with dropout in the penultimate layer ranging from a strength of 0.1 all the way to 0.8. We also train another ResNet50 model with the DeCov regularizer added to the usual crossentropy loss and put a weight of 0 . 0001 on the regularizer to ensure that its numerical range is similar to that of the cross entropy loss term.

Results in Figures 21 & 22 suggest that such regularizers have almost no effect on diffused redundancy. Trends across datasets remain consistent regardless of the strength of dropout or the weight given to DeCov regularizer. This observation further adds to the evidence that diffused redundancy is likely to be a natural property of representations learned by DNNs.

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

Classwise Accuracies (mean)

0.08

0.06

0.30

0.42

0.38

0.39

0.39

0.50

0.63

0.65

0.73

0.74

0.80

0.81

0.85

0.83

0.88

0.87

0.90

0.91

0.92

0.92

0.00

0.02

0.02

0.15

0.22

0.23

0.36

0.56

0.68

0.73

0.79

0.86

0.87

0.89

0.91

0.90

0.92

0.91

0.93

0.94

0.94

0.94

0.19

0.09

0.28

0.31

0.34

0.33

0.40

0.45

0.54

0.55

0.62

0.63

0.71

0.70

0.76

0.75

0.79

0.78

0.81

0.83

0.84

0.83

0.17

0.26

0.17

0.13

0.16

0.17

0.23

0.27

0.36

0.40

0.50

0.49

0.59

0.60

0.69

0.65

0.72

0.71

0.76

0.79

0.79

0.80

0.01

0.17

0.19

0.19

0.24

0.24

0.27

0.32

0.40

0.44

0.57

0.58

0.68

0.69

0.77

0.75

0.80

0.79

0.83

0.85

0.85

0.86

0.06

0.04

0.07

0.23

0.23

0.30

0.38

0.43

0.55

0.62

0.68

0.72

0.75

0.77

0.80

0.80

0.82

0.83

0.85

0.86

0.86

0.86

0.01

0.34

0.40

0.33

0.36

0.35

0.51

0.65

0.68

0.72

0.77

0.81

0.86

0.87

0.91

0.91

0.92

0.92

0.94

0.95

0.95

0.95

0.10

0.09

0.10

0.25

0.25

0.29

0.39

0.62

0.64

0.69

0.74

0.74

0.80

0.82

0.85

0.84

0.87

0.87

0.89

0.89

0.90

0.90

0.00

0.03

0.14

0.09

0.12

0.18

0.37

0.53

0.58

0.61

0.73

0.79

0.84

0.85

0.87

0.87

0.89

0.90

0.92

0.93

0.93

0.93

0.60

0.42

0.19

0.22

0.22

0.26

0.40

0.49

0.66

0.74

0.82

0.83

0.87

0.87

0.90

0.89

0.91

0.91

0.91

0.92

0.93

0.93

0.0

0.2

0.4

0.6

0.8

1.0

(a) CIFAR10, Standard

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

airplane

automobile

bird

cat

deer

dog

frog

horse

ship

truck

Classwise Accuracies (mean)

0.11

0.02

0.14

0.14

0.24

0.31

0.48

0.62

0.71

0.76

0.77

0.81

0.84

0.85

0.87

0.87

0.88

0.88

0.90

0.90

0.91

0.91

0.00

0.04

0.19

0.28

0.38

0.37

0.61

0.73

0.79

0.82

0.84

0.86

0.88

0.89

0.90

0.90

0.91

0.91

0.92

0.92

0.92

0.92

0.14

0.11

0.04

0.04

0.09

0.11

0.27

0.42

0.52

0.57

0.63

0.65

0.72

0.73

0.78

0.78

0.80

0.80

0.82

0.84

0.84

0.84

0.01

0.03

0.07

0.17

0.17

0.16

0.27

0.38

0.45

0.49

0.54

0.57

0.61

0.64

0.66

0.67

0.69

0.69

0.72

0.73

0.73

0.73

0.00

0.02

0.16

0.14

0.16

0.21

0.32

0.42

0.52

0.61

0.65

0.71

0.76

0.78

0.81

0.81

0.82

0.83

0.84

0.85

0.85

0.86

0.14

0.27

0.20

0.24

0.32

0.36

0.50

0.63

0.64

0.68

0.71

0.74

0.76

0.77

0.79

0.79

0.81

0.81

0.82

0.83

0.83

0.83

0.34

0.32

0.27

0.39

0.45

0.48

0.56

0.74

0.80

0.82

0.84

0.88

0.89

0.90

0.91

0.91

0.92

0.93

0.93

0.93

0.93

0.94

0.18

0.41

0.35

0.36

0.34

0.38

0.50

0.64

0.69

0.74

0.78

0.82

0.84

0.86

0.87

0.87

0.88

0.87

0.88

0.89

0.90

0.89

0.07

0.02

0.15

0.17

0.20

0.37

0.53

0.71

0.79

0.86

0.88

0.90

0.90

0.92

0.92

0.93

0.92

0.93

0.93

0.93

0.93

0.93

0.33

0.34

0.65

0.61

0.59

0.54

0.60

0.73

0.78

0.78

0.81

0.85

0.87

0.88

0.89

0.89

0.90

0.90

0.92

0.92

0.92

0.92

0.0

0.2

0.4

0.6

0.8

1.0

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

apple

aquarium_fish

baby

bear

beaver

bed

bee

beetle

bicycle

bottle

bowl

boy

bridge

bus

butterfly

camel

can

castle

caterpillar

cattle

chair

chimpanzee

clock

cloud

cockroach

couch

crab

crocodile

cup

dinosaur

dolphin

elephant

flatfish

forest

fox

girl

hamster

house

kangaroo

keyboard

lamp

lawn_mower

leopard

lion

lizard

lobster

man

maple_tree

motorcycle

mountain

mushroom

oak_tree

orange

orchid

otter

palm_tree

pear

pickup_truck

pine_tree

plain

plate

poppy

porcupine

possum

rabbit

raccoon

ray

road

rocket

rose

sea

seal

shark

shrew

skunk

skyscraper

snail

snake

spider

squirrel

streetcar

sunflower

sweet_pepper

table

tank

telephone

television

tiger

tractor

train

trout

tulip

turtle

wardrobe

whale

willow_tree

woman

worm

Classwise Accuracies (mean)

0.000.000.120.110.120.120.120.270.290.320.450.480.610.600.720.700.780.810.830.850.840.84

0.000.000.000.020.020.010.180.170.240.230.370.350.480.480.600.610.670.670.700.740.750.73

0.000.000.000.000.000.000.040.140.170.190.340.380.460.430.520.490.540.550.560.580.590.56

0.000.000.000.000.000.000.050.020.030.030.130.100.220.160.350.270.400.340.510.580.600.62

0.000.000.000.030.040.010.010.010.010.020.080.040.100.100.190.160.260.230.370.450.470.47

0.000.000.010.030.030.010.050.070.090.090.210.190.400.330.540.480.580.570.630.660.660.68

0.000.000.000.000.000.020.020.040.060.070.110.080.260.190.360.310.510.500.600.620.630.63

0.000.000.000.000.040.050.110.100.090.120.240.170.410.330.610.510.660.610.700.730.730.71

0.000.000.000.060.070.090.140.070.060.100.320.230.540.480.710.730.800.810.840.860.870.85

0.000.000.000.000.000.000.070.060.250.320.590.580.710.700.760.760.760.780.790.800.810.79

0.000.000.000.000.000.000.020.020.010.030.080.070.250.250.410.320.450.430.520.560.580.61

0.000.000.000.000.000.000.040.030.090.040.170.150.270.280.350.350.390.390.430.440.450.45

0.000.110.100.080.090.080.030.040.040.100.210.280.540.500.630.580.690.660.750.790.800.79

0.000.000.000.000.000.000.000.070.050.170.260.180.400.240.450.360.520.470.640.670.660.66

0.000.000.000.000.000.000.020.040.120.110.200.250.400.330.500.380.530.460.620.690.710.68

0.000.000.000.000.010.020.030.060.060.070.260.270.410.360.500.380.610.580.680.710.720.71

0.000.000.020.020.020.020.040.110.320.320.470.410.620.620.740.680.770.780.800.810.810.80

0.030.100.190.130.130.090.120.330.330.400.550.460.630.630.720.740.740.740.790.800.810.83

0.000.000.000.000.000.000.040.070.150.210.360.390.490.510.570.580.590.620.620.660.670.68

0.080.050.040.040.060.030.040.170.220.260.310.390.520.530.620.590.650.630.710.750.760.77

0.010.000.080.060.160.210.380.310.550.580.750.720.800.850.840.860.850.870.850.870.870.88

0.420.150.210.110.070.060.160.290.300.390.520.510.680.660.730.730.740.770.790.810.810.80

0.000.000.150.230.180.230.130.370.510.500.530.520.630.610.700.680.750.720.790.800.800.80

0.000.000.020.040.050.020.090.050.140.130.230.170.270.230.360.330.450.450.530.560.560.56

0.000.040.010.000.040.010.040.170.290.340.490.560.740.800.770.800.770.800.790.790.800.82

0.000.000.010.010.010.010.010.000.010.010.030.000.040.030.230.090.340.210.450.540.580.55

0.000.020.030.030.020.020.120.090.100.100.190.180.260.200.400.360.500.450.550.590.600.60

0.000.000.000.000.000.020.010.020.020.070.120.100.190.170.290.250.360.310.430.480.510.47

0.000.000.000.010.090.090.230.260.270.320.450.520.640.740.750.780.760.790.790.840.840.84

0.000.000.000.000.000.000.000.050.170.210.290.310.440.440.550.510.560.550.610.660.660.66

0.000.000.010.020.030.030.060.020.060.090.180.190.280.240.360.330.420.400.440.470.490.47

0.000.000.000.000.000.000.000.010.060.080.140.150.390.360.590.550.630.670.690.700.710.70

0.000.000.000.000.000.000.000.000.000.000.010.000.050.050.160.090.260.190.390.470.490.49

0.000.000.000.020.030.010.020.030.090.120.150.230.230.250.270.270.330.310.400.460.470.46

0.000.000.000.000.000.020.020.290.230.260.400.420.540.540.620.680.680.680.710.740.750.73

0.000.000.000.000.000.000.070.040.040.040.110.100.190.230.290.310.320.330.370.410.420.40

0.000.000.000.130.140.140.240.230.350.390.430.470.620.720.700.760.700.740.700.710.720.71

0.000.010.000.010.000.050.110.080.110.150.200.200.280.190.440.370.530.490.620.660.660.65

0.000.000.050.080.100.100.090.100.140.120.220.180.290.260.390.380.500.450.560.570.560.56

0.140.140.140.140.150.180.340.370.430.430.600.610.730.740.780.760.820.800.840.860.870.87

0.000.000.000.000.000.000.020.030.020.030.080.200.340.280.490.410.560.540.590.620.630.61

0.000.040.030.030.130.120.310.350.390.370.550.520.650.610.720.670.760.770.800.800.800.80

0.000.000.090.070.040.020.010.010.070.150.330.330.470.410.590.600.640.640.680.760.780.79

0.000.000.030.000.050.030.190.120.240.290.410.560.590.650.630.630.640.670.680.700.700.70

0.000.000.010.010.000.000.010.020.010.020.070.110.190.140.320.240.450.450.520.570.560.57

0.000.040.020.030.030.020.040.040.060.120.140.140.140.150.210.170.260.230.370.440.460.47

0.000.000.000.000.050.040.060.050.080.180.250.340.500.530.560.550.560.580.620.650.640.64

0.000.000.000.010.010.010.020.000.010.060.120.070.220.170.350.260.390.340.420.450.450.44

0.000.000.020.190.310.300.260.330.480.460.540.590.610.720.740.770.760.750.780.840.850.87

0.000.000.000.000.020.040.060.040.050.070.410.340.570.550.690.670.720.720.770.790.790.79

0.000.000.010.000.000.000.000.000.000.000.000.000.020.050.150.080.240.140.320.360.390.40

0.000.000.000.000.000.000.040.160.180.260.320.340.500.520.620.580.660.630.690.730.730.71

0.000.000.000.020.090.110.170.260.290.280.350.380.370.430.550.610.620.630.620.630.640.66

0.000.020.020.020.020.070.100.280.450.460.620.760.800.780.890.890.930.920.960.970.980.98

0.000.000.020.070.080.050.300.440.390.400.490.580.600.580.640.660.710.720.760.810.820.81

0.000.000.000.000.000.000.000.010.000.000.030.010.030.010.080.040.140.090.240.310.320.28

0.000.000.000.000.010.010.080.040.100.110.290.160.350.420.600.490.630.570.710.780.790.80

0.000.000.000.000.000.000.000.060.130.200.220.200.320.240.480.410.580.530.660.710.720.73

0.000.000.000.000.000.000.100.110.310.390.610.710.750.810.730.790.730.740.720.750.750.74

0.000.000.000.000.000.000.050.040.040.040.060.030.080.090.190.120.250.190.350.430.430.41

0.000.000.000.000.000.120.190.230.460.500.490.570.530.520.600.620.640.620.670.680.690.68

0.090.080.110.150.160.170.370.320.370.410.500.520.610.700.650.720.660.690.650.650.660.66

0.000.030.040.050.040.030.120.130.220.300.310.360.430.470.540.500.560.540.630.700.690.66

0.000.000.010.010.040.020.030.000.070.090.170.110.210.400.470.420.490.460.560.610.620.60

0.000.000.030.000.010.000.020.020.040.050.080.060.150.170.300.230.360.310.430.470.470.46

0.000.000.000.000.000.000.000.000.000.000.000.000.110.110.250.190.340.260.490.570.590.59

0.000.130.000.130.080.080.060.050.100.100.190.220.330.280.390.350.460.440.530.590.610.59

0.000.000.070.020.000.000.050.010.010.020.050.040.160.080.270.220.330.280.430.490.510.51

0.000.150.150.140.140.150.160.160.190.190.250.260.510.450.720.710.760.770.840.870.870.87

0.000.000.000.060.000.000.090.180.200.240.470.460.600.590.650.640.690.680.720.730.720.72

0.000.000.000.060.060.040.110.130.100.180.240.340.410.430.600.540.630.600.690.740.750.77

0.000.160.030.020.210.100.230.450.420.400.440.480.460.390.570.530.620.610.670.690.710.71

0.000.000.160.290.290.280.270.200.240.220.290.170.240.130.230.170.280.230.330.350.360.35

0.000.000.000.000.000.000.000.120.260.250.270.400.390.400.380.380.430.430.500.540.560.55

0.000.230.000.040.060.030.020.010.010.070.090.090.140.120.190.120.240.190.300.340.360.34

0.000.000.000.000.010.000.090.050.180.220.410.400.610.520.690.650.750.750.800.820.830.82

0.010.010.040.010.010.170.040.050.110.110.280.180.430.390.600.570.680.660.750.770.780.77

0.000.000.000.000.000.000.000.160.170.170.200.320.380.340.500.430.570.510.610.650.650.63

0.000.000.000.000.000.010.040.130.090.120.280.320.490.610.680.690.710.720.740.750.750.73

0.000.000.000.000.000.000.100.120.110.130.250.200.450.470.560.520.610.570.600.630.640.63

0.240.210.080.130.070.040.030.110.100.090.150.120.210.170.360.280.390.350.420.460.470.48

0.080.040.050.080.060.040.100.200.240.410.400.430.450.440.520.450.610.590.660.700.700.71

0.020.250.300.060.040.170.300.350.340.340.530.600.730.790.820.830.850.850.870.880.880.89

0.010.000.010.060.010.060.020.050.040.140.240.210.380.480.570.540.630.610.660.690.690.69

0.000.020.060.050.050.100.030.030.120.150.150.130.180.180.310.200.400.300.550.630.630.61

0.000.000.000.000.010.040.010.030.060.050.200.080.330.380.550.450.620.580.670.690.710.71

0.000.000.240.220.250.240.420.520.480.540.570.600.700.700.730.710.750.750.790.820.820.82

0.000.000.010.040.020.030.160.130.270.290.460.480.670.720.820.780.830.790.860.880.890.87

0.000.000.000.010.010.010.090.150.260.290.430.430.520.510.610.570.610.600.640.670.670.67

0.000.000.000.000.000.070.090.130.190.220.370.260.570.510.770.650.810.750.840.860.860.87

0.000.000.000.000.000.000.000.000.000.000.080.040.190.110.390.250.460.440.610.660.670.67

0.000.000.000.010.000.000.010.130.110.110.170.190.310.250.450.310.520.450.620.680.680.67

0.000.000.000.000.000.000.040.060.090.140.180.180.270.270.390.320.440.400.470.530.520.53

0.000.000.000.000.000.000.030.030.060.060.130.080.230.140.300.190.360.290.460.560.560.53

0.170.190.230.220.300.340.420.610.620.590.580.640.660.700.690.730.740.720.780.830.840.83

0.160.000.100.070.080.090.130.220.220.250.350.430.500.520.570.590.610.610.640.660.660.63

0.000.020.060.150.120.250.150.090.140.180.120.100.130.100.130.170.180.180.280.320.310.30

0.000.010.090.050.070.110.110.150.190.250.310.290.420.380.560.550.630.620.700.710.710.72

0.000.000.000.000.000.000.010.010.010.020.100.050.170.170.310.240.360.320.450.530.550.55

0.000.000.000.010.010.010.130.130.170.290.500.460.650.640.720.710.740.740.790.810.810.81

0.0

0.2

0.4

0.6

0.8

1.0

(c) CIFAR100, Standard

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

apple

aquarium_fish

baby

bear

beaver

bed

bee

beetle

bicycle

bottle

bowl

boy

bridge

bus

butterfly

camel

can

castle

caterpillar

cattle

chair

chimpanzee

clock

cloud

cockroach

couch

crab

crocodile

cup

dinosaur

dolphin

elephant

flatfish

forest

fox

girl

hamster

house

kangaroo

keyboard

lamp

lawn_mower

leopard

lion

lizard

lobster

man

maple_tree

motorcycle

mountain

mushroom

oak_tree

orange

orchid

otter

palm_tree

pear

pickup_truck

pine_tree

plain

plate

poppy

porcupine

possum

rabbit

raccoon

ray

road

rocket

rose

sea

seal

shark

shrew

skunk

skyscraper

snail

snake

spider

squirrel

streetcar

sunflower

sweet_pepper

table

tank

telephone

television

tiger

tractor

train

trout

tulip

turtle

wardrobe

whale

willow_tree

woman

worm

Classwise Accuracies (mean)

0.000.000.020.010.050.050.320.440.570.640.700.730.790.800.830.820.840.840.850.860.850.86

0.060.060.050.050.090.090.230.310.480.510.560.600.650.630.700.690.730.720.750.760.770.75

0.050.070.060.080.100.100.190.250.260.340.430.420.530.520.580.580.600.610.600.620.620.64

0.000.000.000.000.000.000.030.030.100.230.300.330.430.420.490.430.530.460.530.540.540.53

0.000.000.000.000.000.000.030.010.060.080.190.180.280.260.340.310.380.350.430.470.490.49

0.000.000.010.030.050.050.120.180.270.340.480.460.560.550.630.580.650.620.670.680.690.70

0.000.000.050.050.040.100.090.120.190.340.490.500.600.600.660.650.690.660.730.740.750.74

0.010.020.070.170.250.220.310.350.570.590.690.650.710.700.770.730.780.780.810.810.840.86

0.000.000.000.000.000.010.050.110.290.390.540.560.660.670.720.710.740.720.750.760.770.76

0.000.140.140.190.170.170.300.570.670.680.700.720.740.760.770.770.770.780.780.780.780.78

0.000.000.000.000.000.010.080.150.280.300.410.400.450.440.460.460.450.470.460.450.460.47

0.000.010.030.060.060.080.230.230.330.330.350.370.390.410.430.430.460.450.490.490.500.49

0.000.000.000.010.000.000.010.260.310.380.520.540.650.640.730.700.760.710.780.800.800.80

0.000.040.100.110.100.120.200.270.290.340.390.400.470.450.520.480.550.510.560.580.600.57

0.000.000.070.050.080.090.170.170.330.430.500.550.650.650.710.680.730.710.740.750.750.75

0.000.000.000.000.000.000.050.070.200.260.430.480.600.590.660.650.680.650.690.710.710.72

0.000.000.000.020.030.040.280.440.610.610.670.670.740.730.780.750.810.780.810.830.840.86

0.000.170.130.070.120.140.310.360.540.630.650.700.720.730.730.750.750.740.750.750.750.75

0.000.000.020.030.040.110.260.330.410.420.430.490.530.560.590.590.590.610.620.620.620.63

0.010.000.010.000.000.010.030.150.250.370.470.480.560.570.620.610.640.610.660.670.670.67

0.000.120.080.160.170.380.540.780.830.820.800.830.810.830.840.850.840.840.830.830.830.84

0.140.070.030.030.130.200.380.580.640.680.690.740.770.790.810.810.820.810.810.840.850.84

0.000.000.020.050.070.070.240.430.550.560.590.610.680.680.730.720.750.740.760.790.780.78

0.000.060.030.080.160.180.270.320.410.390.530.530.640.640.660.660.660.670.670.670.680.67

0.130.130.260.250.350.420.550.760.770.740.720.740.740.750.750.760.760.760.760.750.760.76

0.000.000.000.000.000.000.070.080.100.150.310.280.440.400.530.460.560.540.580.580.560.57

0.010.060.060.090.030.040.160.260.290.370.450.470.540.560.620.590.670.650.700.710.720.71

0.000.000.010.010.010.010.030.030.090.190.320.340.430.430.490.490.540.550.540.530.520.54

0.000.010.020.020.060.080.390.510.590.590.650.700.720.730.780.760.790.790.770.790.790.79

0.000.020.050.070.020.160.240.390.420.470.520.500.540.540.600.580.620.600.650.660.680.67

0.000.000.010.020.050.050.240.260.330.360.380.390.460.450.510.470.520.490.520.530.550.54

0.000.000.000.010.010.000.050.120.200.270.360.400.530.520.610.560.630.610.640.660.660.66

0.030.000.000.010.000.000.010.010.060.120.270.260.400.360.480.410.510.500.520.550.560.54

0.000.120.150.140.160.170.100.140.290.340.390.400.450.470.510.480.530.490.540.560.590.57

0.020.000.000.100.140.120.230.280.400.480.550.570.660.630.720.680.740.720.750.750.750.74

0.000.000.010.010.030.020.060.060.080.080.190.170.310.280.380.310.400.370.440.440.450.46

0.000.000.050.100.100.100.220.300.370.430.450.460.520.520.550.520.570.540.590.600.600.60

0.000.000.050.030.000.000.100.160.260.300.420.430.570.570.620.640.640.650.690.690.690.66

0.000.000.000.000.000.000.030.090.160.240.360.350.450.430.490.480.500.520.540.520.540.56

0.000.090.080.090.020.050.350.500.570.630.690.700.740.750.780.780.790.780.800.800.810.80

0.000.000.000.000.000.000.090.260.390.410.510.520.600.610.670.650.680.680.700.690.690.68

0.000.000.110.200.310.300.570.630.710.710.750.780.790.800.820.820.830.820.840.840.840.84

0.000.000.000.000.010.010.070.140.350.400.460.530.560.590.610.620.630.640.650.640.650.64

0.000.000.020.040.120.140.280.350.440.470.470.500.570.540.590.570.600.580.600.620.620.62

0.000.000.000.000.000.000.070.150.180.230.320.390.460.440.500.490.530.510.560.580.580.57

0.000.000.010.020.000.000.030.090.160.210.250.270.340.320.430.400.470.410.490.520.530.55

0.170.320.250.210.190.220.270.430.480.540.500.540.520.540.520.540.520.510.500.500.500.51

0.000.020.010.010.010.020.060.050.180.250.350.360.420.390.490.440.530.470.540.560.580.59

0.030.040.150.180.220.190.360.460.600.680.690.700.750.750.800.780.810.780.840.850.850.85

0.000.000.000.010.000.020.160.230.460.540.690.710.800.810.830.830.830.840.840.840.840.83

0.010.000.000.000.000.000.000.010.020.020.140.110.230.220.310.260.350.300.370.400.420.40

0.000.000.010.040.050.080.110.240.270.330.430.470.620.600.710.680.730.690.760.770.780.77

0.150.000.000.010.170.220.490.480.560.610.710.730.710.720.710.720.710.720.710.690.680.66

0.000.010.070.100.160.270.450.620.700.750.830.840.860.880.890.900.900.920.930.920.930.92

0.000.020.040.100.080.150.250.330.390.510.610.570.670.680.720.700.740.750.760.740.750.74

0.000.000.000.000.000.000.000.010.020.040.100.130.210.190.270.220.300.280.310.310.310.32

0.000.000.010.030.000.050.110.210.370.370.460.480.600.590.640.620.660.630.680.720.730.73

0.000.000.010.000.010.010.080.190.300.340.460.470.630.600.700.680.720.700.720.730.730.73

0.000.000.030.040.150.120.410.550.590.600.620.660.700.720.760.730.780.770.790.800.800.79

0.000.010.000.010.010.020.040.050.070.100.230.210.380.350.470.420.510.480.530.530.550.53

0.000.000.070.050.010.050.310.420.570.630.710.730.800.790.840.830.850.860.860.870.870.88

0.000.000.000.010.050.050.190.340.440.470.520.540.580.600.610.610.640.640.640.670.650.66

0.000.050.110.130.180.270.380.480.510.500.530.570.640.610.690.670.710.690.740.750.750.75

0.000.000.000.000.000.000.040.020.080.190.290.250.430.360.480.420.520.500.540.570.560.56

0.030.000.000.040.020.020.080.100.160.240.240.290.330.300.380.340.400.400.410.430.440.42

0.000.000.000.000.000.000.030.020.040.100.180.150.290.260.350.320.390.340.410.440.430.44

0.000.000.000.000.030.030.090.140.280.370.430.430.530.520.580.580.600.590.620.650.640.64

0.000.000.070.050.040.080.140.240.280.340.380.400.480.450.530.510.550.540.560.570.570.57

0.000.000.020.040.060.050.180.290.410.550.710.670.790.760.830.800.850.820.860.850.860.86

0.000.000.040.120.090.120.210.400.520.640.670.700.750.790.780.790.790.790.780.780.780.78

0.000.300.210.230.220.200.240.300.400.470.620.620.670.690.720.730.730.730.740.740.740.75

0.000.020.040.030.050.080.120.310.420.500.590.650.690.720.720.730.740.730.720.730.730.71

0.000.000.000.000.000.000.000.010.030.060.140.130.220.220.300.260.340.280.380.430.430.44

0.000.000.070.130.260.240.290.300.380.410.440.420.510.460.580.490.620.560.640.650.630.66

0.000.050.000.010.010.030.050.020.070.130.230.230.300.310.380.370.410.390.440.440.440.42

0.000.000.000.130.370.410.540.650.720.750.790.790.830.850.860.850.870.870.870.890.900.89

0.010.050.060.110.190.190.240.400.430.480.580.560.670.660.730.710.760.760.800.800.800.79

0.000.000.000.000.000.000.010.040.090.170.340.370.480.460.540.510.560.560.560.570.560.56

0.180.030.070.220.240.250.320.380.400.490.610.610.650.660.680.670.680.680.680.690.700.70

0.090.000.000.000.000.000.010.070.170.250.400.330.480.440.530.500.570.520.590.610.600.60

0.100.010.020.010.000.000.020.010.010.020.080.060.160.120.270.190.330.260.370.400.420.40

0.190.110.200.200.110.140.220.310.400.450.490.490.570.550.610.620.640.600.650.650.650.64

0.020.020.060.070.110.310.530.610.700.770.770.800.860.860.880.880.880.880.900.890.880.89

0.020.000.000.040.010.080.240.340.410.420.510.530.570.580.630.600.670.650.710.710.730.72

0.000.000.000.000.000.000.060.120.180.210.350.330.460.450.540.510.570.540.590.610.600.59

0.000.000.000.010.000.010.110.200.400.490.640.660.740.720.800.770.810.800.820.820.820.84

0.000.000.110.150.130.160.340.470.570.590.610.630.680.680.710.720.720.730.730.730.730.72

0.000.020.070.140.160.200.340.440.530.620.660.690.750.760.800.790.810.820.810.820.820.83

0.000.000.020.130.120.130.190.310.450.480.490.540.570.570.610.590.630.600.640.640.630.63

0.000.120.070.050.050.050.120.170.240.390.570.500.720.640.770.720.800.770.800.830.840.83

0.000.000.000.000.010.010.060.100.230.310.450.390.600.540.690.640.720.720.740.750.750.74

0.000.000.000.000.010.020.040.170.290.420.530.510.620.610.680.630.710.670.730.750.750.75

0.000.000.020.060.020.040.130.190.210.210.310.300.410.400.490.430.520.490.540.580.590.61

0.000.000.000.000.010.020.080.110.200.270.370.430.550.540.640.630.660.670.700.700.710.70

0.180.050.250.230.340.410.500.510.630.710.770.830.860.860.860.860.860.870.850.860.850.82

0.000.000.010.090.160.130.330.470.530.550.600.630.640.660.690.660.700.670.730.750.750.78

0.000.170.030.010.010.020.040.030.080.090.150.140.240.200.280.260.300.280.340.350.350.41

0.000.000.190.210.190.260.260.350.420.450.500.480.570.570.640.590.680.630.720.740.750.74

0.000.000.000.030.050.050.160.200.210.220.290.270.320.310.360.320.370.350.390.420.420.42

0.160.070.240.290.250.280.440.540.610.660.720.730.730.760.790.770.790.790.800.810.810.80

0.0

0.2

0.4

0.6

0.8

1.0

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

pink primrose

hard-leaved pocket orchid

canterbury bells

sweet pea

english marigold

tiger lily

moon orchid

bird of paradise

monkshood

globe thistle

snapdragon

colt's foot

king protea

spear thistle

globe-flower

purple coneflower

peruvian lily

balloon flower

giant white arum lily

fire lily

pincushion flower

fritillary

red ginger

grape hyacinth

corn poppy

prince of wales feathers

stemless gentian

artichoke

carnation

garden phlox

love in the mist

mexican aster

ruby-lipped cattleya

cape flower

great masterwort

lenten rose

daffodil

poinsettia

bolero deep blue

wallflower

marigold

common dandelion

petunia

wild pansy

primula

sunflower

pelargonium

bishop of llandaff

gaura

geranium

orange dahlia

pink-yellow dahlia?

cautleya spicata

japanese anemone

black-eyed susan

silverbush

californian poppy

osteospermum

spring crocus

windflower

tree poppy

gazania

azalea

water lily

rose

thorn apple

morning glory

passion flower

lotus

toad lily

anthurium

frangipani

clematis

hibiscus

columbine

desert-rose

tree mallow

magnolia

cyclamen

watercress

hippeastrum

foxglove

bougainvillea

camellia

mallow

mexican petunia

bromelia

blanket flower

trumpet creeper

blackberry lily

Classwise Accuracies (mean)

0.000.000.000.000.000.000.010.060.030.360.040.000.130.030.140.160.230.290.470.540.520.55

0.000.000.000.000.010.000.000.000.050.030.050.020.210.170.560.370.720.690.860.880.910.95

0.000.000.000.000.000.000.000.000.000.030.000.000.000.000.000.000.000.000.000.000.000.00

0.000.000.000.000.000.000.010.000.000.100.100.020.060.010.130.130.190.140.220.270.240.22

0.000.000.000.000.000.000.000.000.000.000.030.000.050.030.060.070.110.120.160.200.200.29

0.000.000.000.000.000.000.000.000.000.000.430.300.340.500.780.670.870.900.940.940.940.96

0.000.000.000.000.000.000.080.000.000.000.020.160.120.030.430.500.610.630.800.880.880.90

0.000.000.000.000.000.070.000.000.000.000.090.010.470.340.720.400.720.710.910.981.001.00

0.000.080.000.000.000.000.000.000.020.020.020.070.310.130.360.230.380.550.680.730.750.73

0.000.000.000.000.000.000.000.000.000.000.130.080.070.050.200.110.200.180.390.440.430.48

0.000.000.000.000.000.000.000.000.000.000.070.120.060.010.070.070.070.110.140.240.260.27

0.000.000.000.000.000.070.000.000.000.000.250.000.330.250.360.190.320.370.410.550.650.67

0.000.000.000.260.000.100.000.000.000.390.070.000.030.080.180.060.260.120.480.770.820.86

0.000.000.000.000.650.000.120.000.000.000.180.150.660.270.890.560.480.770.840.920.870.93

0.000.000.000.000.170.000.100.000.000.110.030.010.060.020.280.250.460.430.790.880.910.93

0.070.000.000.000.000.000.000.000.000.040.170.000.020.250.160.360.420.240.540.620.630.62

0.000.000.000.000.000.000.000.000.000.030.180.210.260.180.390.220.330.310.700.720.750.77

0.000.000.060.000.000.000.000.000.000.000.020.170.060.030.050.070.040.020.100.280.330.40

0.000.000.000.000.000.000.000.000.000.010.000.010.060.000.060.120.120.120.370.570.590.55

0.000.000.090.000.160.000.000.010.000.020.020.000.160.150.200.090.430.200.540.630.670.75

0.000.000.000.000.000.000.020.000.000.000.000.020.040.020.010.050.240.140.220.360.430.50

0.000.000.000.000.000.000.000.000.000.010.000.010.250.100.340.340.630.480.660.860.870.90

0.600.000.000.000.000.000.000.000.000.000.340.200.680.240.750.580.820.750.860.860.860.87

0.040.000.000.000.000.000.040.000.000.000.050.000.090.110.140.190.230.250.480.510.550.59

0.000.000.000.000.000.000.000.000.050.000.020.030.190.140.500.350.570.460.760.780.860.86

0.000.000.000.000.090.140.010.020.000.000.030.050.160.150.500.250.590.490.640.700.740.81

0.000.000.000.080.000.000.000.000.000.000.100.120.210.170.520.480.630.400.750.991.001.00

0.000.000.000.000.000.000.000.000.000.000.000.000.090.040.320.280.230.280.370.370.410.43

0.000.000.000.000.000.000.010.000.000.170.120.040.400.480.830.850.970.890.980.980.990.98

0.000.000.000.000.000.120.220.040.290.070.370.400.700.580.770.730.850.890.920.900.890.89

0.000.000.000.000.000.000.000.000.000.030.020.040.010.060.240.170.320.190.430.750.770.78

0.000.000.000.000.000.380.060.200.000.010.000.000.150.180.140.050.120.140.260.400.470.48

0.000.000.000.000.000.000.000.070.010.000.060.000.120.030.230.360.400.380.500.650.700.81

0.000.000.000.000.000.000.000.000.000.010.000.000.010.000.020.010.030.020.080.140.100.20

0.000.000.000.000.000.000.000.000.000.000.190.000.230.170.540.290.720.660.790.830.830.83

0.000.000.000.080.000.000.000.000.000.000.000.000.010.000.050.000.100.020.120.230.340.36

0.000.000.000.000.000.000.000.000.000.000.090.000.270.320.560.470.740.600.910.980.980.99

0.000.000.000.000.000.000.000.000.020.000.000.000.040.010.070.090.190.050.250.430.480.58

0.000.000.000.000.010.000.000.000.000.000.010.000.030.000.000.000.020.000.010.010.010.00

0.000.000.000.000.000.000.020.200.010.000.000.000.000.000.000.000.030.010.060.160.190.23

0.000.000.000.000.000.000.000.140.000.000.230.000.090.400.460.240.650.780.800.800.790.79

0.000.000.000.000.000.000.000.000.040.020.000.000.040.020.180.000.110.090.650.570.580.54

0.000.000.000.000.000.000.110.010.060.010.000.000.080.080.080.140.080.110.170.240.260.31

0.000.000.000.000.000.000.180.000.070.000.110.060.230.460.560.630.700.620.870.950.960.96

0.000.000.000.000.000.000.000.020.000.010.000.000.000.000.000.000.010.030.020.110.150.20

0.000.000.000.000.000.000.000.000.000.020.000.000.110.080.210.070.290.260.560.690.750.78

0.000.000.000.000.000.020.000.000.000.030.030.000.250.060.370.160.490.430.940.990.990.98

0.000.000.000.780.000.000.000.000.070.000.000.000.050.010.020.140.380.170.450.540.560.55

0.000.000.000.000.000.000.000.000.000.000.000.210.390.150.810.770.850.831.001.001.001.00

0.000.000.000.000.000.000.000.000.000.000.070.110.140.240.220.330.520.490.800.940.920.94

0.000.000.000.000.030.070.000.230.000.040.090.110.110.040.210.190.310.280.370.450.440.45

0.000.000.000.000.000.000.030.000.000.000.000.000.000.000.020.030.010.070.220.250.320.28

0.000.000.000.000.000.000.000.000.000.000.070.080.020.210.070.020.100.040.290.360.380.32

0.000.000.000.000.000.000.000.000.000.000.020.000.100.150.170.050.170.060.250.350.360.37

0.000.000.650.000.000.450.000.000.000.000.040.010.110.070.190.070.320.090.470.710.760.82

0.000.000.000.000.000.000.010.000.090.000.050.070.390.360.520.480.580.510.860.990.990.99

0.000.000.000.000.000.010.000.020.000.230.000.040.140.040.200.110.310.290.560.660.710.74

0.000.000.000.000.000.000.140.000.000.000.100.030.300.040.460.410.700.500.890.970.980.98

0.000.000.000.000.000.000.100.000.000.000.160.000.090.200.310.140.580.290.670.850.880.89

0.000.000.000.000.000.000.000.000.020.160.480.630.640.330.870.690.960.980.980.980.990.99

0.000.000.000.000.000.000.000.000.000.010.290.000.350.500.790.680.850.830.991.001.001.00

0.000.000.000.000.000.190.060.000.430.000.010.010.020.010.000.020.020.030.110.100.130.20

0.000.000.000.000.000.000.000.000.000.000.060.040.200.450.090.130.310.190.250.390.400.50

0.000.000.000.000.000.000.000.000.000.080.090.040.210.260.560.440.960.780.941.001.001.00

0.000.000.000.000.000.000.000.030.000.060.020.000.020.040.100.070.360.340.600.710.730.74

0.000.000.000.000.000.000.000.000.000.000.030.010.110.020.320.010.430.410.670.960.970.95

0.000.000.000.000.000.000.000.000.000.000.000.000.010.020.100.020.140.040.220.350.330.36

0.000.000.000.000.000.000.000.000.000.000.000.000.030.010.030.000.060.040.130.180.210.18

0.000.000.000.000.000.000.000.020.000.000.190.030.360.120.690.410.750.710.880.890.890.88

0.000.000.190.000.000.000.000.000.000.000.110.010.170.130.460.230.530.500.690.860.880.90

0.000.000.000.000.000.000.000.000.010.000.010.000.110.050.220.010.270.110.270.380.420.40

0.000.000.000.000.000.000.000.000.000.000.020.010.010.030.070.090.090.060.180.270.280.41

0.000.000.000.000.000.000.010.000.000.000.010.000.070.070.170.160.220.190.500.620.690.74

0.000.000.000.140.000.000.000.000.000.000.010.000.000.000.000.010.020.010.020.040.060.06

0.000.000.000.000.000.010.000.000.000.000.090.080.260.110.310.230.470.460.720.780.790.78

0.000.000.050.000.000.000.060.000.000.000.100.000.020.010.180.040.340.090.520.610.630.67

0.510.000.000.000.000.000.000.000.000.000.020.000.160.210.300.120.480.290.550.700.770.82

0.000.000.000.000.310.000.000.000.000.000.000.110.010.010.020.030.080.040.130.350.350.38

0.000.000.000.000.000.010.040.120.000.000.100.000.330.070.440.380.520.550.800.930.910.95

0.000.000.000.000.000.000.000.000.000.000.120.010.030.070.090.140.300.340.480.680.670.66

0.000.000.000.000.000.000.000.000.020.000.010.070.060.060.120.120.240.260.590.700.730.68

0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.010.000.030.020.03

0.000.000.000.000.000.000.000.300.010.180.010.000.010.060.010.060.040.140.210.310.370.40

0.000.000.000.000.000.000.000.000.000.070.000.000.010.050.020.030.030.050.100.250.280.24

0.000.000.000.000.000.000.190.060.110.050.010.000.230.400.240.100.190.290.560.520.540.49

0.000.950.000.000.000.000.020.010.000.000.100.000.140.250.350.180.310.170.430.690.720.74

0.000.000.000.000.000.000.000.000.190.000.080.050.040.030.240.140.310.240.540.660.730.79

0.000.000.000.000.000.000.000.000.000.000.000.000.010.020.010.000.040.040.160.220.250.26

0.000.000.000.000.000.010.000.000.000.000.000.030.010.120.080.210.210.230.490.600.650.66

0.000.000.000.000.000.000.040.000.000.010.140.000.200.250.380.050.380.310.500.600.580.60

0.000.000.000.000.000.000.000.000.000.000.000.040.020.080.130.070.220.310.480.600.610.64

0.000.000.000.000.000.000.000.000.000.000.010.000.070.000.080.020.090.090.150.250.290.33

0.000.000.000.000.000.000.040.000.000.230.130.090.150.070.400.350.480.250.620.810.850.85

0.000.000.000.000.000.000.000.000.180.000.020.000.040.010.160.070.320.160.450.580.640.68

0.000.000.000.000.000.000.000.000.000.010.000.000.000.000.020.070.090.060.100.160.180.20

0.000.000.000.000.000.000.000.000.000.000.000.130.020.010.030.050.030.050.070.120.130.11

0.000.000.000.000.000.000.000.020.000.000.010.000.010.000.170.020.220.180.390.490.520.57

0.000.000.000.000.000.000.000.000.000.000.040.000.010.000.040.000.050.130.150.290.290.34

0.000.000.000.000.250.000.000.000.000.000.140.000.260.230.350.290.390.590.810.870.880.88

0.000.000.000.000.000.000.000.000.000.000.120.000.170.170.240.180.300.210.420.460.430.41

0.000.000.000.000.000.000.030.000.040.000.000.000.000.010.020.020.020.060.170.180.300.32

0.000.000.000.000.000.000.000.000.000.010.390.040.490.410.690.370.810.690.910.960.960.96

0.0

0.2

0.4

0.6

0.8

1.0

(a) Flowers, Standard

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

pink primrose

hard-leaved pocket orchid

canterbury bells

sweet pea

english marigold

tiger lily

moon orchid

bird of paradise

monkshood

globe thistle

snapdragon

colt's foot

king protea

spear thistle

globe-flower

purple coneflower

peruvian lily

balloon flower

giant white arum lily

fire lily

pincushion flower

fritillary

red ginger

grape hyacinth

corn poppy

prince of wales feathers

stemless gentian

artichoke

carnation

garden phlox

love in the mist

mexican aster

ruby-lipped cattleya

cape flower

great masterwort

lenten rose

daffodil

poinsettia

bolero deep blue

wallflower

marigold

common dandelion

petunia

wild pansy

primula

sunflower

pelargonium

bishop of llandaff

gaura

geranium

orange dahlia

pink-yellow dahlia?

cautleya spicata

japanese anemone

black-eyed susan

silverbush

californian poppy

osteospermum

spring crocus

windflower

tree poppy

gazania

azalea

water lily

rose

thorn apple

morning glory

passion flower

lotus

toad lily

anthurium

frangipani

clematis

hibiscus

columbine

desert-rose

tree mallow

magnolia

cyclamen

watercress

hippeastrum

foxglove

bougainvillea

camellia

mallow

mexican petunia

bromelia

blanket flower

trumpet creeper

blackberry lily

Classwise Accuracies (mean)

0.000.000.000.010.000.000.080.050.050.100.350.230.630.540.640.640.630.630.640.630.670.65

0.000.000.000.140.100.120.490.540.690.730.910.870.890.950.840.950.840.880.830.840.830.80

0.000.000.000.000.010.000.000.000.060.000.060.030.070.020.150.060.210.090.270.320.310.35

0.000.000.060.000.000.070.010.010.010.030.060.040.170.090.180.140.180.170.170.210.210.19

0.130.010.000.020.000.080.010.010.030.110.260.250.400.330.640.560.690.620.720.730.740.69

0.000.020.000.040.050.000.040.130.120.590.790.610.860.800.890.890.900.920.920.930.920.92

0.000.000.020.030.000.050.020.050.300.270.640.470.770.710.740.750.770.780.750.700.690.70

0.000.000.000.000.100.070.010.110.250.570.950.960.980.980.980.990.980.980.980.980.980.98

0.000.320.000.000.050.000.000.040.120.290.270.510.480.330.630.550.700.580.690.720.720.73

0.000.030.010.000.130.200.200.320.340.550.630.620.860.880.870.900.850.870.890.870.890.92

0.000.000.000.000.000.000.090.000.020.010.060.030.220.160.360.250.410.340.500.550.540.54

0.000.000.000.040.020.010.020.010.000.190.450.480.690.670.660.680.670.800.770.790.810.81

0.000.000.000.010.000.010.160.100.540.110.790.630.890.900.900.880.920.920.930.930.930.93

0.000.000.090.070.030.200.100.040.420.080.760.510.910.670.900.820.920.880.960.981.001.00

0.000.000.000.000.030.000.120.030.070.380.610.700.700.740.760.860.770.850.780.790.820.79

0.230.000.000.000.000.000.000.000.210.120.540.370.760.780.840.880.860.860.860.860.860.86

0.000.000.000.000.020.010.040.120.140.250.620.480.790.770.830.810.830.830.830.830.820.83

0.000.000.040.000.020.000.020.000.010.050.040.020.160.100.310.180.390.290.490.560.590.60

0.000.000.000.000.000.000.030.000.110.170.370.230.480.500.480.540.450.480.460.460.420.41

0.000.000.090.000.030.010.040.060.070.180.480.240.550.510.630.590.690.620.710.720.720.72

0.000.130.000.000.000.110.060.060.080.220.490.480.750.660.860.850.850.900.880.940.880.90

0.010.000.000.010.000.040.010.010.010.030.300.080.660.510.870.700.910.880.930.940.950.95

0.310.000.000.000.010.000.020.050.160.310.690.580.830.850.860.870.870.870.860.870.870.87

0.180.000.000.190.200.230.250.140.410.670.880.700.850.870.860.880.850.890.920.970.950.95

0.000.000.000.030.150.190.360.360.200.210.860.850.930.920.920.910.900.910.900.940.940.95

0.000.000.000.080.090.030.110.210.410.500.790.730.810.790.760.800.820.770.810.780.770.76

0.000.000.000.030.010.000.000.020.060.210.420.540.900.820.990.931.000.971.001.001.001.00

0.000.000.000.000.000.000.010.000.340.630.740.470.830.770.850.900.890.920.870.880.890.89

0.000.000.020.150.000.010.150.110.260.330.710.680.890.890.940.890.920.920.920.910.900.91

0.000.040.000.000.000.000.070.140.260.320.630.540.700.760.630.720.620.640.590.590.580.58

0.000.020.000.020.000.080.060.010.010.000.160.090.310.240.560.400.670.510.750.820.840.88

0.000.000.000.000.000.020.020.100.020.050.170.180.300.230.380.290.430.390.400.550.580.56

0.000.000.000.000.000.040.060.120.050.020.370.070.580.510.760.710.820.850.900.920.920.92

0.000.000.000.000.000.020.000.000.040.080.040.040.060.020.350.050.470.220.630.650.660.65

0.000.000.140.210.100.040.380.270.320.570.750.670.940.900.960.970.970.970.970.940.940.91

0.000.190.000.130.000.000.000.010.020.090.190.080.360.210.440.240.570.320.680.730.750.76

0.000.000.000.000.120.000.120.020.130.120.680.510.940.910.970.970.980.980.970.960.950.95

0.000.000.010.000.020.000.060.050.040.160.320.190.480.510.720.680.740.720.810.820.810.83

0.000.050.000.000.030.000.000.020.010.040.060.030.160.140.450.170.550.350.700.740.730.76

0.000.000.000.010.010.000.060.070.070.090.200.090.430.310.600.550.660.570.700.730.720.77

0.000.000.000.000.010.040.010.010.000.060.030.070.160.200.320.240.420.330.460.520.510.52

0.000.000.000.010.000.000.020.030.010.100.380.420.650.350.720.500.790.610.790.820.770.77

0.000.000.000.000.000.000.040.020.020.010.020.070.040.060.130.120.210.170.280.310.320.32

0.000.000.000.000.000.000.000.010.020.010.370.070.500.540.710.690.760.760.810.820.840.85

0.000.000.000.000.000.000.000.020.070.120.310.220.340.260.540.470.620.550.640.620.660.65

0.000.010.000.000.000.000.060.050.210.070.440.290.780.710.890.810.900.870.930.950.950.95

0.000.000.000.150.010.170.060.010.010.050.660.390.950.980.980.980.980.980.980.980.980.98

0.000.000.000.080.000.000.090.020.010.330.670.340.580.600.730.890.840.830.850.910.930.90

0.000.000.000.170.190.040.230.030.190.250.970.921.001.001.001.001.001.001.001.001.001.00

0.000.000.000.000.040.220.000.000.220.230.710.170.850.710.950.890.960.910.950.940.940.94

0.000.000.000.000.010.000.010.080.030.010.040.050.240.230.380.330.430.440.450.430.430.43

0.000.000.000.000.000.000.000.020.050.070.280.250.520.440.700.600.720.610.800.820.830.83

0.000.000.000.000.020.000.020.040.000.040.100.070.220.210.280.220.290.250.360.390.420.41

0.000.000.000.000.000.000.000.020.060.260.180.240.600.410.750.640.810.770.870.900.900.90

0.000.000.020.000.000.130.110.110.180.230.490.440.640.700.840.730.860.870.920.920.910.92

0.000.000.000.010.000.040.200.080.110.300.660.570.950.930.980.980.980.980.970.970.970.97

0.000.000.000.000.000.020.010.060.030.040.340.330.720.650.830.800.870.850.890.910.910.91

0.000.000.080.000.100.050.090.070.010.010.680.160.880.850.910.890.920.910.950.960.960.97

0.000.000.160.000.040.000.010.000.130.090.380.110.880.661.000.921.001.001.001.001.001.00

0.000.000.000.000.280.020.150.060.070.180.760.290.920.960.970.970.960.970.970.980.980.98

0.000.000.000.000.010.010.320.090.210.340.780.850.980.970.990.990.991.000.990.991.001.00

0.000.000.000.000.000.010.010.040.030.010.110.030.300.170.410.300.590.440.640.730.750.74

0.000.000.150.000.050.010.180.220.280.440.960.711.001.001.000.991.001.001.001.001.001.00

0.000.000.000.000.000.000.320.470.060.360.940.801.000.990.990.990.980.990.950.940.940.94

0.000.000.000.000.000.000.020.040.090.040.180.270.500.240.740.530.810.780.870.900.890.87

0.000.000.000.000.000.100.470.530.630.800.990.980.991.001.001.001.000.991.001.001.001.00

0.000.000.000.000.000.010.020.000.030.050.130.130.380.200.680.400.780.660.860.950.950.91

0.000.000.000.000.000.000.000.050.040.090.140.020.250.190.480.340.550.410.620.680.680.71

0.000.000.020.000.040.010.230.090.190.320.760.500.980.920.991.000.990.990.980.960.940.97

0.000.000.190.030.060.000.050.020.130.400.600.900.880.820.910.970.900.980.900.910.910.90

0.000.000.020.000.000.000.250.220.350.170.810.700.940.860.980.940.970.960.990.990.991.00

0.000.150.000.000.000.000.010.020.030.040.140.030.120.080.270.170.330.270.390.450.440.49

0.000.040.000.000.000.000.010.030.050.090.340.120.390.330.680.530.790.640.860.910.920.92

0.000.000.000.010.000.000.010.020.010.040.060.030.190.150.390.240.500.300.620.670.680.70

0.000.000.000.000.040.010.030.130.200.530.740.600.790.860.840.870.800.830.760.750.750.75

0.000.010.110.000.030.000.080.070.120.160.430.340.670.650.750.720.740.750.760.750.740.74

0.450.000.000.000.000.000.000.010.020.030.080.060.510.420.710.570.750.710.810.830.840.84

0.000.000.000.000.000.010.020.050.070.090.250.060.480.400.500.480.510.500.490.500.530.53

0.000.000.000.000.000.000.010.010.100.270.490.430.850.790.960.900.940.960.940.950.950.95

0.000.000.000.010.000.010.040.010.030.110.400.240.670.640.780.750.780.780.780.780.790.79

0.000.000.000.000.000.000.070.050.130.270.630.240.800.780.900.900.930.930.930.940.950.95

0.000.000.000.000.000.000.000.000.000.010.020.010.050.020.120.050.190.150.300.350.390.38

0.000.000.000.000.060.030.010.000.000.020.070.010.100.080.290.100.400.240.500.520.520.52

0.000.000.000.000.000.000.010.010.010.000.040.050.100.040.310.140.440.290.510.550.550.53

0.000.000.000.000.000.000.000.030.070.160.200.230.450.430.650.580.690.660.740.770.770.79

0.000.190.000.000.010.020.020.080.090.070.240.190.420.280.490.460.520.500.590.630.640.66

0.000.000.000.000.000.150.000.040.020.060.190.120.450.430.590.530.670.580.730.820.870.86

0.000.000.000.000.000.000.000.020.010.010.040.030.150.070.190.150.200.200.300.390.430.45

0.000.000.000.000.000.000.000.010.060.040.210.090.280.320.400.380.420.430.460.460.450.46

0.000.000.000.000.000.000.030.000.040.050.160.110.320.160.430.310.540.410.590.610.630.65

0.000.000.000.000.000.000.030.050.020.010.150.050.310.230.430.360.480.410.530.610.640.64

0.000.000.000.010.000.000.000.040.070.000.100.030.310.210.680.490.770.700.800.820.830.83

0.000.000.000.000.000.010.000.000.020.020.380.080.530.400.720.580.830.850.880.880.890.92

0.000.000.000.000.000.000.000.030.000.010.070.030.120.050.280.160.430.310.570.670.680.68

0.000.000.000.000.000.010.000.020.020.010.020.060.060.030.160.070.250.140.340.350.360.37

0.000.000.000.000.000.000.000.020.000.020.010.000.030.020.080.030.110.070.150.210.210.21

0.000.000.000.000.000.000.040.000.070.020.100.020.300.160.430.290.500.440.530.550.570.57

0.000.000.000.000.000.000.020.010.060.050.170.060.190.210.430.290.450.360.500.530.540.58

0.000.000.000.010.070.000.120.100.190.220.410.280.670.610.790.730.830.820.810.800.820.79

0.000.000.000.000.040.010.020.120.170.500.630.590.880.860.940.900.950.940.960.970.970.97

0.000.000.010.000.000.010.010.010.030.050.110.050.290.250.480.310.490.410.580.680.710.68

0.000.000.000.010.000.130.250.110.150.140.620.420.860.820.930.970.960.970.981.001.001.00

0.0

0.2

0.4

0.6

0.8

1.0

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

Abyssinian

American Bulldog

American Pit Bull T errier

Basset Hound

Beagle

Bengal

Birman

Bombay

British Shorthair

Chihuahua

English Cocker Spaniel

English Setter

German Shorthaired

Great Pyrenees

Havanese

Leonberger

Maine Coon

Newfoundland

Persian

Pomeranian

Ragdoll

Russian Blue

Samoyed

Scottish T errier

Shiba Inu

Siamese

Sphynx

Staffordshire Bull T errier

Wheaten Terrier

Yorkshire T errier

Classwise Accuracies (mean)

0.000.000.000.000.040.000.020.150.100.100.220.200.540.570.770.690.830.820.870.900.910.93

0.000.000.000.010.020.110.150.380.470.430.580.670.770.790.870.900.890.890.900.910.910.92

0.000.000.000.000.030.000.010.050.080.090.240.330.520.460.580.570.610.560.610.620.620.63

0.000.000.070.050.040.290.400.270.580.660.820.850.930.920.920.920.930.930.930.940.930.93

0.000.000.000.010.010.010.150.150.200.210.510.540.820.760.960.970.991.000.990.990.990.99

0.000.080.080.000.090.290.330.230.400.370.340.340.490.530.720.660.790.750.840.910.910.91

0.000.000.000.000.000.000.170.000.090.120.170.220.360.330.540.400.670.650.730.780.790.81

0.000.000.000.180.180.260.320.260.550.660.740.750.870.850.900.940.950.950.950.970.960.95

0.000.000.000.130.040.160.220.210.350.430.680.730.890.880.910.900.920.910.930.930.920.92

0.000.010.000.000.000.000.200.260.220.230.380.360.480.510.570.590.580.600.620.650.660.67

0.930.200.260.300.290.390.170.560.910.780.800.840.920.920.930.930.940.940.950.960.970.97

0.000.000.000.000.000.000.030.030.160.560.750.840.820.860.790.820.790.800.790.800.810.80

0.000.000.000.000.100.010.320.270.450.490.730.740.940.960.970.970.970.970.980.980.980.98

0.000.000.000.000.000.030.090.060.240.260.640.730.950.950.970.950.980.980.980.980.980.98

0.000.030.000.020.000.060.110.560.740.800.950.950.980.980.990.991.001.001.001.001.001.00

0.000.000.000.000.000.000.140.230.290.540.700.800.940.970.970.970.970.980.970.980.980.98

0.240.490.270.450.150.270.220.580.630.670.860.810.960.960.970.980.980.991.001.001.001.00

0.000.000.000.000.000.000.000.110.350.310.720.730.960.970.970.980.990.991.001.001.001.00

0.000.000.000.000.110.230.300.310.370.500.870.830.960.960.960.970.970.970.980.980.980.98

0.200.200.200.220.380.290.200.290.520.630.850.840.970.930.980.980.990.980.991.001.001.00

0.000.010.010.000.010.060.060.040.140.390.580.650.760.730.810.840.840.850.830.860.850.84

0.000.000.000.190.210.260.320.240.280.450.770.720.910.920.930.930.940.940.950.940.940.94

0.010.020.010.070.060.110.380.350.510.810.930.970.980.990.991.000.990.991.001.001.001.00

0.000.000.000.010.000.000.000.350.600.540.680.650.830.870.900.900.900.910.900.910.910.91

0.000.000.000.010.190.210.290.200.290.370.670.800.910.940.980.980.980.990.991.001.001.00

0.000.000.050.000.010.000.150.190.470.530.790.680.950.930.980.970.980.980.990.990.990.99

0.000.000.070.000.010.080.150.130.190.360.600.610.680.750.690.710.730.700.720.750.730.75

0.000.240.170.110.150.030.080.170.150.240.290.320.530.480.630.590.710.690.750.770.770.77

0.010.000.070.160.290.070.670.760.770.900.950.980.970.980.980.980.990.990.990.990.990.99

0.000.000.000.000.010.010.240.220.290.630.891.001.001.001.001.000.990.990.990.990.990.99

0.000.000.370.160.150.190.290.330.550.570.660.880.960.980.990.990.991.001.001.001.001.00

0.000.000.050.010.030.010.190.250.180.310.600.720.930.930.991.001.001.001.001.001.001.00

0.000.000.000.000.000.010.350.440.630.580.710.680.870.890.950.950.950.950.960.960.960.96

0.000.010.020.310.300.510.590.390.580.530.690.660.810.870.920.940.930.930.930.930.940.96

0.000.000.000.040.000.180.250.320.450.380.600.540.660.640.720.730.740.760.760.770.790.80

0.000.000.000.230.140.310.440.450.450.490.760.770.850.890.880.890.910.900.930.960.970.97

0.000.000.000.200.350.370.490.350.630.590.810.820.930.950.970.970.990.990.990.990.990.99

0.0

0.2

0.4

0.6

0.8

1.0

(a) Oxford-IIIT-Pets, Standard

0.0005

0.001

0.002

0.003

0.004

0.005

0.01

0.02

0.03

0.04

0.05

0.075

0.1

0.15

0.2

0.25

0.3

0.4

0.5

0.8

0.9

1.0

Fraction/number of neurons

Abyssinian

American Bulldog

American Pit Bull T errier

Basset Hound

Beagle

Bengal

Birman

Bombay

British Shorthair

Chihuahua

English Cocker Spaniel

English Setter

German Shorthaired

Great Pyrenees

Havanese

Leonberger

Maine Coon

Newfoundland

Persian

Pomeranian

Ragdoll

Russian Blue

Samoyed

Scottish T errier

Shiba Inu

Siamese

Sphynx

Staffordshire Bull T errier

Wheaten Terrier

Yorkshire T errier

Classwise Accuracies (mean)

0.000.000.020.030.010.060.180.280.440.490.650.670.720.740.770.740.770.780.770.780.770.78

0.070.090.080.090.140.150.290.420.500.590.660.660.720.740.760.770.780.790.780.750.750.74

0.000.000.000.000.010.010.090.160.230.320.420.400.480.480.530.500.540.510.530.520.520.53

0.030.090.030.040.150.280.500.570.740.780.800.800.820.820.830.820.820.810.810.820.810.81

0.090.000.010.020.130.130.280.320.450.600.730.720.780.800.840.830.840.860.850.850.850.85

0.070.160.220.180.170.190.250.490.600.630.700.700.780.780.790.800.810.770.800.790.790.78

0.000.060.060.060.110.120.190.320.390.440.570.560.650.650.700.690.710.770.730.700.710.69

0.000.070.010.000.050.070.320.530.660.710.730.800.800.820.810.820.820.810.820.840.830.83

0.000.000.070.250.140.090.240.310.540.630.720.770.800.780.820.820.840.860.870.870.860.87

0.000.000.000.090.070.120.260.310.460.430.470.510.540.550.600.600.610.640.620.680.690.70

0.360.040.060.100.060.080.230.400.480.610.680.680.710.710.740.740.760.750.770.760.760.76

0.120.000.110.210.280.230.370.550.620.640.740.760.800.810.840.830.850.870.870.870.860.87

0.000.000.000.010.030.040.120.200.340.470.700.690.830.830.880.870.900.880.880.870.860.86

0.000.000.080.090.040.050.120.310.360.490.690.730.780.780.850.820.850.850.870.890.900.91

0.000.000.030.100.110.160.240.630.790.820.870.900.920.940.960.970.980.960.970.960.960.96

0.000.040.080.160.270.260.460.590.640.720.830.780.870.890.920.910.910.920.920.910.910.91

0.050.210.110.150.100.090.260.390.470.520.700.710.860.870.890.890.910.900.930.930.940.93

0.070.000.210.370.370.410.530.650.770.790.850.850.900.900.920.920.940.930.940.950.950.95

0.000.260.180.230.380.520.580.790.800.860.900.930.940.940.950.950.960.960.960.960.950.96

0.000.000.000.020.050.170.380.570.670.740.850.880.940.940.960.950.960.940.960.950.950.94

0.000.020.000.090.150.170.250.270.410.460.550.550.610.600.680.660.700.730.720.740.750.74

0.000.050.070.170.200.160.460.520.550.620.720.720.790.790.840.820.850.850.850.870.870.88

0.000.080.120.150.290.400.480.600.770.830.860.870.900.910.910.910.920.910.910.900.900.90

0.000.180.250.170.080.140.230.320.470.510.570.570.600.610.650.620.660.640.690.700.700.71

0.000.000.000.050.090.120.370.550.690.750.810.820.840.850.880.870.870.870.890.890.890.89

0.000.010.050.110.070.200.510.690.820.850.870.900.910.920.920.950.950.950.950.950.950.94

0.080.040.040.100.170.090.200.290.330.420.580.550.640.620.670.660.690.650.690.720.720.74

0.000.160.140.080.150.160.290.320.450.490.580.660.700.700.730.730.760.730.770.740.740.73

0.180.190.220.390.550.530.640.820.820.850.880.910.900.910.920.920.920.930.930.930.940.94

0.190.160.260.400.430.560.770.860.890.900.930.920.940.950.960.960.960.950.950.950.950.95

0.000.040.110.190.240.140.400.430.590.660.760.770.890.860.920.940.950.940.950.950.950.95

0.000.000.100.100.230.260.340.500.700.800.830.870.870.900.900.900.890.890.880.880.890.89

0.000.130.120.110.200.210.520.720.760.770.800.830.850.850.880.870.870.840.870.870.870.87

0.010.030.060.250.260.360.400.510.580.650.720.730.770.790.830.840.850.850.850.870.890.87

0.210.030.170.100.250.240.220.320.350.340.400.390.410.420.450.440.470.480.480.490.490.49

0.000.020.170.180.180.250.410.460.550.620.720.760.810.810.820.830.830.830.830.830.830.84

0.040.000.040.090.100.220.550.720.790.790.830.860.870.860.880.870.890.880.890.910.910.91

0.0

0.2

0.4

0.6

0.8

1.0

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(a) CIFAR10

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(b) CIFAR10

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(c) Flowers

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(d) Oxford-IIIT-Pets

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50-robustlinfeps4-random

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(a) CIFAR10

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(b) CIFAR100

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(c) Flowers

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(d) Oxford-IIIT-Pets

vgg16_bn-nonrob-random

vgg16_bn-robustl2eps3-random

resnet18-nonrob-random

resnet18-robustl2eps3-random

resnet50-nonrob-random

resnet50-robustl2eps3-random

wide_resnet50_2-nonrob-random

wide_resnet50_2-robustl2eps3-random

vit_small_patch16_224-nonrob-random

vit_small_patch32_224-nonrob-random

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(a) CIFAR10

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(b) CIFAR100

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(c) Flowers

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(d) Oxford-IIIT-Pets

vit_small_patch16_224-imagenet

vit_small_patch16_224-imagenet21k

vit_small_patch32_224-imagenet

vit_small_patch32_224-imagenet21k

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(a) CIFAR10

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(b) CIFAR100

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(c) Flowers

0.75

0.80

0.85

0.90

0.95

1.00

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(d) Oxford-IIIT-Pets

resnet50-nonrob-random

resnet50-robustl2eps3-random

resnet50_mrl-nonrob-random

resnet50_mrl-nonrob-first

0

1000

2000

3000

4000

Number of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) CIFAR10

0

1000

2000

3000

4000

Number of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) CIFAR100

0

1000

2000

3000

4000

Number of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(c) Flowers

0

1000

2000

3000

4000

Number of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) Oxford-IIIT-Pets

vgg16_bn-nonrob-random

vgg16_bn-robustl2eps3-random

resnet18-nonrob-random

resnet18-robustl2eps3-random

resnet50-nonrob-random

resnet50-robustl2eps3-random

wide_resnet50_2-nonrob-random

wide_resnet50_2-robustl2eps3-random

vit_small_patch16_224-nonrob-random

vit_small_patch32_224-nonrob-random

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) layer1.0.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) layer1.1.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) layer2.0.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(e) layer2.1.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(f) layer2.2.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(g) layer2.3.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(h) layer3.0.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(i) layer3.1.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(j) layer3.2.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(k) layer3.3.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(l) layer3.4.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(m) layer3.5.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(n) layer4.0.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(o) layer4.1.act3

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of neurons

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(p) layer4.2.act3

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of Neurons

0.0

0.2

0.4

0.6

0.8

1.0

(a) ResNet50 nonrobust

0.0

0.2

0.4

0.6

0.8

1.0

Fraction of Neurons

0.0

0.2

0.4

0.6

0.8

1.0

(b) ResNet50 robust

ℓ

2

ϵ

= 3

Imagenet

ImagenetV2

Places365

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(a) Dropout =

0

.

1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(b) Dropout =

0

.

2

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(c) Dropout =

0

.

3

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(d) Dropout =

0

.

4

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(e) Dropout =

0

.

5

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(f) Dropout =

0

.

6

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(g) Dropout =

0

.

7

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

(h) Dropout =

0

.

8

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Fraction of Neurons (Total=2048)

0.0

0.2

0.4

0.6

0.8

1.0

Transfer Accuracy

0

.

0001

)

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(a) Dropout =

0

.

1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(b) Dropout =

0

.

2

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(c) Dropout =

0

.

3

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(d) Dropout =

0

.

4

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(e) Dropout =

0

.

5

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(f) Dropout =

0

.

6

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(g) Dropout =

0

.

7

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

(h) Dropout =

0

.

8

cifar10

cifar100

flowers

oxford-iiit-pets

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Diffused Redundancy (DR)

0

.

0001

)

cifar10

cifar100

flowers

oxford-iiit-pets

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors argue that pre-trained representations exhibit a degree of \"diffused redundancy\", i.e. the accuracy on downstream tasks with linear probing is similar when we use the whole layer compared to a randomly subset of neurons. This claim is validated using representations pre-trained using ImageNet-1k and Imagent-21k and probe these representations on downstream tasks like CIFAR-10 and CIFAR100. The authors look at different factors that influence diffused redundancy like regularization or training loss--including variants like Matryoshka learning or adversarial training. Finally, they evaluate the potential impact on different class accuracies to understand the impact of exploiting diffused redundancy.\n\nSTRENGTHS:\nThe authors identify a new phenomenon that seems to occur when we pre-train representations on large datasets.  The manuscript methodically studies diffused redundancy on a number of different downstream tasks, regularization mechanisms, layer width and loss functions and identify that it occurs across all these scenario but to different extents. They also observe that diffused redundancy increases with adversarial training which is interesting and find that the class accuracies are impacted unevenly if we use a subset of the neurons. Overall, I believe that this is an interesting finding with no complete explanation for when and why it happens.\n\nWEAKNESSES:\n**Does diffused redundancy occur for any random subset?**\n\nThe authors claim diffused redundancy holds for *any* subset of neurons (L5 and L53). However, this claim is only validated by averaging over 5 subsets of neurons (L174). Is it possible that the redundancy is true for only some subset of neurons? For example, if we consider the representation projected onto the last $k$ principal components, does the diffused redundancy observation still hold? Alternately, we could select $k$ neurons with the least variance and test if the linear probe accuracy is similar. I believe that the claim, that this holds for *any* subset of neurons, is not sufficiently validated.\n\nIf I understand the results correctly, there exists an alternative explanation that isn't completely ruled out. As an alternative, let us assume that most of the information is diffused among an *informative subset* of neurons. Since the experiments consider only 5 random subsets of neurons, it is possible that each random subset contains some neurons from this informative subset. Hence the information may not be diffused among *all* neurons in the final layer but only among some of them.\n\nOn a related theme, in Figure 2b, the PCA projected representations has an accuracy that is higher by as much as 20% compared to a random subset of neurons. However in L56, the authors claim that the representation performs as well as PCA. Am I misinterpreting these plots? This seems to indicate that most of the information exists in a smaller subset of neurons.\n\n**Is this specific to pre-training on ImageNet ?**\n\nI also wonder if these observations are specific to models pre-trained on ImageNet and it would be nice to investigate this further. How prevalent is this phenomenon and does it occur if we pre-train models on Places-365 or iMaterialist? I think another important experiment is to pre-train models on smaller datasets with very few classes (like CIFAR10) and see if diffused redundancy occurs in this scenario.\n\n**Definition of diffused redundancy**\n\nI also have some minor concerns regarding the definition of diffused redundancy. The definition (Eqn. 1), requires the accuracy averaged over different subsets of neurons ($M_f$) to be greater than some threshold $\\delta$. But this doesn't require *every* subset of neurons to have high accuracy. Maybe, a more appropriate definition is to require all subsets to have accuracy greater than some threshold if that is the central claim of the paper.\n\nQUESTIONS:\nThe abstract (L17) claims that \"entire layers might not be necessary\". The authors could perhaps clarify that this is only true for the final layer, since their experiments are restricted to this layer (which doesn't discredit the importance of their findings).\n\nIn Figure 1(b) and Figure 1(d) the diffused redundancy plot seems to take only 7 different values which seems rather strange to me. Could the authors explain why this is the case? How many different $D_f$ sets are considered when estimating the diffused redundancy?\n\nCan these observations be extended to random projections of representations? Selecting a random subset of neurons can be viewed as looking at restricted class of random projections.\n\nLIMITATIONS:\nThe authors have studied diffused redundancy under different settings and have adequately addresses limitations. Section 4 also addresses some potential limitations if future methods were to exploit diffused redundancy.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper empirically investigates the various factors that affect how the features of a pretrained representations are encoded. They find that the features exhibit a degree of diffuse redundancy and randomly chosen subsets of the features in the penultimate layer have similar accuracy on downstream tasks and score high in common similarity metrics. They find that the loss and datasets used to pretrain the model affect the number of features required to achieve most of the performance on a given dataset. They show potential applications for efficient transfer learning and implications for unequal performance across classes when neurons are removed.\n\nSTRENGTHS:\n- Using adversarial training to create more diffuse feature representations is an interesting observation and potentially useful for efficient retrieval or efficient inference when target datasets have many classes. In general, understanding how adversarial training affects the feature space of a pre-trained model is an interesting direction. \n- The experiments are relatively thorough with multiple architectures and datasets. \n- The paper is well written and organized.\n\nWEAKNESSES:\n- The applications and implications of the diffused redundancy hypothesis given in the paper are not not well motivated, or lacking experimental evidence. \n\n  - One of the major stated contributions is decreased training time on downstream tasks. I think there are a few weaknesses in this argument. First, from my understanding, to determine the optimal number of neurons to drop for a given target dataset, you would need to perform multiple training runs with varying levels of neurons being dropped. Searching for this hyper-parameter would require additional FLOPS rather than reduce them. Second, I didn’t see analysis in the main paper or appendix about how many FLOPS are saved by using the reduced-length feature vector. Unless most of the FLOPS are in the final layer, the reduction in compute during training will be minor. Similar works, such as Slimmable Neural Networks, reduce parameters throughout the whole network for the purpose of efficiency, but at inference. Third, for most downstream datasets, fine-tuning is computationally cheap as the dataset and number of training epochs are small. Most of the work on efficiency is focused on pre-training or inference. \n\n  - The other stated implication of the diffused redundancy hypothesis is the accuracy inequality between classes when reducing length of the feature vector. The experiments in figure 7 don’t rule out the possibility that the gini-coefficient is merely correlated with the accuracy. It would be more convincing to see additional models on the scatter plot in figure 7 to show that this isn’t the case, and dropping neurons in the final layer are the main cause of the increase of the gini- coefficient\n\n- In the abstract it is stated that: “We find that learned representations in a given layer exhibit a degree of diffuse redundancy”, but from what I understand, the main experimental findings are only for the penultimate layer, so this claim is misleading. \n\n- CIFAR, OxfordPets, and Flowers are relatively small and less diverse than the other datasets in the appendix (ImageNetV2 and Places365). It would have been useful to see more analysis of how diversity and scale affect the conclusions.\n\nQUESTIONS:\n- For the adversarial training experiment, is the test set the same as standard training? (i.e. are the adversarial perturbations only done on the train split.)\n\n- The keys in the legend in figure 7 are unclear. For example, what does “nonrob” or “first” mean exactly?\n\n- For figure 5 it would be easier to draw conclusions if the x-axis was in length of the feature vector rather than fraction of used neurons so that models could be compared. For example, 20% of a model with a 256-length feature vector isn’t comparable to 20% of a model with a 64-length feature vector. If I wanted to know the best model to use for a given feature vector length, it would be difficult to see from the figure.\n\nLIMITATIONS:\nThe authors adequately address the limitations of their work. They discuss how removing features from the feature vector could unequally affect the class accuracy. I think further experiments are needed to convincingly show that dropping features is the root cause of this phenomenon, but the authors bring up an interesting limitation of shortening the feature vector.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce the concept of `diffused redundancy', which is the idea that a random sample of neurons reproduces the performance of the whole layer in many downstream tasks. The authors support the existence of this phenomenon with experiments on a variety of architectures pre-trained on ImageNet and explain it with the similarity between these random samples past a certain threshold size. The observed phenomenon is argued to be an 'inevitable' property of wide deep networks, as it seems to persist even when the training method should avoid it. Finally, the authors discuss the implications of diffused redundancy for fairness.\n\nSTRENGTHS:\nThe work proposes a novel property of the hidden representations of deep neural networks which furthers our understanding of such representations. This new hypothesis is accurately framed within the existing literature and convincingly backed up by extensive numerical experiments. Overall, the manuscript is well-written and easy to follow.\n\nWEAKNESSES:\nI think this paper would benefit from a comparison of the diffused redundancy hypothesis with neural collapse (Papyan et al, 2020). In simple terms, if the neurons of the last layer are clustered into a number of groups much smaller than the width, then there will naturally be a threshold size $k$ such that if I pick $k$ neurons at random I will pick at least one from each cluster with large probability. If the number of clusters is taken as the number of classes, this would lead to a prediction for the behaviour of DR. This prediction would be the same, in principle, for different downstream tasks, but maybe this difference is only due to different downstream tasks having different classes. If true, this difference could be eliminated with an appropriate normalisation (see question 1 below).\n\nAnother criticism concerns the size of the figures. I would recommend the authors use the extra page for increasing the size of the figures.\n\nQUESTIONS:\n1. In line 186, the authors mention that some tasks might require more neurons simply because they have more classes. Would it be possible to find a normalised measure that takes this effect into account?\n\n2. In Figure 2, panels (c) and (d), what is the cause of the discrepancy between dashed and solid curves for a low fraction of neurons?\n\n3. In section 3.2, is the dependency on the width simply due to the fact that there are not enough neurons? For instance, if the fraction of random neurons required to keep performance high with width 10^3 is 10%, then of course there cannot be diffused redundancy for width 10^2. If this is the case, then I would personally demote the discussion on the width to the appendix, either to make more space for the figures or to make space for the methods that decorrelate neurons, which might be more interesting for the general audience.\n\nFew additional comments/suggestions:\n\n4. I believe that the related works section should also discuss the neural collapse phenomenon.\n\n5. It would be helpful, for the sake of clarity, to have a definition of DR in words (i.e. $1$ minus the fraction of neurons required to have at least $\\delta$ accuracy in downstream tasks).\n\nLIMITATIONS:\nthe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes investigating the emergence of redundant representations in pre-trained models. The setup is pretraining of various architectures (VIT, vgg, resnet) on Imagenet1k. The study pertains to how much of the representation is redundant and this is estimated by dropping features and finetuning on smaller datasets (cifar 10, 100, flowers and Pets). The paper proposes a measure of redundancy based on centred kernel alignment. The experiments show that this is true under this measure for random subsets of a certain size as long as the model has enough parameters. They also show that there is an apparent drop in fairness that would have to be addressed if features are to be pruned in the proposed way for some datasets like pets and flowers.\n\nSTRENGTHS:\n* It is an interesting phenomenon to discover and a good study.\n* The paper is generally well written.\n* The comparison across models and losses is interesting and may ultimately lead to improvements in model training and or design.\n\nWEAKNESSES:\n* Not sure how this is going to impact practical aspects of ML in the future. In particular not sure how meaningful the values of CKA are here. The measure (1) seems sensible though.\n* I think section 2.2 comes short of the state goal of explaining why any random subset works. It explains that this is happening but not why this is the result of the optimization.\n* I find the paragraph describing the \"correlation\" between the pruning and PCA confusing and misleading.\n\nQUESTIONS:\n* I am not sure how meaningful fig 7 is if the fairness is computed on the same labels as the quality. Do I understand correctly that the claim is Gini for say cifar 10 drops less than accuracy meaning it is dropped uniformly but pet's and flowers the Gini index goes up more quickly with accuracy thus is less fair.\n\nLIMITATIONS:\nnone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFocusing on the (penultimate layer) representation learned by the pre-training model, this paper proposes a \"diffused redundancy hypothesis\" that includes two main concepts:\n\n- Redundancy: A small subnet can achieve comparable performance with the full model.\n- Diffused redundancy: The selection of the small subnet can be random.\n\nTo validate this hypothesis, this paper conducts experiments across various models, datasets, and training paradigms (e.g., standard training or adversarial training) to demonstrate the diffused redundancy phenomenon. Furthermore, this paper also investigates the possible factors that influence the degree of this phenomenon. Finally, this paper shows the trade-off between fairness (class-wise accuracy) and efficiency (pruning rate).\n\nSTRENGTHS:\n1. The proposed diffused redundancy hypothesis is an unrealized phenomenon and may have a strong impact on the research community.\n2. This paper leverages several metrics, such as Centered Kernel Alignment (CKA), to interpret and understand this phenomenon, which is reasonable and enhances the soundness of the proposed hypothesis.\n3. This paper systematically investigates the underlying mechanism and influencing factors of the diffused redundancy hypothesis.\n\nWEAKNESSES:\n1. Given that model pruning and compression techniques can achieve comparable performance with only a few neurons, the diffuse redundancy phenomenon seems to be expected and less appealing. Intuitively, a wide network inevitably results in diffuse redundancy in the representations.\n2. This paper only investigates the representation on the penultimate layer and does not analyze this phenomenon in terms of middle layers.\n3. This paper only conducts empirical experiments to validate the proposed hypothesis but does not provide theoretical analysis of why this phenomenon exists.\n4. This paper does not provide an intuitive or theoretical understanding of the fairness-efficiency trade-off in Section 4.\n5.  Line 163, Equation 1. $DR(g, T, \\delta)$ is defined with the average $\\frac{T(m \\odot g)}{T(g)}$ for $m\\in M_f $.\nHowever, the diffuse redundancy hypothesis claims that any randomly chosen subset of neurons can achieve similar performance. Therefore, I think using the worst representation similarity instead of the average is more appropriate: $$DR(g,T,\\delta)=\\frac{1-\\min f, \\text{s.t.}\\min_{m\\in\\mathcal{M}_f} \\frac{T(m\\odot g)}{T(g)}\\ge \\delta}{|g|}$$\n\nQUESTIONS:\n1. Line 178: How about $\\ell_\\infty$ adversarial training and other perturbation bounds with $\\epsilon$?\n2. Lines 312-314: This paper claims that diffused redundancy can be used to reduce training and storage costs. Will model pruning or compression techniques serve a similar function? Will carefully pruning $k$% of the neurons result in better performance than randomly dropping $k$% of neurons? ($0\\le k \\le 100$)\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}