SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corresponding diffused redundancy (DR) ablations for Figures 4,5,7. These are shown in Figures 15,16,17 respectively. This should allow for easy comparison of diffused redundancy (lines that are more outside have higher DR). For example, Figure 16 clearly shows higher diffused redundancy in models trained on larger upstream datasets (here ImageNet21k) since these curves lie more on the outside of the same model's curves for ImageNet1k.
Additionally, we show numbers on x-axis for Figure 4 in Figure 18. Figures 5 and 7 compare models with same number of neurons in the final layer and hence trends shown with fraction on the x-axis will be exactly the same with absolute numbers on the x-axis. However, Figure 18 allows a direct comparison of the performance of the same absolute number of neurons across different models.
We report results for  $\ell_{\infty}$  robust models (with  $\epsilon = 4 / 255$ ) in Fig 14 and find that  $\ell_{2}$  model generally shows a greater degree of diffused redundancy.
E Results on Intermediate Layers
We additionally ran our experiment on other intermediate layers and report the results in Fig 19. We present results for a ResNet50 pretrained on ImageNet1k using the standard CrossEntropy loss. The
(b) CIFAR100
(d) Oxford-IIIT-Pets
Figure 10: [Coefficient of Variation As We Drop Neurons] We see a similar trend as reported in Fig 8 of the main paper where inequality increases as we drop neurons for all models on all datasets.
intermediate layers considered are characterized as activations following each residual connection within distinct ResNet blocks. layerX.Y.act3 means the  $Y$ th residual connection in the  $X$ th ResNet block and act3 indicates that we're taking the value after the activation (ReLU) has been applied.
F Results on Harder Downstream Tasks: ImageNetV2 and Places365
We report results on harder downstream tasks such as ImageNet1k, ImageNetV2, and Places365 in Figure 20. We find that when randomly dropping neurons, the model is still able to generalize to ImageNet1k and Places365 with very few neurons, i.e., the phenomena of diffused redundancy observed for smaller datasets, also holds for harder datasets. Interestingly we also observe that the accuracy gap between ImageNet1k and ImageNetV2 is maintained even as we drop neurons.
G Effects of Explicitly Preventing Co-adaptation of Neurons: Analysis of Dropout and DeCov
Regularizers such as dropout and DeCov, force different parts of the representations to not be correlated. Thus these regularizers can be seen as explicitly requiring different, compact parts of the representation to be self-contained for the downstream task. Thus, intuitively, such methods should increase diffused redundancy. Here we investigate if our observation about diffused redundancy is influenced by such regularizers. We evaluate ResNet50 pre-trained on ImageNet1k with dropout in the penultimate layer ranging from a strength of 0.1 all the way to 0.8. We also train another ResNet50 model with the DeCov regularizer added to the usual crossentropy loss and put a weight of 0.0001 on the regularizer to ensure that its numerical range is similar to that of the cross entropy loss term.
Results in Figures 21 & 22 suggest that such regularizers have almost no effect on diffused redundancy. Trends across datasets remain consistent regardless of the strength of dropout or the weight given to DeCov regularizer. This observation further adds to the evidence that diffused redundancy is likely to be a natural property of representations learned by DNNs.
Figure 11: [Error Distributions As A Function of Fraction of Neurons] We see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left.
(a) Flowers, Standard
(b) Flowers, Adversarial Training
Figure 12: [Error Distributions As A Function of Fraction of Neurons - continued] We see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left.
(a) Oxford-IIIT-Pets, Standard
Figure 13: [Error Distributions As A Function of Fraction of Neurons - continued] We see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left.
(b) Oxford-IIT-Pets, Adversarial Training
(b) CIFAR10
(d) Oxford-IIIT-Pets
Figure 14: [Results for  $\ell_{\infty}$  robust model] We show results for ResNet50 trained with 3 different losses: CrossEntropy (nonrob), Adversarial Training with  $\ell_{2}$  threat model (robl2eps3), and with the  $\ell_{\infty}$  threat model (roblinfeps4). We see that the  $\ell_{2}$  threat model shows the most diffused redundancy. All models are trained on ImageNet1k.
(a) CIFAR10
(c) Flowers
Figure 15: [Comparisons Across Architectures For Downstream Task Accuracy] All models shown here are pre-trained on ImageNet1k. This Figure shows corresponding diffused redundancy values for Figure 4 different  $\delta$  values. We see that diffused redundancy exists across architectures, and the trend observed in Figure 1c&1a regarding adversarially trained models also holds here as models curves that are more "inside" are the ones trained with standard loss.
(b) CIFAR100
(d) Oxford-IIIT-Pets
(d) Oxford-IIIIT-Pets
Figure 16: [Comparison Across Upstream Datasets] We see that degree of diffused redundancy depends a great deal on the upstream training dataset, in particular models trained on ImageNet21k exhibit a higher degree of diffused redundancy, although the differences in the degree of diffused redundancy are downstream task dependent
vit_small_batch16_224-imagenet
vit_small.patch32_224-imagenet
vit_small.patch16_224-imagenet21k
vit_small.patch32_224-imagenet21k
Figure 17: [Comparison of Diffused Redundancy in MRL vs other losses] Here we compare ResNet50 trained using multiple losses including MRL [26]. Red line shows results for part of the representation explicitly optimized in MRL, whereas green line shows results for parts that are picked randomly from the same representation. Even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. This figure shows diffused redundancy (DR) for all plots in Figure 7.
resnet50-nonrob-random
resnet50_mrl-nonrob-random
resnet50-robustl2eps3-random
resnet50_mrl-nonrob-first
Figure 18: [Comparisons Across Architectures For Downstream Task Accuracy] This shows the same plots as Figure 4, except showing absolute number of neurons on the x-axis
(a) layer1.0.act3
(b) layer1.1.act3
(c) layer1.2.act3
(d) layer2.0.act3
(e) layer2.1.act3
(f) layer2.2.act3
(g) layer2.3.act3
(h) layer3.0.act3
(i) layer3.1.act3
(m) layer3.5.act3
(j) layer3.2.act3
(n) layer4.0.act3
(k) layer3.3.act3
(o) layer4.1.act3
(1) layer3.4.act3
(p) layer4.2.act3
Figure 19: [Middle Layers; ResNet50, trained with CrossEntropy loss on ImageNet1k] We see that as we go deeper in the network, accuracy progressively increases. We see even middle layers exhibit diffused redundancy, and accuracy plateaus very quickly for earlier layers. layerX.Y.act3 refers to the  $Y$ th residual connection in the  $X$ th ResNet block and act3 indicates that we're taking the value after the activation (ReLU) has been applied.
(a) ResNet50 nonrobust
(b) ResNet50 robust  $\ell_2\epsilon = 3$
Figure 20: [Performance on ImageNet1k, ImageNetV2, and Places365] We check for the performance of randomly chosen subsets of neurons on harder tasks like ImageNet1k, ImageNetV2, and Places365. We find that diffused redundancy holds for all these harder tasks as well. Additionally, we see that randomly dropping neurons still preserves the accuracy gap between ImageNet1k and ImageNetV2.
(a) Dropout  $= 0.1$
(e) Dropout  $= 0.5$
(b) Dropout  $= 0.2$
(c) Dropout  $= 0.3$
(d) Dropout  $= 0.4$
(f) Dropout  $= 0.6$
(g) Dropout  $= 0.7$
(h) Dropout  $= 0.8$
(i) DeCov Regularization (0.0001)
Figure 21: [Dropout and DeCov regularizer's effect on Diffused Redundancy]
(f) Dropout  $= 0.6$
(g) Dropout  $= 0.7$
(h) Dropout  $= 0.8$
(i) DeCov Regularization (0.0001)
Figure 22: [Dropout and DeCov regularizer's effect on Diffused Redundancy] Same results as Figure 21, but showing  $DR$  estimates (Eq 1). Lines that are more towards the right (i.e. more on the "outside") mean they exhibit more diffused redundancy.