On the Origins of the Block Structure Phenomenon in Neural Network Representations
Authors: Thao Nguyen
Maithra Raghu
Simon Kornblith
Venue: Accepted by TMLR ()

Abstract: Recent work by Nguyen et al. (2021) has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block structure has two seemingly contradictory properties: on the one hand, its constituent layers exhibit highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure in relation to the data and training methods. By analyzing properties of the dominant PCs, we find that the block structure arises from dominant datapoints — a small group of examples that share similar image statistics (e.g. background color). However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds. Thus, the block structure reflects meaningful dataset statistics, but is simultaneously unique to each model. Through studying hidden layer activations and creating synthetic datapoints, we demonstrate that these simple image statistics dominate the representational geometry of the layers inside the block structure. We explore how the phenomenon evolves through training, finding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant datapoints continue to change substantially. Finally, we study the interplay between the block structure and different training mechanisms, introducing a targeted intervention to eliminate the block structure, as well as examining the effects of pre-training and Shake-Shake regularization.

Published in Transactions on Machine Learning Research (11/2022)
On the Origins of the Block Structure Phenomenon in Neural Network Representations
Thao Nguyen Paul G. Allen Center for Computer Science Engineering University of Washington'
Naattnm@os..ushhinggaan
Maithra Raghu
maithrur @gmaai com
Samaya A1
Simon Kornblith
skornblith www.:biltt@@gggge..ccooo
Google Research, Brain Team
Reviewed OpenReview: https: oppnrevveee.. net/ forum? id= Lt662)LS
Abstract
Recent work by Nguyen et al. (2021 has uncovered striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar rep- resentations. This block structure has two seemingly contradictory properties: on the one hand, its constituent layers exhibit highly similar dominant first principal components (PCs), but on the other hand, their represenlations, and their common first. PC, are highly dissimi- lar across different random seeds. Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure in relation to the data and training methods. By analyzing properties the dominant PCs, we find that the block structure arises from dominant. datapoints
small group of examples that share similar image statistios (e-g- background color). However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds. Thus, the block structure reflects meaningful da datasel statistics, but is simultaneously unique to each model. Through studying hidden layer acti- vations and creating synthetic datapoints, we demonstrate that these simple image statistics dominate the representational geometry of the layers inside the block structure. We explore how the phenomenon evolves through training, finding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant dat- apoints continue to change substantially. Finally, we study the interplay between the block structure and different training mechanisms, introducing largeted intervention to elimi- nate the block structure, well as examining the effects of pre-training and Shake-Shake regularization.
1 Introduction
Many modern successes of deep neural networks have adopted simple techniques to systematically increase model capacity, often through scaling architecture depth and width (Tan Le 2019). These large capacity models typically also maintain strong performance even in tasks with small amounts of training data. This has led to their widespread across many different applications, including data-scarce, high-stakes settings such medical imaging (Wang et al. 2016 Liu et al. 2017
However, recent work has shown that representational structures of large capacity models trained on small datasots exhibit distinctive properties that are not present shallower haallooee/narower networks. Specifically, Work done primarily while Google Research.

---

Published in Transactions on Machine Learning Research (11/2022)
(a)
Deeper
(b)
CIFAR-100
ResNet-26 1x
ReeNet-Id 1x
ResNat-110 1x
Re5Net-164 1x
ResNat-26 1x
ResNet-218
ResNet-26 10x
75
75
Layer
Layer
50
25
25
75
100 150
150
300
250
500
50
300
800
25 50
Wider
Patch Camelyon ResNet-80
ResNet-38
ResNet-38
ResNet-38 ResNet-38
ResNet-20 1x
ResNet-20
Layer
100
Layer
50-
50
50
100
50 100 Layer
50 100 Layer
SD 100 Layer
50
125 Layer
250
25
50
Similarity
Layer
Layer
Layer
Figure 1: (a): Block structure arises in wide and deep networks. Heatmaps show similarity between layer representations, measured with linear CKA, for models varying widths and depths trained on CIFAR-10. models become wider or deeper, "blocks" of consecutive layers share similar representations. (b): Block structure arises on many datasets. Models trained on CTFAR-100 and medical histopathology dataset Patch Camelyon show similar behavior to those in panel (a). Block structure also appears when networks trained on CIFAR-10 are evaluated on OOD images; see Appendix
(a)
ResNet-38 (2x)
ResNet-38 (10x)
(b)
ResNet-32 1x
Seed
Seed
Seed 0 vs.
75
30
30
Layer
50
Layer
20
20
25
50
75
25 60 75
25 50 75
10
10
ResNet- 164 x
Seed
Seed
Seed
Frac. Var. Exp.
20
30
20
Layer
20
30
250 Layer
500
250 500
500
Layer
Layer
Layer
Layer
Figure 2: (a): The first principal component of representations inside the block structure explains the majority of the variance in representations. Top: Linear CKA similarity heatmaps for networks that do and do not exhibit block structure. Bottom: Fraction of variance in representations explained by first principal component. See Appendix C for similar findings with deep networks. (b): Representations inside the block structure highly unique to each seed. 2 leftmost panels show similarity of representations within networks trained with 2 different random seeds. Rightmost panel shows similarity between the networks.
when using linear centered kernel alignment (CKA) Kornblith et al. 2019 to measure similarity between hidden representations of neural network layers, Nguyen et al. (2021) show that large set of contiguous layers share highly similar representations. This is visible as a clear block structure in the heatmap of pairwise linear CKA similarities between layers (see Figure [h). The block structure phenomenon robust, appearing both in models trained on natural image datasets and those trained on medical imaging datasets (Figure
[p), as well as varicty of CNN architcctures (Nguyen al. 2021 Cianfarani al. 2022 Lu ct al. 2021
Lange al. 2022
The emergence of the block structure with increasing model capacity is accompanied by significant changes in other propertics of the individual layers. As shown in Figure 2 (left), in Mhallow/nrroon networks, the first principal component of each layer exhibits relatively little variance. However, in higher-capacity networks (Figure 2h (right)), the layers that make up the block structure have dominant first principal components that explain most of the variance their hidden representations. Naïvely, one might expect this collapse rank impair generalization (Daneshmand al. 2020) Dong et al. 2021). but these networks nonetheless perform well the distributions they trained The representations of the layers that make up the
2

---

Published in Transactions on Machine Learning Research (11/2022)
block structure are also dissimilar across seeds (Figure pp), which is surprising in light of previous work suggesting that increased capacity and better generalization is typically associated with more consistent solutions across seeds (Morcos et al. 2018).
Thus, the block structure is arguably the single largest representational change associated with overparam- eterization, manifested through changes in similarity between layers as well as the singular value spectra of individual layer representations. It consists of low-rank representations that are similar across layers, but differ across initializations. Having previously been associated with impaired generalization, these prop- erties motivate a pressing question - is the block structure sign of harmful overfitting to the data, is it harmless consequence of overparameteriaation? In this paper, we uncover answers to this question, exploring the origin of the block structure in relation to the data and training methods, and reconciling its contradictory behaviors. Specifically, our results are as follows:
The dominant first principal components of the layers that make up the block structure arise from a small number of dominanl dalapoints that share similar characteristics (e.g. background color). Like the block structure phenomenon, dominant datapoints are found in large-capacity neural networks.
The set of dominant datapoints (and their shared characteristics) can vary across training runs, which explains the observed block structure dissimilarities across seeds.
The block structure is caused by these dominant datapoints. When dominant datapoints are excluded from the dataset used for representational analysis, the block structure disappcars.
Dominant datapoints produce high activation norms inside the hidden layers corresponding to the block structure. By constructing synthetic examples based off of the shared image characteristics of the dominant datapoints, we show that these image charactcristics are indeed responsible for the high activation norms.
The block structure emerges early in training, but early block structures have different representations and yield different dominant datapoints than the final block structure found at the end of training.
We show that it is possible to eliminate the block structure from the internal representations using a novel principal component regularization method. However, this intervention provides only minor accuracy improvemenss, suggesting that the block structure has, at most, minor impact on networks' generalization abilities.
2 Related Work
Previous work has studied certain propensities of deep neural networks in the standard training setting, such as their simplicity bias (Huh et al.)
2021 Valle-Perez et al. 2018 Nakkiran et al.) 2019). texture bias [aker
iet al. ee al. Inspired by the findings of Nguyen et al. (2021) that deep and wide networks learn many layers with similar
2018
Geirhos et al. 2018
Hermann ct al. 2019 and reliance on spurious correlations (McCoy
2019 Geirhos et al. 2020:
Ribeiro el al.] 2016 Jo & Bengio, 2017 Hosseini & Poovendran 2018
reprosentations, we seek to characterize what signals these layers convey and may be overfitting to. To do
so, we analyze the behavior of the internal representations of models of varied depths and widths, using
methods for measuring similarity of neural network hidden representations (Kornblith et al. 2019 Raghu
et al. 2017 Morcos et al.) 2018), as well as standard tools of linear algebra. Representations similarity techniques have previously shed light on model training procedures (Gotmare et al.) 2018 Neyshabur et al. 2020), features (Resnick et al. 2019 Thompson et al.) 2019: Hermann & Lampinen 2020p, and dynamics (Mahoswaranathan et al., 2019). and has also furthered understanding of network internals in applications of deep learning, such as medicine (Raghu al., 2019) and machine translation (Bau al. 2019).
Our work also relates to previous attempts to understand the properties of overparameterizedd models (Zhang) ct al. 2016). Thcoretical work in this area has focused on lincar models, models with random features, or kernel settings (Belkin et al.] 20188: Hastie et al.) 2019 Liang el al., 2020 Bartlett et al. 2020), all of which lack intermediate features, or involves linear networks (Advani et al. 2020). Our results suggest that the behavior of intermodiate features of practical neural networks changes dramatically with increasing overparameterization, in ways that are not obvious from previous analysis, and cannot be easily characterized by properties at initialization.
3

---

Published in Transactions on Machine Learning Research (11/2022)
3
Experimental Setup and Background
Measuring Representation Similarity with CKA: Centered kernel alignment (CKA) (Konbbithh et al. 2019 Cortes et al. 2012) addresses several challenges in measuring similarity between neural network hidden representations including (i) their large size; (ii) neuronal alignment between different layers; and (iii) features being distributed across multiple neurons in layer. Like Nguyyen et al. (2021), we use the minibatch implementation of linear CKA with batch size of 256 sampled without replacement from the test dataset, and accumulate statistics over 10 epochs to allow the minibatch estimalor to converge.
More concretely, given k minibatches of examples, and two layers having Pi neurons and P2 neurons cach, minibatch CKA takes as inputs k pairs of centered activation matrices (X,, (Y,Y (X, Y)) where X, E _nxpi and Y, E xx* reflect the activations of these layers to the same minibatches. It produces a scalar similarity score between and by averaging the scores of Hilbert-Schmidt independence criterion (HSIC), computed with a linear kernel, over the minibatches:
E ISICI ) E
1
LCKAminibatch
E ISSICI(,, X
& E HSI Y.Y,)
where HSIC, is the unbiased estimator of HSIC from Song et al. (012)). This estimator of linear CKA converges to the same value regardless of the batch size.
We compute CKA between all layer representations, including before and after batch normalization, acti- vations, and residual connections. In experiments that involve tracking how a model's internal properties (principal components of activations, representation similarity, etc.) change across epochs, we set batch normalization layers to be in training mode, to reduce the difficulty of adapting to batch statistics of the test set when the model has not converged.
The Block Structure Phenomenon: Nguyen et al. (021] use lincar CKA to compute the representation similarity for all pairs of layers within the same model and visualizes the result heatmap (with x and y axes indexing the layers from input to output). They find a contiguous range of hidden layers with very high representation similarity (yellow squares on heatmaps in Figure [b) in very deep or wide models, and call this phenomenon the block structure. The block structure ariscs in networks that are large relative to the size of the training set. while small works may not exhibit block structure when trained on CIFAR-10, they do exhibit block structure on smaller datasets.
Representations of the layers making up the block structure exhibit different representational geometry than the rest of the layers. For layers inside the block structure, the first principal component explains a large fraction of the variance in representations; this is not the case for the other layers or for networks without the block structure (Nguyen et al. 2021). We replicate this observation in Figure 2h. The similarity between layers inside the block structure reflects the alignment of their first principal components, as can be seen from the following decompasition of linear CKA for centered activation matrices X RRPpP, Y E Raxx:;
Σ En 82 X X (u'x ut
YCKA(X,Y)-
2
P2 2 E}
where uix E R" and uy E R" are the 2h normalized principal components of X and Y, and Axx and A are the amounts of variance that these principal components explain (Kornblith et al. 2019). As the fraction of variance explained by the first principal component of cach representation approaches 1, CKA bocomes a measure of the squared cosine similarity between first principal components (u², )2 Nguyen al. (2021) conclude that the block structure preserves and propagates a dominant first principal component across many hidden layers.
Our setup closely follows that of Nguyen et al. 2021 and analyzes ResNets of varying depths and widths, trained on common image classification datasets CIFAR- 10 and CIFAR-100 (Krzhevsky et al. 2009), as well as the modical imaging datasct Patch Camelyon (Veeling et al., 2018). These datascts are chosen to reflect the image statistics found in different domains, and all easily induce a block structure in reasonably sized ResNets.
Datasets & Models:
4

---

Published in Transactions on Machine Learning Research (11/2022)
The ResNet architecture design follows Zagoruyko & Komodakis (2016), with the layers distributed evenly between three stages - each marked by different. feature map size and the number of channels is doubled after each stage. To scale the model depth and width, we increase the number of layers and channels respectively. In experiments involving Shake-Shar regularization (Gastaldi 2017), the network is modified to have branches that are combined stochastic fashion. More information on hyperparameters can be found Appendix A
4 Dominant Datapoints and How They Shape the Block Structure
4.1
Datapoints that Activate the First PC
Seed Layer 250
Seed 2 Layer 250
# of Example
7500 5000
5000
7500
2500
2500
200
200
100
100
Projected Value (log)
Projected Value (log)
Seed Layer 500
Seed 2 Layer 500
#f Examples
7500
5000
5000
7500
300
200
100
-3
-2
-1
-5
-3
-2
-1
Projected Value (log)
Projected Value (log)
Figure 3: Visualization of the distribution of projected values onto the first principal component by CIFAR-10 test inputs. There exist small number of datapoints that yield significantly larger projected values than the rest and dominate the first principal component of the network. Here we show examples of those dominant images for two different seeds of ResNet-164 (1x) (columns). Each seed's dominant images share similar background colors, but these background colors differ between seeds. Further visualization of dominant datapoints across different layers for instance, layers 250 and 500 in this case show that they are consistent across layers making up the block structure. See Appendix D for analysis of other models and tasks.
Motivated by previous evidence that the block structure propagates a dominant principal component across constituent layers, we examine the distribution of the projections of each example's activations onto the first principal component. We find that the distribution is bimodal. Most examples have small projections, but some are orders of magnitude larger than the median (Figure 3
We call these datapoints with large projections dominant datapoints, and find that they are consistent across the range of layers making up the block structure, as seen from each column of images in Figure 3 showing the dominant datapoints for two different layers in the same block structure. This explains why the first principal components of different layer activations inside the block structure are highly similar (Figure 2 an observation made carlier in (Nguyen et al.) 2021). Moreover, this dominant datapoint phenomenon is present only in networks that also exhibit block structure. As shown in Appendix Figure 15 in networks without block structure, projections the first principal component are unimodally distributed and the corresponding datapoints differ between layers.
Dominant datapoints are visually similar. In the left column of Figure we observe that all datapoints have a blue background, although the precise shade of blue varies. However, the visual signals that the first principal component picks up on depend on the random seed used to train the model. The right column of Figure 3 shows the corresponding properties of an architecturally identical model trained from different seed, where the dominant. datapoints share white backgrounds instead. We observe that dominant. datapoints also arise
5

---

Published in Transactions on Machine Learning Research (11/2022)
# of Examples
Layer 8D
2000
1000
Figure 4: Visualization of the distribution of pro- jected values onto the first principal component by test imputs, for ResNet-50 (8x) trained on 1/16 of ImageNett Top row shows histograms of the projected values on the first PC. Bottom rows show im- ages with the largest projections on the first PC.
400
200
-2
Projected Value (log)
0% Removed
1% Removed
10% Removed
400
400
400
Frac. Seed
200
Var Exp.
0
400
Frac. Seed 2
200
Var Exp.
0
400
Frac. Seed 3
200
Var Exp.
250
500
0
250
500
250
500
Layer
Layer
Layer
Figure 5: Removing small number of dominant datapoints eliminates the block structure. Plots show the effect of removing examples with the largest projections on first PC of layer 300 ResNet-164 (1 x) models trained on CIFAR-10. Columns reflect different numbers of examples removed; rows reflect models trained from different seeds. Within each group, the top left plot shows linear CKA heatmaps, the bottom left shows the fraction of variance explained by the first PC, and the images reflect the examples with the largest projection on the first PC after data removal.
in ImageNet-trained ResNets (Figure 4 Refer to Appendix D for visualizations of dominant images found in other lasks (CIFAR-100, Patch Camelyon) and model architectures (wide ResNet). Besides background color, the dominant datapoints can also reflect other simple image patterns that are prevalent in the dataset, such the appearance of large dark spots in histopathologic scans (Appendix Figure 17..
Finally, the block structure observed in linear CKA heatmaps arises solely from dominant datapoints. As seen in Figure 5 when the 10% most dominant datapoints excluded from evaluation, the block structure is completely climinated in all 3 training runs of ResNet-164 (1x) that we examined, and the fraction of variance explained by the first PCs is substantially reduced. In fact, for one training run, removing only the 1% most dominant datapoints is sufficient to achieve this effect. Thus, the block structure is completely determined by the dominant images, and is sensitive to the frequency of the dataset statistics that captures.
4.2 Dominant Datapoints Have Large Activation Norms
We investigate what happens to the activations dominant example as propagates through the network, and observe that it strongly activates the parts of the network with visible block structure. Figure G shows of dominant datapoints, for ResNet trained on CIFAR-10 that has a preference for white background images (top row), and for another ResNet trained on Patch Camelyon that responds to clear pink backgrounds

---

Published in Transactions on Machine Learning Research (11/2022)
CIFAR-10
1.00
Activation Norm
4000
Activation Norm
400
0.75
0.50
2000
200
500
0.25
250
500
0.00
100
300
500
Layer
Layer
Projected Value
Patch Camelyon
1.00
750
Dominant Datapoint
Activation Norm
400
200
0.75
Batch Median
0.50
Activation
500
200-
100
250
0.25
WAY
100 Layer
50
100
150
200
250
0.0
0.2
Layer
Projected Value First PC
Figure 6: Datapoints that dominate the first principal components of the block structure also strongly activate the corresponding lavers. We explore the relationship between dominant dalapoiuls and activation norms for ResNet-164 (1x) trained on CIFAR-10 (top row) and ResNet-80 (1x) trained on Patch Camelyon data (bottom row). For layers inside the block structure (left column), dominant datapoints (inset) produce much larger activation norms than the median of a randomly selected minibatch (middle column). Moreover, within these layers, the norms of the activations of different datapoints are highly correlated with the magnitudes of their projections the first principal component (right column).
(bottom row). Both models contain block structure in their internal representations, and we find that in the corresponding layers, the activations of the dominant datapoints are substantially larger in norm than the median activations of the minibatches they are a part of. Morcover, the magnitude of the projection on the first PC correlated with the activation norm. We conclude that dominant datapoints evoke activations with large norms, and activations of different dominant datapoints point. similar directions.
In Appendix J inchude additional results mcasuring representational similarity in networks with block structure using CKA with different kernels. All kernels we have tested sensitive to differences between dominant and non-dominant datapoints, but the linear kernel produces particularly strong "blocks" compared to RBF or cosine kernels. Across all kernels, the removal of dominant datapoints consistently eliminates blocks in reproscntational similarity heatmaps.
4.3 Image Backgrounds As Dominant Property
In this section, through data and training manipulations, we further characterize how dominant datapoints arise from specific colors in images.
First, we examine the connection between the background colors of dominant datapoints, which vary across random seeds, and layer activation norms. We begin with dominant examples and repeat only the top left pixel in each image across the entire dimensions of the image, obtaining solid color images. These synthetic images indeed yield even larger activations compared to the dominant examples they are taken from, and different initializations of architocturally identical networks respond to different synthetic images. For instance, given the Nes (1x) model that has dominant datapoints containing a blue background (Figure 3), its hidden layers are further activated when all image pixels replaced with the same shade of blue, but solid white image produces considerably smaller activations (Figure 7 In the same figure, we observe the opposite trend for another ResNet-164 (1x) seed, which has been shown to pick up on white backgrounds (see Figure Refer to Appendix for similar analysis on Patch Camelyon.
Intervention: Color Augmentation: Given carlier insights, a natural intervention to prevent the network from potentially picking up background color signal is adding color augmentation. This includes randomly dropping color channels and jittering brightness, contrast, saturation, and hue of training images (Howard
7

---

Published in Transactions on Machine Learning Research (11/2022)
Seed
Activation Norm
3000
Solid Green
Solid
Solid White
2000
Dominant Datapoint
1000
100
200
300
400
500
Seed 2
Activatiion
2000
400
Layer
Figure 7: Solid color images strongly activate intermediate layers. Rows show ResNet-161 (1x) models that are trained from different random initializations. The top model's dominant datapoints consist of images with blue backgrounds (see inset images), whereas the bottom model prefers white background images. Layers of the top model are strongly activated by solid blue images, but not solid white images, whereas the bottom model shows the opposite pattern. Layers both models are strongly activated by their respective dominant datapoints (red lines), but other solid colors (e.g. green) do not yield strong activations in either. To improve readability of the plot, plot only representations at end each ResNet block. Appendix similar findings on Patch Camelyon models.
Without Color Augmentation
ResNet- 1x
ResNet-164 1x ResNet-38 8x
ResNet-38 10x
1.0
Layer
300
500
100
100
Figure 8: Simply adding color augmen- tation helps reduce the block structure effect. Having established that the activations and representational components of some large- capacity models pick up on common back- ground colors the inputs, we experiment with color dropping and color jittering during training to counter this effect. Indeed, color augmentation minimizes the block structure appearance in the internal representations.
200
250
0.5
0
100
0.0
100
300
250
500
50
100
50
100
With Color Augmentation
ResNet-110
ResNet- 164 1x
ResNet-38 8x
ResNet-38 10x
1.0
Layer
300
100
100
200
250
0.5 O
100
0.0
100
300
250 500 Layer
50 100 Layer
50 100 Layer
Layer
Szegedy et al., 2015). As shown in Figure training with this data augmentation reduces the block structure in large capacity models.
2013
5 Evolution of Block Structure during Training
In the previous section, we characterize the signals the block structure propagates across its layers, and explore their implications on other aspects of the network internals. Informed by these findings, we next explore what happens the block structure and the dominant images over time, from initialization until the model converges, and how this process varies across different training runs.
Figure 9 shows the evolution of the internal representations of ResNet-110 (1x) model as it is trained for 300 epochs on CIFAR-10, and tracks how similar each checkpoint is to the final model. We obscrve that some structure in the CKA heatmap is already present by the first epoch, and the heatmap undergoes little qualitative change past epoch 20 (top set plots). However, when we inspect the corresponding dominant datapoints and compare the hidden representations between intermediate chockpoints and the final model (bottom rows of plots), we find that the block structure does not always carry the same information. Instead, the representations, and corresponding groups of dominant. datapoints, only stabilize much later in training.
8

---

Published in Transactions on Machine Learning Research (11/2022)
WWthin-Checkpoli Similarity
Ep.
Ep.
Ep.
Ep.
Ep.
Ep. 20
Ep. 50
100
200
Ep.
Layer
300
200
100
100
300
100
300
100
300
100
300
100
300
100
300
100
300
100
300
100
300
100
300
Similarity Final Checkpaint
Ep.
Ep.
Ep.
Ep.
Ep.
Ep.
Ep.
Ep.
Layer
200
OK
100
100 300
100 300
100 300
100 300
100 300
100 300 Layer
100 300 Layer
100 300
100 300
100
300
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Dominant Datapoints
Epoch
Epoch 9
Epoch 50
Epoch 300
Figure 9: Block structure phenomenon arises early during training, but the corresponding dominant datapoints continue to change substantially. We compute the CKA between all pairs of layers within ResNet- 110 (1x) model at different stages of training, and find that the shape of the block structure is defined early in training (top row). However, comparing these different model checkpoints the fully-trained model reveals that the block structure representations at different epochs are considerably dissimilar to the final representations, especially during the first half of the training process (middle row). The corresponding dominant datapoints also vary significantly over the course of training, even after the block structure is clearly visible in the heatmaps (bottom row). See Appendix F for similar plots with greater granularity, different seeds and architectures.
We observe similar behavior for other models in Appendix F and note that the differences in block structure representations across random seeds already take shape near the start of training as well. Overall these findings suggest that the uniqueness of the block structure representations in large-capacity models can be attributed both initialization parameters and the image minibatches the models receive throughout training.
To further explore the link between dominant datapoints and fluctuations in network representations, in Figure 10 we track the magnitude of the projected value on the first PC of single dominant datapoint found at the end of training, and find that the value plummets at epochs when the internal representation structure diverges from that of the fully trained model. these epochs, the dominant datapoint does not produce large activations either (bottom set of plots). This illustrates that the precise set of dominant datapoints can vary significantly over the course of training for large-capacity models.
6 Block Structure and Training Mechanisms
Having observed how the internal representation structures - specifically, the dominant PCs of layer rep- resentations could vary significantly during training, we turn examining the interplay between the block structure and the training mechanism. Although the block structure arises naturally with standard training, previous work has suggested that the block structure may be an indication of redundant modules in the corresponding networks (Nguyen et al., 2021). Thus, it is natural to ask whether it is possible to train large-capacity models without a block structure, and how such models perform compared to those with block structures.
Since the block structure reflects the similarity a dominant first principal component, propagated across wide range of hidden layers (soc Section 3). we study whether regularizing the first principal components of layer activations would eliminate the block structure. More specifically, we estimate the fraction of variance explained by the first principal component of each layer using power iteration and penalize it in
9

---

Published in Transactions on Machine Learning Research (11/2022)
Projected Value
0.5
0.0
50
100
150 Epoch
200
250
Epoch 0
Epoch 30
Epoch 60
Epoch 90
2
Activation Norm (x103)
Epoch 120
Epoch 150
Epoch 180
Epoch 210
2
2
2
2
0
0
Epoch 240
Epoch 270
Epoch 300
Dominant Datapoint
2
2
Batch Average
0
0
250
500
250
500
Layer
Layer
Layer
Figure 10: Example of how the magnitudes of layer activations and the projected values onto the first principal component for a dominant image vary across different epochs, for ResNet-164 (Ix). Given dominant datapoint for ResNet-164 (1x) model, we track the magnitude of its projected value onto the first principal component the block structure representations (top left), as well as its activation norms at each layer the network (bottom set of plots), over time. Notice the correspondence between these 2 metrics, especially when their values drop at epochs (initialization), 120 and 240. This also aligned with the measurement CKAs across different epochs (see Figure 22 where we find that the model checkpoints at these 3 epochs are highly dissimilar from the final model in terms of the hidden representations). See Appendix Figure 24 for the corresponding visualization of dominant example of wide ResNet (ResNet-38 (10x)).
Standard Training ResNet: 164 1x
Principal Component Regularization
ResNett 110 1x
Res 38 10x
ResNet: 110 tx
ResNet 184 1x
ResNet 38 10x
Layer
400
100
400-
100
200
200
200
100
100
100
500
50 Layer
Layer
Layer
Layer
Layer
Layer
Shake-Shate Regularization
CKA
Transfer Learning
ResNet-110 1x
ResNei-164 1x
ReeNet-38 10x
ResNet-1 110 1x
ResNel-164 1x
ReeNet-38 10x
300
1000
200
400-
100
200
500
100
100
200
800
500
1000
100
300
100
300
250
500
50
Layer
Layer
Layer
Layer
Layer
Layer
Figure 11: Training with principal component regularization, transfer learning and Shake-Shake reg- ularization helps eliminate the block structure. We directly regularize the first PC of each layer activations given that this component explains large fraction of variance in block structure representations, and find that this eliminates the block structure. Full algorithm details can be found in Appeudix C Shake-Shake regulariza- tion Gastaldii 2017; has similar effect. We also find that transfer learning reduces the appearance of the block structure, although it is still present in the largest network. These results demonstrate that the block structure phenomenon is dependent on the training mechanism. See Appendix | for implications of these training methods on representations across random seeds.
the training objective when exceeds 20%. We provide full implementation details in Appendix G The resulting heatmap, in Figure right, shows that not only does this eliminate the block structure from the internal representations, but there is also detrimental effect on performance (Table 1
We even observe
10

---

Published in Transactions on Machine Learning Research (11/2022)
Depth
Width
Accuracy (%)
Accuracy (%) (PC regularization)
(standard training)
CIFA 10 subsampled (6% of the full dataset):
56
77.8 + 0.429
79.2 0.188
26
80.1 0.354
81.1 0.185
26
10
80.3 + 0.306
81.2 = 0.194
80.2 0.362
80.9 0.264
38
10
80.3 + 0.412
81.4 + 0.350
CIFA R- 10:
110
94.3 0.078
94.4 + 0.063
164
94.4 + 0.075
94.5 0.063
26
10
95.8 + 0.087
96.0 + 0.051
38
95.7 0.091
95.8 0.080
38
10
95.7 + 0.157
95.9 = 0.067
CIFA 10000:
218
0.310
75.1 0.132
224
74.0 + 0.350
75.2 - 0.131
38
79.8 0.149
80.6 0.306
38
10
80.5 0.174
81.1 0.241
Table Comparison performance of large-capacity models on CIFAR-10 and CIFAR-100, with and without principal component regularization. We observe that our proposed principal component regularizer consistently yields accuracy improvements for large capacity models that contain the block structure. The perfor- mance gains particularly significant in the case of CIFAR-100 and subsampled CIFAR-10 datasets. Numbers are calculated based on 10 training runs.
Depth
Width
Accuracy (%) (standard training)
Accuracy (%) (Shake-shake)
Accuracy (%) (transfer learning)
110
94.3 +: 0.078
94.5 0.051
94.4 0.080
164
1
94.4 + 0.075
94.6 1 0.066
94.5 1 0.102
26
10
95.8 + 0.087
95.9 + 0.024
95.9 + 0.099
38
8
95.7 + 0.091
95.9 + 0.033
95.8 + 0.024
38
10
95.7 + 0.157
96.0 + 0.106
95.8 + 0.033
Table 2: Comparison of performance of large-capacity models CIFAR-10, between standard training training with Shake-shake regularization and transfer learning. observe that the latter training techniques (last 2 columns) yield higher accuracies compared to standard training, especially in the case of Shake- shake regularization. Numbers are calculated based on 10 training runs.
small accuracy improvements on CIFAR-100 and in the low-data regime, as shown in Table [ Although we designed this regularizer explicitly to eliminate the block structure, it possible that it has other effects on the training dynamics that lead this small accuracy improvements. Overall these experiments demonstrate that the detrimental impact of the block structure on generalization is most minor.
Other standard training practices that are commonly used to boost performance (Table 2 are also effective at reducing or eliminating the block structure effect. Shake-Shake regularization (Gastaldi 2017) eliminates the block structure for all of the network sizes that we examine (Figure 11 bottom left whercas transfer learning (Figure 11 bottom right) and training with smaller batch sizes (Appendix generally reduce the appearance of the block structure, although blocks are still discernible in the largest models that we trained. In addition to regularizing the block structure, these training methods also produce more similar representations across different training runs of the same architecture configuration (Figure Appendix Figure 27
11

---

Published in Transactions on Machine Learning Research (11/2022)
Standard Training
Principal Component Regularization
500
500
Layer
Seed
CKA
250
250
500
500
Layer
Seed
250
250
500
500
Seed 3
Layer
250
250
500
500
500
Seed
Seed
Seed
Seed Layer
Seed 2 Layer
Seed
Layer
Layer
Layer
Layer
Figure 12: Training with principal component regularization yields models that are more similar across different training runs. Each group of plots shows CKA between layers of models with the architecture but different initializations (off the diagonal) or within single model (on the diagonal). Similar to the observation made in Figure 27 while representations across models are highly dissimilar the standard training case, models trained with principal component regularization from different random inilializations show more representational similarity in corresponding layers.
Overall, findings suggest that it is possible to obtain good generalization accuracy in networks with and without block structure. We observe that learning processes that reduce the dominance of the first PC of the representations provide slightly higher accuracies than standard training. However, some caution is warranted in interpreting these performance benefits: it may be difficult to causally determine the connection between the block structure and performance, as any training intervention targeting the block structure may have other distinct ramifications that also affect performance.
7 Discussion
Scope and Limitations: Our work primarily focuses on the behavior of large-capacity networks trained relatively small datasets. This is motivated by domains such as medical imaging where data is expensive relative to the cost of training a large model, and the high-stakes nature makes it important to understand the model's behavior. In general, understanding how the representational properties change with model capacity (relative dataset size) of both scientific and practical interest, as model size continues to grow over the years but there are many domains beyond vision (e.g, see Kaggle) where dataset size does not. Additional exploration is needed to study ttato-of-tho-art settings in c.g. NLP, which use much bigger and heterogeneous datasets.
The block structure phenomenon uncovered in previous work (Nguyen et al. 2021 reveals significant differences in the representational structures of overparameterized neural networks and shal- lower/narooer ones. However, it also exhibits some contradicting behaviors - being unique to each network while propagating a dominant PC across a wide range of layers that respectively suggest the underlying representations could either overfit to noise artifacts or capture relevant signals in the data. Our work seeks to provide explanation for this discrepancy. We find that despite the inconsistency of the block structure across different training runs, it arises not from noise, but real and simple dataset statistics such as back- ground color. We further discover a small set of dominant datapoints (with large activation norms) that are responsible for the block structure. These datapoints emerge early in training and vary across epochs, as well as across random seeds. We show how different training procedures, including color augmentation, transfer learning, Shako-Shake regularization, and a novel principal component regularizer, can reduce the influence of these dominant datapoints, climinating the block structure and leading to more consistent representations across training runs.
Conclusion:
12

---

Published in Transactions on Machine Learning Research (11/2022)
Since the block structure phenomenon has been shown to robustly arise in large-capacity networks under various settings (Nguyen et al., 2021 rigorously characterizing its cause and effects is of great importance to understanding the nuances in the way these networks learn, despite their similarly good performances. We believe that insights into the representational properties of overparameterized models can benefit tech- niques that make direct use of the internal representations, such as transfer learning and interpretability methods. This work also motivates interesting open questions including exploring how dominant datapoints are manifested in other domains and applications of deep learning, as well applying principal component regularization to distribution shift and sule-supen problems.
References
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky, High-dimensional dynamics of generalization error in neural networks. Newral Networks, 132:428 446, 2020.
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip J Kellman. Deep convolutional networks do not classify based on global object shape. PLoS computational biology, 14(11))e1006613, 2018.
Peter I, Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, :17643)330063 30070, 2020.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass, Identifying and controlling important neurons in neural machine translation. In International Conference Learning Represenlations, 2019. URL htttps://openreviirt wttp:///wwwweeeeeeeoorr/eeee-aee
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pp. 541 549. PMLR, 2018.
Christian Cianfarani, Arjun Nitin Bhagoji, Vikash Sehwag, Ben Zhao, and Prateek Mittal. Understanding robust learning through the lens of representation similaritics. arXiv preprint arXiv:2206.098 2022.
Corinna Cortes, Mchryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered aligument. The Journal of Machine Learning Research, 13(1):795 828, 2012.
Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Advonces in Neural Information Processing Systems, 33:18387-18398, 2020.
Yihe Dong, Jean- Baptise Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 2793 2803. PMLR, 2021.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: 774507485 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trainee trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. Xrvi preprint ar Xiv: 1. 12231, 2018.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665-673, 2020.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXio:1810. 22110.122. 13243, 2018
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. arXiv preprint arXiv: 1908. 085600, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-77 2016.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arxie preprint arXiv:2006. 12433, 2020.
Katherine L. Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXie preprint arXiv:1911.09071 2019.
Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1614 11441619,,11 2018
Andrew G Howard. Some improvements on deep convolutional neural network based image classification. arXiv preprint Xiv: 1512.1312.5444 2400, 2013.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit. Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103. 10427, 2021.
Jason Jo and Yoshua Bengio. Measuring the tendency of CNNs to learn surface statistical regularities. arXiv preprint arXii:1711.11561, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Richard D Lange, Jordan Matelsky, Xinyue Wang, Devin Kwok, David S Rolnick, and Konrad Kording. Neural networks as paths through the space of representations. arXiv preprint arXiv:2266. 10909, 2022.
Tengyuan Liang, Alexander Rakhlin, et al. Just interpolate: Kernel "ridgeless" regression can generalize. Annals of Statistics, 48(3):1329 1347, 2020.
Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E Dahl, Timo Kohlberger, Aleksey Boyko, Sub- hashini Venugopalan, Alcksei Timofcev, Philip Q Nelson, Greg S Corrado, et al. Detecting cancer metas- tases on gigapixel pathology images. arXiv preprint
2017.
Yao Lu, Wen Yang, Yunzhe Zhang, Jinhuan Wang, Shengbo Gong, Zhuangzhi Chen, Zuohui Chen, Qi Xuan, and Xiaoniu Yang. Graph modularity: Towards understanding the cross-layer transition of feature repre- sentations in deep neural networks. arXiv preprint arXiv:2111.19483 2021.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. In Advances in neural information processing systems, pp. 15629-15641, 2019.
R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Xiv preprint arXiv:1922.01007 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gen- eralive adversarial networks. In International Conference Learning Representations, 2018. URL
https://opeerevieer wttp://www....etrooooorr////aaaa
Ari S Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. arXiv preprint
2018.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd neural networks learns functions of increasing complexity. arXiv preprint arXiv:1905.11604, 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint "Xtt:2008.11687 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076 6085, 2017.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in neural information processing systems, pp. 3347-3357, 2019.
Cinjon Resnick, Zeping Zhan, and Joan Bruna. Probing the state of the art: A critical look at visual representation ovaluation. arXiv preprint Xiv: 000115, 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 'Why should i trust you?" explaining the predic- lions of any classifier In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. The Journal of Machine Learning Research, 13(1):1393 144-3999-444,,,,111 2012.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference compuler vision and pallern recognilion, pp. 1 9, 2015.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International Conference on Machine Learning, pp. 6105 6105-6114, 2019.
Jessica AF Thompson, Yoshua Bengio, and Mare Schoenwiesner. The effect of task and training on interme- diate representations in convolutional neural networks revealed with modified rv similarity analysis. arXiv preprint 1iv: 022660, 2019.
Guillermo Valle-Perez, Chico Q Camargo, and Ard A Louis. Deep learning generalizes because the parameter- function map is biased towards simple functions. arXiv preprint ar Xre: 08522, 2018.
Bastiaan S. Vecling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cmns for digital pathology. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola- López, and Gabor Fichtinger (eds.), Medical Image Compuling and Computer Assisted Intervention MICCAI 2018, pp. 210 218, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00934-2.
Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun Trshad, and Andrew H Beck. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.08 05718, 2016.
Sergey Zaaoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC), pp. 87.1-87.12, September 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arxiv preprinl ar arxiv: 1611 1.0350000, 2016.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Appendix
A Training Details
For wide ResNets, we look at models with depths of 14, 20, 26, and 38, and width multipliers of 1, 2, 4, 8 and 10. For deep ResNets, we experiment with depths 32, 44, 56, 110 and 164. In CIFAR-100 experiments, the block structure only appears at a greater depth so we also include depths 218 and 224 in our investigation. For Patch Camelyon datasets, we find that depth 80 is enough to induce a block structure in the internal representations. All ResNets follow the architecture design in (He al. 2016 Zagoruyko & Komodakis, 2016).
Unless otherwise specified, train all the models using SGD with momentum of 0.9 for 300 epochs, to- gether with a cosine decay learning rate schedule and batch size of 128. Learning rate is tuned with values (0.005, 0.01, 0.001] and L2 regularization strength with values [0.001, 0.005]. For CIFAR-10 and CIFAR-100 experiments, we apply standard CIFAR-10 data augmentation consisting of random flips and translations of up to pixels. With Patch Camelyon, we use random crops of size 32x32, logether with random flips, obtain the training data, At test time, the networks are evaluated on central crops of the original images. For CKA analysis, each architecture is trained with 10 different sceds and evaluated the full test set of the corresponding domain.
B Block Structure on Out-of-Distribution Data
CIFAR-10
CIFAR-10 Corrupted
RN-26 1x
RN-44 1X
RN-56 1x
RN-110 1x RN-164 1X
RN-26 1x
RN-44 1x
RN-56 1X RN-110 1x RN-164 1x
Layer
75
Layer
50
50
25
25
75
150
150
100
250 500
255075
150
300
250
RN-38 1x
RN-38 2
RN-38 4x
RN-38 8x
RN-38 10x
RN-38 1
RN-38 2
RN-38
RN-38 8x
RN-38 10x
Layer
Layer
100
50
50
50 100
50 100
50 100
50 100
50 100
50 100
50 100
50 100 Layer
50 100
50 100
Layer
CIFAR-100
CKA
Patch Camelyon
RN-26 1x
RN-44 1x
RN-56 1x RN-110 1x RN-164 1x
RN-26 1x
RN-44 1x
RN-56 1
RN-110 1x RN-164 1x
Layer
75
Layer
75
50
50
25
25
50
150
150
100 300
250 500
255075
150
50
150
100 300
250
RN-38
RN-38
RN-38 4x
RN-38 8x
RN-38 10x
RN-38 1
RN-38 2X
RN-38
RN-38 8x
RN-38 10x
Layer
100
Layer
100
50
50
50 100
50 100
50 100 Layer
60 100
50 100
50 100
50 100
50 100
50 100
50 100
Layer
Figure 13: Appearance of block structure depends the data which representations computed. We plot CKA heatmaps for models of varying depths (top rows) and widths (bottom rows) trained on CIFAR-10, evaluated on different datasets ordered by the degree of out-of-distribution. We observe that the block structure representation are robust to small distribution shifts in the data, as evident from CKAs computed on CIFAR-10 corrupted dataset (which adds perturbations the original CIFAR-10 data) and CIFAR-100 dataset (which contains mutually exclusive classes but undergoes the same data collection procedure as CIFAR-10). However, larger shifts, such from CIFAR-10 to Patch Camelyon, produce significantly different representational structures.
16

---

Published in Transactions on Machine Learning Research (11/2022)
C
Block Structure and the First Principal Component
ResNet-38 (10x)
ResNet-110 (1xx x)
ResNet-44 (1x)
ResNet-38 (2x)
CKA
CKA
CKA
100
30
30
Layer
20
20
50
10
20
30
50
100
20
40
10
20
30
Variance explained by first
Variance explained by first PC
Variance explained by first PC
Variance explained by PC
1.00
1.00
1.00
1.00
Fraction of variance
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25-
0.25
0.25
0.25
M
0.00
10
50
100
20
30
Cos. similarity first PCs
Cos. similarity of first PCs
Cos, similarity of first PCs
Cos. similarity of first PCs
100
30
30
Layer
20
20
50
10
10
20
30
50
100
20
40
10
20
30
CKA without first PC
CKA without first PC
CKA without first PC
CKA without first PC
100
40
30
30
Layer
50
20
10
10
10
20 Layer
30
50
100
20
40
20 30
Layer
Layer
Layer
Figure 14: The relationship between block structure and the first principal component. Each column represents a different architecture. In ResNet-110 (1x) and ResNet-38 (10x), we observe a block structure in the CKA plot (top row), and find that the first principal component explains large fraction of the variance the layers that comprise the block structure (second row). We also observe that the cosine similarity of the first PCs (third row) resembles the CKA plot, and removing the first. PC before computing CKA substantially allemaatss the block structure (bottom row). By contrast, in ResNet-44 (1x) and ResNet-38 (2x), which have block structure, the first PC explains only small fraction of the variance, and the CKA plot does not resemble the cosine similarity between the first PCs, but instead resembles CKA computed without the first PCs.
17

---

Published in Transactions on Machine Learning Research (11/2022)
D Additional Visualizations of Dominant Datapoints
ResNet-32
ResNet-38
1.0
1.0
125
0.8
80
100
Layer
60
0.6
Similariy
Layer
75
0.6
Similerity
0.4
40
20
0.2
25
0.2
0.0
0.0
50
100
50
100
Layer
Layer
7500
Layer
7500
Layer 25
of Examples
5000
of Examples
5000
2500
2500
200
200
100
100
Projected Value (log)
Projected Value (log)
of Examples
7500
5000
Layer
of Examplee
5000
7500
Layer
2500
2500
200
200
100
100
-2
Projected Value (log)
Projected Value (log)
+ of Exampees
7500
5000
Layer 75
of Examplee
5000
7500
Layer 75
2500
2500
200
200
100
100
-4
-3
-2
-5
-2 Projected Value (log)
-3
Projected Value (log)
hugh
of Examples
Layer 100
# of Examples
Layer 100
2500
200
200
100
100
-5
-3
-2
-5
-3
-2
Projected Value (log)
Projected Value (log)
Dominant datapoints are not present in networks without block structure. The topmost row shows representational similarity heatmaps from two networks without block structure. The rows below show histograms of the projected values on the first PC, well images with the largest projections. Note that the distributions of projected values are unimodal, unlike bimodal distributions observed networks with a block structure (Figure 3) In addition, the datapoints with the highest projected values are highly dissimilar between layers.
Figure 15:
18

---

Published in Transactions on Machine Learning Research (11/2022)
Layer 400
Layer 450
7500
7500
# of Examples
5000
5000
2500
2500
400
400
200
200
-
-4
-3
-2
5
33
Projected Value (log)
Projected Value (log)
Figure 16: Visualization of the distribution of projected values onto the first principal component by test inputs, for ResNet-224 (1x) trained CIFAR-100. Top shows histograms of the projected values on the first PC. Bottom rows show images with the largest projections on the first PC. See Figure 3 for similar plot. for ResNet-164 (1x) trained on CIFAR-10.
Layer 200
Layer 250
7500
7500
of Examples
5000
5000
2500
2500
200
200
-5
-4
33
2
0
-5
-4
33
-2
-1
0
Projected Value (log)
Projected Value (log)
Figure 17: Visualization of the distribution of projected values onto the first principal component by inputs, for ResNet-80 (1x) trained on Patch Camelyon. Top row shows histograms of the projected values the first PC, Bottom rows show images with the largest projections on the first PC. See Figure for similar plot Re6Net-164 (1x) trained on CIFAR-10.
Layer 45
Layer 60
7500
7500
of Examples
5000
5000
2500
2500
400
400
200
200
-5
-4
-3
-2
-1
0
-5
-4
33
-2
-1
0
Projected Value (log)
Projected Value (log)
Figure 18: Visualization of the distribution of projected values onto the first principal component by inputs, ResNet-26 (8x) trained on Patch Camelyon. Top row shows hislograms of the projected values the first PC. Bottom rows show images with the largest projections on the first PC. See Figure for similar plot 164 (1x) trained on CIFAR-10.
19

---

Published in Transactions on Machine Learning Research (11/2022)
E
Dominant Examples and Layer Activations
1.0
Solid Pink
0.8
Activation Norm
800
Solid White
200
Dominant Datapoint
600
Layer
0.6
0.4
400
100
0.2
200
0.0
100
200
50
100
150
200
250
Layer
Layer
Figure 19: Solid color images strongly activate layers making up the block structure when trained on Patch Camelyon. Top row shows dominant examples for a ResNet-80 (1x) model trained on Patch Camelyon dataset. The CKA heatmap in the bottom left shows location the block structure the internal represcutations of the model. We observe that the dominant images share pink background. When we feed a synthetic image filled with this background color the network, we observe that yields even larger activations compared to the original image, for layers making up the block structure (i.e., after layer 200). See also Figure 7 for similar plot for models trained on CIFAR-10 dataset.
20

---

Published in Transactions on Machine Learning Research (11/2022)
F Evolution of the Block Structure
Within-Checkpoint Similarity
Ep.
Ep.
Ep. 3
Ep.
Ep. 5
Ep. 6
Ep.
Ep. 8
Ep. 9
Ep. 10
Layer
300
200
100
Ep.
30
Ep.
Ep.
Ep.
Ep.
Ep.
0.6
Layer
300
200
0.5
100
0.4
Ep. 120
Ep. 140
Ep. 160
Ep. 180
Ep. 200
Ep. 220
Ep. 240
Ep. 260
Ep. 280
Ep. 300
0.3
Layer
300
0.2
200
0.1
100
100
300
100 300
100 300
100 800
100
300
100
300
100 800
100 300
100 300
100 300
Similarity with Final Checkpoint
Ep.
Ep. 2
Ep. 3
Ep.
Ep.
Ep. 6
Ep.
Ep. 8
Ep. 9
Ep. 10
Layer
300
200
100
Ep.
Ep
Ep.
Ep.50 50
60
Ep.
90
100
Ep. 110
Layer
300
0.5 3
0.6
200
100
0.4
Ep. 120
Ep. 140
Ep. 160
Ep. 180
Ep. 200
Ep. 220
Ep. 240
Ep. 260
Ep. 280
Ep. 300
0.3
Layer
300
0.2
200
0.1
100
100 300 Layer
100 300 Layer
100 300 Layer
100 300
100 300 Layer
100 300 Layer
100 300 Layer
100 300 Layer
100 300 Layer
100 300 Layer
Layer
Dominant Datapoints
Epoch
Epoch
Epoch 10
Epoch 20
Epoch
Epoch
Epoch
Epoch 100
Epoch 150
Epoch 200
Epoch 250
Epoch 300
Figure 20: Fine-grained analysis of the evolution of the block structure in ResNet-110 (1x) model. This plot shows the evolution of block structure for the same network as in Figure g but with greater temporal gramularity. As Figure we find that shape of the block structure defined early training (top row). However, comparing these different model checkpoints to the final, fully-trained model reveals that the block structure representations at different epochs are considerably dissimilar, especially during the half of training process (middle row). The corresponding dominant datapoints also vary over training, even after the block structure is clearly visible the within-checkpoint similarity heatmaps (bottom row).
21

---

Published in Transactions on Machine Learning Research (11/2022)
Within-Checkpoint Similarity
Ep.
Ep. 2
Ep. 3
Ep.
Ep. 5
Ep. 6
Ep.
Ep. 8
Ep. 9
Ep. 10
Layer
300
200
100
Ep. 20
Ep 30
Ep. 40
Ep. 50
Ep. 60
Ep. 70
Ep. 80
Ep. 90
Ep. 100
Ep. 110
Layer
300
0.6
200
0.5
100
0.4
Ep.
Ep.
Ep. 160
Ep.
Ep. 200
Ep. 220
Ep.
Ep. 260
Ep. 280
Ep.
0.3
Layer
300
0.2
200
100
100
300
100 300
100 300
100 300
100 300
100
300
100 300
100 300
100 300
100 300
Similarity with Final Checkpoint
Ep.
Ep. 2
Ep. 3
Ep.
Ep. 5
Ep. 6
Ep.
Ep. 8
Ep. 9
Ep. 10
Layer
300
200
100
Ep. 20
Ep. 30
Ep. 40
Ep. 50
Ep. 60
Ep. 70
Ep. 80
Ep. 90
Ep. 100
Ep. 110
0.7
Layer
300
0.6
200
0.5
0.4
Ep.
Ep.
Ep. 160
Ep.
Ep. 200
Ep. 220
Ep.
26.
Ep. 280
Ep.
0.3
Layer
300
200
100
100
100 300
100 300
300 Layer
100 Layer
300 Layer
100 Layer
Layer
Layer
Layer
Layer
Layer
Layer
Dominant Datapoints
Epoch
Epoch
Epoch 10
Epoch 20
Epoch
Epoch
Epoch
Epoch
Epoch 160
Epoch 200
Epoch 260
Epoch 300
Figure 21: Fine-grained analysis of the evolution of the block structure in different ResNet-110 (1x) model. This plot shows the evolution of block structure for network that is architecturally identical to the Figure but trained with different seed. For this training run, the final shape of the block structure is established slightly later in training (top row), and similarity between early checkpoints and the last checkpoint is very low (middle row). Analysis of the dominant data points shows that they change substantially of training (bottom row), and continue to vary long after the shape of the block structure ceases to change in the within- check-iieekpoint similarity heatmaps.
the
22

---

Published in Transactions on Machine Learning Research (11/2022)
Similarity Between Layers of Same Checkpoint
Ep. 10
Ep. 20
Ep. 50
Ep. 70
Ep. 90
Ep. 100
Ep. 110
Ep. 120
Ep. 130
Ep. 140
1.0
Layer
0.9
250
0.8
0.7
0.6
Ep. 150
Ep. 160
Ep. 170
Ep. 180
Ep.
Ep. 230
Ep. 250
Ep. 270
Ep. 290
Ep. 300
0.5
0.4
500
Layer
0.3
250
0.2
0.1
0.0
250
250500
250500
250 500
250 500
250 500
250500
250500
250500
250
Similarity with Final Checkpoint
Ep. 0
Ep. 10
Ep. 20
Ep. 30
Ep. 40
Ep. 50
Ep. 60
Ep. 70
Ep. 80
Ep. 90
1.0
Layer
500
0.9
250
0.8
0.7
0.6
Ep. 100
Ep. 120
Ep. 140
Ep. 160
Ep. 180
Ep. 200
Ep. 220
Ep. 240
Ep. 260
Ep. 280
0.5
0.4
500
Layer
0.3
250
0.2
0.1
0.0
550 Layer
250500 Layer
250 500
500
250500
250 500
250500 Layer
250500 Layer
250500 Layer
250500
Layer
Layer
Layer
Layer
Layer
Figure 22: Evolution of the block structure over the course of training for a ResNet-164 (1x) model.
We compute the CKA between all pairs of layers within ResNet-161 (1x) model different stages of training, and find that the internal representations already contain a block structure at epoch 10. Comparing these different model checkpoints the final, fully-trained model reveals that the block structure representations at different epochs are cousiderably dissimilar,, especially during the first half of the training process.
23

---

Published in Transactions on Machine Learning Research (11/2022)
Similarity Between Layers of Same Checkpoint
Ep.
Ep.
Ep. 50
Ep. 70
Ep.
Ep. 100
Ep. 110
Ep. 120
Ep. 130
Ep. 140
1.0
Layer
100
0.9
0.8
50
0.7
0.6
0.5 40
Ep. 150
Ep. 160
Ep. 170
Ep. 180
Ep. 210
Ep. 230
Ep. 250
Ep. 270
Ep. 290
Ep. 300
0.4
Layer
100
0.3
0.2
50
0.1
0.0
50 100
50000
50 100
50 100
50 100
50 100
50 100
50 100
50 100
50 100
Similarity with Final Checkpoint
Ep.
Ep. 10
Ep. 20
Ep. 30
Ep. 40
Ep. 50
Ep. 60
Ep. 70
80
Ep. 90
1.0
Layer
100
0.9
0.8
50
0.7
0.6
Ep. 100
Ep. 120
Ep. 140
Ep. 160
Ep. 180
Ep. 200
Ep. 220
Ep. 240
Ep. 260
Ep. 280
0.5
0,4
Layer
100
0.3
0.2
50
0.1
0.0
50 100
50 100
100 Layer
50 100 Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Figure 23: Evolution of the block structure over the course of training for ResNet-38 (10x) model. We compute the CKA between pairs layers within a ResNet-38 (10x) model at different stages of training, and find that the internal representations already contain block structure epoch 10. Comparing these different model checkpoints to the final, fully-trained model reveals that the block structure representations at different epochs are considerably dissimilar, especially during the first half of the training process. See also Figure 22 for similar plot for a deep ResNet (ResNet-164 (1x)).
24

---

Published in Transactions on Machine Learning Research (11/2022)
Projected Value
0.2
0.1
0.0
50
100
150 Epoch
200
250
300
Epoch O
Epoch 30
Epoch 60
Epoch 90
2
2
Activation Norm (1)%))
Epoch 120
Epoch 150
Epoch 180
Epoch 210
2
2
2
MMLLL MUDA
AuN 1000 Am0
1.
A Conh
Epoch 240
Epoch 270
Epoch 300
2
2
Dominant Datapoint
Batch Median
MW MMM.
0
50
100
50
100
50
100
Layer
Layer
Layer
Figure 24: Example of how the activation magnitude and the projected values onto the first principal component for dominant image vary datapoint for Res.ett-3 (10x) model, we track the magnitude of its projected value onto the first principal component of the block structure representations (top left), as well its activation norms each layer in the network (bottom set of plots), over time. We observe that before the block structure represenlations stabilize and show more similarity with those the fully trained model (i.e., epoch 90, see Figure 23 above), the dominant image yields small value when projected onto the first principal component, and also doesn't t strongly activate the layers inside the block structure. This is aligned with the observation made in Figure 10 ResNet-164 (1x).
different epochs, for ResNet-38 (10x). Given a dominant
100
1.0
0.8
Fraction Varlance Explained First PC
60-
Step
40
20
0.2
0.0
50
100
150
200
250
300
350
Layer
Figure 25: Evolution of the first principal components of layer representations over the first 100 steps of training. At every step of training, we measure the proportion of variance explained by the first principal component in each layer of ResNet-110 (1x) network. At around step 30, the first principal component begins to explain the majority of the variance in the layer representations.
25

---

Published in Transactions on Machine Learning Research (11/2022)
G Training with Principal Component Regularization
To regularize the first principal component, we first compute the amount of variance that it explains using power iteration (Miyato et al. 2018 At training step, 4, we compute the batch of n convolutional feature maps with height and width w containing e channels M, E RRnxAuxx, flatten the spatial dimensions to the channels dimension to create matrix of size X; E _R** where p = x w Wxx,, and subtract column means obtain a centered matrix X,. We randomly initialize the stored cigenvoctor Uo E Rº at the beginning of training. At each training step, we perform single step of power iteration initialized from the previous eigenvector:
81 X, X
(3)
A; |1r|12
(4)
W = Vz/Ar /x-
(5)
X approximates the top eigenvalue of X X, and thus the amount of variance explained by the first principal component of the representation. The proportion of variance explained is given by \ We incorporate the regularizer additive term in the loss:
roe_reg ()e, X; 6) = a max
8,0),
(6)
where is the strength of the regularizer and 8 is the threshold proportion of variance explained at which it is imposed. In experiments, tune hyperparameter with values in [0.1, 1, 10), and set å = 0.2 based analysis of the first principal components of models without the block structure. To speed up the training process, we only apply the regularizer to ReLU layers starting from the second stage, where block structure is often found.
26

---

Published in Transactions on Machine Learning Research (11/2022)
H
Effect of Batch Size on the Block Structure
Batch Size 128
ResNet-110
ResNet-164 1x
ResNet-38
ResNet-38
1.0
500
Layer
300
100
100
200
250
50
50
0.5
100
0.0
100
300
250
500
50
100
50
100
Batch Size 16
ReaNet-110 1x
ResNet-164 1x
ResNet-38 8x
ResNet-38
1.0
500
Layer
300
100
100
200
250
50
50
0.5
100
0.0
100
300
250 Layer
100
50 100 Layer
Layer
Layer
Figure 26: Using very small batch sizes during training reduces the appearance of the block structure ure. The top shows the block structure effect in range of very deep and wide networks, trained with standard batch size = 128. We experiment with a drastically smaller batch size of 16 (bottom row) and find that the block structure is highly reduced, especially in deep models.
Impact of Transfer Learning and Shake-Shake Regularization on Similarity of Layers Inside the Block Structure
Standard Training
Transfer Learning
Seake--Saake Regularization
500
500
Layer
1000
Seed
250
250
500
CKA
Seed 2
Layer
500
500
1000
250
500
500
Layer
1000
Seed
250
250
500
500
500
500
500 0
500
500
1000
1000
1000
Seed
Seed 2 Layer
Seed 3 Layer
Seed Layer
Seed
Seed Layer
Seed 1 Layer
Seed 2 Layer
Layer
Layer
Layer
Figure 27: Training with transfer learning and Shake-Shake regularization yields models that are more similar across different training runs. Each group of plots shows CKA between layers of models with the same architecture but different initializatinns (off the diagonal) within single model (on the diagonal). In the standard training case, representations across models are highly dissimilar, especially in the block structure region. In contrast, when similarity in corresponding layers.
transfer learning and Shake-Shake regularization, comparisons across seeds show more
27

---

Published in Transactions on Machine Learning Research (11/2022)
J Block Structure Under Different Kernels
As previously identified by [gguyen et al. 2021), the block structure a phenomenon the linear CKA heatmaps of large (wide or deep) networks. In this section, we investigate whether the block structure phenomenon also arises in CKA heatmaps computed with other kernels, and also examine the effect of removing the dominant datapoints (identified by the magnitudes of their projections the first principal component, in Section 4.1 upon these CKA heatmaps.
To compute CKA heatmaps under alternative kernels, we again use minibatch CKA. The approach in Eq. [ can be easily adapted to nonlinear kernels by replacing X and Y,Y,T the linear Gram matrices formed by minibatch i, with minibatch kernel matrices K; Raxx and K E R The elements of these minibatch kernel matrices are the kernels between pairs of examples in the minibatches, i.e, Ki. k(Xi,, (
and Km kl (Y,,.. : Like linear minibatch CKA, nonlinear minibatch CKA is computed by averaging HSIC₁ across minibatches:
Σ HSIC; (K,, K!)
CK miiibatch
V Σ HSICI (K, K.)
E HSICI K()
We investigate the behavior of CKA under the linear kernel kunear (x. 3) x: y, the cosine kernel Keos (a: y) w //([<=/]] [||x||]|||]] and the RBF kernel kybr (z, 0) exp(-I<< yl / For each layer, we measure the median Euclidean distance d between examples in cach layer and set = od with E {0.2, 0.5, 1,2, 5, 10) of that median Euclidean distance. To reduce variance when computing RBF CKA with small we use a minibatch size 1000 for these experiments.
Figure 28 shows the appcarance of CKA heatmaps of narrow, shallow network (ResNet-38 1x, top), a wide network (ResNet-38 10x, middle), and deep network (ResNet-164 1x, bottom). Although heatmaps computed for small network (ResNet-38 1x) look qualitatively similar regardless of kernels, both wide (ResNet-38 10x) and deep (ResNet-164 1x) networks exhibit significant differences.
ResNet-38
Linear
Cosine
RBF 0.2
RBF 0.5
RBF
RBF 2
RBF 5
RBF 10
100
Layer
50
100
100
100
100
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
ResNet-38 10x
Linear
Cosine
RBF 0.2
RBF 0.5
RBF
RBF 2
RBF 5
RBF 10
Layer
100
50
100
50 Layer
100
50
100
50
100
50
100
50
100
50
100
50 Layer
100
Layer
Layer
Layer
Layer
Layer
Layer
ResNet-164 1x
10
Layer
400
200
250
500
250
500
250 Layer
500
250
500
500
250
500
250
500
250
500
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Figure 28: Appearance of representation heatmaps in wide and deep networks, but not narrow saarrowsshallwe networks, depends on the choice of kernel. Rows reflect different models and columns reflect different kernels, For RBF kernels, the parameter indicates the fraction of the median distance between examples (computed separately for each layer) that is used as standard deviation of the kernel.
28

---

Published in Transactions on Machine Learning Research (11/2022)
Because differences in representational similarity heatmaps ultimately reffect differences in the underlying kernel matrices, in Figure 29 we show kernel matrices of individual layers taken from inside the block structure of each network on random minibatches where the examples have been sorted in descending values of the first principal component. All kernels are sensitive to dominant datapoints, but in different ways and to different degrees. Linear kernel matrices are dominated by the similarity between dominant datapoints. The cosine kernel ignores activation norms, and finds high similarity within groups of dominant and non-dominant datapoints but low similarity between groups. The RBF kernel effectively considers all far away points to be equally dissimilar, and thus indicates that dominant datapoints dissimilar to all other datapoints, including other dominant datapoints, which are typically far in Euclidean distance (because, while aligned in direction, they have different norms).
Note that the prevalence dominant datapoints can differ across models and initializations, as previously demonstrated Figure [ The dominant datapoints are clearly visible as block in the top-left corner of the cosine kernel matrix. For ResNet-38 10x, there are in the minibatch of 128 examples that is shown, but for ResNet-164 (1x), there only 2.
ResNet-38 LLaer 76
9.9
Example
8.0
3.96
100
8.5
100
100
100
1000
100
100
100
100
Example
Example
Example
Example
Example
Example
Example
Example
ResNet-38 38 10x Layer 76
RBF
Example
7.5
7.5
7.5
7.5
100
5.0
2.5
2.5
100
100
Example
Example
Example
Example
Example
Example
Example
Example
ResNet-164
Layer RBF 1e.1
Linear
Cosine
RBF a
RBF '6.1
RBF 2 1x.1
RBF 1x.1
RBF 1e.1
Example
1.0
7.5
7.5
0.5
5.0
5.0
100
2.5
2.5
2.5
100
100
100
Example
Example
Example
Example
Example
Example
Example
Example
Kernels based on dot products, cosine similarity, or Euclidean distance are sensitive to dominant datapoints. Plots show kernel matrices for randomly sampled minibatch of 128 examples, computed from layer inside the block structure, for models that exhibit one. Examples are sorted in descending value of the first principal component of the raw activations; top rows and left columns reflect dominant datapoints.
Figure 29:
What is the effect of removing dominant datapoints upon CKA similarity heatmaps computed with these other kernels? In Figure 30 we show that, once the dominant datapoints are removed from large networks, again see only differences among CKA heatmaps computed with different kernels, in line with the results observed for shallow networks in Figure 28 There are no longer large blocks of many consecutive similar layers in any of the heatmaps. Across all choices of kernel that we have investigated, when blocks appear in CKA heatmaps, they can eliminated by eliminating the dominant datapoints.
29

---

Published in Transactions on Machine Learning Research (11/2022)
ResNet-38 10x
No Datapoints Dropped
Lineer
Cosine
RBF
RBF 0.5
RBF
RBF
RBF 5
RBF
Layer
100
50-
50
100
50
100
100
50
100
50
100
50
100
50
100
50
100
1% Most Dornnnnt Datapoints Dropped
Layer
50
50
100
50
100
100
50
100
50
100
50
100
50
100
50
100
10% Most Dominant Datapoints Dropped RBF
RBF
Layer
50
50
100
50
100
50
100
50
100
50
100
50
100
50
100
50
100
20% Most Dominant Datapoints Dropped
RBF 0.5
RBF
RBF
Layer
50-
50
100
50 100
50
100
50
100
50 100 Layer
50 100 Layer
50 100 Layer
60 Layer
100
Layer
Layer
Layer
Layer
ResNet- 164 1
No Datapoints Dropped
Linear
Cosine
RBF 0.2
RBF 0.5
RBF 1
RBF
RBF 5
RBF 10
Layer
200
250
500
250
800
250
500
250
500
250
500
250
500
250
500
250
500
1% Most Dominant Datapoints Dropped
Linear
Cosine
RBF 0.2
RBF 0.5
RBF
RBF
RBF 5
RBF 10
Layer
250
500
250
500
500
250
250
500
250
500
260
500
10% Most Dominant Datapoints Dropped ABF 0.5
Linear
Cosine
RBF 0.2
RBF
RBF
RBF 5
R8F
Layer
250 Layer
500
250 Layer
500 Layer
250 Layer
500
250 Layer
500
250 500 Layer
250
Layer
Layer
Figure 30: Removing dominant datapoints eliminates both blocks and differences among kernels in CKA representational similarity heatmaps. remove the top k% examples according the magnitudes of their projections the first PC in layer 76 for ResNet-38 10x and layer 276 for ResNet-164 1x. Because 14/128 datapoints are dominant in the minibatch kernel matrix shown in Figure 29 we provide results removing 20% of datapoints for that network, although they look only modestly different from the results obtained by removing 10% of datapoints.
30


Reviews:
[{"review_id": "GQaPVHCdxT", "reviewer": "Reviewer_JN8m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Experiments provide non-trivial observations that may be interesting.\n\n2. The paper is technically sound.\n\n\nWeaknesses:\n\n1. The motivation and the goal of the proposed study are unclear.\n\n2. Conclusions are based on experiments with a relatively limited scope.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper empirically studies the similarities and correlations between network responses at different layers, called the block-structure effect after the work by Nguyen et al., 2021, that emerge in deep neural networks after training them on a classification dataset. The primary finding of the paper is that only a small subset of the training datasets, called dominant data points, contribute to this phenomenon. It is observed that these data points share high-level semantic similarities, e.g., having similar backgrounds. It is hypothesized that since according to empirical observations, these data points change depending on the random seed, the exact block-structure effect will also be different for different initialization random seeds. Experiments are performed by training ResNet-based architectures using CIFAR10, CIFAR100, and Patch Camelyon datasets. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I think this paper is not ready for publication and needs significant improvement: \n\n1. I agree some of the observations are non-trivial but the motivation is unclear. It is unclear to me what is the goal of the explorations in this work. What will be the benefits of the findings for the ML community? I think the the introduction portion of the paper needs to be rewritten. It is not clear to me why the block structure is even important, to begin with. The motivation, importance, and benefits of this study must be stated clearly.\n\n2. Conclusions are based on empirical investigations that are very limited in my opinion. It is not easy to draw conclusions by running experiments on one network and three datasets. At least several architectures should be considered and more complex datasets need to be used in experiments to draw more confident conclusions. It is really non-trivial to me that the findings of experiments are generalizable. \n\n3. The paper lacks theoretical insight. I understand the field of deep learning has a highly empirical nature but when a work is about providing insights into deep neural networks, I expect more justifications.\n\nIn conclusion, this work needs significant improvements to become suitable for publication.", "broader_impact_concerns": "No concern could be detected.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3NtFXi9ZAj", "reviewer": "Reviewer_uytN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* Well written and clear\n* Analysis of the origins of the block-based structure\n* Has some interesting insights (connection of the structure with model complexity, the statistics of the training dataset, used augmentations, training procedure...)\n\n**Weaknesses**\n* One of the claimed contributions is how to eliminate these block structure. However, whether this actually is necessary remains unclear from the introduction. \n* Further, the effect of the proposed interventions on the accuracy of a network is only presented for one of the described methods and only in the Appendix. In my opinion this should be part of the main paper as well. Otherwise it is unclear, whether it is actually necessary to remove/avoid these block structures.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analysis the block structure observed in neural networks. It identifies connections to the dataset statistics, training procedure and the network size. Further, it gives some suggestions on how to reduce these structures. The behaviour has been analyzed for several network architectures and datasets. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Important\n* Adding the effect of the described methods in section 6 on the accuracy, otherwise the made claim that eliminating the block structure is not interfering with generalization is not fully evaluated.\n\nChanges which would strengthen the work\n* Making the motivation clearer on why eliminating the blockstructure is necessary\n", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qnS8JAEv6t", "reviewer": "Reviewer_LF8A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- scope of experiments- I appreciate the scope of experiments in the paper.  The authors experiment with multiple datasets and architectures and find similar patterns across datasets/architectures.  I also liked the design of experiments in the paper since they did a great job of demonstrating that the block structure phenomenon originates from dominant datapoints.  For example, the authors include experiments where the remove a fraction of the dominant datapoints from evaluation and find that the block structure disappears and show that for models without block structure, there are no dominant datapoints.\n- writing is clear\n\nWeaknesses:\n- motivation- while the results are interesting, the motivation for the paper is not very clear to me.  Why should we be interested in the block structure phenomenon?  Can knowledge of the origin of the block structure phenomenon guide us towards improving training methods for high capacity NNs?  While the paper does look at different regularization techniques and their impact on block structure and training accuracy, it isn't very clear to me whether we should be trying to learn models that avoid block structure or not.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at the block structure phenomenon which arises during standard training of high capacity NNs.  They find that this phenomenon arises from what they introduce as dominant datapoints, which are a small portion of datapoints which have significantly larger value when their activation is projected onto the first principle component.  They find that these dominant datapoints often share similar backgrounds.  The authors also look at the evolution of block structure during training and find that the block structure arises early during training, but the set of dominant datapoints varies throughout training.  Additionally, they look at the impact of various forms of regularization on block structure and find that regularization can remove block structure without hurting accuracy.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- clarifying the motivation of the study of the block structure phenomenon (would strengthen work in my view)", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x8WQ0nLCIL", "reviewer": "Reviewer_GD8Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper provides a very comprehensive empirical analysis of the block structure phenomenon, and conducts an extensive battery of experiments to support its claims. The experiments conducted towards validating hypothesis are sound, well-thought, and convincing. In particular, the data augmentation experiments highlight the impact of dominant data points, and the visualisations of activation distributions, prominent data points, and layer-wise similarity heatmaps clearly highlight the role of different data points during model training, as well as the significant change in these data points over time. From a writing perspective, the paper is easy to follow and is very well illustrated. The contributions are accurately stated at the beginning of the manuscript and the main findings are discussed well. \n\n**Weaknesses:** \n\nThough the empirical analysis of this phenomenon is quite extensive and concrete, I feel that the paper lacks a theoretical component/discussion supporting its claims. For instance, the intuitions about dominant data points seem well-justified, and the experimental results showing their effects appear solid. However, it is not clear *why* this phenomenon arises from reading the paper, or what sets of properties make a data point dominant. The paper mentions that dominant data points present high-level features that are statistically meaningful, e.g., background color, but do not investigate this point any further. Moreover, the discussion on variability of dominant data points also necessitates explanation. More specifically, it would be very interesting to explore what conditions enable data points to arise as dominant near the beginning of training, as this could offer an avenue to solving the problem beyond the more patch-like regularisation techniques discussed. Indeed, this ties in nicely with the earlier point on \"What makes a data point dominant?\" Clearly, the answer lies both within the dataset statistics and in the randomness of the ensuing model/training, but this line of thought is not expanded on, and this severely limits the applicability and relevance of the paper's insights more broadly. \n\nFrom a writing perspective, I find that the paper would benefit strongly from re-positioning some of its sections and including many sections from the appendix. Throughout the main body, the appendix is  heavily referenced, and as a result the paper cannot be read in a stand-alone fashion. I therefore recommend that the authors revisit their writing so as to make the main body self-contained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides an improved understanding on the phenomenon of block structures ,i.e. high levels of similarity among consecutive neural network layers, arising within the intermediate representations of deep neural networks. In particular, it reconciles the seemingly contradictory observations that block structures regularly occur (suggesting good fitting to the data and generalization) while the resulting block representations being quite variable across training seeds (suggesting overfitting to noise in the data). Indeed, the paper studies a variety of deep neural networks and proposes an explanation based on *dominant data points*. Dominant data points, which themselves are a small proportion of the overall data provided in training, ultimately yield strong activations at the beginning of training, leading to the block structures empirically observed. These data points typically share several high-level features, e.g., background colors, and are dictated by the statistics of the input dataset. However, these data points are not consistent throughout training, and in fact change significantly, both during training and across different training initializations due to batch and initialization randomness, which in turn explains the variability in block structures. \n\nTo validate the influence of dominant data points on block structures, multiple experimental protocols are proposed. First, dominant data points are shown to yield large activations within neural networks, thus dictating the principal components of their intermediate representations. This is further confirmed through the removal of said data points, following which block structures disappear. The underlying shared properties of these data points are also explored through an augmentation experiment, in which solid color images, based on dominant data points, are introduced, and these in turn lead to even higher activations than the original images. Moreover, the variability of the block structure is analyzed, and shown to change significantly over the course of training: Dominant data points are not static, but rather change substantially over time, likely due to the minibatch training process and to random initial model configurations. Finally, the paper studies regularization techniques, principal component and shake-shake regularisation, showing that they can effectively alleviate block structures without harming model performance (even at times yielding minor improvements), and mentions that transfer learning can also help mitigate this phenomenon.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1) Please provide a more in-depth analysis, ideally supported by probabilistic/theoretical arguments, for the properties of dominant data points. It is currently clear that data points exhibit statistically meaningful high-level features, yet this is a somewhat hand-wavy insight that really should be studied in more depth. Moreover, this becomes more confusing when considering that random seeds and training significantly affect which data points become dominant. Hence, I strongly recommend that such an analysis be conducted, so as to understand jhow dominant data points are brought about, both from a data and model perspective. \n\n2) On a more minor note, I believe that the manuscript should be revised to minimise references to the appendix for essential arguments that are key to the flow of the paper.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has been reviewed by four reviewers. Three of them recommended "leaning accept" or "accept", while one recommended "reject". 

The main concerns include the following. (1) Motivation and consequence. It is unclear what is the motivation of studying block-structure effect, and whether understanding this phenomenon can lead to improved training of deep networks. (2) Theoretical analysis. The paper is mostly empirical, without theoretical analysis. The authors pushed back on these two issues in their replies. 

I think the phenomenon studied in this paper is interesting and noteworthy, and the origin of the phenomenon as revealed by the paper is convincing. So I lean towards accepting the paper. 

In the revision, I hope the authors will continue to strengthen the part on motivation and consequence, and if possible, consider adding some theoretical understanding. 
Decision: Accept (poster)