Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations
Authors: Thao Nguyen
Maithra Raghu
Simon Kornblith
Venue: ICLR 2022 Submitted (2022)
Keywords: representation learning
representational similarity
distributed representations

Abstract: Recent work has uncovered a striking phenomenon in large-capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block structure has two seemingly contradictory properties: on the one hand, its constituent layers have highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure in relation to the data and training methods. By analyzing properties of the dominant PCs, we find that the block structure arises from dominant datapoints — a small group of examples that share similar image statistics (e.g. background color). However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds. Thus, the block structure reflects meaningful dataset statistics, but is simultaneously unique to each model. Through studying hidden layer activations and creating synthetic datapoints, we demonstrate that these simple image statistics dominate the representational geometry of the layers inside the block structure. We also explore how the phenomenon evolves through training, finding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant datapoints continue to change substantially. Finally, we study the interplay between the block structure and different training mechanisms, introducing a targeted intervention to eliminate the block structure, as well as examining the effects of pretraining and Shake-Shake regularization.     

Under review as a conference paper at ICLR 2022
DOMINANT DATAPOINTS AND THE BLOCK STRUC- TURE PHENOMENON IN NEURAL NETWORK HIDDEN REPRESENTATIONS
Anonymous authors Paper under double-bline review
ABSTRACT
Recent work (Nguyen et al. 2021 has uncovered a striking phenomenon in large- capacity neural networks: they contain blocks of contiguous hidden layers with highly similar representations. This block structure has two seemingly contra- dictory properties: on the one hand, its constituent layers have highly similar dominant first principal components (PCs), but on the other hand, their represen- tations, and their common first PC, are highly dissimilar across different random seeds. Our work seeks to reconcile these discrepant properties by investigating the origin of the block structure relation to the data and training methods. By analyzing properties of the dominant PCs, we find that the block structure arises from dominant datapoints - a small group of examples that share similar im- age statistics (e.g. background color). However, the set of dominant datapoints, and the precise shared image statistic, can vary across random seeds. Thus, the block structure reflects meaningful dataset statistics, but is simultaneously unique to each model. Through studying hidden layer activations and creating synthetic datapoints, demonstrate that these simple image statistics dominate the repre- sentational geometry of the layers inside the block structure. We also explore how the phenomenon evolves through training, finding that the block structure takes shape early in training, but the underlying representations and the corresponding dominant datapoints continue to change substantially. Finally, we study the inter- play between the block structure and different training mechanisms, introducing targeted intervention eliminate the block structure, as well as examining the effects of pretraining and Shake-Shake regularization.
INTRODUCTION
Many modern successes of deep neural networks have used simple rules to systematically increase model capacity, often through scaling architecture depth and width (Tan & Le 2019). These large capacity models typically also maintain strong performance even in tasks with small amounts of training data. This has led their widespread data-scarce, high-stakes settings such as medical imaging (Wang et al. 2016 Liu et al. (017))
across different many applications, including
However, recent work has shown that the representational structures of these large capacity models exhibit distinctive properties that are not present in shallower/narrowerr networks. Specifically, when using linear centered kernel alignment (CKA) (Kornblith et al. 2019) measure similarity between hidden representations of neural network layers, Nguyen et al. (2021 I) find that large set of con- tiguous layers that share highly similar representations. This is visible as a clear block structure in the heatmap of pairwise linear CKA A similarities between layers (see Figure This block structure phenomenon is robust, appearing both in models trained on natural image datasets and in models trained medical imaging datasets (Figure[p)..
The emergence of the block structure as model capacity increases does not merely indicate a change in relationships among layer representations; it is also associated with changes in the properties of individual representations. As shown in Figure 2h, layer representation inside the block structure have dominant first principal components (PCs) that explain the majority of variance in activations, whereas the first PCs representations of layers outside the block structure, or in networks without block structure, explain far less variance. But counterintuitively, although layer representations and

---

Under review as a conference paper at ICLR 2022
(a)
Desper
CIFAR-100
ResNet-26 26
ResNet-64 1x
Res 1x
Hestiee 164
HesNett25 1x
ResNet-218 1x
ResNet-22 10x
75
75
25
50
50 100 150
150 300
250
500
25 50 75
800 600
25 60
75
Wider
Patch Camelyon
FasNNet-38 1x
ReeNee-33 2x
ResNiet-30 ex
RusNest336 10x
ReeNet-20 1x
ResNest80
ResNeet-20 10x
100
50
25-
100
50 100
100
50
25 50
125 Layor
25 50
Similanty
Layer
Layer
Layer
Layer
Layer
Layer
Figure 1: (a): Block structure arises in wide and deep networks. Hcatmaps show similarity between layer representations, measured with linear CKA, for models of varying widths and depths trained on CIFAR- 10. As models become wider deeper. "blocks" of consecutive layers share similar representations. (b): Block
structure arises on many datasets. Models trained on CIFAR- --00 the medical histopathology dataset Patch Camelyon show similar behavior to those in panel (a).
(a)
ResNet-38 (2x)
ResNet-38 (10x)
(b)
ResNet-32 1x Seed
Seed
Seed
30
30
Loger
20
20
25 50 76
26 50 75
ReaNat-164
Seed
Exc.
20
30
10
20
30
500
250
20
500
250 500
250 500
Layor
Layor
Layer
Layer
Layer
Figure 2: (a): The first principal component of representations inside the block structure explains the majority the variance representations. Top: Linear CKA similarity heatmaps networks that do and do not exhibit block structure. Bottom: Fraction variance in representations explained by first principal component. See Appendix for more networks. (b): Representations inside the block structure are highly unique to each seed. 2 lefimost panels show similarity of representations within networks trained with 2 different random seeds. Rightmost panel shows similarity between the networks.
their dominant first PCs are similar across the layers that make up the block structure, they are highly dissimilar across random seeds, shown in Figure 2p. Such representational inconsistencies have been linked to overfitting and poor generalization (Morcos et al. 2018).
The prevalence of the block structure, and this discrepancy in its properties, motivate a pressing fundamental question- - is the block structure sign of overfitting to idiosyncrasies of the data and training process, or does pick up meaningful signals? In this paper, we uncover answers this question, investigating the origin of the block structure in relation to the data and training methods, and reconciling its contradictory behaviors. Specifically, our findings are as follows:
The dominant first principal components of the layers that make up the block structure arise from small number of dominant datapoints that share similar characteristics (e.g., background color).
But the set of dominant datapoints and their common characteristies leading to the observed block structure dissimilarities across random seeds.
vary across training rums,
Like the block structure phenomenon, dominant datapoints can only be found in large capacity neural networks. We show that these dominant datapoints are excluded from the dataset used for representational analysis, the block structure effect disappears.
We find that dominant datapoints strongly activate the hidden layers corresponding to the block structure. By constructing synthetic examples based off of their shared image characteristics, we show that these characteristics are indeed responsible for the high activation norms.
The block structure emerges carly in training, but carly block structures have different representa- tions and yield different dominant datapoints than the final block structure at the end of training.
We study whether it possible to eliminate the block structure without interfering with general- ization using novel principal component regularization method. We also show that alternative
2

---

Under review as a conference paper at ICLR 2022
training mechanisms such transfer learning and Shake Shake regularization can reduce the block structure and yield more consistent representations across different training runs.
2 RELATED WORK
Previous work has studied certain propensities of deep neural networks in the standard training set- ting, such as their simplicity bias (Huh et al. [202]] Valle-Perez et al. 2018 Nakkiran et al. 2019), texture bias (Baker al. 2018 Geirhos et al. 2018 Hermann al., 2019 and reliance on spu- rious correlations (McCoy et al. 2019: Geirhos et al.) 2020 Ribeiro et al. 2016 Jo & Bengio, 2017 Hosseini Poovendran 2018). Inspired by the findings of Nguyen al. (2021) that deep and wide networks learn many layers with very similar representations, we seek to characterize the kind of signals these layers may overfit to. Additionally, our work also explores how this sig- nal varies over the course of training instead of focusing analyze the behavior of the internal representations of models of varied depths and widths, using methods for measuring similarity of neural network hidden representations (Kornblith et al.) 2019
single, final model. To do so,
Raghu et al. 2017 Morcos et al., 2018). Besides shedding light on model training procedures Goot mare et al. 2018 Neyshabur et al. 2020), features (Resnick al. 2019 Thompson et al. 2019
Hermann & Lampinen 2020). and dynamics [Maheswaranathan et al. 2019). representational anal-
ysis has also furthered understanding of network internals in applications of deep learning, such as
medicine (Raghu et al. 2019) and machine translation (Bau et al. 2019).
Our work also related previous attempts understand the properties of overparameterized models (Zhang et al. (0016). Most theoretical work in this area has focused on linear models, mod- els with random features, or kernel settings (Belkin et al., 2018: Hastie et al. 2019, Liang et al. 2020 Bartlett et al. 2020). all of which lack intermediate features, or involves networks without nonlinearities (Advani et al. 2020). Our results suggest that the behavior of intermediate features of practical neural networks changes dramatically with increasing overparametterization in ways that are not obvious from previous analysis.
EXPERIMENTAL SETUP AND BACKGROUND
Mcasuring Representation Similarity with CKA: Centered kernel alignment (CKA) (Kornblihh) al., 2019 Cortes al. 2012) addresses several challenges measuring similarity between neural network hidden representations including (i) their large size; (ii) neuronal alignment between dif- ferent layers; and (iii) features being distributed across multiple neurons in a layer. Like Nguyen et al. 20221), we use the minibatch implementation of linear CKA with batch size of 256 sampled without replacement from the test dataset, and accumulate statistics over 10 epochs to allow the minibatch estimator to converge.
More concretely, given minibatches of n examples, and two layers having Pt neurons and P2 neurons each, minibatch CKA takes inputs & pairs of centered activation matrices (X,, . (X,, Y2) where X, E Raxp: and Y, E maxpa reflect the activations of these layers to the same minibatches. It produces a scalar similarity score between 0 and I by averaging HSIC scores over minibatches:
E HSIC (X,X) Y.Y,)) PAN
CKAmiuibatch
V EI= HISICI(X,X) X EL HSIC (Y,Y, Y.Y,))
where HSIC, is the unbiased estimator from Song et al. (2012). The result is an estimator of CKA that converges to the same value regardless of the batch size.
We compute CKA between all layer representations, including before and after batch normaliza- tion, activations, and residual connections. In experiments that involve tracking how a model's in- ternal properties (principal components of activations, representation similarity, etc.) change across epochs, we set batch normalization layers to be in training mode, to reduce the difficulty of adapting to batch statistics of the test set when the model has not converged.
The Block Structure Phenomenon: Nguyen et al. (202I) use linear CKA to compute the represen- tation similarity for all pairs of layers within the same model and visualizes the result as a heatmap (with x and axes indexing the layers from input to output). They find a contiguous range of hidden layers with very high representation similarity (yellow squares heatmaps in Figure TT) in very
3

---

Under review as a conference paper at ICLR 2022
deep wide models, and call this phenomenon the block structure. The block structure arises in net- works that are large relative to the size of the training set - although small networks do not exhibit block structure when trained on CIFAR-10, they do exhibit block structure on smaller datasets.
Representations of the layers that comprise the block structure exhibit different representational geometry than other layers. For layers inside the block structure, the first principal component explains a large fraction of the variance in representations: this is not the case for the rest of the layers or for networks without the block structure (Nguyen et al. 2021). We replicate this observation in Figure Ph. The substantial similarity between layers inside the block structure reflects a high degree of alignment of their first principal components, as can be seen from the following decomposition of linear CKA for centered activation matrices X E Raxm, Y E RaxP2;
CKA(XXT, FEEXT,YYT)A I Ee= Th
X X (u'x )
(2)
VE V ES=
where u'x RR and u E R the jth normalized principal components of X and Y. and X and Xy are the amounts of variance that these principal components explain (Kornblith et al. 2019). As the fraction of variance explained by the first principal component of each representation approaches 1, CKA becomes measure of the squared cosine similarity between first principal components (u'x Y² Nguyen et al. (2021) conclude that the block structure preserves and propagates a dominant first principal component across many hidden layers.
Our setup closely follows that of Nguyen et 2021 and analyzes ResNets of varying depths and widths, trained on common image classification datasets CIFAR-10 and CIFAR- 100 (Kizhevsky et al. 2209),, as well as the medical imaging dataset Patch Camelyon (Veeling et al.) 2018). These datasets are chosen to reflect the image statisties found in different domains, and we observe that they all easily induce block structure in reasonably sized ResNets.
Datasets & Models:
The ResNet architecture design follows Zagoruyko & Komodakis (2016), with the layers distributed evenly between three stages each marked by a different feature map size and the number of channels is doubled after each stage. To scale the model depth and width, we increase the number of layers and channels respectively. In experiments involving Shake-Shake regularization (Gastaldi) 2017), the network is modified to have branches that are combined in a stochastic fashion. More information on hyperparameters can be found in Appendix A
4 DOMINANT DATAPOINTS AND HOW THEY SHAPE THE BLOCK STRUCTURE
4.1 DATAPOINTS THAT ACTIVATE THE FIRST PRINCIPAL COMPONENTS
Motivated by previous evidence that the block structure propagates dominant principal component across its constituent layers, we attempt to interpret the signal carried by this PC by examining the distribution of values after projecting all test examples onto the first PC of each layer's activations. We find that the distribution is bimodal: while most values are concentrated within a certain range (in terms of magnitude), there exist some that are orders of magnitude bigger (Figure 3
We call these datapoints with large projections dominant datapoints, and find that they are consistent across the range of layers making up the block structure, as seen from each column of images in Figure Sshowing the dominant datapoints for two different layers in the same block structure. This explains why the first principal components of different layer activations inside the block structure are highly similar (Figure 2), an observation made earlier in (Nguyen et al. 202T). Moreover, this dominant datapoint phenomenon is present only in networks that also exhibit a block structure. As shown in Appendix C. in networks without block structure, projections on the first principal component are unimodally distributed and the corresponding datapoints differ between layers.
Dominant datapoints are visually similar. In the left column of Figure 3 we observe that all data- points have a blue background, although the precise shade of blue varies. However, the background colors that the first principal component picks up depend on the random seed used to train the model. The right column of Figure shows the corresponding properties of an architecturally iden- tical model trained from different seed, where the dominant datapoints share white backgrounds instead. Refer to Appendix C for visualizations of dominant images found in other tasks (CIFAR- 100, Patch Camelyon) and model architectures (wide ResNet). We find that besides background
4

---

Under review as a conference paper at ICLR 2022
Seed Layer 250
Seed Layer 250
of Brommle
75500 5000
5000 2500
7600
2500
200
200
100
100
-5
2 Projected Value (logi
-5
Projected Value (log)
Seed Layer 50D
Seed 500
7500
7500
8 of Exeamplee
2500
2500
200
200
Projected (log)
Projected Value (log)
Figure 3: Visualization of the distribution of projected values onto the first principal component by test inputs. There exist small number of datapoints that yield significantly larger projected values than the rest and dominate the first principal component of the network. Here show examples of those dominant images for different seeds ResNet- 164 (I)) (columns). Each seed's dominant images share similar background colors, but these background colors differ between seeds. Further visualization of dominant datapoints across different layers for instance, layers 250 and 500 in this case show that they are consistent across layers making up the block structure. See Appendix C for analysis of other models and tasks.
Removed
1% Removed
10% Removed
200
200
200
Frui. Ver Exp.
400
400
400
200
200
200
Ver Exs
Frac.
400
400
400-
200
200
Var Exp.
Fruc.
250
500
250
500
250
500
Figure 4: Removing small number of dominant datapoints eliminates the block structure. Plots show largest projections on the first layer 300 ResNet-164 (1 x)
the effect removing examples models. Columns reflect different numbers examples removed; rows reflect models trained from different seeds. Within each group. the top left plot shows linear CKA heatmaps, the bottom left shows the fraction of variance explained the first PC, and the images reflect the new examples with the largest projection the first PC after data removal.
color, the dominant datapoints can also reflect other simple image patterns that prevalent in the dataset, such as the appearance of large dark spots in histopathologic scans (Appendix Figure
Finally, the block structure observed linear CKA heatmaps arises solely from dominant datapoints. As seen in Figure when the 10% most dominant datapoints are excluded from evaluation, the block structure completely eliminated all training runs of ResNet- 164 (1x) that we examined, and the fraction of variance explained by the first PCs is substantially reduced. For one seed, removing only the 1% most dominant datapoints is already enough achieve this effect. Thus, the block structure is completely determined by the dominant images, and is sensitive to the frequency of the dataset statistics that they capture. We investigate the effect of removing dominant datapoints on CKA heatmaps computed with nonlinear kernels in Appendix Figure
5

---

Under review as a conference paper at ICLR 2022
CIFAR-10
Activation Norm
4000
Activation Norm
400
1000
0.50
2000
200
0,25
250
500
0.00
100
200
300
400
500
0.0
0.5
Leyer
Layer
Projected
Patch Camelyon
1.00
Activatton Norm
750
Dominant Datapoint
Aasileae Saniinnouo Norm
400
200
0.75
Batch Median
0.50
gg
500
200
100
250
WWW
NW
0.00
200
50
100
150
200
250
0.0
0.2 Projected Value on First PC
Layer
Layer
Figure 5: Datapoints that dominate the first principal components of the block structure also strongly activate the corresponding layers. We explore the relationship between dominant datapoints and activation norms for ResNet-164 x1)) trained CIFAR-10 (top row) and ResNet-80 (1 (1x) trained on Patch Camelyon data (bottom row). For layers inside the block structure (left column), dominant datapoints (inset) produce much larger activation norms than median randomly selected minibatch (middle column). Moreover, within these layers, the norms of the activations of different datapoints are highly correlated with the magnitudes of their projections on the first principal component (right column).
4.2 DOMINANT DATAPOINTS HAVE LARGE ACTIVATION NORMS
Having discovered dominant datapoints by projecting onto the first principal component of the un- derlying representations, we next explore their implications on other properties of the internal rep- resentations, such as through layer activations. We investigate what happens the activations of dominant example as it propagates through the network, and observe that it strongly activates the parts of the network with a visible block structure. Figure shows ResNet trained on CIFAR- 10 that has preference for white background images (top row), and for another ResNet trained on Patch Camelyon that responds to clear pink backgrounds (bottom row). Both models contain the block structure in their internal representations, and find that in the corresponding layers, the activations of the dominant datapoints are substantially bigger than the median activations of the minibatches they are part of.
shows two dominant datapoints, for
4.3
CASE STUDY: IMAGE BACKGROUNDS AS A DOMINANT DATASET PROPERTY
As background colors appear to be a common dataset characteristic that is picked up by many large- capacity models, we provide further evidence, through data and training manipulations, to confirm that this is a real property the hidden representations that emerges only with overparameterization.
First, attempt to illustrate the connection between specific background colors, which vary across random seeds, and the layer activations. To approximate this data statistic, we repeat only the top left pixel in each image across the entire dimensions of the image, obtaining solid color images. These synthetic images indeed yield even larger activations compared to the dominant images they taken from, and different initializations of architecturally identical networks respond different synthetic images. For instance, given the ResNet-164 (Ix) model that has dominant datapoints containing a blue background (Figure 3). its hidden layers are further activated when all image pixels are replaced with the same shade of blue, but solid white image produces considerably smaller activations (Figure In the same figure, we observe the opposite trend for another ResNet- 164 (1x x seed, which has been shown to pick up on white backgrounds (see Figure 3). Refer to Appendix for similar analysis on the Patch Camelyon dataset.
Intervention: Color Augmentation: Given earlier insights, a natural intervention to prevent the network from potentially picking up background color signal is adding color augmentation. This includes randomly dropping color channels and jittering brightness, contrast, saturation, and hue of training images (Howard 2013 Szegedy et al., 2015). As shown in Figure training with this data augmentation reduces the block structure effect large capacity models expected.

---

Under review as a conference paper at ICLR 2022
Seed
Activation Norm
3000
Solid Blue
Solid Green
Solid White
2000
Deminant Datapoint
1000
0
100
200
300
400
500
Seed
4000
Activetion
2000
100
200
300
400
500
Layer
Figure 6: Solid color images strongly activate intermediate layers. Rows show models with the same ar- chitecture (ResNet-164 (1 x)) that are trained from different random initializations. The first model's dominant datapoints consist of images with blue backgrounds (see inset images), whereas the second network prefers white background images.
then observe that layers the first model are strongly activated by solid blue images, but not solid white images, whereas the second model shows the opposite pattern. Layers of both mod- els are strongly activated by their respective dominant datapoints (red lines), but other solid colors (c.g., green) do not yield strong activations either. To improve readability of the plot, we plot only the representations for similar findings on Patch Camelyon models.
the end of each ResNet block. See Appendix E
Without Calor Augmentation
ReaNet-110 1x
RasNat-164 1x ResNer-38 ex
ResNet-38 10x
1.0
Figure 7: Simply adding color augmenta- tion helps reduce the block structure ef- fect. Having established that the activations and representational components of some large-capacity models pick up common background colors from the inputs, we periment with color dropping and color jit- tering during training to counter this effect. Indeed, color augmentation minimizes the block structure appearance in the internal representations.
Laye:
300
100
100
200
250
0.5 g
100
0.0
100
300
250 500
50 100
50
100
With Color Augmentation on
ResNst-110
ResNet- 164
ResNet-38
ResNet
1.0
Layer
a00
200
500
100
0.5
3
50
50
0.0
100
300
250 500
50 100
50 Layer
100
Layer
Layer
Layer
EVOLUTION OF THE BLOCK STRUCTURE DURING TRAINING
In the previous section, we characterize the signals the block structure propagates across its layers, and explore their implications on other aspects of the network internals. Informed by these findings, we next explore what happens to the block structure and the dominant images over time, from initialization until the model converges, and how this process varies across different training runs.
Figure g shows the evolution of the internal representations of a ResNet- 1 110 (Ix) model as it is trained for 300 epochs on CIFAR-10, and tracks how similar each checkpoint is to the final model. We observe that some structure in the CKA heatmap already present by the first epoch, and the heatmap undergoes little qualitative change past epoch 20 (top set of plots). However, when we inspect the corresponding dominant datapoints and compare the hidden representations between intermediate checkpoints and the final model (bottom rows of plots), we find that the block structure does not always carry the same information. Instead, the representations, and corresponding groups of dominant datapoints, only stabilize much later training. We observe similar behavior for other models in Appendix [F] and note that the differences in block structure representations across random seeds already take shape near the start of training as well. Overall these findings suggest that the uniqueness of the block structure representations in large-capacity models can be attributed to both initialization parameters and the image minibatches the models receive throughout training.
To further explore the link between dominant datapoints and fluctuations in network representations, in Appendix Figure F.4 we track the magnitude of the projected value on the first PC of a single dominant datapoint found at the end of training, and find that the value plummets epochs when the internal representation structure diverges from that of the fully trained model (i.e., epochs 0, 120 and 240). At these epochs, the dominant datapoint does not produce large activations either (bottom set
7

---

Under review as a conference paper at ICLR 2022
Within- Chinki-the Similarity
Ep.
Ep.3
Ep.
Ep.
Ep.
Ep.
Ep.
Ep. 100
E0.
Ep. 300
CKA
200
100 300
100
300
100
300
100
300
100
300
100
300
100
300
100
300
100 300
100
300
Similarity with Final Checkpoint
Ep.
Ep.
Ep.
Ep.
Ep.
Ep.
Ep.50
Ep. 100
E0.
Ep. 300
300
200
CKA
100
100 300
100 300
100
300
100
300
100 300
100 300
100
100
300
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Dominant Datapoints
Epoch
Epoch
Epoch
Epoch
Figure 8: Block structure phenomenon arises early during training, but the correspondlng dominant datapoints continue change substantially. We compute the CKA between all pairs of layers within ResNet-110 (1x) model different stages of training, and find that shape of block structure is defined carly in training (top row). However, comparing these different model checkpoints to the fully-trained model reveals that the block structure representations at diffcrent epochs are considerably dissimilar to the final rep- resentations, especially during the first half of the training process (middle row). The dominant datapoints also vary significantly over the course of training, even after the block structure clearly visible in the heatmaps (bottom row). Appendix [F similar plots with greater granularity, different seeds and architectures.
of plots). This illustrates the significance of the observed variations in dominant datapoints through the training of large-capacity models.
BLOCK STRUCTURE AND TRAINING MECHANISMS
Having observed how the internal representation structures more specifically the dominant PCs of the layer representations could vary significantly over the course of training, we turn to ex- amining the interplay between the block structure and the training mechanism. Even though the block structure arises naturally with standard training, previous work has suggested that the block structure may an indication of redundant modules the corresponding networks (Nguyen al.] 2021). Thus, it is natural to ask whether it is possible train large-capacity models that do not have a block structure, and how such models perform compared to those with block structures.
Since the block structure reflects the similarity of dominant first principal component, propagated across a wide range of hidden layers (see Section B). we study whether regularizing the first principal components of layer activations would eliminate the block structure. More specifically, we estimate the fraction of variance explained by the first principal component of each layer using power iter- ation and penalize it in the loss when it exceeds 20%. We provide full implementation details in Appendix G The resulting heatmap, in Figure 9 top right, shows that not only does this eliminate the block structure from the internal representations, but surprisingly there is also no detrimental effect on performance (Appendix Table G.T). We even observe small accuracy improvements on CIFAR- 100 and in the low-data regime, as shown in Appendix Table G.1
Other standard training practices that are commonly used to boost performance are also effective reducing or eliminating the block structure effect. Shake-Shake regularization (Gastaldi] 2017) eliminates the block structure for all of the network sizes that we examine (Figure 9 bottom left). whereas transfer learning (Figure bottom right) and training with smaller batch sizes (Appendix(H) appear to reduce the appearance of the block structure, although blocks are still discernible in the largest models that we trained. addition to regularizing the block structure, these training methods also produce more similar representations across different training runs of the same architecture configuration (Appendix Figure [.I
Overall, our findings suggest that it is possible to obtain good generalization accuracy in networks with and without block structure. We observe that learning processes that reduce the dominance of the first PC of the representations provide slightly higher accuracies than standard training. However,
8

---

Under review as a conference paper at ICLR 2022
Standard Training Hastict-18
Principal Component Regularization
Restiet- 10
RasNict 38 10x
ResNet-110
Restict
Hashiet 36 10x
400
100
400
100
230
200
200
50
200
50
100
300
500
100
300
500
Layer
Layer
Layer
Layer
Layer
Layer
Shake: Shake Regularization
CKA
Transfer Learning
ResNiet-110
ResNitt1164
ResNitt38 10x
ResNot-110
ResNiss-164
Reshies-388 10x
300
750
500
1000
200
Laver
300
200
400
500
100
100
200
400
800
500
1000
100 200
300
100
303
250
500
50 100
Layer
Layer
Layer
Layer
Layer
Figure 9: Training with principal component regularization, transfer learning and Shake-Shake regu- larization helps to eliminate the block structure. We directly regularize the first PC of each layer activations given that this component explains a large fraction of variance in block structure representations. and find that this eliminates the block structure. Full algorithm details can be found in Appendix G Shake-Shake regular- ization (Gastaldi 2017) has similar effect. We also find that transfer learning reduces the appearance of the block structure, although still present the largest network we trained. These results demonstrate that the block structure phenomenon is dependent on the training mechanism (see Appendix for implications these training methods on representations across random seeds).
some caution warranted in interpreting these performance benefits: may be difficult to causally determine the connection between the block structure and performance, as any training intervention targeting the block structure may have other distinct ramifications that also affect performance.
DISCUSSION
Scope and Limitations: Our work primarily focuses on the behavior of aage-capacity networks trained on relatively small datasets. This is motivated by domains such medical imaging where data is expensive relative to the cost of training large model, and the high-stakes nature makes important to understand the model's behavior. Additional future exploration is necessary to study state-of-the-art settings in c.g. NLP, which use much bigger and heterogeneous datasets.
Conclusion:
The block structure phenomenon uncovered in previous work (Nguyen et al. 2021 reveals significant differences in the representational structures of overparameterized neural net- works and shallowerharrower ones. However, it also exhibits some contradicting behaviors being unique to each network while propagating a dominant PC across a wide range of layers that suggest the underlying representations could either overfit to noise artifacts capture rele- vant signals in the data. Our work seeks to provide an explanation for this discrepancy. We find that despite the inconsistency of the block structure across different training runs, it arises not from noise, but real and simple dataset statistics such as background color. We further discover a small set of dominant datapoints (with large activation norms) that are responsible for the block structure. These datapoints emerge early in training and vary across epochs, as well as across random seeds. We show how different training procedures, including color augmentation, transfer learning, Shake- Shake regularization, and novel principal component regularizer, can reduce the influence of these dominant datapoints, climinating the block structure and leading to more consistent representations across training runs. This work motivates interesting open questions such exploring how dom- inant datapoints are manifested in other domains and applications of deep learning, and applying principal component regularization distribution shift and self-supervision problems.
REPRODUCIBILITY STATEMENT
Our experiments are performed on standard datasets and architectures that are publicly available. We provide information regarding training hyperparameters in Appendix A and implementation details for the principal component regularizer in Appendix G We plan to release source code for reproducing our results upon acceptance of the paper.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Newworks, 132:428-446, 2020.
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip Kellman. Deep convolutional networks do not classify based on global object shape. PLoS computational biology, 14(12): c1006613, 2018.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117748):30003-33070 2020.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference Learning Representations, 2019, URLhttps: /ttp://///p/eeeii..e.ooo net/ nww./orummf forum?
id=HI2-PsR5KX
Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to under- stand kemee learning. In International Conference Machine Learning, pp. 541-5411 PMLR, 2018.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012.
Xavier Gastaldi. Shake-shake regularization. arXiv preprint arXiv: 1705.07877 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained trained cnns are biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:/811.12233 2018.
Robert Geirhos, Hönn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(1)))665-673, 2020.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.13244 2018
Trevor Hastic, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint reXiv:/903.0 1903.08560, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp- 770-778, 2016.
Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv preprint arXiv:2006. 12433, 2020.
Katherine L Hermann, Ting Chen, and Simon Kornblith. The origins and prevalence of texture bias in convolutional neural networks. arXiv preprint arXiv:19/ 000/////009077 2019.
Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1614-1619, 2018.
Andrew G Howard. Some improvements on deep convolutional neural network based image classi- fication. arXiv preprint arXiv: 1312.5402 2013.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103. 10427, 2021.
Jason Jo and Yoshua Bengio. Mcasuring the tendency of CNNs to learn surface statistical regulari- ties. arXiv preprint arXiv: 1561, 2017.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICMI., 2019.
10

---

Under review as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tengyuan Liang, Alexander Rakhlin, et al. Just interpolate: Kernel "ridgeless" regression can gen- eralize. Annals of Statistics, 48(3):1329-1344) 2020.
Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q Nelson, Greg S Corrado, et al. Detecting cancer metastases on gigapixel pathology images. arXiv preprint arXiv:1703.02442 2017.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Uni- versality and individuality neural dynamics across large populations of recurrent networks. In Advances in neural information processing systems, pp. 11669-15644 2019.
Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic
heuristics in natural language inference. arXiv preprint arXiv: 1902.0007700 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
net. forum?idddsp QRaziT-
Ari S Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. arXiv preprint urXiv:/806.05755 2018.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. arXiv preprint arXiv: 1604, 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang, What is being transferred in transfer learn- ing? arXiv preprint arXiv:2008. 1687, 2020.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Interna- tional Conference on Learning Representations, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085, 2017.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, pp- 3347-3357, 2019.
Cinjon Resnick, Zeping Zhan, and Joan Bruna. Probing the state of the art: A critical look at visual representation evaluation. arXiv preprint 19Xiv:1912.000 2019.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144 2016.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. The Journal of Machine Learning Research, 13(1):1993-1434, 2012.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference Machine Learning, pp. 6105-6114, 2019.
Jessica AF Thompson, Yoshua Bengio, and Marc Schoenwiesner. The effect of task and training intermediate representations in convolutional neural networks revealed with modified rv similarity analysis. arXiv preprint 19te.o:222220000 2019.

---

Under review as a conference paper at ICLR 2022
Guillermo Valle-Perez, Chico Camargo, and Ard A Louis. Deep learning generalizes because the parameter-fume funttttor- map is biased towards simple functions. arXiv preprint 2018.
Bastiaan Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. In Alejandro F, Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Siberola-López and Gabor Fichtinger (eds.). Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pp. 210-218, Cham, 2018. Springer International Publish- ing. ISBN 978-3-030-00934-2.
Dayong Wang, Aditya Khosla, Rishab Gargeya, Humayun Irshad, and Andrew H Beck. Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.00 05711, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC), pp. 12..1. September 2016.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv: /6/ 1611:555 2016.
12

---

Under review as a conference paper at ICLR 2022
Appendix
A TRAINING DETAILS
For wide ResNets, we look at models with depths of 14, 20, 26, and 38, and width multipliers of 1, 2, and 10. For deep ResNets, we experiment with depths 32, 44, 56, 110 and 164. In CIFAR-100 experiments, the block structure only appears greater depth so also include depths 218 and 224 our investigation. For Patch Camelyon datasets, we find that depth 80 is enough induce a block structure in the internal representations. All ResNets follow the architecture design in (He ct al. 2016 Zagoruyko Komodakis 2016).
Unless otherwise specified, we train all the models using SGD with momentum of 0.9 for 300 epochs, together with a cosine decay learning rate schedule and batch size of 128. Learning rate is tuned with values [0.005, 0.01, 0.001] and L2 regularization strength with values [0.001, 0.005]. For CIFAR-10 10 and CIFAR-100 experiments, we apply standard CIFAR-10 10 of random flips and translations of up to 4 pixels. With Patch Camelyon, we use random crops of size 32x32, together with random flips, to obtain the training data. At test time, the networks are evaluated on central crops of the original images. For CKA analysis, each architecture is trained with 10 different seeds and evaluated on the full test set of the corresponding domain.
data augmentation consisting
13

---

Under review as a conference paper at ICLR 2022
B
BLOCK STRUCTURE AND THE FIRST PRINCIPAL COMPONENT
ResNet-38 (10x)
ResNet-110 (1x)
ResNet-44 (1x)
ResNet-38 (2x)
CKA
CKA
CKA
CKA
100
40
30
20
10
10
100
20
20
30
Variance explained by first PC
Variance explained by first PC
explained by first PC
Variance explained by first PC
Fracton of variance
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
10
20
30
50
100
20
40
10
20
30
Gos. similarty of first PCs
Cas, simllaity of first PCs
Cos. simllanty frat PCs
Cos. similarity first PCs
30
30-
Layer
20
20
50-
20
10
10
20
30
50
100
20
40
10
20
30
CKA without first PC
CKA without first PC
CKA without first PC
CKA without first PC
100
30-
20
50
20
10
10
100
20
10
20 Layer
30
Layer
Layer
Layer
Figure B.1: The relationship between block structure and the first principal component, Each column represents different architecture. In ResNet-110 (1x) and ResNet-38 (10x), observe block structure in the CKA plot (toprow) row), and find that the first principal component explains large fraction of variance in the layers that comprise the block structure (second row). We also observe that the cosine similarity of the first PCs (third row) resembles the CKA plot, and removing the first PC before computing CKA substantially attenuates the block structure (bottom row). By contrast, in ResNet-44 (1x) and ResNet-38 (2x), which have block structure, the first PC explains only small fraction the variance, and the CKA plot does not resemble the cosine similarity between the first PCs, but instead resembles CKA computed without the first PCs.
14

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL VISUALIZATIONS OF DOMINANT DATAPOINTS
ResNet-32 x
ResNet-38 1
80
100
Layer
60
0.6
Layer
75
088
Similarity
0.4
0.4
40
25
100
Layer
Layer
# of Exameles de
5.00
7500
Layer 25
of Exametes
7500
5000
Layer
2500
2500
200
200
100
100
-4
-2 Projected Value (log)
Projected Value (log)
of Exampies
7500
5000
Layer
of Examples
7500
5000
Layer
e500
2500
200
200
Projected (log)
Projected (log)
of Exomples
Layer 75
of Exmmpees
7500
Layer 75
2500
200
200
100
100
-4
-2
-4
-2 Projected Value (log)
Projected Value (log)
Buggton
hughte
0 of Examples
5000
7500
Layer 100
Examples
7600
5000
Layer 100
2500
2500
200
200
100
100
-5
-2
-5
-3
-2
Projected Value (log)
Projected Value (lag)
Figure C.1: Dominant datapoints are not present in networks without block structure. The topmost row shows representational similarity heatmaps from two networks without block structure. The rows below show histograms of the projected values the first PC, well images with the largest projections. Note that distributions of projected values are unimodal, unlike bimodal distributions observed in networks with a block structure (Figure 3). In addition, the datapoints with the highest projected values are highly dissimilar between layers.
15

---

Under review as a conference paper at ICLR 2022
Layer 400
Layer 450
7500
7500
Examples
5000
5000
2500
2500
400
400
200
200
0
-3
Projected Value (log)
Projected Value (log)
Figure C.2: Visualization of the distribution of projected values onto the first principal component by test imputs, for ResNet-224 (1x) trained CIFAR-100. Top shows histograms of the projected values on the Bottom rows show images with the largest projections on See Figure 3 for a similar plot for ResNet-164 (1 x) trained on CIFAR-10.
Layer 200
Layer 250
7500
7500
Examples
5000
5000
2500
200
200
-5
-3
-2
0
-5
-3
-2
Projected Value (log)
Projected Value (log)
Figure C.3: Visualization of the distribution of projected values onto the first principal component by test inputs, for ResNet-80 (Ix) trained on Patch Camelyon. Top row shows histograms of the projected values on the first PC. Bottom rows show images with the largest projections on the first PC. See Figure 3 similar plot for ResNet-164 164 (1 x) trained CIFAR-10.
Layer
Layer 60
7500
7500
Example @
2500
2500
400
200
200
-2
-5
-3
-2
-1
Projected Value (log)
Projected Value (log)
Figure C.4: Visualization of the distribution of projected values onto the first principal component by test inputs, for ResNet-26 (8x) trained on Patch Camelyon. Top row shows histograms of the projected values on the first PC. Bottom rows show images with the largest projections on the first PC, See Figure for similar plot for ResNet- 164 (I x) trained on CIFAR-I0.
16

---

Under review as a conference paper at ICLR 2022
D BLOCK STRUCTURE ON OUT-OF-DISTRIBUTION DATA
CIFAR-10
CIFAR-10 Corrupted
RN-26 1x
RN-44 1x
RN-56 1x
RN-110 1x RN-164 1x
RN-26 1x
RN-4 1x
RN-56 1x
RN-110 1x RN-164 1x
Layer
75
Layer
25
256075
50 150
255075
50
300
RN-36
RN-38
RN-38
Layer
100
Layer
100
50 100
50 100
50 100
50 100
50
100
50 100
90 100
50 100 Layer
50 100
90 100
Layer
CIFAR-100
CKA
Patch Camelyon
RN-26 1x
RN-44 1x
RN-56 1x RN-110 1x RN-164 1x
AN-26 1x
RN-M 1x
RN-56 1x RN-110 1x RN-164
Layer
75
Layer
8 8 3x
60
25
256075
255075
50
RN-38 1x
RN-38 2x
RN-38 RN-38 8x RN-38 10x
RN-38 1x
RN-38 2x
RN-38
RN-38 8x
RN-38 10x
Layer
100
Layer
100
.50
50
50
100
50 100
50 100
100
50
100
50 100
50 100
50 100 Layer
50 100
50 100
Layer
Figure D.1: Appearance of block structure depends the data which representations computed. plot CKA heatmaps models varying depths (top rows) and widths (bottom rows) trained on CIFAR- 10, evaluated on different datasets ordered by the degree of out-of--istributin structure representation are robust to small distribution shifts in the data, evident from CKAs computed on CIFAR-10 corrupted dataset (which adds perturbations to the original CIFAR- 10 data) and CIFAR- -00 dataset (which contains mutually exclusive classes but undergoes the However, larger shifts, such as from CIFAR-10 to Patch Camelyon, produce significantly different representa- tional structures.
We observe that the block
data collection procedure CIFAR-10),
E DOMINANT EXAMPLES AND LAYER ACTIVATIONS
1.0
Solid Pink
0.8
Activation Norm
Solid White
200
Dominant Datapoint
Layer
0.6
600
0.4
400
100
0.2
200
0
100
200
100
250
Layer
Layer
Figure E.1: Solid color images strongly activate layers making up block structure when trained on Patch Camelyon. Top row shows dominant examples for a ResNet-80 (1x) model trained on Patch Camelyon dataset. The CKA heatmap in the bottom left shows the location of the block structure in the internal represen- tations of the model. We observe that the dominant images share pink background. When we feed synthetic image filled with this background color into the network, observe that yields larger activations com- pared to the original image, layers making up the block structure (i.c., after layer 200). See also Figure for a similar plot for models trained on CIFAR-10 dataset.
17

---

Under review as a conference paper at ICLR 2022
F EVOLUTION OF THE BLOCK STRUCTURE
Within- Chhnn-Cheeoppint Similarity
Ep.1
Ep.
Ep.
Ep.4
5
Ep.
Ep.
Ep.
Ep.
Ep. 10
1.0
Layer
300
200
Ep. 20
Ep.
Ep.40
Ep.
Ep. 60
Ep.
Ep. 60
Ep.90
Ep. 100
Ep. 110
0.7
Layer
900
200
0.5
100
0.4
Ep. 120
Ep. 140
Ep. 160
Ep. 180
Ep. 200
Ep. 220
Ep. 240
Ep. 260
Ep. 280
Ep. 300
0.3
Layer
300
200
100
0.1
0.0
100
300
100 300
100 300
100
800
100 300
100 300
100 300
100 300
100 300
100 300
Similarity
Checkpoint
Ep.
Ep.
Ep.
5
Ep.
Ep. 7
Ep.
10
Layor
200
900
0.9
100
Layor
300
200
0.5 g
100
Ep.
Ep.
240
Ep.
Layer
300
0.2
200
100
0.1
100
100 300
100
100 300 Layer
100 Layer
300 Layer
100 Layer
300 Layer
100 Layer
Layer
Layer
Layer
Layer
Dominant Datapoints
Epoch 1
Epoch
Epoch 10
Epoch 20
Epach 30
Epoch
Epoch
Epach
Epoch 150
Epech 200
Epoch 250
Epoch 300
Figure F1:: Fine-grained analysis of the evolution of the block structure in ResNet-110 (1x) model. This plot shows the evolution block structure gramularity. As in Figure [ we find that the shape of the block structure is defined early in training (top row). However, comparing these different model checkpoints to the final, fully-trained model reveals that the block structure representations at different epochs considerably dissimilar, especially during the first half of the training process (middle row). The corresponding dominant datappoints also vary over training, even after block structure clearly visible the within-checkkooin similarity heatmaps (bottom row).
same network as in Figure & but with greater temporal
18

---

Under review as a conference paper at ICLR 2022
Within Within-Checken Checkpoint Similarity
Ep.
Ep.3
5
Ep.
10
1.0
Layor
200
300
0.9
100
Ep.
Layer
300
0.6
200
0.5
100
Ep. 120
Ep.
Ep. 160
Ep.
Ep. 200
Ep.
Ep.
Ep.
Ep.
Ep.
0.8
Layer
300
0.2
200
300
100
100
100 300
100
300
100
Similarity with Final Checkpoint
Ep.1
Ep. 2
Ep.
Ep.
Ep. 5
Ep.
Ep. 7
Ep.
Ep.9
Ep. 10
Lager
300
200-
100
0.8
Ep. 20
Ep. 30
Ep. 40
Ep.
Ep. 60
Ep. 70
Ep. 80
Ep. 90
Ep. 100
Ep. 110
0.7
Layer
300
200
100
0
120
Ep. 160
Ep.
200
Ep.
240
Ep. 260
Ep. 280
Ep. 300
Layer
300
200
100
0.1
100 300
100 300
100 300
100 300 Layer
100 300
100 300 Layer
100 300 Layer
100 300
100 300 Layer
100 300 Layer
Layer
Layer
Layer
Layer
Layer
Dominant Datapoints
Epoch
Epoch
Epoch
Epoch 20
Epoch
Epoch
Epoch 100
Epoch 160
Epech 200
Epoch 280
Epoch 300
Figure F2: Fine-grained analysis of the evolution of the block structure in different ResNet-110 (1x) model. This plot shows the evolution of block structure for network that is architecturally identical the one Figure
different seed. this training run, the final shape of the block structure established slightly later in training (top row), and similarity between early checkpoints and the last checkpoint is very low (middle row). Analysis of the dominant data points shows that they change substantially over the course of training (bottom row), and continue to vary long after the shape of the block structure ceases change the within-checkpeint similarity heatmaps.
trained
19

---

Under review as a conference paper at ICLR 2022
Similarity Between Layers of Same Checkpoint
Ep. 10
Ep. 20
Ep. 50
Ep. 70
Ep. 90
Ep. 100
Ep. 110
Ep. 120
Ep. 130
Ep. 140
1.0
500
Layer
250
0.8
0.6
Ep. 150
Ep. 160
Ep. 170
Ep. 180
Ep. 210
Ep. 230
Ep. 250
Ep. 270
Ep. 290
Ep. 300
0.5
500
Layer
0.3
250
0.1
250500
250500
250500
250500
250500
250500
250500
250500
250500
250500
Similarity with Final Checkpoint
Ep.
Ep.
Ep.
Ep. 30
Ep. 40
Ep.
Ep. 60
70
Ep.
Ep.
Layer
500
0.9
250
0.7
0.6
Ep. 100
Ep.
Ep. 140
Ep. 160
Ep. 180
Ep.
Ep. 220
Ep. 240
Ep. 260
Ep.
0.5
0.4
500
Layer
0.2
0.0
250500 Layer
250500
250500
250500
250500
250500
250500 Layer
250500 Layer
250500
250500
Layer
Layer
Layer
Figure F3: Evolution of the block structure over the course of training for ResNet-164 (1x) model. We compute the CKA between all pairs of layers within ResNet-38 (10x) model different stages of training. and find the internal representations already contain block structure at epoch 10. Comparing these different model checkpoints to the final, fully-trained model reveals that the block structure representations at different epochs are considerably dissimilar, especially during the first half of the training process.
Projected Value
0.0
50
100
150 Epoch
200
250
300
Epoch
Epoch 30
Epoch 60
Epoch 90
2
Activation Norm (x100)
0
Epoch
Epoch 150
Epoch
Epoch
2-
2
2
0
Epoch 240
Epoch 270
Epoch 300
Dominant Datapoint
2
Batch Average
0
0
500
250
Layer
Layer
Layer
Figure F:4: Example how the magnitudes layer activations and the projected values onto the first principal component for a dominant image vary across different epochs, for ResNet-164 (1x). Given dominant datapoint for ResNet-164 (1x) model, we track the magnitude its projected value onto the first principal component of the block structure representations (top left), well as its activation norms at each layer the network (bottom of plots), over time. Notice the correspondence hetween these metrics, especially when their values drop at epochs (initialization). 120 and 240. This is also aligned with the measurement of CKAs across different epochs (see Figure F3 where we find that the model checkpoints at these epochs are highly dissimilar from the final model in terms of the hidden representations). See Appendix FigureFF66oo for the corresponding visualization of dominant example of wide ResNet (ResNet-38 (10x)).
20

---

Under review as a conference paper at ICLR 2022
Similarity Between Layers Checkpoint
Ep.
Ep.
Ep. 50
Ep. 70
Ep.
Ep.
110
Ep. 120
Ep.
Ep.
1.0
Layer
100
0.9
50
0.7
Ep. 150
Ep. 160
Ep. 170
Ep. 180
Ep. 210
Ep. 230
Ep. 250
270
290
Ep. 300
0.5
Layer
100
0.3
0.2
0.1
0.0
50 100
50 100
50 100
50 100
50 100
50 100
50 100
50 100
50 100
50 100
Similarity with Final Checkpoint
Ep. 0
Ep. 10
Ep.
Ep. 30
Ep. 40
Ep. 50
Ep. 60
Ep. 70
Ep. 80
Ep. 90
Layer
100
0.8
0.6 g
Ep. 100
Ep. 120
140
Ep. 160
Ep. 180
Ep. 200
220
Ep. 240
Ep. 260
280
0.4 0
0.5
Layer
100
50
0.2
0.1
50 100 Layer
50 100
100
50 100 Layer
50 100 Layer
50 100
100 Layer
50 100 Layer
50 100 Layer
50 100 Layer
Layer
Layer
Layer
Figure F5: Evolution of the block structure over the course of training for a ResNet-38 (10x) model. We compute the CKA hetween all pairs of layers within ResNet-38 (10x) model at different stages of training, and find that the internal representations already contain block structure epoch 10. Comparing these different model checkpoints different epochs are considerably dissimilar, especially during the first half of the training process. See also FiguroFF.31 for similar plot for a deep ResNet (ResNet- 164 (1 x)).
final, fully-trained model reveals
block structure representations
Projected Value
0.2
0.1
0.0
50
100
150 Epoch
200
250
300
Epoch
Epoch 30
Epoch 60
Epoch
2
Activalion Narm (x10%)
Epoch 120
Epoch 150
Epoch 180
Epoch 210
2/
Epoch
Epoch 270
Epoch 300
2
Dominant Datepoint
Batch Median
0
50
100
50
100
0
50
100
Layer
Layer
Layer
Figure F.:: Example of how the activation magnitude and the projected values onto the first principal component for dominant image vary across different epochs, for ResNet-38 (10x). Given a dominant datapoint for a ResNet-38 (10x) model, we track the magnitude of projected value onto first principal component of the block structure representations (top left), as well as its activation norms at each layer in the network (bottom set of plots), over time. We observe that before the block structure representations stabilize and show more similarity with those in the fully trained model (i.e., epoch 90, Figure F55 above), the dominant image yields small value when projected onto first principal component, and also doesn't strongly activate the layers inside the block structure. This is aligned with the observation made Figure F.4 for ResNet- (1 x)).
21

---

Under review as a conference paper at ICLR 2022
100
0.8
Fraction Variaace Explained by First
Stap
40
0.2
100
150
200
250
300
350
Layer
the first 100 steps training. every step training, we measure the proportion of variance explained by the first principal component each layer of a ResNet- 110 (1x) network. At around step 30, the first principal component begins to explain the majority of the variance in the layer representations.
Figure F7: Evolution of the first principal components of layer representations
22

---

Under review as a conference paper at ICLR 2022
G TRAINING WITH PRINCIPAL COMPONENT REGULARIZATION
To regularize the first principal component, we first compute the amount of variance that it explains using power iteration (Miyato al., 2018). At training step, compute the batch of convo- lutional feature maps with height h and width w containing channels Mt E IR R flatten the spatial dimensions
the channels dimension create matrix size X; E R=xp where
p x w X c, and subtract its column means to obtain centered matrix Xt. We randomly initial- ize the stored cigenvector uo E Rp at the beginning of training. At each training step, we perform single step of power iteration initialized from the previous eigenvector:
X, X
(3)
A |url2
(4)
Ur Uz/Dc.
(5)
d approximates the top eigenvalue of X Xr and thus the amount of variance explained by the first principal component of the representation. The proportion of variance explained is given by We incorporate the regularizer as an additive term in the loss:
\// (/ll
Cpc reg (Ne, X; a. 6) a max(A1/1 Xelli 5.00.
(6)
where a is the strength the regularizer and is the threshold proportion of variance explained at which it is imposed. In experiments,
hyperparameter with values in [0.1, 1, 10], and set 5 = 0.2 based on our analysis of the first principal components of models without the block structure. To speed up the training process, we only apply the regularizer to ReLU layers starting from the second stage, where block structure is often found.
tune
Depth
Width
Accuracy (%)
Accuracy (%) (PC regularization)
(standard training)
CIFAR- 10 subsampled (6% of the full dataset):
56
1
77.8 + 0.429
79.2 + 0.188
26
8
80.1 0.354
81.1 + 0.185
26
80.3 0.306
81.2 + 0.194
38
8
80.2 + 0.362
80.9 + 0.264
38
10
80.3 + 0.412
81.4 + 0.350
CIFAR-IO: 10:
110
94.3 1 0.078
94.4 1: 0,063
164
94.4 0.075
94.5 1 0.063
26
95.8 0.087
1 0.051
38
95.7 0.091
95.8 + 0.080
38
10
95.7 + 0.157
95.9 + 0.067
CIFAR-100:
218
74.1 + 0.310
75.1 + 0.132
224
74.0 + 0.350
75.2 + 0.131
38
79.8 0.149
80.6 + 0.306
38
10
80.5 0.174
+ 0.241
Table G.1: Comparison of performance of large capacity models on CIFAR-10 and CIFAR-100, with and without principal component regularization. We observe that our proposed principal component regularizer consistently yields accuracy improvements for large capacity models that contain the block structure. The performance gains particularly significant in the of CIFAR- 1-0 and subsampled CIFAR- datasets.
23

---

Under review as a conference paper at ICLR 2022
H
EFFECT OF BATCH SIZE ON THE BLOCK STRUCTURE
Batch Size 128
ResNel-110 1x
ReeNett184 1x
Re Bx
ReeNet-38 10x
Layer
200
300
500
100
100
50
100
250
50
50
100
300
250
500
50
Batch Size 16
ResNet-110 1x
ReeNet-164
1x
ReeNet-38 Bx
ReeNet-S8 10x
1.0
Laver
200
250
100
100
0.5 g0
100
300
250 500
100 Layer
50 100 Layer
Layer
Layer
Figure H.l: Using very small batch sizes during training reduces the appearance of the block structure. The top row shows the block structure effect in range of very deep and wide networks, trained with standard batch size 128. We experiment with drastically smaller batch size of 16 (bottom row) and find that the block structure is now highly reduced, especially in deep models.
24

---

Under review as a conference paper at ICLR 2022
1 IMPACT OF TRANSFER LEARNING AND ON SIMILARITY OF LAYERS INSIDE THE BLOCK STRUCTURE
REGULARIZATION
Standard Training
Transfer Learning
Shake-Shake Regularization
Seed 1
Layer
500
500
1000
260
250
500
CKA
Seed 2
Layer
500
500
260
250
500
Seed
Layer
500
1000
250
250
500
500
500
500
500
500
500
1000
1000
1000 Seed 3
Seed
Seed 2
Seed 3 Layer
Seed Layer
Seed 2
Seed 3 Layer
Seed 1
Seed
Layer
Layer
Layer
Layer
Layer
Layer
Figure Training with transfer learning and Shake-Shake regularization yields models that are more similar across different training runs. Each group of plots shows CKA between layers of models with the same architecture but different initializations (off the diagonal) or within single model (on the diagonal). In the standard training case, representations across models highly dissimilar, especially in the block structure region. In contrast, when we use transfer learning and Shake-Shake regularization, comparisons across seeds show more similarity in corresponding layers. The same observation can be made for models trained with principal component regularization (see Appendix Figure 1.2
Standard Training
Principal Component Regularization
500
500
Layer
Seed
CKA
250
Seed 2
Layer
250
250
500
500
Seed 3
Layer
250
250
500
500
500
500
500
Seed
Seed
Seed
Seed
Seed
Seed Layer
Layer
Layer
Layer
Layer
Layer
Figure I,2: Training with principal component regularization yields models that different training runs. Each group of plots shows CKA between layers of models the same architecture but different initializations (off the diagonal) or within single model (on the diagonal). Similar to the obser- vation made in Figure [.] while representations across models are highly dissimilar in the standard training case, models trained with principal component regularization from different random initializations show more representational similarity in corresponding layers.
similar
25

---

Under review as a conference paper at ICLR 2022
J BLOCK STRUCTURE UNDER DIFFERENT KERNELS
As previously identified by (Nguyen et al. 2021 the block structure is phenomenon the linear CKA heatmaps large (wide or deep) networks. In this section, we investigate whether the block structure phenomenon also arises in CKA heatmaps computed with other kernels, and also examine effect removing the dominant datapoints (identified by magnitudes of their projections on the first principal component, in Section 4.1 upon these CKA heatmaps.
To compute CKA heatmaps under alternative kernels, we again use minibatch CKA. The approach in Eq. can be easily adapted to nonlinear kemels by replacing x, and Y,Y," the linear Gram matrices formed by minibatch i, with minibatch kernel matrices K; E R and K; E R"** The elements of these minibatch kernel matrices the kernels between pairs of examples in the minibatches, i.e., Ki k(Xi,.> imm.) and Kmm = k' K'YY,, Y...). Like linear minibatch CKA, nonlinear minibatch CKA is computed by averaging HSIC, across minibatches:
- HSIC; KY)
CK CK Aminbatth
V V+Σt. HSIC, (K,, Ki) ΣL HSICI (K), K()
We investigate the behavior of CKA under the linear kernel klinear (æ, y) = x y. the cosine kernel kooa (ax, y) x (/(///= wl)), and the RBF kernel kibr(ar y; exp(-11æ y||² (2l=/(22²)) For each layer, we measure the median Fuclidean distance d between examples in each layer and set od with c E {0.2, 0.5, 1, 2, 10) of that median Euclidean distance. To reduce variance when computing RBF CKA with small c, we use minibatch size of 1000 for these experiments.
Figure shows the appearance of CKA heatmaps of a narrow, shallow network (ResNet-38 1x, top), a wide network (ResNet-38 10x, middle), and a deep network (ResNet-164 1 1x, bottom). Al- though heatmaps computed for small network (ResNet-38 I 1x) look qualitatively similar regardless of kernels, both wide (ResNet-38 10x) and deep (ResNet-166 1 xx) networks exhibit significant dif- ferences.
ResNet 38
100
50
50 Layer
100
50
100
50
100
50
100
50
100
50
100
50
100
50
100 Layer
Layer
Layer
Layer
Layer
Layer
Layer
ReeNet-38 10x
Linear
Cosino
RBF 0.2
REF D.C.
RBF
RBF
RBF 5
RBF
60
50 Layer
100
60 Layer
100
100 Layer
Leyer
Layer
Layer
Layer
Layer
ResNet-164 1
200
250
500
250
500
250 Layer
500
250
500
250
500
250
500
250
500
250
500
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Figure J.l: Appearance of representation heatmaps in wide and deep networks, but not narrow/shallow networks, depends on the choice kernel. Rows reflect different models columns reflect different ker- nels. For RBF kernels, the parameter indicates the fraction of the median distance hetween examples (computed separately for each layer) that is used as the standard deviation of the kemee.
Because differences in representational similarity heatmaps ultimately reflect differences in the un- derlying kernel matrices, in Figure [.2] we show kernel matrices of individual layers taken from inside the block structure of each network random minibatches where the examples have been sorted descending values the first principal component. All kernels are sensitive to dominant datapoints, but in different ways and to different degrees. Linear kernel matrices are dominated by
26

---

Under review as a conference paper at ICLR 2022
the similarity between dominant datapoints. The cosine kernel ignores activation norms, and finds high similarity within groups of dominant and non non-dominant datapoints but low similarity between groups. The RBF kernel effectively considers all far away points to be equally dissimilar, and thus indicates that dominant datapoints are dissimilar to all other datapoints, including other dominant datapoints, which are typically far in Fuclidean distance (hecause, while aligned in direction, they have different norms).
Note that the prevalence of dominant datapoints can differ across models and initializations, as previously demonstrated in Figure 4 The dominant datapoints are clearly visible as a block in the top-left corner of the cosine kernel matrix. For ResNet-38 10x, there are 14 in the minibatch of 128 examples that is shown, but for ResNet-166 (1x), there are only 2.
ResNet- 38 1x Layer 76 FBF
Linear
REF
REF
RBF
RSF
Exampia
8.86
A88
85
9.84
100
100
100
100
100
100
Example
Example
Example
Example
Example
Example
Example
Example
ResNet-30
10x Layer 76
Linear
RBF
RBF
RBF
RBF
RSF
Exampia
7.5
5.0
5.0
2.6
100 Example
100
100
100
100
100
Example
Example
Example
Example
Example
Example
Example
ResNet 164 1x Layer 276
Linear
RBF
RBF
RSF
1.0
Example
7.5
0.5
100
100
100
100
100
100
Exampla
Example
Example
Exampla
Example
Example
Example
Example
Figure J.2: Kernels based on dot products, cosine similarity, or Euclidean distance are sensitive to dom- inant datapoints. Plots show kernel matrices for randomly sampled minibatch of 128 examples. computed from layer inside the block structure, for models that exhibit one. Examples sorted in descending value of the first principal component of raw activations: top rows and columns reflect dominant datapoints.
What is the effect of removing dominant datapoints upon CKA similarity heatmaps computed with these other kernels? Figure J.3, we show that, once the dominant datapoints are removed from large networks, we again see only differences among CKA heatmaps computed with different ker- nels, line with the results observed for shallow networks in Figure There are no longer large blocks of many consecutive similar layers in any of the heatmaps. Across all choices of kernel that we have investigated, when blocks appear in CKA heatmaps, they can be eliminated by eliminating the dominant datapoints.
27

---

Under review as a conference paper at ICLR 2022
ResNet-38 10x
No Datapoints Dropped
Cosine
RBF
1000
50
100
50
100
50
100
100
50
100
50
100
50
100
50
100
1% Most Dominant Datapoints Dropped
100
60
100
50
100
50
100
50
100
50
100
50
100
60
100
50
100
Most Dominant Datapoints Dropped
1.00
60
50
100
50
100
50
100
100
50
100
50
100
50
100
50
100
20% Most Dominant Datapoints Dropped
0.S.
1.00
50
60
100
60 100
50
100 Layer
100 Layer
50 100 Layer
50 100 Layer
50 100
50
100
Layer
Layer
Layer
Layer
ResNet-164 1
No Datapoints Dropped
RBF
RBF
200
250
600
250
600
250
800
260
500
260
500
260
500
260
600
250
300
Most Dominant Datapoints Dropped
200
250
500
250
600
250
500
250
500
260
500
250
500
250
500
250
500
Most Dominant Datapoints Dropped 0.5.
200-
500
250 Layer
500
250
260
500
500 Layer
250
260 Layer
500 Layer
Layer
Layer
Layer
Layer
Figure L3: Removing dominant datapoints eliminates both blocks and differences among kernels in CKA representational similarity heatmaps. We remove the top k% of examples according to the magnitudes their projections on the first in layer 76 for ResNet-38 10x and layer 276 for ResNet- 164 1x. Because 14/128 datapoints are dominant in the minibatch kernel matrix shown in Figure J.2] we provide results for removing 20% of datapoints for that network, although they look only modestly different from the results obtained by removing 10% of datapoints.
28


Reviews:
[{"review_id": "yX1Cqh1-SN", "reviewer": "Reviewer_1vN3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of the paper:\n- investigating the existing problem which is the block structure phenomenon in large-capacity networks trained on relatively small datasets,\n- proposing a novel principal component regularization method.\n\n\nThe weaknesses of the paper:\n- authors focus on the block structure phenomenon in large-capacity networks trained on relatively small datasets. In practice, a neural network is rarely trained on small data sets.\n- the paper present too small a novelty, to eliminate the block structure, the authors studied different existing training mechanisms such as transfer learning and Shake-Shake regularization.\n", "summary_of_the_paper": "Authors study phenomenon in large-capacity neural networks which contain blocks of contiguous hidden layers with highly similar representations. The block structure has two seemingly contradictory properties: on the one hand, its constituent layers have highly similar dominant first principal components (PCs), but on the other hand, their representations, and their common first PC, are highly dissimilar across different random seeds. \n\nThe authors investigated the origin of this block structure with the data and training methods. They found that the block structure arises from dominant datapoints - a small group of examples that share similar image statistics (e.g. background color). ", "summary_of_the_review": "The authors consider the problem of the block structure phenomenon uncovered in previous works but consider large-capacity networks trained on relatively small datasets. They try to answer the question: \"is the block structure a sign of overfitting to idiosyncrasies of the data and training process, or does it pick up meaningful signals?\" They tried to eliminate the block structure by a novel principal component regularization method, or alternative existing training mechanisms such as transfer learning and Shake-Shake regularization. In my opinion, the novelty of this paper is too small for the conference ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "w_6J-fIwgOW", "reviewer": "Reviewer_9NUZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While several facts mentioned in the paper are interesting, one of the key dissatisfying experiences of reading this paper is that the authors do not place this phenomenon in larger context or really examine its implication. \n\nOn the issue of surprise and arising of the block structure:\n1. It does not seem entirely surprising to me that subsequent layers have similar representations, after all this is why we need deep neural networks, to slowly evolve the representations towards something that correctly predicts the output. \n2. Since the authors mainly examine resnets, which have block structure architecture, this is even less surprising. Perhaps the authors could try to vary the residual connections. \n3. Can the block structure be seen from an information bottleneck viewpoint?\n\n\nOn the implications of block structure:\n1. The authors don't address in their paper why this block structure \"is bad\". The authors have no results showing that it affects results or generalization. \n2.  The main issue with it seems to be that the layers have redundant representation and perhaps one could reduce depth if the representation evolved more quickly, but this is never mentioned or brought up in the paper. The paper has no experiments showing the ability to prune or reduce networks when they eliminate block structure. \n\n\nOn the differences in block structure that arise:\n1. There actually seems to be an order-dependence to the particular representation that arises, thats why different random seeds result in different representations. This could be because the network learns what to do on a the initial set of inputs first and then on other sets of inputs, so the authors could try batching groups of similar inputs which are shown first and which are shown later to the network. But having said that it is not clear why the order dependence is bad again. \n2. This would explain the transfer learning helping as well, because in transfer learning a different dataset is trained initially. \n\nOn eliminating the block structure: \n1. It seems that most attempts to eliminate block structure still leaves some of it in tact. \n2.  Can it be removed by explicitly penalizing for betewen layers similarity? Would this affect generalization?\n\n\n", "summary_of_the_paper": "Building on the work by Nyugen et al. 2020  the authors examine the block structure phenomenon where subsequent layers of a neural network have very similar internal representations. They show that a small number of datapoints load very highly to the dominant principle components of the block structure.  They also examine the dynamics of the block structure and the fact that it changes through training. Finally the authors show that some regularizations such as Shake-shake improves the phenomenon. ", "summary_of_the_review": "The paper does not deeply examine the phenomenon at hand, it makes assumptions that are not explained, and does not explore implications well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VnR8UHAUsRU", "reviewer": "Reviewer_83My", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n(1) The paper uncovers the origin of the block structure phenomenon. \n\n(2) The experiments are insightful and thorough. \n\n(3) The paper's analysis of dominant data points can potentially be useful for training big networks on relatively small data. \n\nWeaknesses: \n\n(1) The paper lacks a rigorous theoretical analysis. \n\n(2) The idea of influential points is not very novel. ", "summary_of_the_paper": "This paper studies the block structure phenomenon in training large capacity neural networks. The paper provides an explanation that the phenomenon is caused by dominant data points. The paper also investigates methods to mitigate this issue. ", "summary_of_the_review": "This paper provides a reasonable explanation to the block structure phenomenon. It is a useful empirical contribution. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HCRvJYcBwoU", "reviewer": "Reviewer_4Z4y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The claims made by the authors in the paper seem correct and there is no obvious error I can see in the paper. The paper provides a number of experimental results that provide interesting insights into the appearance of block structures in large deep networks. The work builds on Nguyen et al. 2020 and therefore is not strikingly novel, the block structure itself and using PCA to investigate it are both coming from the original paper. There are some new insights coming out of the experiments carried out by the authors on this paper and therefore the paper has some novelty to it. The paper is very clearly written and the figures are informative and easy to parse. The relevance or impact of the work is in my mind small. Even though the paper provides interesting insights it is not clear to me what are the benefits of removing the block structure.   ", "summary_of_the_paper": "The authors investigate the block structure present in large deep networks and relate it to structure in the data and investigate trainings approaches that can reduce the presence of block structure.  ", "summary_of_the_review": "Interesting paper following up on Nguyen et al 2020. Some nice insights and overall a well-written paper, but probably impact is low. My recommendation is still leaning towards accepting the paper in the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-qa_jwLBuef", "reviewer": "Reviewer_yvqE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n\n### [Strength1]\nThe authors conducted exhaustive experiments on standard datasets (CIFAR-10, CIFAR-10, Patch Camelyon) and on different model architectures (several levels of shallow/deep ResNets with narrow/wide layers).\nAccording to (Nguyen et al., 2020), the block structure can emerge beyond ResNet as well (they conducted experiments on ResNet without skip connections).\nThese observations imply that the results reported in the paper will be ubiquitous (at least for images).\nRevealing general phenomena underling deep models would be valuable to the community.\n\n### [Strength2]\nThe observation that dominant datapoints tend to share some common image patterns seems to be nontrivial.\nThis phenomenon itself can be a topic of interest for further researches.\nThis phenomenon might also imply that deep models can be vulnerable to a slight change of the colors of the entire images.\nThat is, if we modify an input image to have a color close to that of dominant points, its intermediate representations might be large (e.g., Fig5), which will lead to different outputs (i.e., the model is vulnerable). \n\n\n## Weakness\n\nIn the paper, the authors reported that the block structure is induced by the dominant datapoints.\nThe authors then discussed the experimental results based on this connection.\nThe results are therefore connected to block structure through dominant datapoint, i.e., [block structure] -- [dominant datapoints] -- [results].\nBecause of this structure, there are two topics underlying the current paper: (i) the connection [block structure] -- [dominant datapoints] discussing the origin of the block structure, and (ii) the connection  [dominant datapoints] -- [results] discussing the emergence and dynamics of dominant datapoints.\n\nMy concern here is that these topics (i) and (ii) would be irrelevant (or, at least the connection would be far looser than expected).\nThe current paper claims that [block structure] and [results] are tightly connected through [dominant datapoints].\nHowever, the validity of the connection (i) [block structure] -- [dominant datapoints] is questionable, as I describe below.\nIn particular, I suspect it can be an artifact induced by the use of KCA with linear kernel for the similarity measure.\n\n### [Detail]\nEssentially, KCA with linear kernel measures the squared Frobenius norm of the covariance matrix (with some normalization).\nIt is known that the covariance is vulnerable to the existence of outliers.\nIndeed, as reported in (Nguyen et al., 2020, Fig3) as well as in Fig4 of the current paper, removing outliers (i.e., PCs or dominant datapoints) eliminate the similarity between layers.\nThese observations imply that \"dominant datapoints are kept being outliers across layers ((ii) above)\" rather than \"the representations are similar across layers ((i) above)\" because (at least some fraction of) the similarity is the consequence of the existence of such outliers.\n\nIn Appendix J, the authors reported the similarity under some other kernels.\nThere, we can find that linear kernel exhibits the strongest similarity (possibly because it is vulnerable to the existence of outliers).\nThis raises a question on the adequacy of the current study which rely solely on linear kernel.\nThe removal of outliers will lead to smaller KCA under linear kernel (and less blocks) as reported in the paper.\nHowever, I wonder whether the similar can hold for KCA under some other kernels.\nFor example, we can reduce the effect of outliers with large activations for block structure by adopting the feature map $\\phi(x) = \\frac{x}{\\\\| x \\\\|}$ which induces the cosine kernel $k(x, y) = \\cos(x, y)$.\nThen, the question is whether the removal of dominant datapoints lead to reduction of KCA under cosine kernel (and some other kernels as well) and elimination of the block structure.\nIf the significant reduction of KCA is not observed in variety of kernels, the connection (i) [block structure] -- [dominant datapoints] would be an artifact induced by the use of linear kernel.\n\nHere, I would like to remind that my comment above is all about the validity of the connection (i) [block structure] -- [dominant datapoints].\nIf the connection (i) still holds despite the selection of kernels, that would be far nontrivial, and would be an interesting phenomenon with further investigations needed.\nI would also remark that the second connection (ii) [dominant datapoints] -- [results] can be of independent interest irrelevant to the validity of the connection (i). (unless there are related studies in this direction)\n\n", "summary_of_the_paper": "The study of (Nguyen et al., 2020) reported that wide and deep neural networks tend to have similar feature representations across layers (called \"block structure\").\n(Nguyen et al., 2020) also revealed that the block structure is essentially induced by the first principal component (PC) of the representation in each layer.\n\nThis paper provides more detailed empirical analysis of the block structure.\nTo measure the similarity, the authors used KCA (with linear kernel) as the similarity measure, which was also used in (Nguyen et al., 2020).\nThis paper reported that\n* The dominant datapoints, which lie in the direction of PC, share some common image patterns such as background colors.\n* The dominant datapoints are different across different model initialization.\n* Although the block structure emerges in the very first stage of the training, the dominant data points differ between the beginning and at the end of the training.\n* The block structure can be suppressed by modifying training, e.g., by regularizing PCs, by using Shake-Shake regularization, and by using transfer learning.", "summary_of_the_review": "The current paper mixes the two connections (i) [block structure] -- [dominant datapoints] and (ii) [dominant datapoints] -- [results].\nTo make the paper rigid, these two connections need to be distinguished.\nThe results in the current paper implies the connection (ii).\nHowever, the validity of the connection (i) remains questionable.\nI suspect it can be an artifact induced by the use of KCA with linear kernel for the similarity measure.\nIf this is the case, the paper needs to be reorganized with its main focus on (ii) but not (i).\nIf the authors are willing to connect (i) and (ii), it would be essential to confirm that the reported phenomena are not artifact induced by the use of KCA with linear kernel, and demonstrate they are ubiquitous irrelevant to the selection of similarity measures.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experimentally shows that the block-structure of similarities between layers typically appears for different models and such a structure is mainly induced by small set of dominant datapoints. Moreover, the dominant datapoints are not just noisy artifacts but represent some common image patterns such as background colors. The authors also found that the block structure can easily disappear by removing  the dominant datapoints, and the authors also proposed a method to suppress the block structure by regularizing PCs, Shake-Shake regularization, and transfer learning.

This paper gives thorough experiments that clarify the mechanism of appearance of a block structure. However, its significance is a bit minor. Indeed, the block structure does not affect the generalization ability very much, and it can be removed without changing the predictive performance. I agree that investigating the behavior of the internal representation is of scientific interest as the authors pointed out, but on the other hand, its significance would not be convincing. Indeed, this concern was pointed out by several reviewers. Next, the main focus of this study is about the setting of large model with small data size. It is not clear whether it is universal across different model size relative to the dataset size. There is no theoretical investigation (for example, the block-structure phenomenon could be explained by a high dimensional random matrix theory). 

In summary, this paper investigates a somehow interesting phenomenon but its significance is not convincing. Thus, it would be a bit below the threshold of the acceptance.
Decision: Reject