Dominant Datapoints and the Block Structure Phenomenon in Neural Network Hidden Representations

1 INTRODUCTION . Many modern successes of deep neural networks have used simple rules to systematically increase model capacity , often through scaling architecture depth and width ( Tan & Le , 2019 ) . These large capacity models typically also maintain strong performance even in tasks with small amounts of training data . This has led to their widespread use across different many applications , including data-scarce , high-stakes settings such as medical imaging ( Wang et al. , 2016 ; Liu et al. , 2017 ) . However , recent work has shown that the representational structures of these large capacity models exhibit distinctive properties that are not present in shallower/narrower networks . Specifically , when using linear centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ) to measure similarity between hidden representations of neural network layers , Nguyen et al . ( 2021 ) find that a large set of contiguous layers that share highly similar representations . This is visible as a clear block structure in the heatmap of pairwise linear CKA similarities between layers ( see Figure 1a ) . This block structure phenomenon is robust , appearing both in models trained on natural image datasets and in models trained on medical imaging datasets ( Figure 1b ) . The emergence of the block structure as model capacity increases does not merely indicate a change in relationships among layer representations ; it is also associated with changes in the properties of individual representations . As shown in Figure 2a , layer representation inside the block structure have dominant first principal components ( PCs ) that explain the majority of variance in activations , whereas the first PCs of representations of layers outside the block structure , or in networks without block structure , explain far less variance . But counterintuitively , although layer representations and their dominant first PCs are similar across the layers that make up the block structure , they are highly dissimilar across random seeds , as shown in Figure 2b . Such representational inconsistencies have been linked to overfitting and poor generalization ( Morcos et al. , 2018 ) . The prevalence of the block structure , and this discrepancy in its properties , motivate a pressing fundamental question — is the block structure a sign of overfitting to idiosyncrasies of the data and training process , or does it pick up meaningful signals ? In this paper , we uncover answers to this question , investigating the origin of the block structure in relation to the data and training methods , and reconciling its contradictory behaviors . Specifically , our findings are as follows : • The dominant first principal components of the layers that make up the block structure arise from a small number of dominant datapoints that share similar characteristics ( e.g. , background color ) . • But the set of dominant datapoints and their common characteristics can vary across training runs , leading to the observed block structure dissimilarities across random seeds . • Like the block structure phenomenon , dominant datapoints can only be found in large capacity neural networks . We show that if these dominant datapoints are excluded from the dataset used for representational analysis , the block structure effect disappears . • We find that dominant datapoints strongly activate the hidden layers corresponding to the block structure . By constructing synthetic examples based off of their shared image characteristics , we show that these characteristics are indeed responsible for the high activation norms . • The block structure emerges early in training , but early block structures have different representations and yield different dominant datapoints than the final block structure at the end of training . • We study whether it is possible to eliminate the block structure without interfering with generalization using a novel principal component regularization method . We also show that alternative training mechanisms such as transfer learning and Shake-Shake regularization can reduce the block structure and yield more consistent representations across different training runs . 2 RELATED WORK . Previous work has studied certain propensities of deep neural networks in the standard training setting , such as their simplicity bias ( Huh et al. , 2021 ; Valle-Perez et al. , 2018 ; Nakkiran et al. , 2019 ) , texture bias ( Baker et al. , 2018 ; Geirhos et al. , 2018 ; Hermann et al. , 2019 ) and reliance on spurious correlations ( McCoy et al. , 2019 ; Geirhos et al. , 2020 ; Ribeiro et al. , 2016 ; Jo & Bengio , 2017 ; Hosseini & Poovendran , 2018 ) . Inspired by the findings of Nguyen et al . ( 2021 ) that deep and wide networks learn many layers with very similar representations , we seek to characterize the kind of signals these layers may overfit to . Additionally , our work also explores how this signal varies over the course of training instead of focusing on a single , final model . To do so , we analyze the behavior of the internal representations of models of varied depths and widths , using methods for measuring similarity of neural network hidden representations ( Kornblith et al. , 2019 ; Raghu et al. , 2017 ; Morcos et al. , 2018 ) . Besides shedding light on model training procedures ( Gotmare et al. , 2018 ; Neyshabur et al. , 2020 ) , features ( Resnick et al. , 2019 ; Thompson et al. , 2019 ; Hermann & Lampinen , 2020 ) , and dynamics ( Maheswaranathan et al. , 2019 ) , representational analysis has also furthered understanding of network internals in applications of deep learning , such as medicine ( Raghu et al. , 2019 ) and machine translation ( Bau et al. , 2019 ) . Our work is also related to previous attempts to understand the properties of overparameterized models ( Zhang et al. , 2016 ) . Most theoretical work in this area has focused on linear models , models with random features , or kernel settings ( Belkin et al. , 2018 ; Hastie et al. , 2019 ; Liang et al. , 2020 ; Bartlett et al. , 2020 ) , all of which lack intermediate features , or involves networks without nonlinearities ( Advani et al. , 2020 ) . Our results suggest that the behavior of intermediate features of practical neural networks changes dramatically with increasing overparameterization , in ways that are not obvious from previous analysis . 3 EXPERIMENTAL SETUP AND BACKGROUND . Measuring Representation Similarity with CKA : Centered kernel alignment ( CKA ) ( Kornblith et al. , 2019 ; Cortes et al. , 2012 ) addresses several challenges in measuring similarity between neural network hidden representations including ( i ) their large size ; ( ii ) neuronal alignment between different layers ; and ( iii ) features being distributed across multiple neurons in a layer . Like Nguyen et al . ( 2021 ) , we use the minibatch implementation of linear CKA with a batch size of 256 sampled without replacement from the test dataset , and accumulate statistics over 10 epochs to allow the minibatch estimator to converge . More concretely , given k minibatches of n examples , and two layers having p1 neurons and p2 neurons each , minibatch CKA takes as inputs k pairs of centered activation matrices ( X1 , Y1 ) , . . . , ( Xk , Yk ) where Xi ∈ Rn×p1 and Yi ∈ Rn×p2 reflect the activations of these layers to the same minibatches . It produces a scalar similarity score between 0 and 1 by averaging HSIC scores over minibatches : CKAminibatch = 1 k ∑k i=1 HSIC1 ( XiX T i , YiY T i ) √ 1 k ∑k i=1 HSIC1 ( XiX T i , XiX T i ) √ 1 k ∑k i=1 HSIC1 ( YiY T i , YiY T i ) , ( 1 ) where HSIC1 is the unbiased estimator from Song et al . ( 2012 ) . The result is an estimator of CKA that converges to the same value regardless of the batch size . We compute CKA between all layer representations , including before and after batch normalization , activations , and residual connections . In experiments that involve tracking how a model ’ s internal properties ( principal components of activations , representation similarity , etc . ) change across epochs , we set batch normalization layers to be in training mode , to reduce the difficulty of adapting to batch statistics of the test set when the model has not converged . The Block Structure Phenomenon : Nguyen et al . ( 2021 ) use linear CKA to compute the representation similarity for all pairs of layers within the same model and visualizes the result as a heatmap ( with x and y axes indexing the layers from input to output ) . They find a contiguous range of hidden layers with very high representation similarity ( yellow squares on heatmaps in Figure 1a ) in very deep or wide models , and call this phenomenon the block structure . The block structure arises in networks that are large relative to the size of the training set — although small networks do not exhibit block structure when trained on CIFAR-10 , they do exhibit block structure on smaller datasets . Representations of the layers that comprise the block structure exhibit different representational geometry than other layers . For layers inside the block structure , the first principal component explains a large fraction of the variance in representations ; this is not the case for the rest of the layers or for networks without the block structure ( Nguyen et al. , 2021 ) . We replicate this observation in Figure 2a . The substantial similarity between layers inside the block structure reflects a high degree of alignment of their first principal components , as can be seen from the following decomposition of linear CKA for centered activation matrices X ∈ Rn×p1 , Y ∈ Rn×p2 : CKA ( XXT , Y Y T ) = ∑p1 i=1 ∑p2 j=1 λ i Xλ j Y 〈uiX , u j Y 〉2√∑p1 i=1 ( λ i X ) 2 √∑p2 j=1 ( λ j Y ) 2 ( 2 ) where uiX ∈ Rn and uiY ∈ Rn are the ith normalized principal components of X and Y , and λiX and λiY are the amounts of variance that these principal components explain ( Kornblith et al. , 2019 ) . As the fraction of variance explained by the first principal component of each representation approaches 1 , CKA becomes a measure of the squared cosine similarity between first principal components 〈u1X , u1Y 〉2 . Nguyen et al . ( 2021 ) conclude that the block structure preserves and propagates a dominant first principal component across many hidden layers . Datasets & Models : Our setup closely follows that of Nguyen et al . ( 2021 ) and analyzes ResNets of varying depths and widths , trained on common image classification datasets CIFAR-10 and CIFAR100 ( Krizhevsky et al. , 2009 ) , as well as the medical imaging dataset Patch Camelyon ( Veeling et al. , 2018 ) . These datasets are chosen to reflect the image statistics found in different domains , and we observe that they all easily induce a block structure in reasonably sized ResNets . The ResNet architecture design follows Zagoruyko & Komodakis ( 2016 ) , with the layers distributed evenly between three stages — each marked by a different feature map size — and the number of channels is doubled after each stage . To scale the model depth and width , we increase the number of layers and channels respectively . In experiments involving Shake-Shake regularization ( Gastaldi , 2017 ) , the network is modified to have 3 branches that are combined in a stochastic fashion . More information on hyperparameters can be found in Appendix A .

Summary: The study of (Nguyen et al., 2020) reported that wide and deep neural networks tend to have similar feature representations across layers (called "block structure"). (Nguyen et al., 2020) also revealed that the block structure is essentially induced by the first principal component (PC) of the representation in each layer.  This paper provides more detailed empirical analysis of the block structure. To measure the similarity, the authors used KCA (with linear kernel) as the similarity measure, which was also used in (Nguyen et al., 2020). This paper reported that * The dominant datapoints, which lie in the direction of PC, share some common image patterns such as background colors. * The dominant datapoints are different across different model initialization. * Although the block structure emerges in the very first stage of the training, the dominant data points differ between the beginning and at the end of the training. * The block structure can be suppressed by modifying training, e.g., by regularizing PCs, by using Shake-Shake regularization, and by using transfer learning.