Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
Authors: Thao Nguyen
Maithra Raghu
Simon Kornblith
Venue: ICLR 2021 Poster (2021)
Keywords: Representation learning

Abstract: A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there is limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models. We demonstrate that this block structure arises when model capacity is large relative to the size of the training set, and is indicative of the underlying layers preserving and propagating the dominant principal component of their representations. This discovery has important ramifications for features learned by different models, namely, representations outside the block structure are often similar across architectures with varying widths and depths, but the block structure is unique to each model. We analyze the output predictions of different model architectures, finding that even when the overall accuracy is similar, wide and deep models exhibit distinctive error patterns and variations across classes.

Published as a conference paper at ICLR 2021
Do WIDE AND DEEP NETWORKS LEARN THE SAME THINGS? UNCOVERING How NEURAL NETWORK REPRESENTATIONS VARY WITH WIDTH AND DEPTH
Thao Nguyen, Maithra Raghu, Simon Kornblith
Google Research
{thaotn, maithra, skormbbithh
ABSTRACT
A key factor in the success of deep neural networks is the ability to scale models improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model hidden representations, finding characteristic block structure in the hidden rep- resentations of larger capacity (wider or deeper) models. We demonstrate that this block structure arises when model capacity is large relative the size of the training set, and is indicative of the underlying layers preserving and propagating the dominant principal component of their representations. This discovery has important ramifications for features learned by different models, namely, repre- sentations outside the block structure are often similar across architectures with varying widths and depths, but the block structure is unique to each model. We analyze the output predictions of different model architectures, finding that even when the overall accuracy is similar, wide and deep models exhibit distinctive error patterns and variations across classes.
INTRODUCTION
Deep neural network architectures are typically tailored to available computational resources by scaling their width and/or depth. Remarkably, this simple approach model scaling can result in state-of-the-art networks for both high- and low-resource regimes (Tan & Le) (001)). However, despite the ubiquity of varying depth and width, there is limited understanding of how varying these properties affects the final model beyond its performance. Investigating this fundamental question critical, especially with the continually increasing compute resources devoted to designing and training new network architectures.
More concretely, we can ask, how do depth and width affect the final learned representations? Do these different model architectures also learn different intermediate (hidden layer) features? Are there discernible differences detailed analysis of family of ResNet models with varying depths and widths trained on CIFAR-10 Krizhevsky et al. 2009). CIFAR-100 and ImageNet (Deng et al. 2209)).
the outputs?
this paper, we study these core questions, through
We show that depth/width variations result in distinctive characteristics in the model internal rep- resentations, with resulting consequences for representations and outputs across different model initializations and architectures. Specifically,
contributions as follows:
We develop a method based on centered kernel alignment (CKA) to efficiently measure the simi- larity of the hidden representations of wide and deep neural networks using minibatches.
We apply this method different network architectures, finding that representations in wide or deep models exhibit characteristic structure, which we term the block structure. We study how the block structure varies across different training runs, and uncover a connection between block
Work done member of the Google AT Residency program.

---

Published as a conference paper at ICLR 2021
structure and model overparametrization block structure primarily appears in overparameter- ized models.
Through further analysis, we find that the block structure corresponds to hidden representations having a single principal component that explains the majority of the variance in the representa- tion, which is preserved and propagated through the corresponding layers. We show that some hidden layers exhibiting the block structure can be pruned with minimal impact on performance.
With this insight on the representational structures within a single network, we turn to compar- ing representations across different architectures, finding that models without the block structure show reasonable representation similarity in corresponding layers, but block structure representa- tions are unique each model.
Finally, we look at how different depths and widths affect model outputs. We find that wide and deep models make systematically different mistakes at the level of individual examples. Specif- ically, on ImageNet, even when these networks achieve similar overall accuracy, wide networks perform slightly better on classes reflecting scenes, whereas deep networks are slightly more ac- curate on consumer goods.
2 RELATED WORK
Neural network models of different depth and width have been studied through the lens of univer- sal approximation theorems (Cybenko 1989, Hornik] 1991 Pinkus, 1999 Lu et al.) 2017: Hanin & Sellke, 2017; Lin & Jegelka| 2018) and functional expressivity (Telgassy 2015 Raghu et al. 2017b). However, this line of work only shows that such networks can be constructed, and provides neither a guarantee of learnability nor characterization of their performance when trained on finite datasets. Other work has studied the behavior of neural networks in the infinite width limit by relat- ing architectures to their corresponding kernels (Matthews al.) 20188 Lee et al. 2018 Jacot al. 2018), but substantial differences exist between behavior in this infinite width limit and the behavior of finite-width networks (Novak et al. 2018 Wei et al. 2019 Chizat et al. 2019 Lewkowyez al., 2020). In contrast to this theoretical work, we attempt to develop empirical understanding of the behavior of practical, finite-width neural network architectures after training on real-world data.
Previous empirical work has studied the effects of width and depth upon model accuracy in the con- text of convolutional neural network architecture design, finding that optimal accuracy is typically achieved by balancing width and depth ZZagoruyko & Komodakis 2016)] Tan & Le 2019). Further study of accuracy and error sets have been conducted in (Hacohen & Weinshall 2020) (error sets over training), and (Hooker et al. 2019) (error after pruning). Other work has demonstrated that it often possible for narrower or shallower neural networks to attain similar accuracy to larger net- works when the smaller networks are trained to mimic the larger networks' predictions (Ba & Caru- ana 2014 Romero et al., 2015). We instead seek to study the impact of width and depth on network internal representations and (per-example) outputs, by applying techniques for measuring similarity of neural network hidden representations (Kornblth et al. 2019 Raghu et al. 2017a Morcos et al., These techniques have been very successful in analyzing deep learning, from properties of neura network training (Gotmare et al. 2018 Neyshabur et al. 2020), objectives (Resnick et al.
2018).
2019
Thompson et al. [2019] Hermann & Lampinen (0200), and dynamics (Maheswaranathan al.
2019
to revealing hidden linguistic structure in large language models (Bau et 20199 Kudugunta
al.
2019
Wu al. 2019: 2020) and applications in neuroscience (Shi et al.) 2019 Li al. 2019
Merel
et al.
2019 Zhang & Bellec 2020) and medicine (Raghu et al. 2019).
3
EXPERIMENTAL SETUP AND BACKGROUND
Our goal is to understand the effects of depth and width on the function learned by the underlying neural network, in setting representative of the high performance models used practice. Re- flecting this, our experimental setup consists of family of ResNets (He et al.) 2016:; Zagoruyko & Komodakis, 2016) trained on standard image classification datasets CIFAR- 10, CIFAR- 100 and ImageNet.
For standard CIFAR ResNet architectures, the network's 's layers are evenly divided between three stages (feature map sizes), with numbers of channels increasing by factor of two from stage to the next. We adjust the network's 's width and depth by increasing the number of channels and layers respectively in each stage, following Zagoruyko & Komodakis (2016). For ImageNet ResNets,
2

---

Published as a conference paper at ICLR 2021
ResNet-50 and ResNet- 101 architectures differ only by the number of layers in the third (14 x 14) stage. Thus, for experiments on ImageNet, we scale only the width or depth of layers in this stage. More details training parameters, as well the accuracies of all investigated models, can be found in Appendix
We observe that increasing depth and/or width indeed yields better-performing models. However, we will show in the following sections how they exhibit characteristic differences in internal repre- sentations and outputs, beyond their comparable accuracies.
3.1 MEASURING REPRESENTATIONAL SIMILARITY USING MINIBATCH CKA
Neural network hidden representations are challenging to analyze for several reasons including (i) their large size; (ii) their distributed nature, where important features in a layer may rely on multiple neurons; and (iii) lack of alignment between neurons in different layers. Centered kernel alignment (CKA) (Kornblith et al. [2019] Cortes et al. 2012) addresses these challenges, providing robust way to quantitatively study neural network representations by computing the similarity between pairs of activation matrices. Specifically, we use linear CKA, which Kornblith et al. (2019) have previously validated for this purpose, and adapt it so that it can be efficiently estimated using minibatches. We describe both the conventional and minibatch estimators of CKA below.
Let X E RM and Y E Rmxpa contain representations of two layers, one with P1 neurons and another P2 neurons, the same of examples. Each element of the x Gram matrices K = XXT and = YYT reflects the similarities between a pair of examples according to the representations contained in X or Y. Let H In _11 T be the centering matrix. Then K' = НKK H and L' HH reflect the similarity matrices with their column and row means subtracted. HSIC measures the similarity of these centered similarity matrices by reshaping them to vectors and taking the dot product between these vectors, HSIC, (K L,L) L) vee (') "xe')vee(L''), (') 1)? HSIC is invariant to orthogonal transformations of the representations and, by extension, to permutation of neurons, but it is not invariant to scaling of the original representations. CKA further normalizes HSIC to produce similarity index between and that invariant isotropic scaling,
HSIC K, L)
CKA(K, L)
HHSICo(K, K, K KHHSSC (L, L)
Kornblith et al. (2019) show that, when measured between layers of architecturally identical net- works trained from different random initializations, linear CKA reliably identifies architecturally corresponding layers, whereas several other proposed representational similarity measures do not. However, naive computation of linear CKA requires maintaining the activations across the entire dataset in memory, which is challenging for wide and deep networks. To reduce memory consump- tion, we propose to compute linear CKA by averaging HSIC scores over & minibatches:
Te Σ HSIC, (X,X, Y,Y,)
CK CKAminibttch
E (X.XX, & E (YY,,
where X; E Rnxpi and Y, E RRax are now matrices containing activations of the 7th minibatch of n examples sampled without replacement. In place of HSIC,. which is biased estimator of HSIC, we unbiased estimator of HSIC (Song et al. 2012) so that the value of CKA is independent of the batch size:
1 KKILLL
HSIC, HSICI(K,L)- n(n - 3)
(n(KL)+
(n - 1)(n -) n-2 I
(3)
where K and İ are obtained by setting the diagonal entries of similarity matrices K and L to zero.
This approach of estimating HSIC based on minibatches is equivalent approach of Yamada et al. (2018), and converges to the same value as if the entire dataset were considered as a single minibatch, as proven in Appendix A We use minibatches of size = 256 obtained by iterating over the test dataset 10 times, sampling without replacement within each time.
the bagging block HSIC
4 DEPTH, WIDTH AND MODEL INTERNAL REPRESENTATIONS
We begin our study by investigating how the depth and width of a model architecture affects its in- ternal representation structure. How do representations evolve through the hidden layers in different
3

---

Published as a conference paper at ICLR 2021
architectures? How similar different hidden layer representations each other? To answer these questions, we use the CKA representation similarity measure outlined Section 3.1
We find that as networks become wider and/or deeper, their representations show characteristic block structure: many (almost) consecutive hidden layers that have highly similar representations. By training with reduced dataset size, pinpoint connection between block structure and model overparametrization block structure emerges in models that have large capacity relative to the training dataset.
4.1
INTERNAL REPRESENTATIONS AND THE BLOCK STRUCTURE
CIFAR-10
Deeper
ReaNet-26 1x
ResNet-44 1x
ResNet-56 1x
ResNet-118 10 1x
ResNet-164 1x
75
Layer
50
25
0-
25
50
75
50
100
150
0
50
100
150
100
200
300
250
600
Wider
ResNet-38 1x
ResNet-38 2x
ResNet-38 4x
ResNett38 8x
ResNet-38 10x
100
50
100
0
100
100
Layer
CIFAR-100
1/16 ImageNet
ResNet-38 (1x)
ResNet-38 (10x)
0.0
0.5
1.0
ResNet-50 (1x)
ResNet-50 (8x)
CKA
100
Layer
Layer
50
50
50
100
50
100
50
100
50
100
Layer
Layer
Layer
Layer
Figure Emergence of the block structure with increasing width or depth. As we increase the depth or width of neural networks, we see the emergence of large, contiguous set of layers with very similar represen- tations - the block structure. Fach of the panes of the figure computes the CKA similarity hetween all pairs of layers in single neural network and plots this a heatmap, with and axes indexing layers. See Appendix Figure C. for block structure in wide networks without residual connections.
In Figure I we show the results of training ResNets of varying depths (top row) and widths (bottom row) on CIFAR-10. For each ResNet, we use CKA to compute the representation similarity of all pairs of layers within the same model. Note that the total number of layers is much greater than the stated depth of the ResNet, the latter only accounts for the convolutional layers in the network but we include all intermediate representations. We can visualize the result as heatmap, with the and y axes representing the layers of the network, going from the input layer to the output layer.
The heatmaps start off as showing a checkerboard-liks representation similarity structure, which arises because representations after residual connections are more similar other post-residual rep- resentations than representations inside ResNet blocks. As the model gets wider deeper, the emergence of a distinctive block structure - a considerable range hidden layers that have very high representation similarity (seen yellow square the heatmap). This block structure mostly appears in the later layers (the last two stages) of the network. We observe similar results in networks without residual connections (Appendix Figure C.1
4

---

Published as a conference paper at ICLR 2021
All Training Data ResNet-38 4x
ResNet-38 1x
ResNet-38 2x
ResNet-38 8x
ResNet-38 10x
Layer
1.0
100
100
0
100
50
100
1/4 Training Data
0.8
ResNet-38 1x
ResNet-38 2x
ResNet-38 4x
ResNet-38 8x
ResNet-38 10x
100
0.6
Layer
0.4
100
100
50
100
50
100
0.2
1/16 Training Data
ResNet-38
ResNet-38 2x
ResNet- 4x
ResNet-38
ResNet-38 10x
0.0
100
Layer
50
50
100
100
0
50
100
50
100
100
Layer
Figure 2: Block structure emerges in narrower networks when trained on less data. We plot CKA simi- larity heatmaps we increase network width (going right along each row) and also decrease the dataset size (down each column). As result of the increased model capacity (with respect the task) from smaller dataset size, smaller (narrower) models now also exhibit block structure.
Block structure across random seeds: In Appendix Figure D.1 multiple random seeds of a deep network and a wide network. We observe that while the exact size and position of the block structure vary, it is present across all training runs.
plot CKA heatmaps across
4.2
THE BLOCK STRUCTURE AND MODEL OVERPARAMETRIZATUU
Having observed that the block structure emerges models get deeper and/or wider (Figure [ next study whether block structure is result of this increase in model capacity namely, is block structure connected the absolute model size, or the size of the model relative the size of the training data?
Commonly used neural networks have many more parameters than there are examples in their train- ing sets. However, even within this overparameterized regime, larger networks frequently achieve higher performance on held out data (Zagoruyko Komodakis 2016]] Tan Le 2019). Thus, to explore the connection between relative model capacity and the block structure, we fix model ar- chitecture, but decrease the training dataset size, which serves to inflate the relative model capacity.
The results of this experiment with varying network widths are shown in Figure 2 while the cor- responding plot with varying network depths (which supports the same conclusions) can found in Appendix Figure D.2. Each column of Figure shows the internal representation structure of fixed architecture as the amount training data is reduced, and we can clearly see the emergence of the block structure in narrower (lower capacity) networks less training data is used. Refer to Figures [D.3] and D.. in the Appendix for a similar set of experiments on CIFAR-100. Together, these observations indicate that block structure in the internal representations arises in models that are heavily overparameterized relative to the training dataset.
PROBING THE BLOCK STRUCTURE
In the previous section, we show that wide and/or deep neural networks exhibit a block structure in the CKA heatmaps of their internal representations, and that this block structure arises from the large

---

Published as a conference paper at ICLR 2021
ResNet-110 (1x)
ResNet-38 (10x)
Similarity of First PCs
CKA
Similarity First PCs
CKA
100
30
30
Layer
Layer
20
20
50
10
10
0
0-
Varlance Explained
CKA w/o First PC
Varlance Explained
CKA w/o First PC
1.00
1.00
Fraction of Variance
Fraction of Varlance
0.75
0.75
0.50
50
0.50
20
0.25
0.25
10
0.00
0-
0.00
50
100
50
100
10
20
30
10
20
30
Layer
Layer
Layer
Layer
Figure 3: Block structure arises from preserving and propagating the (dominant) first principal compo- of four plots, for layers of deep network (left) and
nent of the layer representations. Above representations (top right), shows block structure in both networks. By
a wide network (right). CKA comparing this to the variance explained by the top principal component each layer representation (bottom left), we see that layers in the block structure have a highly dominant first principal component, This principal component is also preserved throughout the block structure, seen by comparing the squared cosine similarity of the first principal component across pairs of layers (top left), the CKA representation similarity (top right). Compared to the latter, after removing the first principal component from the representations (bottom right), the block structure is highly reduced- the block structure arises from propagating the first principal component.
capacity of the models in relation to the learned task. While this latter result provides some insight into the block structure, there remains a key open question, which this section seeks to answer: what happening to the neural network representations they propagate through the block structure?
Through further analysis, we show that the block structure arises from the preservation and propaga- tion of the first principal component of its constituent layer representations. Additional experiments with linear probes (Alain & Bengio 2016) further support this conclusion and show that some layers that make up the block structure can be removed with minimal performance loss.
5.1 THE BL.OCK STRUCTURE AND THE FIRST PRINCIPAL COMPONENT
For centered matrices of activations X RuXPI Y R linear CKA may be written
E EH 2
CKA(XXT, YYT)
(4)
1Eth V (X))2
where u'y E R" and uiy F are the 7th normalized principal components of X and Y and X and Xi are the corresponding squared singular values (Kornblith et al. 2019). As the fraction of the variance explained by the first principal components approaches I, CKA reflects the squared alignment between these components (utx, ) We find that, in networks with visible block structure, the first principal component explains large fraction of the variance, whereas in networks with visible block structure. does not (Appendix FigureDD.).. suggesting that the block structure reflects the behavior of the first principal component of the representations.
Figure explores this relationship between the block structure and the first principal components of the corresponding layer representations, demonstrated on a deep network (left group) and wide network (right group). By comparing the variance explained by the first principal component (bot- tom left) to the location of the block structure (top right) observe that layers belonging to the block structure have a highly dominant first principal component. Cosine similarity the first prin- cipal components across all pairs of layers (top left) also shows similarity structure resembling the block structure (top right), further demonstrating that the principal component is preserved through- out the block structure. Finally, removing the first principal component from the representations

---

Published as a conference paper at ICLR 2021
ResNet- 26 (2x)
ResNet-26 (2x)
ResNet 26 (10x)
ResNet- 26 (10x)
1.0
Figure 4: Linear probe accu- racy. Top: CKA between lay- of individual ResNet mod- els, different architectures and initializations. Bottom: Accuracy of linear probes for each of the layers hefore (or- ange) and after (blue) the resid- ual connections.
75
0.8
Layer
0.6
50
0.4
25
0.2
0
0.0
100
pooie ggoe
yoo Aag
Accuracy
Residual Pre-Reeiduul
50
25
50
75
25 50
75
0 25 50
75
80
76
Layer
Layer
Layer
Layer
Figure 5: Effect of deleting blocks on accuracy for mod- els with and without block structure. Blue lines show effect of deleting blocks backwards one-hy-one within each ResNet stage. (Note the plateau at the block structure.) Vertical green lines reflect boundaries between ResNet stages. Horizontal gray line reflects accuracy of the full model.
ResNet-38 (1x)
ResNet-38 (10x)
1.0
100
0.8
Layer
0.6 g
50
0.4.
0.2
0.0
100
Accuracy (96)
50
0
50
100
60
100
0
50
100
0
50
100
Layer
Layer
Layer
Layer
nearly eliminates the block structure from the CKA heatmaps (bottom right). A full picture of how this process impacts models of increasing depth and width be found in Appendix Figure F.6
In contrast, for models that do not contain the block structure, we find that cosine similarity of the first principal components across all pairs of layers bears little resemblance the representation similarity structure measured by CKA, and the fractions of variance explained by the first principal components across all layers are relatively small (see Appendix Figure D.7p. Together these re- sults demonstrate that the block structure arises from preserving and propagating the first principal component across its constituent layers.
Although layers inside the block structure have representations with high CKA and similar first principal components, each layer nonetheless computes nonlinear transformation of input. Ap- pendix Figure shows that the sparsity of ReLU activations inside and outside of the block struc- ture similar. In particular, ReLU activations the block structure are sometimes the linear regime and sometimes in the saturating regime, just like activations elsewhere in the network.
5.2 LINEAR PROBES AND COLLAPSING THE BLOCK STRUCTURE
With the insight that the block structure preserving key components of the representations, we next investigate how these preserved representations impact task performance throughout the network, and whether the block structure can be collapsed in a way that minimally affects performance.
In Figure 4 we train linear probe (Alain & Bengio 2016) for each layer of the network, which maps from the layer representation to the output classes. models without the block structure (first 2 panes), the block structure (last 2 panes), linear probe accuracy shows little improvement inside the block structure. Comparing the accuracies of probes for layers pre- and post-residual connections, find that these connections play an important role in preserving representations in the block structure.
monotonic increase in accuracy throughout the network, but in models with
Informed by these results, we proceed to pruning blocks one-by-one from the end of each residual stage, while keeping the residual connections intact, and find that there is little impact on test ac- curacy when blocks are dropped from the middle stage (Figure 5). unlike what happens in models without block structure. When compared across different seeds, the magnitude of the drop in accu- racy appears to be connected the size and the clarity of the block structure present. This result suggests that block structure could be an indication of redundant modules in model design, and that the similarity of its constituent layer representations could be leveraged for model compression.

---

Published as a conference paper at ICLR 2021
ResNet-38 (1x)
ResNet-38 (10x)
ResNet-164 (1x)
Layer
100
100
500
Seed
250
Seed 2
Layer
100
100
500
250
500
Seed Layer
100
100
50
50
250
0
0
100 Seed 3 Layer
100
100
500 0
500
500
Seed 1
Seed 2 Layer
Seed Layer
Seed Layer
Seed 3 Layer
Seed Layer
Seed
Seed Layer
Layer
Layer
Figure Representations within "block structure" differ across initializations. Each group of plots shows CKA between layers of models with the same architecture but different initializations (off the diagonal) or within single model (on the diagonal). For narrow, shallow models such as ResNet-38 (1x x). there is no block structure, and CKA across initializations closely resembles CKA within single model. For wider (middle) deeper (right) models, representations within the block structure are unique each model.
6 DEPTH AND WIDTH EFFECTS ON REPRESENTATIONS ACROSS MODELS
The results of the previous sections help characterize effects of varying depth and width on a (single) model's internal representations, specifically, the emergence of the block structure with increased capacity, and its impacts on how representations are propagated through the network. With these insights, we next look at how depth and width affect the hidden representations across models. Concretely, are learned representations similar across models of different architectures and different random initializations? How is this affected as model capacity is changed?
We begin by studying the variations in representations across different training runs of the same model architecture. Figure Figurr@@jllustraaa iillustrates CKA heatmaps for smaller model (left), wide model (mid- dle) and deep model (right), trained from random initializations. The smaller model does not have the block structure, and representations across seeds (off' diagonal plots) exhibit the same grid-like similarity structure as within a single model. The wide and deep models show block structure in all their seeds (as seen in plots along the diagonal), and comparisons across seeds (off-diagona plots) show that while layers not in the block structure exhibit some similarity, the block structure representations are highly dissimilar across models.
Appendix Figure T.T shows results of comparing CKA across different architectures, controlled for accuracy. Wide and deep models without the block structure do exhibit representation similarity with each other, with corresponding layers broadly being of the same proportional depth in the model. However, similar what we observe in Figure 6 the block structure representations remain unique to each model.
7 DEPTH, WIDTH AND EFFECTS ON MODEL PREDICTIONS
To conclude our investigation on the effects of depth and width, we turn to understanding how the characteristic properties of internal representations discussed the previous sections influence the outputs of the model. How diverse are the predictions of different architectures? Are there examples that wide networks are more likely to do well on compared to deep networks, and vice versa?
By training populations of networks on CIFAR-10 and ImageNet, we find that there is considerable diversity in output predictions at the individual example level, and broadly, architectures that are more similar in structure have more similar output predictions. On ImageNet we also find that there are statistically significant differences in class-level error rates between wide and deep models, with the former exhibiting small advantage in identifying classes corresponding to scenes over objects.
Figure P compares per-example accuracy for groups of 100 architecturally identical deep models (ResNet-62) and wide models (ResNet- 14 (2x)) )), all trained from different random initializations CIFAR-10. 10. Although the average accuracy of these groups is statistically indistinguishable. they tend to make different errors, and differences between groups substantially larger than expected
8

---

Published as a conference paper at ICLR 2021
1
honeycomb
ResNet- (2x)
Accuracy (56)
75
seashore
library
Bouvier des
50
Accuracy Difference (%)
bookshop
Flandres
25
2-
01
ResNet-E2 (1x) b
100
Accuracy (%)
75
-2-
50
mud turtle
Lebrador retriever black and gold garden spider
25
ResNet-83 93 ResNet-83 (Controi)
trash
ResNet-50 (2.8x) ResNet-83
01
rar
0
100
20
60
80
100
ResNet-62 (1x) Accuracy (%6)
ResNet-83 Accuracy (%)
Figure 7: Systematic per-example and per-class performance differences between wide and deep models. a: Comparison of accuracy on individual examples for 100 ResNet-62 (1 x) and ResNet-14 14 (2x) models, which have statistically indistinguishable accuracy the CIFAR- 10 test set. b: Same as (a), for disjoint sets of 100 architecturally identical ResNet-62 models trained from different initializations. See Figure [F.Tforr for a similar plot for ResNet-14 (2x) models. c: Accuracy differences on ImageNet classes for ResNets between models with increased width (y-axis) or depth (x-axis) in the third stage. Orange dots reflect difference between two sets of 50 architecturally identical deep models (i.e., different random initializations of ResNet-83).
by chance (Figure p). Examples of images with large accuracy differences are shown in Appendix Figure F. while Appendix Figures F2 and F3 further explore patterns of example accuracy for networks different depths and widths, respectively. As the architecture becomes wider or deeper, accuracy on many examples increases, and the effect is most pronounced for examples where smaller networks were often but not always correct. At the same time, there are examples that larger net- works less likely get right than smaller networks. We show similar results for ImageNet networks in Appendix Figure F.4
We next ask whether wide and deep ImageNet models have systematic differences in accuracy at the class level. As shown in Figure 77. there are small but statistically significant differences in accuracy for 419/1000 classes (p < 0.05, Welch's f-test), accounting for 11% of the variance in the differences in example-level accuracy (see Appendix F.3). Three of the top 5 classes that are more likely to be correctly classified by wide models reflect scenes rather than objects (seashore, library, bookshop). Indeed, the wide architecture is significantly more accurate on the 68 ImageNet classes descending from "structure" or "geological formation" (74.9% + 0.05 vs. 74.6% + 0.06, = 6 x 10-5, Welch's 's t-test). Looking at synsets containing > 50 ImageNet classes, the deep architecture is significantly more accurate on the 62 classes descending from "consumer goods" (72.4% + 0.07 vs. 72.1% + 0.06, 0.001; Table F.2). In other parts of the hierarchy, differences are smaller; for instance, both models achieve 81.6% accuracy on the 1 118 dog classes (p = 0.48).
CONCLUSION
In this work, we study the effects of width and depth neural network representations. Through experiments on CIFAR-10, CIFAR-100 and ImageNet, we have demonstrated that as either width or depth increases relative to the size of the dataset, analysis of hidden representations reveals the emergence a characteristic block structure that reflects the similarity of a dominant first principal component, propagated across many network hidden layers. Further analysis finds that while the block structure is unique to each model, other learned features are shared across different initializa- tions and architectures, particularly across relative depths of the network. Despite these similarities in representational properties and performance of wide and deep networks, we nonetheless observe that width and depth have different effects on network predictions the example and class levels. There remain interesting open questions on how the block structure arises through training, and using the insights on network depth and width to inform optimal task sase-spe model design.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We thank Gamaleldin Elsayed for helpful feedback on the manuscript.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint 16xiv:1610.01644, 2016.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural informa- fion processing systems, pp. 2654-2662, 2014.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference Learning Reprexentations, 2019. URLhttps: /ttps///ppneevieeeee n forum?
idd=Hlz-PRR5a
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, pp. 2937-2947, 2019.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Bradley Efron. Regression and anova with zero-one data: Measures residual variation. Journal of the American Statistical Association, 73(361):113-11,, 1978.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.13243. 2018
Guy Hacohen and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. ICMI., 2020.
Boris Hanin and Mark Sellke. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv: 1710. 1278, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016
Katherine L Hermann and Andrew K Lampinen. What shapes feature representations? exploring datasets, architectures, and training. arXiv preprint arXiv:2006. 1e0000661443. 12433, 2020.
Sara Hooker, Aaron Courville, Yann Dauphin, and Andrea Frome. Selective brain damage: Mea- suring the disparate impact of model pruning. arXiv preprint arXiv:1911.05248 2019.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2):251-257, 1991.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018. 2018.
Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICMI., 2019.
10

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Sneha Reddy Kudugunta, Ankur Bapna, Isaac Caswell, Naveen Arivazhagan, and Orhan Firat. In- vestigating multilingual nmt representations at scale. arXiv preprint arXiv:1909.0219,, 2019.
Jachoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Confer- ence on Learning Representations, 2018. URL bictps://openreett ew. www.nee//eee fort/forum?in
B 1-AA---2 M-CZ
Aitor Lewkowyzz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.022 022.18 2020.
Zhe Li, Wieland Brendel, Edgar Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Matthias Bethge, Fabian Sinz, Zachary Pitkow, and Andreas Tolias. Learning from brains how regularize machines. In Advances in Neural Information Processing Systems, pp. 9529-9539, 2019.
Hongzhou Lin and Stefanie Jegelka. Resnet with one-neuron hidden layers universal approxi- mator. In Advances in neural information processing systems, pp. 6169-6178, 2018.
Zhou Lu, Hongming Pu, Feicheng Wang. Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Advances in neural information processing systems, pp. 6231-6239, 2017.
Niru Maheswaranathan, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Uni- versality and individuality neural dynamics across large populations of recurrent networks. In Advances in neural information processing systems, pp. 1 (1629-1564)) 2019.
Alexander G de Matthews, Mark Rowland, Jiri Hron, Richard Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv: 11711, 2018.
Josh Merel, Diego Aldarondo, Jesse Marshall, Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology virtual rodent. arXiv preprint 1911.9911 2019.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, pp- 5727-5736, 2018.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What being transferred in transfer learn- ing? arXiv preprint arXiv:2008. 1687, 2020.
Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXiv:1810.05145: 2018.
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta numerica, 8(1): 143-195, 1999.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085, 2017a.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the ex- pressive power of deep neural networks. In international conference machine learning, pp- 2847-2854. PMLR, 2017b.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in neural information processing systems, pp. 3347-3357, 2019.
11

---

Published as a conference paper at ICLR 2021
Cinjon Resnick, Zeping Zhan, and Joan Bruna. Probing the state of the art: A critical look at visual representation evaluation. arXiv preprint arXiv:9912.00215, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Jianghong Shi, Eric Shea-Brown, and Michael Buice. Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. In Advances in Neural Information Processing Systems, pp. 5764-5774, 2019.
Le Song, Alex Smola, Arthur Gretton, Justin Bedo, and Karsten Borgwardt. Feature selection via dependence maximization. The Journal of Machine Learning Research, 13(1):1393-1434, 2012.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, 2019.
Matus Telgarsky. Representation benefits of deep feedforward networks. arXiv preprint arXiv: 15..1159900000 2015.
Jessica AF Thompson, Yoshua Bengio, and Marc Schoenwiesner. The effect of task and training on intermediate representations in convolutional neural networks revealed with modified rv similarity 2019.
analysis. arXiv preprint
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. In Advances in Neural Information Processing Systems, pp. 9712-9724, 2019.
John M Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Simi- larity analysis of contextual word representation models. arXiv preprint arXiv:2005.01172, 2020.
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross- lingual structure in pretrained language models. arXiv preprint 1xXiv:1911.01464 2019.
Makoto Yamada, Yuta Umezu, Kenji Fukumizu, and Ichiro Takeuchi. Post selection inference with kernels. In International Conference on Artificial Intelligence and Statistics, pp. 152-160. 152.LRR. 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Muchine Vision Conference (BMVC), pp. 87.1-87.13 12, September 2016.
Yu Zhang and Pierre Bellec. Transferability of brain decoding using graph convolutional networks. bioRxiv, 2020.
12

---

Published as a conference paper at ICLR 2021
Appendix
A CONVERGENCE OF MINIBATCH HSIC
Proposition Let K Rmxm and E Rm be two kernel matrices constructed by apply- ing kernel functions k and respectively to all pairs of examples in a dataset D. Form random partitionings p of 'D into m/n minibatches b of size n, and let K E Rax= and L'P E gx be kernel matrices constructed by applying kernels i and to all pairs of examples within each minibatch. Define Up -HSIC1 (K, L), the value of HSIC₁ applied to the full dataset, and UP m "0 HSTCI (KUP,, LDD), the average value of HSIC1 over the minibatches parti- tioning (epoch) p. Then 2/ Ep Up _ Uo as c + co.
Proof. Let i be the set of all 4-tples of indices between and m where each index occurs exactly once. As proven in Theorem 3 of Song al. (2012). Up is U-statistic:
Uo = HSIC (K, L) (m 4)!
h(Ks,
(5)
m!
SERT"
where ((s.j.g.)) = (Ki,j: Kigg Kiyr Ki Kj Kg,r) and the kernel of the U-statistic h is defined in Song et al. (2012). Let &3 be if the 4-tuple of dataset indices S is selected in minibatch b and otherwise. Then:
m/n
(n 4)!
Up
Σ 8gh(Ks, Ls).
(6)
n!
b
SEET
Taking the expectation with respect to 8, and noting that independent of h (KKs, Iis);
m/n
Es [.[ūp] = (n
4)!
Es [osh(Ks,
n!!
b=
SE4T"
(n 4)!
m/n
Es 6' h(Ks, Ls).
(8)
m
SE
By symmetry, E& [6%] is the same for all example and batch indices. Specifically, there are nl/(n 4)! 4-tuples that be formed from each batch and ml/(m 4)! 4-tuples that be formed from the entire dataset, so the probability that a given 4-tuple is in given batch is Es [6%] (n!/(n- 4)l)/(ml//m/ 4)!). Thus:
Eg|ū,| (m 4)!
Σ N(Ks, Ls) = Up-
m!
SEET
The minibatch indicators 8g are either O or 1, so their variances and covariances are bounded, and the weighted sum in Eq. has finite variance. Thus, by the law of large numbers, 1 c Ep- Up P, Uo as p + DC.
B TRAINING DETAILS
Our CIFAR-10 and CIFAR-100 networks follow the same architecture as He et al. (016); Zagoruyko & Komodakis (2016). We train set of models where we fix the width multiplier of deep networks to and experiment with models of depths 32, 44, 56, 110, 164. On CIFAR- the block structure only appears at greater depth we also include depths 218 and 224 in our investigation. For wide networks, we examine width multipliers of 1, 2, 4, 8 and 10 and depths of 14, 20, 26, and 38. We use SGD with momentum of 0.9, together with cosine decay learning rate schedule and batch size of 128, to train each model for 300 epochs. Models are trained with standard CIFAR-10 data augmentation comprising random flips and translations of up to 4 pixels.
13

---

Published as a conference paper at ICLR 2021
Each depth and width configuration is trained with 10 different seeds for CKA analysis, and 200 seeds for model predictions comparison.
On ImageNet, we start with the ResNet-50 architecture and increase depth width in the third stage only, following the scaling approach of (He et al.] (016). We train for 120 epochs using SGD with momentum of 0.9 and cosine decay learning rate schedule batch size of 256. We use 100 seeds for model prediction comparison.
For experiments with reduced dataset size, subsample the training data from the original CIFAR training set by the corresponding proportion, keeping the number of samples for each class the same. All CKA results are then computed based the full CIFAR test
Table B.I: Accuracy of examined neural networks CIFAR-10 and CIFAR-I00.
Depth
Width
CIFAR-TO Test
CIFAR- 100 Test
Accuracy (%)
Accuracy (%)
32
93.5
71.2
44
I
94.0
72.0
56
I
94.2
73.3
110
I
94.3
74.0
164
94.4
73,9
14
92.0
67.8
14
2
94.1
72.9
4
95.4
77.0
14
8
95.9
80.0
14
10
96.0
80.2
20
92.8
69.4
20
2
94.6
74.4
20
4
95.4
77.6
20
8
96.0
80.2
20
10
95.8
80.8
26
93.3
70.5
26
2
94.9
75.8
26
4
95.6
79.3
26
8
95.9
80.9
26
10
95.8
81.0
38
93.8
72.3
38
2
95.1
75.9
38
4
95.5
78.6
38
8
95.7
79.8
38
10
95.7
80.5
14

---

Published as a conference paper at ICLR 2021
C
BLOCK STRUCTURE IN A DIFFERENT ARCHITECTURE
CNN-38 1
CNN-38 2x
CNN-38 4x
CNN-38 8x
CNN-38 10x
1.00
100
0.75
Layer
0.50
50
0.25
0-
0.00
100
100
100
100
100
Layer
Figure C.1: Block structure also appears models without residual connections. We remove residual connections from existing CIFAR- 10 ResNets and plot CKA heatmaps for layers in the resulting architecture after training. Since the lack of residual connections prevents deep networks from performing well on the task, here we only show the representational similarity for models of increasing width. As previously observed in Figure the block structure emerges higher capacity models.
D PROBING THE BLOCK STRUCTURE
ResNet
1.0
300
Layer
200
100
0.6
100
100
200
300
100
200
100
200
100
200
ResNet-33 10x
0.4
100
Layer
50
0
0.0
0
50
100
0
50
100
50
100
50
100
50
100
Layer
Figure D.1: Block structure varies across random initializations. We plot CKA heatmaps as in Figure for 5 random seeds of deep model (top row) and wide model (bottom row) trained CIFAR-10 10. While the size and position vary, the black structure is clearly visible in all seeds.
15

---

Published as a conference paper at ICLR 2021
All Training Data
ResNot 32 1x
ResNet 44 1
ResNot 1x
ResNet 110 1x
RosNet 1iMeet-164 164 1x
Layer
50
0-
50
100
0
50
100
150
50
100
150
100 200 300
250
500
1/4 Training Data
0.8
ResNet-32 1x
ResNet-44 44 1
ResNet- 1x
ResNNe- 110 1x
ResNet- 164
0.8
Layer
50
0
50
100
0
50
100
150
50
100
150
100 200 300
250
500
0.2
1/16 Training Data
ResNet-32 1x
ResNet-44
ReaNet-S8 1x
ResNett1 1x
ResNet-1 64
0.0
Layer
50
0
50
100
0
50
100
150
50 Layer
100
150
0 100 200 300
250
500
Figure D.2: Block structure emerges in shallower networks when trained on less data (CIFAR-10). We increase network depth (going right along each row) and also decrease
plot CKA similarity heatmaps the size (down each column) training data. Similar to the observation made in Figure 2 as result of the increased model capacity (with respect to task) from smaller dataset size, smaller (shallower) models now also exhibit the block structure.
All Training Data ResNet-164 1
ResNet-56 1
ReaNet-1 -10 1x
ReaNet-218 x
ResNet-224 1x
Layer
100
50
0
50
100
150
100 200 300
250
500
250
500
750
0
250
500
750
1/2 Training Data
0.8
ResNet-56 1x
ResNet-110 1x
ResNet-164 1x
ResNet-218 1x
ResNet-224 1x
Layer
100
50
0.4
50
100
150
100 200 300
250
500
250
500
750
250
500
750
0.2
1/4 Training Data
ResNet-56 1x
ResNet-110 1x
ResNet-164 1x
ResNet-218 1x
ResNet-224 1x
Layer
100
50
50
100
150
100 200 300
250
500
250
500
750
500
750
Figure D.3: Block structure emerges shallower networks when trained on less data (CIFAR-100), We plot CKA similarity heatmaps as we increase network depth (going right along each row) and also decrease the size of training data (down each column). Similar to the observation made in Figure 2 result of the increased model capacity (with respect the task) from smaller dataset size, smaller (shallower) models now also exhibit the block structure.
16

---

Published as a conference paper at ICLR 2021
All Training Data
ResNat-38
ResNet-38
100
Layer
50
100
0
50
100
50
100
60
100
0
50
100
1/2 Training Data
0.8
ResNet-38
ResNet-38
ResNet-38
ReaNet-38
ResNet-38
Layer
50
0.4
50
100
0
50
100
0
50
100
50
100
0
50
100
0.2
1/4 Training Data
ResNet-38
ResNet-38
ResNet-38
ResNet-38 8x
ResNet-38
Layer
50
0
60
100
0
50
100
50
100
50
100
0
50
100
Layer
Figure D.4: Block structure emerges in narrower networks when trained on less data (CIFAR-100) We plot CKA similarity heatmaps as we increase network width (going right along each row) and also decrease result of the
the size (down each column) of training data. Similar to the observation made in Figure increased model capacity (with respect to the task) from smaller dataset size, smaller (narrower) models now also exhibit the block structure.
17

---

Published as a conference paper at ICLR 2021
=
A1///h 0.05
Av/lAla 0.05
As//Ala 0.05
Av/Mhh
RN-32 (1x)
100
50
/ 0.04
A,/44 0.04
A.AAA 0.04
>./Ah 0.04
A./AA - 0.04
RN-44 (1x)
150
100
50
0.06
4 0.06
A./2 0.06
A,/ 0.05
A 0.04
150
RN-58
100
50
7ı/M4 0.22
A 0.22
A 0.15
A 0.13
A./Ai -
300
RN-110
200
100
A3/M/1 0.85
A1/0 0.65
A:/1A/a 0.39
A3/M1/1 0.20
1r//1 0.11
RN-164 (1x)
500
1.0
250
0.8
0
0.6
A-/4/1= 00.6
A1/// = 0.05
A:/1/:= = 0.05
> - 0.05
A1/M4 - 0.05
RN-38 (1x)
0.4
100
50
0.2
As/lA12 0.11
4444 0.05
AURA 0.05
> 0.04
Av/A 0.04
0.0
100
RN-38
50
Aa/1Ah 0.14
A 0.08
AvIA 0.07
Av/4 0.04
A 0.04
RN-38 (4x)
100
50
>./121 = 0.54
A./M 0.49
A./121 0.36
/ 0.27
A.//a 0.22
RN-388 (8x)
100
50
An///h
A1/Ah
A:/12/
A./Mh
Av/Ah
RN-38 (10x)
100
Seed
Seed
Seed Seed
Seed
Figure D.5: Top principal component explains a large fraction variance in the activations of models with block structure. Each row shows a different model configuration that is trained on CIFAR-10, with the first 5 rows showing models of increasing depth, and the last 5 rows models of increasing width. Columns correspond different seeds. Each heatmap labeled with the fraction of variance explained by the top principal component of activations combined from the last 2 stages the model (where block structure often found). Rows (seeds belonging to the same architecture) are sorted by decreasing value of the proportion of variance explained. We observe that this variance measure is significantly higher in model seeds where the block structure is present.
18

---

Published as a conference paper at ICLR 2021
ResNet-26
ResNet-44
ResNet-66
ResNet-110
ResNet-164
1.0
Layer
0.8
%
0.6
150
100
$
ResNot-38 1
ResNet-38 2x
ResNet-38 4x
ResNet-38 8x
ResNet-38 10x
0
0.4
100
Layer
0.2
0.0
0 50
50
50
50
100
50
100
Layer
Figure D.6: How the representational structure evolves with increasing depth and width when the first principal component is removed, We plot CKA similarity heatmaps models become deeper (top row) and wider (bottom row), with the top principal component of their internal representations removed. Compared Figure[] we observe that while this process significantly eliminates block structure large capacity models (as also shown in Figure 3), has negligible impact on the representational structures of smaller models (where no block structure is present). The latter is not surprising, since the first principal component docsn't 't account for large fraction of the variance in representations of these models, demonstrated in Appendix Figure D.5 above.
ResNet-44 (1x)
ResNet-38 (2x)
Similarity of First PCs
CKA
Similarity of First PCs
CKA
40
40
30
30
Layer
20
20
20
Variance Explained
CKA w/o First PC
Variance Explained
CKA w/o First PC
1.00
1.00
Fraction of Variance
40
Fraction of Varrlance
0.75
0.75
30
0.50
0.50
20
20
0.25
0.25
10
0.00
0.00
Layer
Layer
Layer
Figure D.7: Relationship between the representation similarity structure nent in networks without block structure. Above are two sets of four plots, for layers of a deep network (left) and wide network (right). that don't contain block structure. In contrast to Figure 3 the first principal component of each layer representation only accounts for small fraction of variance in the representation (bottom left). Comparing the squared cosine similarity of the first principal component across pairs layers (top left), to the CKA representation similarity (top right), we find that these two structures don't resemble cach other. Last but not least, the representational structure of each model remains mostly unchanged after the first principal component is removed (bottom right).
first principal compo-
19

---

Published as a conference paper at ICLR 2021
ResNet-38 1x
ResNet-38 2x
ResNet-38 4x
ResNet-38 8x
ResNet-38
1.00
0.75
Layer
0.50
0.25
10
20
30
10
20
30
10
20
30
10
20
30
10
20
30
0.00
100
100
100
100
100
% Zero Activations
M
80
80
80
80
80
M by
60
20
30
10
30
10
10
20
10 20
Layer
Figure D.8: ReLU activations inside and outside the block structure are similarly sparse. To rule out the possibility that the block structure arises because layers inside behave linearly, measured the sparsity of the ReLU activations. We observe that significant proportion activations always non-zero, and the proportion similar inside outside block structure. Thus, although layers inside the block structure have similar representations, each layer still applies nonlinear transformation to its input.
20

---

Published as a conference paper at ICLR 2021
E
REPRESENTATIONS ACROSS MODELS
RN-56 Layer RN-14 (2x) Layer
RN-14 (2 2x) Layer
OKA
200
RN- RN-164 164 Layer
150
400
100
200
50
25
50 D
100 150
25
50 0
250
RN-14 (2x) Layer
RN-56 Layer
RN-14 (2x) Layer RN-164 Layer
Figure E.1: Representations align between models of difffernn widths and depths when no block struc- ture is present. In each group heatmaps, top left and bottom right show CKA within single model trained on CIFAR-10. Bottom left shows CKA for all pairs of layers between these (non-au architecturally-iiie -ddenticca)))i models, which have similar test performance. In the absence of block structure (left group), representations the relative depths similar models. But when comparison involves models with block structure (right group), representations within the block structure are dissimilar to those of other model.
F EXAMPLE- AND CLASS-LEVEL ACCURACY DIFFERENCES
F.1 EFFECT OF VARYING WIDTH AND DEPTH CIFAR-10 PREDICTIONS
Restet-1 Example (14)
100
Ras (1x) Exampla (9)
100
Restert114 14 Example (96)
100
60
60
60
25
50
75
100
25
50
75
100
25
75
100
ResNet-6 62
Example Acc.
ResNet 62
Example Acc.
ResNet 14 (2x) Example Acc. (%)
b
ResNet-62 (1 X):
74%
71%
60%
ResNet-14 (2x):
Easier for
ResNet-62 (1x)
ResNet-62 (1 x): 20% ResNet-14 (2X): 90%
26%
28% 79%
22% 71%
23% 96%
32% 85%
39% 86%
19% 76%
11% 50%
13%
92%
55%
Easier for ResNet-14 14 (2x)
Figure F.1: Systematic per-example performance differences between wide and deep models CIFAR-
10. Comparison predictions of 200 ResNet-62 (1x) and ResNet-14 (2x) models, which have statistically indistinguishable accuracy on the CIFAR-I0 test set (mean + SEM 94.09 1 0.01 vs. 94.08 1 0.01, (199) 0.73. 0.47). a Scatter plots of per-example accuracy for 100 ResNet-14 (2x) models vs. 100 ResNet- 62 (1x) models (left) show substantially higher dispersion than corresponding plots for disjoint architecturally identical models trained from different initializations (middle and right). Examples with the highest accuracy differences between the two types of models. Accuracies are reported on a subset of models that is disjoint from those used to select the examples.
of 100
21

---

Published as a conference paper at ICLR 2021
ReaNat-14
ResNet-20
ResNet-26 (93.09%)
ReaNet-32 (93.44%)
ReeNet-38 (93.67%)
ResNet-56 (34.02%)
ReaNet-164 ResNat-110 (94.33%)
(91.69%)
(92.61 96)
(94.37%)
100
ReeNet- 14
(96)
100
ResNet-20
Acc. (96)
100
ResNet-29
Acc. (%6
100
ReaNet-32
(96)
100
RegNet-38
1%6)
100
ReaNet-58
(%6)
ReaNet-110
ReeNet- 164
100
Acc. (%6)
1000 D
1000
1000
100
1000
100 0
1000
100
ResNet-14 14
ResNet-20 Acc. (96)
ResNet-26 Acc. (96)
ResNet-32 (%6)
ResNet-38 (96)
ResNet-56 Acc. (96)
ResNet- 110 Acc. (96)
ResNet-164 Acc.
Acc. (96)
Figure Effeect depth on example accuracy. Scatter plots per-example accuracies ResNet models with different depths on CIFAR-10. Blue dots indicate per-example accuracies of two groups of 100 networks each with different architectures indicated by axes labels. Orange dots show the distribution for groups of architecturally identical models, copied from the plot on the diagonal above. Accuracy of each model is shown the top.
22

---

Published as a conference paper at ICLR 2021
RN-26 1.0x
RN-26 1.4x
RN-26 1.7x
RN-26 2.2x (94.949)
RN-26 2.4x (95.05%)
RN-26 3.0x
RN-26 4.0x
RN-26 8.0x (95.81%)
(33.09%)
(94.10%)
(94.53%)
(95.30%)
(95.53%)
RN-26 1.0x
100
(96)
RN-26 1.4X
100
Acc.
RN-26 1.7x
100
(%6
2.2x
100
(96)
RN-26
244
100
(%6
RN-25
3.0x
100
RN-28
RN-26 4,0x
FIN-25 8,0x
100
Acc. (%6)
1000 D
1000
100
100 0
100 0
100 0
100 0
100
RN-26 1.0x
RN-26 1 ,,x
RN-26 1.7x
RN-26 2.2x
RN-26 2.4x
AN-26 3.0x
RN-26 4.0x
RN-26 8.0x Acc. (96)
Acc. (96) Acc. (%)
Acc. (96)
Acc. (%)
Acc. (96)
Acc. (%6)
Acc. (96)
Figure F3: Effeet width on example accuracy. Scatter plots per-example accuracies ResNet models with different widths on CIFAR-10. Blue dots indicate per-example accuracies of two groups of 100 networks each with different architectures indicated by axes labels. Orange dots show the expected distribution for groups of architecturally identical models, copied from the plot on the diagonal above. Accuracy of each model shown the top.
23

---

Published as a conference paper at ICLR 2021
F.2
EFFECT OF VARYING WIDTH AND DEPTH ON IMAGENET PREDICTIONS
ResNet-50 (76.97%)
ResNet-50 (2.8x) (77.97%)
ResNet-89 83 (78.00%)
100
ResNett 50
Acc. (%)
50
0
ResN 50 (2. (2.8x)
100
Acc. (%)
100
ResNet-88 8-3
Acc. (96)
100 0
100
100
ResNet-50 50
ResNet- 50 (2.8x)
ResNet-89 83
Acc. (%)
Acc. (%6
Acc. (96)
Figure F4: Systematic per-example performance differences between wide and deep models Ima- geNet. Scatter plots of per-example accuracy averaged across vanilla ResNet-50 (1x) models versus that for groups of 50 models with increased depth (6 -> 17 blocks, "ResNet-83") or width (2.8x wider) in the 3rd stage. Orange dots in plots show the expected distribution for two groups of 50 architecturally identical models, copied from the plot on the diagonal above. The deeper and wider models have very similar but statis- tically distinguishable accuracy (mean + SEM for deeper model: 78.00 0.01, wider model: 77.97 0.01, t(99)= = 2.0, p 0.017).
F.3 EFFECT SIZES FOR CLASS-LEVEL EFFECTS
We measure how much of the difference between the example-level predictions of the wide and deep ImageNet ResNets in Section 7 could be explained by the classes to which they belonged by fitting a set of three models. Model A attempts to model whether each pre- diction was correct incorrect as linear combination factors corresponding to the exam- ple ID and whether the prediction came from a wide or deep model (stat smodels is for- (wide_or_deep))
mula y C (example sponding to the example ID as well as factors corresponding to the interaction between the class ID and the type of model the prediction came from (stat smodels formula C Cexample _id + C (wide _or deep) * C(class_ ______d).. ). Model C includes the interaction between the example ID and the type of model, and corresponds to simply mea- suring the average accuracy separately for both types of models (stat.smode s formula Note that model C is nested inside model y C (texample_id) * C (wide_or_deep) B, which is nested inside model A.
Model B includes a factor corre-
We seek to measure how much of the variability in predictions that can be explained by model but not model A can be explained by model B. We fit these models with logistic regression and measure the residual variance Varq = >m=1(0 (1/ T2)² /n, where n is the number of examples x the number of models, y/ is either or 0 depending and T; is the output probability from logistic regression model Q. We then compute the squared differences between y and the predictions of the logistic regression model:
whether the CNN's prediction was correct or incorrect,
Var Var
0.11.
(10)
Var Varc
This approach is analogous the pseudo- R² of Efron 1978
24

---

Published as a conference paper at ICLR 2021
Finally, compare the AIC values of the logistic regression models, shown in the table below. Because the models are nested GLMs, we can also test for statistical significance using a x² test, which is highly significant for each pair of nested models. We do not report p- p-values because they are O to within machine precision.
Table F.1: AIC for models A, B, and C, described above
Model
AIC
A: Example Model
3011367
B: Example + Class Moodel
3006889
C: Example * Model
2969410
F.4 PERFORMANCE DIFFERENCES AMONG IMAGENET SYNSETS
Table F2: Differeces between wide and deep architectures on ImageNet synsets with many classes. Com- parison of accuracy of wide (ResNet- 50 with 2.8x width in 3rd stage) and deep (ResNet-83) ImageNet models synsets with >50 classes. Note that some synsets are descendants (hyponyms) of others. p-values are com- puted using t-test with multiple testing (Holm-Sidak) correction. Results for the of models used generate blue dots Figures F.4 and Post-selection effect sizes and testing the main text use disjoint set of models.
Class
# Classes
Wide Acc.
Deep Acc.
Diff.
p-value
entity
1000
1 0.01
78.0 0.01
0.89
physical entity
997
78.0 1 0.01
78.0 0.01
-0.03
0.89
object
958
78.1 + 0.01
7..1
-0.04
0.76
whole
949
7
78.2
-0.05
0.48
artifact
522
0.02
73.8
-0.01
living thing
410
8.06
-0.10
0.023
organism
410
83.5
83.6
-0.10
0.023
animal
398
+
83.1
0.09
0.032
instrumentality
358
73.9
7..0
0.02
vertebrate
0.02
8.02
0.22
chordate
337
8.02
0.08
0.22
mammal
218
+..03
82.1 0..03
0.09
0.47
placental
212
81.9
0.09
0.08
0.66
carnivore
158
81.1
0.03
0.09
0.73
device
130
72.9
7.09
0..0
1
canine
130
81.3 1.0
8..4
0.04
domestic animal
123
8.0M
81.0
0.00
dog
118
0.06
8..6
0.01
container
100
0.05
7.07
0.00
covering
90
0.05
7.02
-0.19
0.13
conveyance
72
8..0
8..4
0.13
0.65
vehicle hunting dog
67
8..2
83.1
0.11
0.76
63
0.05
8.02
0.01
1
commodity
63
+
7..6
-0.22
5.1
x
10
consumer goods
62
7.03
72.7
-0.41
6.7
10
invertebrate
83,6
83.8
-0.16
0.37
bird
59
9..5
92.7
-0.21
0.0018
structure
58
75.9
75.5
0.42
5.7
10
matter
50
7.06
7.04
0.17
0.74
25


Reviews:
[{"review_id": "iZiAM4GPYof", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper，the author studies the effects of width and depth on neural network representation.\n\n\nIn this paper，the author studies the effects of width and depth on neural network representation. This paper conducts lots of experiments on CIFAR-10, CIFAR-100 and ImageNet with different network architectures and apply the CKA to measure the similarity between representation of each layer. As a result, they find a characteristic block structure in the hidden representations of larger capacity models which is also dependent to the size of dataset. This work has the following advantages:\n1、\tWell-arranged and detailed experiments which strongly support the final conclusion.\n2、\tExquisite figures that well displays the experiment results.\n3、\tThe concept of “block structure” in ResNet is novel. And all of the experiments and analysis illustrate there really exists blocks with similar representation in overoptimization models. And this phenomenon can guide researchers to design networks well.  \nHowever, there are some disadvantages or doubts in my opinion:\n1、\tThis paper lacks of further explanations about the CKA or HSIC tools. I can’t fully understand how the similarities between representations of each layer are measured.\n2、I wonder if the block structure arises dependent to the residual blocks. I want to see more experiments with other network architectures.\n3、I think the analysis of effects of width and depth on neural network representation can well guide researchers to design networks. So，I expect to see an modified network architecture or a method to balance the network size and accuracy . However, this paper is just about theoretical analysis based on experiment phenomenon. Thus, I think this paper is lack of some innovation.\n\n4. Some previous related works is better to be appreciated: \n\nC. L. Philip Chen et al Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture，IEEE Transactions on Neural Networks and Learning Systems，2017 \nC.L.P. Chen, Z. Liu, and S. Feng, Universal approximation capability of broad learning system and its structural variations， IEEE transactions on neural networks and learning systems 30 (4), pp. 1191-1204.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "dlMMMBw6nPO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores characteristics of resnet networks that arise with different capacity.  In particular, a blockwise structure of behavioral similarity arises when the network has large capacity, regardless of whether that capacity was introduced with greater depth or with greater width (similarity is measured using CKA, which compares the pattern of pairwise comparisons among elements in a minibatch between two different sets of layers/activations).\n\nThis is an interesting method and characterization of resnet behavior, with thorough experiments that tie together different aspects of the approach:  CKA is used to show a type of blockwise similarity, much of which is subsequently explained, and related experimentally to classification performance using linear probes through the layers.  Finally, it opens questions for future research, such as: despite qualitatively similar layerwise behavior, why does larger depth vs larger width produce different types of classification errors?  Might any of the differences in patterns that develop for deep vs wide in CKA relate to the spatial scale of regions that most influence each layer (the effective receptive fields)?\n\nOverall this looks like a rather thorough investigation.  I have a few questions of things I don't quite know how to interpret, below, which I feel could be made clearer.\n\n\n\nQuestions:\n\n- Fig 1:  In the last two plots of each of the top two rows, it appears the block starts to develop from the \"middle\" out, in two sections, as opposed to from the first or last layer in a section.  Is there a reason for this and what does it correspond to?\n\n- Sec 5.1:  Is there an intuitive description of the first principal component in R^n, as opposed to R^p1 --- it seems easier to think of the latter, being the activation vector that captures the most variance over n datapoints, but the one used here is from the transpose, with the vector in R^n (i.e. coefficients of data samples):  What does this mean and how does it relate to the block structure and/or residual connections?\n- Given that much of the blockwise structure is due to the first principal component, can anything be said about the structures that arise after removing it?  How do these behave when increasing depth vs width and do these relate at all to the different classification results?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TuKst880uy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to get a better understanding of differences between wide and deep neural networks through an empirical evaluation. It does it through a similarity analysis, and an analysis of which errors wide and deep residual networks do.\n\nThe paper is well written and should not be difficult to reproduce. Most of the questions I got while reading where addressed in the paper, in an extensive evaluation. The most interesting and somewhat surprising finding is that even though two networks with different number of parameters and layers but with the same accuracy make very different mistakes, and there is a pattern to it. The weakest part is the similarity analysis, which does not seem to reveal much new. It has already been known that deep networks have layers which do not contribute significantly to final accuracy, can be proved or even forced to learn more useful representation. Would be interesting to apply the similarity analysis in a network with reinitialized layers. Also, the choice of specific similarity function and it's benefits/drawbacks are not discussed. Could the same analysis be achieved with a simpler similarity function? Is it computationally efficient to compute?\n\nOverall, there are definitely valuable contributions in the paper, so I propose lower score only due to the unclear choice of similarity function, as described above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Eeue4NSTACR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores core questions related to how depth and width of deep neural networks affect the learned representations. The authors attempt to develop an empirical understanding of the behavior of neural network architectures after training on real-world data. In my humble opinion, the paper is very clearly written, presenting at the beginning of each section the scientific question they try to answer. The questions that the authors try to answer are pertinent, the conclusions are consistent with the results obtained, and statistical tests are employed to check the existence of statistically significant differences between the different approaches tested. I have several questions that the authors should take more as comments or suggestions than criticisms:\n\n- The paper focuses on the family of ResNet models. Do the authors have solid reasons to believe that their findings generalize to other neural models (other ConvNets, recurrent, generative,...) and problems (regression, dense prediction,...?\n\n- Apart from the fact that \"some layers that make up the block structure can be removed with minimal performance loss\", what are the main practical consequences of the findings of this research? \n\n- The authors state that \"on ImageNet, [...] wide networks perform slightly better on classes reflecting scenes, whereas deep networks are slightly more accurate on consumer goods\". Do the authors have any intuition about the reason for this behavior? What more general conclusions could be extracted about the performance of deep vs wide neural networks? \n\n- Related to the previous point, in the last years, the recommendation (for instance, in [1]) was to go as deep as possible, and not care too much about wide neural networks. However, if I'm not mistaken, according to the results obtained by the authors \"the average accuracy of these groups [deep and wide models] is statistically indistinguishable\". Could the authors comment a bit more on this? How all these experimental results corroborate, complement or refute some of the already existing theoretical results regarding depth and width in neural nets?\n[1] Eldan, Ronen, and Ohad Shamir. \"The power of depth for feedforward neural networks.\" Conference on learning theory. 2016.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies whether neural networks with different architectures, especially different width and depth, learn similar representations. All reviewers agree that the investigations are thorough and the experimental discoveries are convincing and well explained. Good work. I recommend accept.
Decision: Accept (Poster)