Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance
Authors: Yiting Chen
Jiazi Bu
Junchi Yan
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural Network Layer width.+Neural Network Training Dynamics
Primary area: other

Abstract: The trade-off between cost and performance has been a longstanding and critical issue for deep neural networks. 
One key factor affecting the computational cost is the width of each layer. 
However, in practice, the width of layers in a neural network is mostly empirically determined. In this paper, we show that a pattern regarding the variance of weight norm corresponding to different channels can indicate whether the layer is sufficiently wide and may help us better allocate computational resources across the layers.
Starting from a simple intuition that channels with larger weights would have larger gradients and the difference in weight norm enlarges between channels with similar weight, we empirically validate that wide and narrow layers show two different patterns with experiments across different data modalities and network architectures. 
Based on the two different patterns, we identify three stages during training and explain each stage with corresponding evidence. We further propose to adjust the width based on the identified pattern and show that conventional layer width settings for CNNs could be adjusted to reduce the number of parameters while boosting the performance.

Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance
Yiting Chen, Jiazi Bu, Junchi Yan
Dept. of CSE& & School of AI & MoE Key Lab of AI, Shanghai Jiao Tong University
[situcyt. bujiazi001, yanjunchi @ @sjtu.edu.cn
https://ggt http://ccc///tteeeee htt.:///cccheeeeeeaeaeeeeeaaeeee Mataaameeelaathh___Fee__eee
Abstract
The trade-off' between cost and performance has been a longstanding and critical issue for deep neural networks. One key factor affecting the computational cost is the width of each layer. However, in practice, the width of layers in neural network is mostly empirically determined. In this paper, we show that a pattern regarding the variance of weight norm corresponding to different channels can indicate whether the layer is sufficiently wide and may help us better allocate computational resources across the layers. Starting from a simple intuition that channels with larger weights would have larger gradients and the difference in weight norm enlarges between channels with similar weight, we empirically validate that wide and narrow layers show two different patterns with experiments across different data modalities and network architectures. Based on the two different patterns, we identify three stages during training and explain each stage with corresponding evidence. We further propose to adjust the width based on the identified pattern and show that conventional layer width settings for CNNs could be adjusted reduce the number of parameters while boosting the performance.
Introduction
The cost-accuracy trade-off has been longstanding and critical issue for deep neural networks. As one key factor that affects the computational cost, the width of each layer mostly empirically determined or extensively searched over architecture space in neural network architecture search
literature [43 22). Despite the research on over-parameterization [24] 23 and empirical evidence [37
15 31 41 showing that the wider network leads to better performance, there are few works about how we allocate the computational resources across the layers. On understanding the layer width requirement in deep neural network, (2 theoretically proves the lower bound of layer width for the model to approximate any Lebesgue-integrabe function while 44] improves the width lower bound with dynamic systems. However, practical indicator of sufficient layer width is still lacking.
In this paper, we identify a simple pattern difference between wide and narrow layers regarding the weight norm variance between different channels during training. A simple observation is that larger- weight channels would have larger gradient. the weight norm increases during training [30, 11], we show the disparity of weight norm between similar channels also increases during training, which we call the Matthew effect between channels. It motivates us to investigate the weight norm variance across the channels in each layer during training. With experiments image, graph, and text datasets (including CIFAR-10 10 [8]. cora [35 and enwiki8 (22) with various model structures including
*Corresponding author. This work was in part supported by NSFC (92370201, 62222607) and Shanghai Municipal Science and Technology Major Project under Grant 2021SH2DZX000
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(GCN 177. GRU (,. VIT 19), we show that wide and narrow layers show two different patterns during training. As we change the layer width, we show that narrow layers show a decrease until the saturate (DS) pattern, such that the weight norm variance first increases and then decreases. For wider layers, the weight norm variance keeps increasing until saturate (IS). We conjecture that, for sufficiently wide (or ) layers, channels with small weight norm would always be surpassed by similar channels with larger weights, and therefore, the variance keeps increasing until convergence. On the other hand, for narrow layers with few channels, the channels with small weights could be orthogonal to channels with large weights, and the variance decreases.
We further show that the training from random initialization to convergence can be divided into three stages for neural networks showing the identified IS or DS pattern. In the first stage, typically a few epochs after random initialization, since the gradient is almost orthogonal to the weight vector, the variance between weight norms corresponding to different neurons does not increase or even decrease. In the second stage, both the performance of the network and the variance between neurons drastically increase for a dozen epochs. indicates a fast learning process with the weight norm of a few neurons increasing drastically. In the third stage, the wide layers show the IS pattern, such that the weight norm variance keeps increasing and staying high, while the narrow layers show the DS pattern, such that the weight norm variance starts to decrease. We provide our explanations for each stage of training with corresponding empirical evidence on VGGs [29] trained on CIFAR1O.
Furthermore, we show that the conventional layer width setting for CNNs such as VGG or ResNet might not be optimal. Generally, with the conventional layer width setting, the layers in the middle show the DS pattern while the former and latter layers show the IS pattern. It indicates that the layers in the middle could use more width, which also cohere with the intrinsic dimension firstly increasing then decreasing and reaching the high point at the middle (2]. We propose adjusting the width of each layer of widely used networks to boost the performance and reduce the number of parameters. We summarize the contributions of this paper in the following:
To our best knowledge, we are the first identify patterns regarding the variance of weight norm between different channels during training that can indicate whether a layer is wide enough. We have observed the pattern with different model structures on different data modalities (including CNNs and ViTs for image classification (29 14 9. GCN for graph node classification [17), and
GRU for language modeling (6).
Based on the identified pattern, we further show that there are, in general, three training stages from random initialization to convergence. We provide explanations with empirical evidence on VGG models trained for image classification that supports our explanation.
We verify that with the identified pattern, the conventional width setting for CNNs could be adjusted for fewer parameters and better performance on CIFARIO and TinyImageNet.
2 Related Works
Researches on the Width of Neural Networks. By simply widening the network, early works B7 15] empirically show that the performance can be im- proved. On the other hand, structured pruning meth-
Description
Measred by lesting aoas- racy after con- vergence.
The critica
Prwiding cornupted dats the early training plase
pesiod
in
training I1
lends to inrewcable damuge
final perfommance
ods 33 36 [6] propose to reduce the width for ef- ficiency. Compared to the static pruning methods, another branch of research focusing on dynamically changing the width of the network conditioned the input by skipping neurons in linear layer (3) branches in Mixture of Experts (MoE) [10] and channels in
Froguency-
Gradieni norm
principle
2x
frequency information and then higuefeequeccy in- formation.
on different frequency
components.
Crokking M
Neural networks show pe- ried of sear cear-perfect ing performance nearly randoen guessing test pei- formance before generaliza- tice.
Training and testing
ray.
CNN [12 19] at inference time. Readers interested in dynamic networks could refer to [3] for more details. Notably, a recent work [5] proposes to merge neurons to reduce the computational cost where lay- ers at the middle have fewer neurons that could be merged. Similar results are also reported regarding investigating the intrinsic dimension 2 across the layers. Unlike these previous works focusing on the trained model, we show that one could identify the
Domble- desccur
The testing accuracy firstly and evenually improves model capacity incresses ur the training goes, un. Sco deinls in Fig [xby
Testing 8o00- recy.
improwes, tien
Ours
Weight norm variance in layer
Table 1: We list some different training dy- namics phenomena.

---

w!!l will
Skge
Supe3
Trugi katuauo daccs
14th koyer
Triig matazoo dardialyy ticeusss
Ratee a > Pagel
Page Raye
jabkes Ne eoneue kusmooo liyer vide sesugh.
w/) u] betwren seailler chanrel
Do tire Meigh dane
qut dan law and, Edlasio
willa
(a) Illustration of the Matthew effect between channels with (b) Two different patterns we identified of weight vectors of similar direction.
how weight norm variance change during training regarding layer width
Figure I: Illustration of our motivation and the identified weight norm variance pattern. Fig, (a): For two channels with weight vectors of similar direction, the gradient of the former layer is proportional to the weight norm of the latter layer and vice versa. As empirical evidence indicated, the weight norm increases during training, and the weight norm disparity between channels increases. Fig. (b): The two different patterns For wide layers, the weight norm variance keeps increasing, which we call the increase to saturate pattern (IS). For narrow layers, the weight norm starts to decrease after reaching high point, which we call the decrease to saturate pattern (DS).
identified of how weight norm variance would change during training.
layers with insufficient width during training. We further propose a simple width-wiss streamlining technique to boost the network performance and reduce the number of parameters. Note that the objective of our width-wise streamlining technique differs from the pruning method and we make the FLOPs of the adjusted model close to the original network.
Dynamics of Network Training Investigating the training process of neural networks has always been an active research topic. The early phase of neural network training is investigated in [[] and timeline for the early phase of training (the first several epochs) is provided while other works [24 g
25 have focused on the later training phase investigating the effect of preepaarmtttiizttionn on the generalization and the training heuristics that lead simpler solutions. From the frequency spectrum perspective, F-principle (27
34 32 shows that wwwffrequncy components play more important role in the generalization and that the training of CNN could be divided into two phases where the CNN firstly learn now-frequency components then learn high- frequency components. The frequency principle theoretically proves that the gradient on low-frequency components will larger [21]. In this paper, we analyze the training dynamics from the weight norm variance perspective and show that the training of neural networks could be summarized into three stages general. To our knowledge, this is the first work to investigate the variance between weight norms corresponding to different neurons. We list some well-known phenomena Table [[] While a recent work 77] correlates grokking [26] with double-descent 23) hope the relationship between the three stages of training and the other phenomena could be further explored in future works. For example, the difference between the second stage and the third stage could be the result of network learning different frequency components, easy or hard data samples [42 etc.
3 Different Patterns between Wide and Narrow Layers
In this section, we first provide a simple analysis showing the motivation for us to investigate the weight norm variance across the channels of layer. We then provide empirical results from experiments across different data modalities and model structures to show that there are two different patterns between wide and narrow layers.
3.1 Simple Analysis Regarding Channels in Layer During Training
Let us consider an MLP with ReLU activation for simplicity. Suppose we have d channels in the (-th layer. Let 2/1 E Rdi.. and z/+1 E Rdout denote the features at the - 1-th layer and the + 1-th layer. The weight for [-th layer and l + 1-th layer is W' E Rddnxd and W/+1 E [Rdxdour, Then we have:
z ++
W oW''2')
(I)
3

---

where (-)) is the ReLU activation function, Let w Rh denotes the i-th row vector of W and w/t1 E Reduu denotes the i-th column vector of Wi++, the output corresponding to the i-th channel is (w =²-1) w/( and the output a/ ! is the combination of the outputs of all the d channels
Cow w
(2)
Suppose the loss function is L. and the gradient for +1 is V x L. then we have
whit = - w z' V Vettih
w/ T z/-1 2 0 W x 0
(3)
V w = { (w/
V gi+ L) z1-1
w! x w; z'-1 0
(4)
0,
For two channels m, n [1, d]. suppose
w w
and
w Iw
then w
z111__1
and
1|
w
[T g/ would have the same sign and we have
1|w
V
wht
1+1 C
~
|ww]
||
V
w
C.
(5)
|wt+1| Vw! L.
(6)
[ [+1
It means that for channels with weights of similar direction, the larger ||w leads to larger V + C
and the larger ||w[+]|| leads to larger W. While empirical evidence in previous works 30 11
has shown that the weight norm increases during training. It means that, for two channels with similar weight direction, the weight norm disparity between them would generally keep increasing during training, which we call the Matthew effect between similar channels. For the channel with a larger weight, the larger weight norm at the I ++1 layer leads to larger gradient of the corresponding weight at the / layer, which increases the weight norm at the I layer. In turn, a larger weight norm at the / layer further speeds up the increase of the weight norm at the |+ layer. According to [5], sufficiently wide layers may learn many similar channels after training, therefore. we expect the variance of weight norm across different channels would higher in the wide layer than in the narrow layer. motivates us to investigate further the weight norm variance between different channels in layer.
3.2 Empirical Evidence of Two Different Patterns for Wide and Narrow Layers.
In this subsection, we investigate the variance of weight norms corresponding to different channels. For the i-th channel in Eq. 2 with ReL.U as activation functions we have
(W z
1+1
z 2
z 1-1
0
0,
2
Therefore, we define the weight norm corresponding to the i-th channel as ||w||| ||w conduct our experiments several datasets across different data modalities with different model architectures and record the variance of corresponding weight norms during training.
We
We report the weight norm variance results for Graph Convolutional Neural Network (GCN) [17
trained on graph dataset cora (35), Gated Recurrent Units (GRU) (6] trained on text dataset en- wiki8 (22] and Vision Transformers (ViT) (9] trained on image dataset CIFAR10 [8] As we change the width of a certain layer (e.g. the MLP module in the transformer), we show that the wide layer and the narrow layer show two different patterns. For the wide layer, the weight norm variance firstly increases and stays at a high level until convergence, which we call the increase to saturate (IS) pattern. For the narrow layer, the weight norm variance decreases after the initial increase, which we call the decrease to saturate (DS) pattern.
It is similar for other ReLU-like activation functions such leaky ReLU.

---

width 16 (acc 76.20%)
width 32 (ace 79.90%)
width 64 (ace 80.30%) (a) GCN on cora
width 128 (ace 79.80%)
Payl
input-hidden, width 500 input-hidden, width 800 input-hidden width 1000 input-hidden width 1150
(b) GRU on enwiki8
Figure 2: Weight norm variance change during training of GCN 17 on cora [35 and GRU 6 on enwiki8 [22] We change the width of the second layer in GCN and the input-hidden weight of the second layer in GRU. As the width increases, the weight norm variance during training shows two different patterns. For the wide layer, the weight norm variance increases and stays at a high level until convergence, which we call an increase to saturate (IS) pattern. For the narrow layer, the weight norm variance decreases after the initial increase, which we call the decrease to saturate (DS) pattern.
As shown in Fig. present the results of GCN
Accr80.915
Acc=77
17
on cora 135 and GRU [6] on en-
In Fig. 2(a) the results show the weight norm variance change of the second layer with different widths. As the width increases from 16 to 64, the weight norm variance changes from DS pattern IS pattern. Notably, increas- ing the width when the DS pattern is identified (from 16 64) leads an accuracy boost, while increasing the width when the IS pattern is iden- tified (from 64 to 128) does not. In Fig. [(b] we report the results of the weight norm variance change of the input-hidden gate of the second layer of the GRU. Though the recurrent neu- ral networks (RNN) have different architectures compared to the multi-layer ReLU network we discussed before, the weight norm variance of the narrow and wide layers also shows DS and IS patterns, respectively.
wiki8
22
Opoets
Epoche
Figure 3: Weight norm variance between different channels of the MLP at the 5th layer of tiny VIT trained on CIFAR-10. The left figure corresponds to the result where the MLP size is set as 512, and the right figure corresponds to the result where the MLP size is set 32.
Fig. B shows the results of two VITs trained on CIFARIO. We use VIT with 6 layers where each layer has 8 heads and the hidden size for each attention layer is 512. The width of the MLP at the 5th layer is set at 32 and 512 respectively. For the layer of width 512, the weight norm variance always increases and stays a high level following the IS pattern. As we reduce the width of the MIP to 32, the weight norm variance starts to decrease after reaching a high point following the DS pattern. For more details the GCN, GRU, and ViT training, please refer Appendix B
Our Conjecture About the Different Patterns Regarding Layer Width. As shown in Eq. 2 each layer is like "mixture of experts" where each channel has the same weight. The only difterence between channels is the random initialization. For wide layers with large number channels, the chance for two channels learning weight of similar direction is much higher than that for narrow layers with limited channels. Therefore, the weight norm variance among channels does not decrease while the weight norm variance among channels of a narrow layer decreases after reaching a high point. In the next section, we provide a more detailed analysis with empirical evidence about the three training stages we identified.
wide layer
5

---

(a) Isrlayer VGG
(b) 5th loyer VGG
(c) 13th layer of VGG
(d) loyer 3, ResNet
(c) layer 10, ResNer
(f) layer 17, ResNer
Figure 4: The variance of weight norm || ||w different layers of VGG-16 and 8 88 trained on CIFAR-IO. The y-axis corresponds to the variance while the x-axis corresponds to the epochs trained after initialization. From left to right is the result for the first layer, the layer in the middle, and the last layer. For more results please refer to Appendix
(a) layer
(h) Layer 3
(c) Layer 7
(d) Layer 11
(c) Layer 17
Figure 5: Density plots of the weight norm of different layers of a ResNet-18 18 trained on CIFAR1O. The x-axis corresponds to the weight norm, and the y-axis corresponds the density. For clarity, we omit the label for the y-axis. From top to bottom, each density plot corresponds to different epochs, we demonstrate how the weight norm distribution different layers changes during training.
Three training Stages from Random Initialization to Convergence
Based on the two patterns we identified regarding the weight norm variance, we further provide our explanation for this phenomenon. We show that, generally, we can divide the training of the neural networks showing the identified pattern into three stages. We provide our explanation for each stage with corresponding empirical evidence on VGG-16 (9) and ResNet-18 [4 trained on CIFAR10 [8]. As shown in Fig. 4 we report the weight norm variance of some of the layers in VGG-16 and ResNet-18. In Fig. [ also present the density plots of the weight norm for different layers of a ResNet-18 8 trained on CIFAR10 at different epochs. Generally, different layers show different patterns. The layers the middle typically follow the DS pattern and the former and latter
layers follow the IS pattern. We will discuss this phenomenon in the next section Sec. 5
4.1 Stage Weight Variance Stay Low Shortly after Initialization
Starting from a random initialization, the first stage of neural network training lasts for a few epochs. this stage, the variance of the weight norm between neurons does not increase. At cer-
weight sion changed rato (%)
tain layers, the variance even decreases between neurons. Starting from random initialization, due to the high-dimensional weight vectors of each neuron, the gradient is more likely to be or- thogonal to the weight vectors corresponding to different neurons. When measuring the cosine similarity between the gradient and the weight vector across all the channels across all the lay- ers VGG16 bn trained on CIFAR-10, the largest absolute value is 0.0003 at initialization, which provides an explanation for the weight variance not increasing.
epoch
Figure 6: Ratio of weight elements where the sign changed compared to the initialization during training different models on CIFAR-10. We show the result for the first 10 epochs. After the first epochs, the sign of at least 10% weights changes, indicating dramatic change the weight space.
In some scenarios, we also find weight norm variance decreases at the first stage. We conjec-

---

Pearson corclation: 0.8891
tedd
0.50
102
0.00
50
23
2.5
Pearson comelation: 0,4254
weightoomm space
2.04-
2.5
2.00
epochs
0.0
wiightt shorthe cooch
2.5
(a) The loss, accuracy, and weight norm variance of (b) Correlation between the weight norm at the sec- the 5-th layer of a VGGI6 trained on CIFAR-10 ond stage and the weight norm after convergence
Figure 7: (a): The loss, accuracy, and weight norm variance between neurons at 5-th layer of VGGI6 trained on CIFAR-10. The model is trained for 150 epochs, after the second stage of training mentioned in Sec. 4.2 the model has achieved a high performance at around 20 epoch. For more details please refer to Appendix B (b): We demonstrate the correlation between the weight norm at the 6-th epoch and the weight norm after the last epoch. The x-axis corresponds to the weight norm after the last epoch (150-th epoch). For the upper plot, the y-axis corresponds to the change of weight norm from 6-th epoch to 4-th epoch. For the lower plot, the y-axis corresponds to the weight norm at the 6-th plot. Each point represents a neuron at the first layer of VGG16.
ture that results from the fact that the direction of weight vectors is drastically changed in the first stage. In fact, the first stage of neural network training is an extremely chaotic stage where the direction of each weight vector is dramatically changed. Fig. shows that, for different networks, the sign of at least 10% weight elements changes after the first 5 epochs. As result of the dramatic change weight space, the direction of many weight vectors reversed and the weight norm firstly decreases and then increases.
4.2 Stage 2: Both Performance and Variance Drastically Increase
In the second stage, the variance of weight norm among channels starts to increase drastically. This phenomenon indicates that for some of the neurons, whose corresponding weight vectors are close to the gradient direction, the corresponding weight norm increases much faster than other neurons. At the same time, the performance also increases rapidly as shown in Fig. ((a) The test performance after this stage is very close to the final performance, e.g. the testing accuracy for the VGG16 on CIFAR-I0 at 20 epoch in Fig. 7( is at 84.65%, which is relatively close to the final testing accuracy 92.94%. We conjecture that to reach a fair performance, the network only needs to learn a few significant features, channels whose weight direction is close the direction of the gradient would have advantages.
The second stage generally determines the relative weight chaotic first stage, the second stage is much more stable, where the weight norm of those neurons corresponding the "significant features" discussed above constantly increases faster than other neurons. Notably, at the beginning of the second stage, the tendency of how the weight norm would change provides more information than the weight norm itself. As shown in Fig. 7bb) we demonstrate the weight norm at the beginning of the second stage and the weight norm after convergence. The result is from channels at the first layer of a VGG16 trained on CIFAR-I0. The x-axis corresponds to the weight norm after convergence. For the upper subplot, the y-axis corresponds to the difference between the weight norm at the 4-th epoch and the weight norm the 6-th epoch. For the lower subplot, the y-axis corresponds to the weight norm at the 6-th epoch. The Pearson correlation between the weight norm difference and the final weight norm is 0.8891 while the Pearson correlation between the weight norm at 6-th epoch and the final weight norm is only 0.4254. We conjecture that the channels whose weight norm increases rapidly this stage correspond to those easy features such as the low- www-frequency information in prequency-principle [27 34 32..
after convergence. Compared the
7

---

(a) The first layer (IS pattern)
(b) The 5-th layer (DS pattern)
(c) The 13-th layer (IS pattern)
Figure 8: The cosine similarity between weight vectors at the same layer of a VGG16 trained on CIFAR-10. The value at i-th row and j-th column correspond to the cosine similarity between the weight vectors of the i-th neuron and the j-th neuron. Note that we sort the neurons by their weight norm in descending order which means the 0-th neuron is the neuron with the largest weight norm. The results show that weights for neurons at the 5-th layer of VGG16 are almost orthogonal.
4.3 Stage 3: Variance Decreases or Increases to Saturation at Different Layers
The third stage is the longest stage where the performance slowly increases. In the third stage, layers show two distinct patterns which indicates whether this layer learns similar neurons. For the variance of weight norm between neurons, there are two different patterns across the layers:
Increase to Saturate (IS): The variance continues increase slowly and stay high.
Decrease Saturate (DS): The variance decreases after reaching high value.
Interestingly, as shown in Table 2 we find that for layers where neu- rons (or channels) could be merged using the algorithm proposed in [5]. the variance between neurons follows the pattern that increases and stays at high level. On the other hand, for layers where no neurons could be merged, the variance between neurons follows the DS pattern and decreases. Fig. [8 shows the cosine similarity of weight vectors between different neu- rons. For layers following the DS pat- tern, the cosine similarity of weight vectors between most different neu- rons is almost zero while the cosine similarity between neurons for lay- ers following the IS pattern is much higher. It indicates that the weight vec- tors of channels in layers showing the DS pattern are nearly orthogonal while layers following the IS pattern contain many similar channels. Note that the neurons are sorted by the weight norm where the O-th neuron is the neuron with the largest weight norm. Fig. ((a) and Fig. [(C) show that neurons with different weight norms could be in similar direction. Further results of the weight norm distribution are presented in Appendix A
Width alter neuron merging
Layer
Third stage pattern
ongin
Layer
64
IS pattern
Layer
64
63
DS pattern
Layer
128
128
DS pattern
Layer
128
128
DS pattern
Layer
256
DS pattern
Layer
256
256
DS pattern
Layer 256
256
DS pattern
Layer 512
504
IS pattern
Layer 512
IS pattern
Layer
512
509
IS pattern
Layer
512
506
IS pattern
Layer 12 512
420
IS pattern
Layer 512
IS pattern
Table 2:
We employ IFM 5 to merge the neurons (channels) across the layers of VGG16 trained on CIFAR-10 10. We list the original width, the width after merging, and the weight norm variance pattern in the third stage.
Width Modification on CNNs for Fewer Parameters and Better Performance
As discussed in Sec. 4.3 the layers in VGG-16 show different patterns. In this section, we investigate the weight norm variance change during the training of CNNs. We empirically show that for widely used CNNs such VGG and ResNet, the middle layers show the DS pattern and other layers

---

Table 3: In this table, we report the number of parameters, FLOPs, and the testing aceuracy of the original VGG and ResNet models and our width width-adjusted models. Each result is averaged over 10 runs with a different random seed. For a fair comparison, we make the FLOOPs of the width-adjusted models close to the original model with nearly 40% parameters reduced.
Top-I Accuracy
Model
Params
% of params
FLOPs % of FLOPS
CIFAR T0
CIFAR- 100
VGG16
origin
14.73M
314.3M
91.02 0.35
66.17 = 0.71
streamline width
9.06M
61.55%
525.55M
103.89%
93.41 1 0.15
68.63
0.31
VGGI9
origin
20.04M
399.35M
91.84 1 0.46
65.53 1 0.66
streamline width
11,50M
57.39%
904.
101.40%
92.38 + 0.09
66.99
+ 0.52
ResNet18
origin
11.17M
89M
92,99 - 0.23
72.84
= 0.27
streamline width
6.26M
55.99%
604.22M
108.30%
93.60 1 0.13
73.38
0.30
ResNet50
origin
23, 52M
131159M
92.47 1 0.43
73.23
1 0.45
streamline width
14.91M
63.40%
1404.87M
107.11%
93.12 0.33
73.53 0.50
show the IS pattern. We then adjust the layer width for VGG and ResNet, reducing the number of parameters while boosting the performance compared to the conventional layer width setting.
5.1 Investigating Weight Norm Variance Across the Layers of CNNs
As shown in Fig. 4 we report the variance of weight norm ||w|]| ||w/ different layers of VGG-16 and ResNet18 trained on CIFAR-10. Generally, we find that the first several layers follow the IS pattern while the layers in the middle follow the DS pattern, and the layers at last follow the IS pattern again. It reflects the intrinsic nature of the neural network training such that layers in the middle learn orthogonal neurons. One may naturally associate this phenomenon with the results that intrinsic dimension firstly increases then decreases and reaches high point in the middle (2]. Intrinsic dimension describes the minimum dimension needed solve a certain problem precision level. We conjecture that intrinsic dimension is possible explanation for the two different kinds of layers where high intrinsic dimension leads to orthogonal weights. For more results, please refer to Appendix A It further indicates that the conventional layer width setting for CNNs might not be optimal. In the conventional setting (such as in VGGs and ResNets) the layer width increases through the first several layers and then stays large through the following layers. Since the layers in the middle could use more width and the layers in the front and back could use less width, a better setting might be that the width first increase and then decrease across the layers with the widest layer in the middle just like a streamline.
certain
5.2 Streamlining the Width of Popular CNNs
To verify that setting the middle layer to be the widest is better than setting the last layer to be the widest, we adjust the width of each layer for widely used VGG and ResNet networks. For both VGG and ResNet, the width starts at 64 and ends at 512. We increase the width of the layers showing the DS pattern and decrease the width of the last several layers showing the IS pattern. For a fair comparison, we control the number of FLOPs of the width-adjusted model to be similar to the original model. As result of the larger feature map in the middle compared to the last several layers, the number of parameters is reduced by nearly half. We train these models on CIFAR-10. 10, CIFAR-100, 100, and TinyImageNet. Each model was trained for 120 epochs with SGD. The learning rate is 0.1 at the beginning and is reduced by 0.1 every 40 epochs. The weight decay is set at le- - 4, and momentum is 0.9. For more details about the training recipe and the width- adjuuted architecture please refer Appendix B
Model on TinyImageNet
Validation accuracy
origin
49.78 0.92
VGG16
streamline width
54.40 0.38
ResNet18
origin
54.46 0.41
streamline width
56.56 + 0.27
Table 4: The result of VGGI6 and ResNet18 trained on Tiny-ImageNet. Each result is averaged over 10 runs. Each model is trained for 90 epochs. More details are in Appendix B
As shown in Table 3 and Table we report the experimental results of our adjusted version of VGG and ResNet on CIFAR-10 10, CIFAR- 100, and Tiny- ImageNet. Each result is averaged over 10 runs.

---

The results show that with similar FLOPs, by adjusting the width across the layer, we could boost the performance and largely reduce the parameters.
Conclusion
In this paper, we propose to investigate the weight norm variance among channels in the same layer and identify two different patterns between narrow and wide layers with much empirical evidence (Sec. 3) We further show that training neural networks with the identified pattern could be divided into three stages from random initialization to convergence. We provide our explanations of the three stages with corresponding empirical evidence (Sec. 4 Using the identified pattern
indicator, in Sec. [ we propose to adjust the conventional layer width setting of VGG and ResNet to reduce the number of parameters and boost the performance. The main limitation of the width adjustment is that we manually adjust the layer width to control the FLOPs of the adjusted model. Another limitation is that we can not determine which pattern (IS or DS) the layer follows until convergence. A more automatic and efficient width adjustment method could be proposed regarding the new weight norm variance perspective, which we leave for future works. In assessing the potential broader impact, this work provides a new perspective to evaluate the layer width setting of deep neural networks, which have the potential to advance the neural network architecture design. This work has significant negative potential impact.
References
[1] Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. CoRR, abs/1711.08855 2017.
[2] Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data representations deep neural networks. NeurIPS, pages 6109-6119, 19, 2019.
[3] Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computa- tion in neural networks for faster models. CoRR, abs/1511 2015.
[4] Vaggos Chatziafratis, Sai Ganesh Nagarajan, and Ioannis Panageas. Better depth-width trade- offs for neural networks through the lens of dynamical systems. International Conference on Machine Learning, pages 1478-478. PMLR, 2020.
[5] Yiting Chen, Zhanpeng Zhou, and Junchi Yan. Going beyond neural network feature simi- larity: The network feature complexity and interpretation using category theory. CoRR, abs/2310.06756, 2023.
[6] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555, 2014.
[7] Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent. CoRR, abs/2303.06177 2023.
[8] Benoit Dherin, Michael Munn, Mihaela Rosca, and David Barrett. Why neural networks find simple solutions: The many regularizers of geometric complexity. In NewrIPS, 2022.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[10] David Eigen, Mare' Aurelio Ranzato, and Πlya Sutskever. Learning factored representations in deep mixture of experts. In ICLR (Workshop Poster), 2014.
[11] Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In ICLR. OpenReview.net, 2020.
[12] Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert D. Mullins, and Cheng-Zhong Xu. Dynamic channel pruning: Feature boosting and suppression. In ICLR (Poster). OpenReview.net, 2019.
10

---

[13] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 44(11):7436-7435 7456, 2022.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image the recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, 1bw0/704..448111 2017.
[16] Gao Huang. Shichen Liu, Laurens der Maaten, and Kilian Q. Weinberger. Condensenet: An efficient densenet using learned group convolutions. In CVPR, pages 2752-2761. Computer Vision Foundation IEEE Computer Society, 2018.
[17] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. αrXiv preprint arXiv: (609,00900000000 2016.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[19] Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In CVPR, pages 8607-8617. Computer Vision Foundation / IEEE, 2021.
[20] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In NIPS, pages 6231-6233 2017.
[21] Tao Luo, Zheng Ma, Zhi-Qin John Xu, and Yaoyu Zhang. Theory of the frequency principle for general deep neural networks. CoRR, abs/1906.09253 09235, 2019.
[22] Matt Mahoney. Large text compression benchmark, 2011.
[23] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Hyya Sutskever. Deep double descent: Where bigger models and more data hurt. In ICLR. OpenReview.net, 2020.
[24] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks. CoRR, hass/1805.1207,, 2018
[25] Mike Nguyen and Nicole Mücke. How many neurons do need? A refined analysis for shallow networks trained with gradient descent. CoRR, 0bb//2339008044, 2023.
[26] Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. CoRR, 035/22010021777 2022.
[27] Nasim Rahaman, Devansh Arpit, Aristide Baratin, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of deep neural networks. CoRR, abs/1806.08734, 2018.
[28] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Aging evolution for image classifier architecture search. In AAAI conference on artificial intelligence, volume 2, page 2019.
[29] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556,.59..,,,,,,. 1556, 2014.
[30] Abhishek Sinha, Aahitagni Mukherjee, Mausoom Sarkar, and Balaji Krishnamurthy. Introspec- tion: Accelerating neural network training by learning weight evolution. In 5th International Conference on Learning Representations, ICLR 20/7, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.

---

[31] Pengyu Song, Chunhui Zhao, and Biao Huang. Sfnet: A slow feature extraction network for parallel linear and nonlinear dynamic process monitoring. Neurocomputing, 488:359-380, 2022.
[32] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. High-frequency component helps explain the generalization of convolutional neural networks. In CVPR, pages 8681-8691. Computer Vision Foundation/ / IEEE, 2020.
[33] Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, pages 2074-2082, 2016.
[34] Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural network in frequency domain. In ICONIP (1), volume 11953 of Lecture Notes in Computer Science, Springer, 2019.
pages
[35] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning, pages 40-48. PMLR, 2016.
[36] Jianbo Ye, Xin Lu, Zhe Lin, and James Z. Wang. Rethinking the maaller-nomm-ess--nfommaiii assumption in channel pruning of convolution layers. ICLR (Poster). OpenReview.net, 2018.
[37] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC. BMVA Press, 2016.
[38] Shaofeng Zhang. Meng Liu, Junchi Yan, Hengrui Zhang, Lingxiao Huang. Xiaokang Yang. and Pinyan Lu. M-mix: Generating hard negatives via multi-sample mixing for contrastive learning. In SIGKDD, 2022.
[39] Shaofeng Zhang, Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao, Hongyang Li, and Xiaokang Yang. Align representations with base: A new approach to self-supervised learning. In CVPR, 2022.
[40] Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2022.
[41] Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper LLM networks are fairer LLM evaluators. CoRR, abs/2308.01862. 2023.
[42] Xiaoling Zhou and Ou Wu. Which samples should be learned first: Easy or hard? CoRR, 2021.
[43] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for scalable image recognition. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA. June 18-22, 2018, pages 6697-8710. Computer Vision Foundation / IEEE Computer Society, 2018
12

---

A More Results
A.1 The Weight Norm Variance of Each Layer During Training
In this section, we present the weight norm variance Var(I|w]l ||w(+ of each layer of VGG16, ResNet18 during training.
(a) layer
1, layer
(,(c) layer
(d) layer
4.(c) layer
5,
VGG16
VGG16
VGGI6
VGG16
VGG16
(1) layer
(g) layer
7. layer
8,() layer
9.() layer
10.
VGG16
VGG16
VGG16
VGG16
VGG16
(k) layer
11,())
layer
(m) layer
13,
VGG16
VGG16
VGG16
Figure 9: The weight norm variance between neurons at each layer of VGG16 during its training on CIFAR10.
(a) layer
(b) Layer 3
(c) Layer 7
(d) Layer 11
(e) Layer 17
Figure 10: The results of weight variance for ResNet18.
A.2
The weight Norm Distribution of Each Layer
In this section, provide the distribution of ||]]l ||w|+1|| each layer during training. As shown in Fig. [1 the layers following the IS pattern and DS pattern show different norm distributions.
A.3 Cosine Similarity between Neurons
We provide the cosine similarity between the weight vectors corresponding to different neurons at the same layer in Fig. 12
A.4 Results with small ResNet-20
We report weight norm variance change during training ResNet-20 and widened ResNet-20 following WRN. ResNet-20 has three blocks, with each block containing 6 layers. Since ResNet-20 a relatively small network when trained on CIFAR10, the first 15 layers follow the DS pattern, indicating the layers require more width, and only the last layers show the IS pattern.
13

---

(a) layer
1.(b) layer
3,(c) layer
VGG16
VGGI6
VGG16
(d)
layer
7,(e)
layer
9,
VGG16
VGG16
Figure 11: The weight norm distribution of each layer of VGG16 trained on CIFARIO. The number on the left indicates the epoch.
(a) layer
1.(b) layer
(.(c) layer
(d) layer
4. layer
5,
VGG16
VGG16
VGG16
VGG16
VGG16
(f) layer
(g) layer
7.(h) layer
(,) layer
9.() layer
10,
VGG16
VGG16
VGG16
VGG16
VGG16
(k) layer
11.())
layer
(m) layer
13,
VGG16
VGGI6
VGG16
Figure 12: The cosine similarity of weight vectors corresponding to different neurons at each layer of VGG16 trained on CIFARIO.
(a) layer 1
(b) Layer 4
(c) Layer 8
(d) Layer 15
(c) Layer 16
Figure 13: The results of weight variance for ResNet20.
14

---

(a) layer 1
(b) Layer 4
(c) Layer
(d) Layer 15
(e) Layer 16
Figure 14: The results of weight variance for ResNet20x4.
(a) layer
(b) Layer 4
(c) Layer
(d) Layer 15
(e) Layer
Figure 15: The results weight variance for ResNet20x8.
B Experimental Details
B.1 Details of Network Structure after Our Width-wise Streamlining Technique
In this section, we report the width of each model after applying our width-wise streamlining technique. We change the width of each layer of VGG and ResNet models. The width of each layer is manually set to keep the FL.OPs similar to the original model. Table shows the detailed width of each layer VGG16, VGG19, ResNet18, and ResNet50.
agaazat
1S6, I2E
TIA 2EL
+gazat vid8
ISL DA DO
KE with
Nocveb3 Inu stdda
vice
e4
11e 130 1E--11 IO0 Itbs 196,
1224
Modlectitat MiMacls
aijanalii wide
1004 331 334 12b a04 264 344 10t
Jah not 1004 255,
1224
1004 256 256, 1004
NoW 1n2 rob0
Table 5: The original width and the adjusted width with our width wise streamlining technique for
each layer in VGG and ResNet Models.
B.2 Training Details
For all the models trained in this paper, train each model with SGD, where the learning rate is firstly set at 0.1, momentum is set at 0.9 and weight decay is set at 0.0001. For models trained on CIFAR datasets, the model is trained for 160 epochs, and the learning rate decays 80-th epoch and 120-th epoch by 0.1. For models trained on TinyImageNet, the model is trained for 90 epochs and the learning rate is decayed at the 30-th and 60-th epoch by 0.1. Note that the results reported in Table 3 are averaged 10 runs where we train each model for 120 epochs and the learning rate is decayed the 40-th and 80-th epoch by 0.1. For the proposed width streamlining method, we mainly conduct experiments with supervised learning, we hope our method could further expand used in
more scenarios such as seff-super learning 40 39 38
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have summarize the main contribution of this paper in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitation the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

The theoretical results in this paper is simple analysis over the backpropaga- tion of each channel, where the formulas are caff-exllnnatoy.. There is no other theorems or theoretical assumptions in this paper.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have described the details of every experiments in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The source code will be made publically available upon acceptance. Since the time is tight for us to prepare the paper, we need more time to refactor our code for public release.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
sooolioicciiiiiiiiiooooooooiiiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the full details in Appendix
B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars in results shown in Table and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.coo
Answer: [Yes]
Justification: The research is conducted the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential broader impact in the conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code packages datasets.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "0WitiakwhK", "reviewer": "Reviewer_Q6z7", "summary": "This paper investigates the relationship between the differences in weight norms across channels and the adequacy of layer widths. The authors suggest that knowing these patterns(IS/DS) can help set layer widths better, leading to better resource use, fewer parameters, and improved performance in different network designs. The experiment shows that narrow-wide-narrow streamline network would boost the performance.", "questions": "1. The experiment includes both ResNet and VGG16. How should we define the weight norm of convolutional layers? Chapter 3 provides a simple analysis of the Matthew effect between similar channels. Is there any theoretical analysis for convolutional layers?\n2. DS patterns suggest less similar neurons, indicating we should increase the width. IS patterns suggest redundant neurons, meaning we can decrease the width. In practice, why don't we adjust the width exactly according to the IS/DS patterns? For example, in the appendix, Figure 9 shows IS/DS patterns for 13 layers of the VGG network. Layers 2, 3, 4, 5, 6, and 7 show DS patterns, so we should increase the width of these layers. However, in Table 13, only the widths of layers 2, 4, and 7 are increased. Would adjusting all the layers accordingly result in better performance?\n3. How do we determine the exact width of a layer based on the DS pattern? Is there an empirical boundary for the width of layers? I noticed that in the third stage of weight norm variance, DS patterns exhibit different drop ratios. For instance, in Figure 9, layers 2, 3, and 7 drop to a higher level, while layers 4, 5, and 6 drop to a lower level. Does this indicate that we should increase the width more for layers 4, 5, and 6?\n4. In table3, after adjusting to streamline width, all networks would have a higher FLOPs than origin. Would the performance improvement due the FLOPs increase?", "limitations": "No apparent limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: Yes, this paper is the first work that studies the width of network from the perspective of variance of weight norm. \n\nQuality. Good, clear figures and enough experiments.\n\nClarity. Good, the organization of the paper is well\n\nSignificance. Yes, it provide a new metric to adjust the width of the network.", "weaknesses": "To determine the appropriate width, your method should involve training the model and observing various patterns of weight norm variance to see if the width is sufficient. However, retraining the model to decide on the width can be time-consuming. Considering that IS/DS indicate whether layers learn similar neurons, could we use the cosine similarity of a pretrained model to directly assess if the width is sufficient?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Q6z7, \n\nThank you for your valuable feedback. We address each of your concerns and questions in the following. \n\n\n> W1: Retraining is time-consuming. Considering that IS/DS indicates whether layers learn similar neurons, could we use the cosine similarity of a pre-trained model to directly assess if the width is sufficient?\n\nYes, for pre-trained models, using cosine similarity could be a more efficient way to assess if the width is sufficient. However, when it comes to inspecting a model on a new dataset, using weight norm variance indicates whether the layer width is sufficient way before convergence.\n\n> Q1: The experiment includes both ResNet and VGG16. How should we define the weight norm of convolutional layers? Chapter 3 provides a simple analysis of the Matthew effect between similar channels. Is there any theoretical analysis for convolutional layers?\n\nThe difference between a convolutional layer and a linear layer is that the scalar multiplication in a linear layer becomes a convolution with the kernel as a matrix. Like in a linear layer, the larger kernel produces a larger output. In practice, for the weight norm, we reshape the kernel matrix into a vector and use the l2 norm of the vector to further calculate the weight norm of each channel. Thanks for the good question, we will add more analysis in our paper and clarify it.\n\n> Q2: DS patterns suggest less similar neurons, indicating we should increase the width. IS patterns suggest redundant neurons, meaning we can decrease the width. In practice, why don't we adjust the width exactly according to the IS/DS patterns? \n\nYes, we can adjust the width exactly according to the pattern. In the paper, we manually adjust the layer width, and we want to make the model into a streamlined shape while the layer width is neat. To answer the question, we have made a VGG16_muted version 2 and tested it. The result is in the response to Q4. \n\n> Q3: How do we determine the exact width of a layer based on the DS pattern? Is there an empirical boundary for the width of layers? I noticed that in the third stage of weight norm variance, DS patterns exhibit different drop ratios. For instance, in Figure 9, layers 2, 3, and 7 drop to a higher level, while layers 4, 5, and 6 drop to a lower level. Does this indicate that we should increase the width more for layers 4, 5, and 6?\n\nYou are right. According to results in Sec. 3.2, the DS pattern exhibits different drop ratios. As we increase the width, the pattern moves from DS to IS. The perfect scenario would be finding the smallest width leading to an IS pattern.\n\n> Q4: In Table 3, after adjusting to streamline width, all networks would have higher FLOPs than the origin. Would the performance improvement due to the FLOPs increase?\n\nSince the shallow layers take much more calculation than the deep layers in CNN, for the streamlined networks, the increase in FLOPs is not significant compared to the decrease in the number of parameters. \nWe have also made a new version of VGG16_muted with lower FLOPs than the original VGG16 (99.68%).\n\n| model           | FLOPs (%) | params (%) | CIFAR10 accuracy | CIFAR100 accuracy |\n| --------------- | --------- | ---------- | ---------------- | ----------------- |\n| VGG16           | 100%      | 100%       | 94.03$\\pm$ 0.09  | 71.25$\\pm$ 0.25   |\n| VGG16 steamline | 103.89%   | 61.55%     | __94.19$\\pm$ 0.10__  | __71.90$\\pm$ 0.23__   |\n| VGG16 streamline v2 |99.68%|   58.53%    |     94.16$\\pm$ 0.11|71.68$\\pm$ 0.29|\n\nFor this experiment, we add normalization and random rotation in the preprocess of training data as suggested by reviewer iwdV. We will provide more experimental results and add them to our paper. Generally, VGG16 streamlined version 2 reduces both the FLOPs and the number of parameters while gaining performance increases.\n\nWe hope our rebuttal addresses your concerns, and we look forward to your message. Please feel free to make any comments; we are more than happy to respond to any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "34SEJjs4FI", "reviewer": "Reviewer_iwdV", "summary": "This paper tries to address an issue in deep neural networks: the trade-off between computational cost and performance, particularly focusing on the width of each layer. Traditionally, layer widths in neural networks are determined empirically or through extensive searches. The authors propose a novel approach by examining the variance of weight norms across different channels to determine if a layer is sufficiently wide.", "questions": "Please see the weakness.", "limitations": "Addressed in conclusion.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors identify patterns regarding the variance of weight norm between different channels during training. Some layers exhibit an \"increase to saturate\" (IS) pattern and other layers show a \"decrease to saturate\" (DS) pattern. These patterns have some connection regarding the inter-channel similarities given a layer.\n2. They redesign the width of classical CNN models like ResNets and VGGs according to their findings. And the empirical results provide some support for their arguments.", "weaknesses": "1. The empirical justification of the proposed method is somehow not convincing enough. From Table 2, we can see that channels within layers with IS patterns are also hard to merge, like layers 8 to 11 in Table 2. From Figure 12 in the Appendix, we can see that the similarity gradually decreased first and then increased for middle layers with DS patterns, and it is hard to say there is a hard cutoff.\n2. Suppose the proposed argument is correct, then it seems challenging to use this method in practice. The width of each layer depends on the IS or DS pattern of the original model, so you need to train the original model first and then the model with the streamline width. This always increases the total training cost especially when scaling up the model. \n3. The authors claim that they have similar observations for ViTs in lines 58-62, but no results are given. Since the original ViT has a uniform width, I am wondering whether the arguments still hold for models with uniform width.\n4. The experiment setup is not very convincing. For CIFAR-10/100 scales of datasets, ResNet-18 or ResNet-50 are large and they have many redundant parameters. In the original ResNet paper, they used different architectures for CIFAR datasets with a much smaller number of parameters (0.27M to 1.7M with ResNet-20 to ResNet-110), please see Table 6 in \"Deep Residual Learning for Image Recognition\". The performance of ResNets and VGGs is much lower than public baselines for CIFAR-10 and CIFAR-100. Please see these two repo: https://github.com/weiaicunzai/pytorch-cifar100 and https://github.com/kuangliu/pytorch-cifar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer iwdV, \n\nThank you for your valuable feedback. We address each of your concerns in the following.\n\n> W1: The empirical justification of the proposed method is somehow not convincing enough. It is hard to say there is a hard cutoff.\n\nWith all due respect, our results indicate that there is no hard cutoff between the DS pattern and the IS pattern. As shown in Sec. 3.2, as the layer width increases, the pattern gradually changes from the DS pattern to the IS pattern. Similar to Fig. 9 and Fig. 12 in the Appendix, the pattern and the similarity gradually change from IS to DS and from DS to IS again. In our hypothesis, the perfect width would be that the pattern is at the edge between the DS and IS pattern.\n\n> W2: Suppose the proposed argument is correct, then it seems challenging to use this method in practice.\n\nGenerally, whether the weight norm variance follows the IS or DS pattern is shown way before convergence, which makes a more efficient method possible.  \nStill, we agree that finding a more efficient method is challenging. However, the community has witnessed the development of more efficient methods for finding lottery tickets [1], where retraining is also required in the first paper. We hope our findings in this paper could innovate future works in adjusting layer width more efficiently.\n\n> W3: The authors claim that they have similar observations for ViTs in lines 58-62, but no results are given.\n\nSorry for the confusion. The results regarding ViTs are presented in Sec. 3.2 and Fig. 3 [lines 153- 167]. As shown in Fig. 3, as we change the width of the MLP in a small ViT, the narrower MLP show a DS pattern, while the wider MLP shows an IS pattern. This phenomenon aligns with our observations on GCN, GRU, and CNNs, in that wider layers show the IS pattern while narrow layers show the DS pattern.\n\n> W4: In the original ResNet paper, they used different architectures for CIFAR datasets with a much smaller number of parameters (0.27M to 1.7M with ResNet-20 to ResNet-110).\n\nWe conduct experiments with ResNet-20 on CIFAR10.  Since it is a much smaller network, we show that the layers in the first two blocks follow the DS pattern, and only the layers in the third block follow the IS pattern. By widening the small ResNet-20 by 8 times, we show that the layers in the first block and the third block show the IS pattern, and the layers in the second block show the DS pattern.\n\nSince these results are in figures, they are in the PDF attached to the joint rebuttal.\n\n> W5: The performance of ResNets and VGGs is much lower than public baselines for CIFAR-10 and CIFAR-100.\n\nThe main difference between our implementation and the two mentioned repos is that we only use random flip in the training data preprocess while the two repos use normalization, and the repo for CIFAR100 uses random rotation in the data preprocessing.\n\nWe report the results using the code of the two mentioned repos:\n\n**CIFAR10**\n| Model               | accuracy       |\n| ------------------- | -------------- |\n| VGG16 | 94.03$\\pm$ 0.09|\n| VGG16 streamline    | __94.19$\\pm$ 0.10__ |\n|ResNet18| 95.12$\\pm$ 0.12|\n| ResNet18 streamline | __95.60$\\pm$ 0.09__ |\n**CIFAR100**\n\n| Model | accuracy  |\n| ------------------- | --------------- |\n| VGG16 | 71.25$\\pm$ 0.25|\n| VGG16 streamline    | __71.90$\\pm$ 0.23__ |\n|ResNet18 | 74.33 $\\pm$ 0.21|\n| ResNet18 streamline | __74.94$\\pm$ 0.23__ |\n\nEach result is averaged over $5$ runs. We report the mean and std of the accuracy. \n\nWe hope our rebuttal addresses your concerns, and we look forward to your reply. Thanks for your time and effort during the reviewing process. We are more than happy to answer any further questions.\n\n[1] Frankle, Jonathan, and Michael Carbin. \"The lottery ticket hypothesis: Finding sparse, trainable neural networks.\" ICLR 2019", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qSunMzNMV3", "reviewer": "Reviewer_nV2M", "summary": "The paper proposes a method to optimize neural network layer width by analyzing the variance of weight norms across channels during training. This approach helps determine if a layer is sufficiently wide. Empirical validation shows that adjusting layer widths based on these patterns can reduce parameters and improve performance across various models and datasets.", "questions": "See Weaknesses.", "limitations": "The limitations have been adequately discussed in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper proposes a novel way to assess layer width sufficiency using weight norm variance.\n\n2. The paper shows robust experimental evidence across multiple datasets and model architectures.\n\n3. Besides the application on the model width optimization, the paper also offers a deeper understanding of training dynamics through the identification of distinct patterns in weight norm variance, which could be valuable for other related areas.", "weaknesses": "1. The layer width optimization is related to channel pruning and NAS-based channel number search, which have achieved significant success in finding optimal layer widths. I think these methods should be discussed and compared in the paper.\n\n2. Measuring the weight norm would introduce additional computation cost, can the authors discuss the complexity of the method and report how much additional time does the method introduce?\n\n3. Can the authors provide more detailed and theoretical analysis on the choice of the metric? For example, there are other metrics such as gradient norm, Hessian matrix, and absolute weight value to evaluate the weight importance and training statistics, why the proposed metric is better?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer nv2M, \n\nThank you for your valuable feedback. We will address each of your concerns and questions in the following section.\n\n> W1: The layer width optimization is related to channel pruning and NAS-based channel number search, which have achieved significant success in finding optimal layer widths. I think these methods should be discussed and compared in the paper\n\nThanks for the suggestion. We will add more discussion and comparisons to our paper.  Generally, our approach differs from the two kinds of methods mentioned above. \n* For channel pruning, the objective is to reduce the computational cost, which differs from the objective of this paper. Notably, most pruning methods would manually set a pruning ratio for each layer.\n* For NAS-based search, the model width is searched over a model structure space. However, in this paper, we want to provide a more principled indicator for whether one layer in a network is sufficiently wide.\n\n> W2: Measuring the weight norm would introduce additional computation costs. Can the authors discuss the complexity of the method and report how much additional time the method introduces?\n\nThe computation cost for measuring the weight norm is less than that of inferencing once with the model. We report the time used for measuring the weight norm on the CPU (AMD EPYC 7302 16-Core Processor). \n| model    | time used(s)       |\n| -------- | ------------------ |\n| VGG16    | 0.4580$\\pm$ 0.3560 |\n| ResNet18 | 0.3150$\\pm$ 0.2433 |\n\nThe results are averaged over 10 runs, and we report the mean and the std of the results. Therefore, the time required to measure the weight norm is neglectable. We will add a more comprehensive computational cost analysis in our paper.\n\n> W3: Can the authors provide a more detailed and theoretical analysis of the choice of the metric? For example, there are other metrics such as gradient norm, Hessian matrix, and absolute weight value to evaluate the weight importance and training statistics. Why is the proposed metric better?\n\nSince we are focusing on the channels in one layer, the output of each channel is combined (mostly added up) and becomes the layer's output. For one channel, the weight norm generally determines how influential the channels are to the layer's output, whereas a larger weight norm produces a larger output. \n\nFor other metrics, we have shown in Sec. 3.1 that the gradient norm is positively correlated to the weight norm, which also requires backward propagation on certain losses to be measured. While absolute weight value and Hessian matrix do not fit our purpose.\n\nWe hope our rebuttal could address your concerns. We are looking forward to your reply. Thanks for your time and effort during the reviewing process. We are more than happy to answer any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yaIoCxzHzx", "reviewer": "Reviewer_nGxF", "summary": "The paper studies the effects of layer widths on neural network performance. By studying the affects of the weight norm in different channels, the work discovers several distinct stages of training, which are apparent across different modalities and architectures. The authors also show how the insights could help in improving network performance with the same computational costs.", "questions": "Some possible interesting references which could be relevant:\n\n1. https://proceedings.neurips.cc/paper_files/paper/2023/hash/b63ad8c24354b0e5bcb7aea16490beab-Abstract-Conference.html", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper has great merit, and discovers interesting dynamics in neural network training. Further, the dynamics are consistent, and come from an intuitive intuition of NN training. It is exhibited across different architectures and datasets, with an ability to also improve networks using these insights.", "weaknesses": "The paper is really great! It could benefit from more experiments and archs, but the current experiments are convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer nGxF,\n\nThank you very much for your thoughtful and valuable feedback. Your recognition of the intuitiveness of our findings and the potential for our insights to further improve neural networks is very encouraging and motivating for us. We are committed to further advancing this line of research by conducting more experiments and arches.\n\n> Some possible interesting references which could be relevant.\n\nThank you for introducing the references, which are indeed interesting in providing insights into the representations learned in self-supervised learning.\n\nOnce again, thank you for your valuable comments and support. We are more than happy to respond to any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear AC and Reviewers,

We sincerely thank you for the time and effort you dedicated to the reviewing process. We are delighted to hear that reviewers find the paper to be well-written (Q6z7, nV2M, nGxF), novel (Q6z7, nV2M), and interesting (nGxF). To further address the comments and questions posed by reviewers, we have also conducted additional experiments as required by the reviewers, including:
* Measuring the time required to calculate the weight norm.
* Train the networks with normalization added to the preprocessing of training data.
* Conduct experiments with smaller ResNet-20. (**The results in figures are provided in the attached PDF**)

For each reviewer, we have posed a rebuttal addressing the concerns. We look forward to your reply and are more than happy to respond to any further comments. Once again, thank you for your valuable comments and support.

Meta-review: The paper proposes a method to optimize neural network layer width by analyzing the variance of weight norms across channels during training. This approach helps determine if a layer is sufficiently wide. Empirical validation shows that adjusting layer widths based on these patterns can reduce parameters and improve performance across various models and datasets.

Most reviewers find the paper is well-written, and the results are interesting.
Please make sure to incorporate the feedback in the camera-ready version, especially addressing the reviewers' concerns on the experimental setup and more comprehensive studies on larger datasets.
Decision: Accept (poster)