Unveiling The Matthew Effect Across Channels: Assessing Layer Width Sufficiency via Weight Norm Variance

Authors: yiting  chen
jiazi  bu
junchi  yan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 12:17:00

Abstract: The trade-off between cost and performance has been a longstanding and critical issue for deep neural networks. 
One key factor affecting the computational cost is the width of each layer. 
However, in practice, the width of layers in a neural network is mostly empirically determined. In this paper, we show that a pattern regarding the variance of weight norm corresponding to different channels can indicate whether the layer is sufficiently wide and may help us better allocate computational resources across the layers.
Starting from a simple intuition that channels with larger weights would have larger gradients and the difference in weight norm enlarges between channels with similar weight, we empirically validate that wide and narrow layers show two different patterns with experiments across different data modalities and network architectures. 
Based on the two different patterns, we identify three stages during training and explain each stage with corresponding evidence. We further propose to adjust the width based on the identified pattern and show that conventional layer width settings for CNNs could be adjusted to reduce the number of parameters while boosting the performance.

Main paper: 
 introduction: 
 The cost-accuracy trade-off has been a longstanding and critical issue for deep neural networks. As one key factor that affects the computational cost, the width of each layer is mostly empirically determined or extensively searched over an architecture space in neural network architecture search literature [43, 28] . Despite the research on over-parameterization [24, 23] and empirical evidence [37, 15, 31, 41] showing that the wider network leads to better performance, there are few works about how we allocate the computational resources across the layers. On understanding the layer width requirement in a deep neural network, [20] theoretically proves the lower bound of layer width for the model to approximate any Lebesgue-integrable function while [4] improves the width lower bound with dynamic systems. However, a practical indicator of sufficient layer width is still lacking.In this paper, we identify a simple pattern difference between wide and narrow layers regarding the weight norm variance between different channels during training. A simple observation is that largerweight channels would have a larger gradient. As the weight norm increases during training [30, 11] , we show the disparity of weight norm between similar channels also increases during training, which we call the Matthew effect between channels. It motivates us to investigate the weight norm variance across the channels in each layer during training. With experiments on image, graph, and text datasets (including CIFAR-10 [18] , cora [35] , and enwiki8 [22] ) with various model structures including (GCN [17] , GRU [6] , ViT [9] ), we show that wide and narrow layers show two different patterns during training. As we change the layer width, we show that narrow layers show a decrease until the saturate (DS) pattern, such that the weight norm variance first increases and then decreases. For wider layers, the weight norm variance keeps increasing until saturate (IS). We conjecture that, for sufficiently wide (or over-parameterized) layers, channels with a small weight norm would always be surpassed by similar channels with larger weights, and therefore, the variance keeps increasing until convergence. On the other hand, for narrow layers with few channels, the channels with small weights could be orthogonal to channels with large weights, and the variance decreases.We further show that the training from random initialization to convergence can be divided into three stages for neural networks showing the identified IS or DS pattern. In the first stage, typically a few epochs after random initialization, since the gradient is almost orthogonal to the weight vector, the variance between weight norms corresponding to different neurons does not increase or even decrease. In the second stage, both the performance of the network and the variance between neurons drastically increase for a dozen epochs. It indicates a fast learning process with the weight norm of a few neurons increasing drastically. In the third stage, the wide layers show the IS pattern, such that the weight norm variance keeps increasing and staying high, while the narrow layers show the DS pattern, such that the weight norm variance starts to decrease. We provide our explanations for each stage of training with corresponding empirical evidence on VGGs [29] trained on CIFAR10. Furthermore, we show that the conventional layer width setting for CNNs such as VGG or ResNet might not be optimal. Generally, with the conventional layer width setting, the layers in the middle show the DS pattern while the former and latter layers show the IS pattern. It indicates that the layers in the middle could use more width, which also cohere with the intrinsic dimension firstly increasing then decreasing and reaching the high point at the middle [2] . We propose adjusting the width of each layer of widely used networks to boost the performance and reduce the number of parameters. We summarize the contributions of this paper in the following:• To our best knowledge, we are the first to identify patterns regarding the variance of weight norm between different channels during training that can indicate whether a layer is wide enough. We have observed the pattern with different model structures on different data modalities (including CNNs and ViTs for image classification [29, 14, 9] , GCN for graph node classification [17] , and GRU for language modeling [6] ). • Based on the identified pattern, we further show that there are, in general, three training stages from random initialization to convergence. We provide our explanations with empirical evidence on VGG models trained for image classification that supports our explanation. • We verify that with the identified pattern, the conventional width setting for CNNs could be adjusted for fewer parameters and better performance on CIFAR10 and TinyImageNet. 
related works: 
 Phenomenon Description Measured by The critical period in training [1] Providing corrupted data in the early training phase leads to irrevocable damage to final performance Testing accuracy after convergence.Frequencyprinciple [27, 34] Neural networks first learn low-frequency information and then high-frequency information.Gradient norm on different frequency components. Grokking [26] Neural networks show a period of near-perfect training performance and nearly random guessing test performance before generalization.Training and testing accuracy.The testing accuracy firstly improves, then worsens, and then eventually improves as the model capacity increases or the training goes on.Testing accuracy.See details in Fig. Researches on the Width of Neural Networks. By simply widening the network, early works [37, 15] empirically show that the performance can be improved. On the other hand, structured pruning methods [33, 36, 16] propose to reduce the width for efficiency. Compared to the static pruning methods, another branch of research focusing on dynamically changing the width of the network conditioned on the input by skipping neurons in linear layer [3] , branches in Mixture of Experts (MoE) [10] and channels in CNN [12, 19] at inference time. Readers interested in dynamic networks could refer to [13] for more details. Notably, a recent work [5] proposes to merge neurons to reduce the computational cost where layers at the middle have fewer neurons that could be merged. Similar results are also reported regarding investigating the intrinsic dimension [2] across the layers. Unlike these previous works focusing on the trained model, we show that one could identify the For wide layers, the weight norm variance keeps increasing, which we call the increase to saturate pattern (IS). For narrow layers, the weight norm starts to decrease after reaching a high point, which we call the decrease to saturate pattern (DS).layers with insufficient width during training. We further propose a simple width-wise streamlining technique to boost the network performance and reduce the number of parameters. Note that the objective of our width-wise streamlining technique differs from the pruning method and we make the FLOPs of the adjusted model close to the original network.Investigating the training process of neural networks has always been an active research topic. The early phase of neural network training is investigated in [11] and a timeline for the early phase of training (the first several epochs) is provided while other works [24, 8, 25] have focused on the later training phase investigating the effect of over-parameterization on the generalization and the training heuristics that lead to simpler solutions. From the frequency spectrum perspective, F-principle [27, 34, 32] shows that low-frequency components play a more important role in the generalization and that the training of CNN could be divided into two phases where the CNN firstly learn low-frequency components then learn high-frequency components. The frequency principle theoretically proves that the gradient on low-frequency components will be larger [21] .In this paper, we analyze the training dynamics from the weight norm variance perspective and show that the training of neural networks could be summarized into three stages in general. To our knowledge, this is the first work to investigate the variance between weight norms corresponding to different neurons. We list some well-known phenomena in Table 1 . While a recent work [7] correlates grokking [26] with double-descent [23] , we hope the relationship between the three stages of training and the other phenomena could be further explored in future works. For example, the difference between the second stage and the third stage could be the result of network learning different frequency components, easy or hard data samples [42] etc. 
different patterns between wide and narrow layers: 
 In this section, we first provide a simple analysis showing the motivation for us to investigate the weight norm variance across the channels of a layer. We then provide empirical results from experiments across different data modalities and model structures to show that there are two different patterns between wide and narrow layers.Let us consider an MLP with ReLU activation for simplicity. Suppose we have d channels in the l-th layer. Let z l-1 ∈ R din and z l+1 ∈ R dout denote the features at the l -1-th layer and the l + 1-th layer. The weight for l-th layer and l + 1-th layer is W l ∈ R din×d and W l+1 ∈ R d×dout . Then we have:EQUATIONwhere σ(•) is the ReLU activation function. Let w l i ∈ R din denotes the i-th row vector of W l and w l+1 i ∈ R dout denotes the i-th column vector of W l+1 , the output corresponding to the i-th channel is σ(w l⊤ i z l-1 ) • w l+1 i and the output z l+1 is the combination of the outputs of all the d channels asz l+1 = d i=1 σ(w l⊤ i z l-1 ) • w l+1 i (2)Suppose the loss function is L, and the gradient for z l+1 is ∇ z l+1 L, then we haveEQUATIONEQUATIONFor two channels m, n ∈ [1, d], supposew l+1 m ∥w l+1 m ∥ ≈ w l+1 n ∥w l+1n ∥ andw l m ∥w l m ∥ ≈ w l n ∥w ln ∥ then w l⊤ m z l-1 and w l⊤ n z l-1 would have the same sign and we haveEQUATIONEQUATIONIt means that for channels with weights of similar direction, the larger ∥w l ∥ leads to larger ∇ w l+1 L and the larger ∥w l+1 ∥ leads to larger ∇ w l L. While empirical evidence in previous works [30, 11] has shown that the weight norm increases during training. It means that, for two channels with similar weight direction, the weight norm disparity between them would generally keep increasing during training, which we call the Matthew effect between similar channels. For the channel with a larger weight, the larger weight norm at the l + 1 layer leads to a larger gradient of the corresponding weight at the l layer, which increases the weight norm at the l layer. In turn, a larger weight norm at the l layer further speeds up the increase of the weight norm at the l + 1 layer. According to [5] , sufficiently wide layers may learn many similar channels after training, therefore, we expect the variance of weight norm across different channels would be higher in the wide layer than in the narrow layer. It motivates us to investigate further the weight norm variance between different channels in a layer.In this subsection, we investigate the variance of weight norms corresponding to different channels.For the i-th channel in Eq. 2, with ReLU as activation functions we haveEQUATIONTherefore, we define the weight norm corresponding to the i-th channel as ∥w l i ∥ • ∥w l+1 i ∥ 1 . We conduct our experiments on several datasets across different data modalities with different model architectures and record the variance of corresponding weight norms during training.We report the weight norm variance results for Graph Convolutional Neural Network (GCN) [17] trained on graph dataset cora [35] , Gated Recurrent Units (GRU) [6] trained on text dataset en-wiki8 [22] and Vision Transformers (ViT) [9] trained on image dataset CIFAR10 [18] . As we change the width of a certain layer (e.g. the MLP module in the transformer), we show that the wide layer and the narrow layer show two different patterns. For the wide layer, the weight norm variance firstly increases and stays at a high level until convergence, which we call the increase to saturate (IS) pattern. For the narrow layer, the weight norm variance decreases after the initial increase, which we call the decrease to saturate (DS) pattern. [17] on cora [35] and GRU [6] on enwiki8 [22] . We change the width of the second layer in GCN and the input-hidden weight of the second layer in GRU. As the width increases, the weight norm variance during training shows two different patterns. For the wide layer, the weight norm variance increases and stays at a high level until convergence, which we call an increase to saturate (IS) pattern. For the narrow layer, the weight norm variance decreases after the initial increase, which we call the decrease to saturate (DS) pattern. As shown in Fig. 2 , we present the results of GCN [17] on cora [35] and GRU [6] on en-wiki8 [22] . In Fig. 2 Fig. 3 shows the results of two ViTs trained on CIFAR10. We use a ViT with 6 layers where each layer has 8 heads and the hidden size for each attention layer is 512. The width of the MLP at the 5th layer is set at 32 and 512 respectively. For the layer of width 512, the weight norm variance always increases and stays at a high level following the IS pattern. As we reduce the width of the MLP to 32, the weight norm variance starts to decrease after reaching a high point following the DS pattern. For more details on the GCN, GRU, and ViT training, please refer to Appendix B.Our Conjecture About the Different Patterns Regarding Layer Width. As shown in Eq. 2, each layer is like a "mixture of experts" where each channel has the same weight. The only difference between channels is the random initialization. For wide layers with a large number of channels, the chance for two channels learning weight of similar direction is much higher than that for narrow layers with limited channels. Therefore, the weight norm variance among channels of a wide layer does not decrease while the weight norm variance among channels of a narrow layer decreases after reaching a high point. In the next section, we provide a more detailed analysis with empirical evidence about the three training stages we identified.  
three training stages from random initialization to convergence: 
 Based on the two patterns we identified regarding the weight norm variance, we further provide our explanation for this phenomenon. We show that, generally, we can divide the training of the neural networks showing the identified pattern into three stages. We provide our explanation for each stage with corresponding empirical evidence on VGG-16 [29] and ResNet-18 [14] trained on CIFAR10 [18] . As shown in Fig. 4 , we report the weight norm variance of some of the layers in VGG-16 and ResNet-18. In Fig. 5 , we also present the density plots of the weight norm for different layers of a ResNet-18 trained on CIFAR10 at different epochs. Generally, different layers show different patterns. The layers in the middle typically follow the DS pattern and the former and latter layers follow the IS pattern. We will discuss this phenomenon in the next section Sec. 5. Starting from a random initialization, the first stage of neural network training lasts for a few epochs. In this stage, the variance of the weight norm between neurons does not increase. At certain layers, the variance even decreases between neurons. Starting from random initialization, due to the high-dimensional weight vectors of each neuron, the gradient is more likely to be orthogonal to the weight vectors corresponding to different neurons. When measuring the cosine similarity between the gradient and the weight vector across all the channels across all the layers in a VGG16_bn trained on CIFAR-10, the largest absolute value is 0.0003 at initialization, which provides an explanation for the weight variance not increasing.In some scenarios, we also find weight norm variance decreases at the first stage. We conjec- ture that it results from the fact that the direction of weight vectors is drastically changed in the first stage. In fact, the first stage of neural network training is an extremely chaotic stage where the direction of each weight vector is dramatically changed. Fig. 6 shows that, for different networks, the sign of at least 10% weight elements changes after the first 5 epochs. As a result of the dramatic change in weight space, the direction of many weight vectors is reversed and the weight norm firstly decreases and then increases.In the second stage, the variance of weight norm among channels starts to increase drastically. This phenomenon indicates that for some of the neurons, whose corresponding weight vectors are close to the gradient direction, the corresponding weight norm increases much faster than other neurons. At the same time, the performance also increases rapidly as shown in Fig. 7 (a). The test performance after this stage is very close to the final performance, e.g. the testing accuracy for the VGG16 on CIFAR-10 at 20 epoch in Fig. 7 (a) is at 84.65%, which is relatively close to the final testing accuracy 92.94%. We conjecture that to reach a fair performance, the network only needs to learn a few significant features, channels whose weight direction is close to the direction of the gradient would have advantages.The second stage generally determines the relative weight norm after convergence. Compared to the chaotic first stage, the second stage is much more stable, where the weight norm of those neurons corresponding to the "significant features" discussed above constantly increases faster than other neurons. Notably, at the beginning of the second stage, the tendency of how the weight norm would change provides more information than the weight norm itself. As shown in Fig. 7 (b), we demonstrate the weight norm at the beginning of the second stage and the weight norm after convergence. The result is from channels at the first layer of a VGG16 trained on CIFAR-10. The x-axis corresponds to the weight norm after convergence. For the upper subplot, the y-axis corresponds to the difference between the weight norm at the 4-th epoch and the weight norm at the 6-th epoch. For the lower subplot, the y-axis corresponds to the weight norm at the 6-th epoch. The Pearson correlation between the weight norm difference and the final weight norm is 0.8891 while the Pearson correlation between the weight norm at 6-th epoch and the final weight norm is only 0.4254. We conjecture that the channels whose weight norm increases rapidly in this stage correspond to those easy features such as the low-frequency information in Frequency-principle [27, 34, 32] . Figure 8 : The cosine similarity between weight vectors at the same layer of a VGG16 trained on CIFAR-10. The value at i-th row and j-th column correspond to the cosine similarity between the weight vectors of the i-th neuron and the j-th neuron. Note that we sort the neurons by their weight norm in descending order which means the 0-th neuron is the neuron with the largest weight norm.The results show that weights for neurons at the 5-th layer of VGG16 are almost orthogonal.The third stage is the longest stage where the performance slowly increases. In the third stage, layers show two distinct patterns which indicates whether this layer learns similar neurons. For the variance of weight norm between neurons, there are two different patterns across the layers:• Increase to Saturate (IS): The variance continues to increase slowly and stay high.• Decrease to Saturate (DS): The variance decreases after reaching a high value. Table 2 : We employ IFM [5] to merge the neurons (channels) across the layers of a VGG16 trained on CIFAR-10. We list the original width, the width after merging, and the weight norm variance pattern in the third stage.Interestingly, as shown in Table 2 , we find that for layers where neurons (or channels) could be merged using the algorithm proposed in [5] , the variance between neurons follows the IS pattern that increases and stays at a high level. On the other hand, for layers where no neurons could be merged, the variance between neurons follows the DS pattern and decreases. Fig. 8 shows the cosine similarity of weight vectors between different neurons. For layers following the DS pattern, the cosine similarity of weight vectors between most different neurons is almost zero while the cosine similarity between neurons for layers following the IS pattern is much higher. It indicates that the weight vectors of channels in layers showing the DS pattern are nearly orthogonal while layers following the IS pattern contain many similar channels. Note that the neurons are sorted by the weight norm where the 0-th neuron is the neuron with the largest weight norm. Fig. 8 (a) and Fig. 8 (c) show that neurons with different weight norms could be in a similar direction. Further results of the weight norm distribution are presented in Appendix A. 
width modification on cnns for fewer parameters and better performance: 
 As discussed in Sec. 4.3, the layers in VGG-16 show different patterns. In this section, we investigate the weight norm variance change during the training of CNNs. We empirically show that for widely used CNNs such as VGG and ResNet, the middle layers show the DS pattern and other layers show the IS pattern. We then adjust the layer width for VGG and ResNet, reducing the number of parameters while boosting the performance compared to the conventional layer width setting.As shown in Fig. 4 , we report the variance of weight norm ∥w l i ∥ • ∥w l+1 i ∥ at different layers of VGG-16 and ResNet18 trained on CIFAR-10. Generally, we find that the first several layers follow the IS pattern while the layers in the middle follow the DS pattern, and the layers at last follow the IS pattern again. It reflects the intrinsic nature of the neural network training such that layers in the middle learn orthogonal neurons. One may naturally associate this phenomenon with the results that intrinsic dimension firstly increases then decreases and reaches a high point in the middle [2] . Intrinsic dimension describes the minimum dimension needed to solve a certain problem to a certain precision level. We conjecture that intrinsic dimension is a possible explanation for the two different kinds of layers where high intrinsic dimension leads to orthogonal weights. For more results, please refer to Appendix A. It further indicates that the conventional layer width setting for CNNs might not be optimal. In the conventional setting (such as in VGGs and ResNets) the layer width increases through the first several layers and then stays large through the following layers. Since the layers in the middle could use more width and the layers in the front and back could use less width, a better setting might be that the width first increase and then decrease across the layers with the widest layer in the middle just like a streamline. To verify that setting the middle layer to be the widest is better than setting the last layer to be the widest, we adjust the width of each layer for widely used VGG and ResNet networks. For both VGG and ResNet, the width starts at 64 and ends at 512. We increase the width of the layers showing the DS pattern and decrease the width of the last several layers showing the IS pattern. For a fair comparison, we control the number of FLOPs of the width-adjusted model to be similar to the original model. As a result of the larger feature map in the middle compared to the last several layers, the number of parameters is reduced by nearly half. We train these models on CIFAR-10, CIFAR-100, and TinyImageNet. Each model was trained for 120 epochs with SGD. The learning rate is 0.1 at the beginning and is reduced by 0.1 every 40 epochs. The weight decay is set at 1e -4, and momentum is 0.9. For more details about the training recipe and the width-adjusted architecture please refer to Appendix B.As shown in Table 3 and Table 4 , we report the experimental results of our adjusted version of VGG and ResNet on CIFAR-10, CIFAR-100, and Tiny-ImageNet. Each result is averaged over 10 runs.The results show that with similar FLOPs, by adjusting the width across the layer, we could boost the performance and largely reduce the parameters. 
conclusion: 
 In this paper, we propose to investigate the weight norm variance among channels in the same layer and identify two different patterns between narrow and wide layers with much empirical evidence (Sec. 3). We further show that training neural networks with the identified pattern could be divided into three stages from random initialization to convergence. We provide our explanations of the three stages with corresponding empirical evidence (Sec. 4). Using the identified pattern as an indicator, in Sec. 5, we propose to adjust the conventional layer width setting of VGG and ResNet to reduce the number of parameters and boost the performance. The main limitation of the width adjustment is that we manually adjust the layer width to control the FLOPs of the adjusted model. Another limitation is that we can not determine which pattern (IS or DS) the layer follows until convergence. A more automatic and efficient width adjustment method could be proposed regarding the new weight norm variance perspective, which we leave for future works. In assessing the potential broader impact, this work provides a new perspective to evaluate the layer width setting of deep neural networks, which have the potential to advance the neural network architecture design. This work has no significant negative potential impact.A More Results 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the potential broader impact in the conclusion section.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original paper that produced the code packages or datasets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: In this section, we present the weight norm variance V ar(∥w In this section, we provide the distribution of ∥w l i ∥ • ∥w l+1 i ∥ at each layer during training. As shown in Fig. 11 , the layers following the IS pattern and DS pattern show different norm distributions.We provide the cosine similarity between the weight vectors corresponding to different neurons at the same layer in Fig. 12 .We report weight norm variance change during training of ResNet-20 and widened ResNet-20 following WRN. ResNet-20 has three blocks, with each block containing 6 layers. Since ResNet-20 is a relatively small network when it is trained on CIFAR10, the first 15 layers follow the DS pattern, indicating the layers require more width, and only the last 5 layers show the IS pattern. In this section, we report the width of each model after applying our width-wise streamlining technique. We change the width of each layer of VGG and ResNet models. The width of each layer is manually set to keep the FLOPs similar to the original model. For all the models we trained in this paper, we train each model with SGD, where the learning rate is firstly set at 0.1, momentum is set at 0.9 and weight decay is set at 0.0001. For models trained on CIFAR datasets, the model is trained for 160 epochs, and the learning rate decays at 80-th epoch and 120-th epoch by 0.1. For models trained on TinyImageNet, the model is trained for 90 epochs and the learning rate is decayed at the 30-th and 60-th epoch by 0.1. Note that the results reported in Table 3 are averaged 10 runs where we train each model for 120 epochs and the learning rate is decayed at the 40-th and 80-th epoch by 0.1. For the proposed width streamlining method, we mainly conduct experiments with supervised learning, we hope our method could further expand used in more scenarios such as self-supervised learning [40, 39, 38] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The source code will be made publically available upon acceptance. Since the time is tight for us to prepare the paper, we need more time to refactor our code for public release.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We provide information on the computer resources in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research is conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).It is similar for other ReLU-like activation functions such as leaky ReLU. 


Decision: The paper proposes a method to optimize neural network layer width by analyzing the variance of weight norms across channels during training. This approach helps determine if a layer is sufficiently wide. Empirical validation shows that adjusting layer widths based on these patterns can reduce parameters and improve performance across various models and datasets.

Most reviewers find the paper is well-written, and the results are interesting.
Please make sure to incorporate the feedback in the camera-ready version, especially addressing the reviewers' concerns on the experimental setup and more comprehensive studies on larger datasets.

Reviews: review: paper_summary: paper_summary: This paper investigates the relationship between the differences in weight norms across channels and the adequacy of layer widths. The authors suggest that knowing these patterns(IS/DS) can help set layer widths better, leading to better resource use, fewer parameters, and improved performance in different network designs. The experiment shows that narrow-wide-narrow streamline network would boost the performance.
strength_weakness: strength_weakness: To determine the appropriate width, your method should involve training the model and observing various patterns of weight norm variance to see if the width is sufficient. However, retraining the model to decide on the width can be time-consuming. Considering that IS/DS indicate whether layers learn similar neurons, could we use the cosine similarity of a pretrained model to directly assess if the width is sufficient?
questions: questions: 1. The experiment includes both ResNet and VGG16. How should we define the weight norm of convolutional layers? Chapter 3 provides a simple analysis of the Matthew effect between similar channels. Is there any theoretical analysis for convolutional layers?
2. DS patterns suggest less similar neurons, indicating we should increase the width. IS patterns suggest redundant neurons, meaning we can decrease the width. In practice, why don't we adjust the width exactly according to the IS/DS patterns? For example, in the appendix, Figure 9 shows IS/DS patterns for 13 layers of the VGG network. Layers 2, 3, 4, 5, 6, and 7 show DS patterns, so we should increase the width of these layers. However, in Table 13, only the widths of layers 2, 4, and 7 are increased. Would adjusting all the layers accordingly result in better performance?
3. How do we determine the exact width of a layer based on the DS pattern? Is there an empirical boundary for the width of layers? I noticed that in the third stage of weight norm variance, DS patterns exhibit different drop ratios. For instance, in Figure 9, layers 2, 3, and 7 drop to a higher level, while layers 4, 5, and 6 drop to a lower level. Does this indicate that we should increase the width more for layers 4, 5, and 6?
4. In table3, after adjusting to streamline width, all networks would have a higher FLOPs than origin. Would the performance improvement due the FLOPs increase?
limitations: limitations: No apparent limitations.
paper_summary: paper_summary: This paper tries to address an issue in deep neural networks: the trade-off between computational cost and performance, particularly focusing on the width of each layer. Traditionally, layer widths in neural networks are determined empirically or through extensive searches. The authors propose a novel approach by examining the variance of weight norms across different channels to determine if a layer is sufficiently wide.
strength_weakness: strength_weakness: 1. The empirical justification of the proposed method is somehow not convincing enough. From Table 2, we can see that channels within layers with IS patterns are also hard to merge, like layers 8 to 11 in Table 2. From Figure 12 in the Appendix, we can see that the similarity gradually decreased first and then increased for middle layers with DS patterns, and it is hard to say there is a hard cutoff.
2. Suppose the proposed argument is correct, then it seems challenging to use this method in practice. The width of each layer depends on the IS or DS pattern of the original model, so you need to train the original model first and then the model with the streamline width. This always increases the total training cost especially when scaling up the model. 
3. The authors claim that they have similar observations for ViTs in lines 58-62, but no results are given. Since the original ViT has a uniform width, I am wondering whether the arguments still hold for models with uniform width.
4. The experiment setup is not very convincing. For CIFAR-10/100 scales of datasets, ResNet-18 or ResNet-50 are large and they have many redundant parameters. In the original ResNet paper, they used different architectures for CIFAR datasets with a much smaller number of parameters (0.27M to 1.7M with ResNet-20 to ResNet-110), please see Table 6 in "Deep Residual Learning for Image Recognition". The performance of ResNets and VGGs is much lower than public baselines for CIFAR-10 and CIFAR-100. Please see these two repo: https://github.com/weiaicunzai/pytorch-cifar100 and https://github.com/kuangliu/pytorch-cifar.
questions: questions: Please see the weakness.
limitations: limitations: Addressed in conclusion.
paper_summary: paper_summary: The paper proposes a method to optimize neural network layer width by analyzing the variance of weight norms across channels during training. This approach helps determine if a layer is sufficiently wide. Empirical validation shows that adjusting layer widths based on these patterns can reduce parameters and improve performance across various models and datasets.
strength_weakness: strength_weakness: 1. The layer width optimization is related to channel pruning and NAS-based channel number search, which have achieved significant success in finding optimal layer widths. I think these methods should be discussed and compared in the paper.

2. Measuring the weight norm would introduce additional computation cost, can the authors discuss the complexity of the method and report how much additional time does the method introduce?

3. Can the authors provide more detailed and theoretical analysis on the choice of the metric? For example, there are other metrics such as gradient norm, Hessian matrix, and absolute weight value to evaluate the weight importance and training statistics, why the proposed metric is better?
questions: questions: See Weaknesses.
limitations: limitations: The limitations have been adequately discussed in the paper.
paper_summary: paper_summary: The paper studies the effects of layer widths on neural network performance. By studying the affects of the weight norm in different channels, the work discovers several distinct stages of training, which are apparent across different modalities and architectures. The authors also show how the insights could help in improving network performance with the same computational costs.
strength_weakness: strength_weakness: The paper is really great! It could benefit from more experiments and archs, but the current experiments are convincing.
questions: questions: Some possible interesting references which could be relevant:

1. https://proceedings.neurips.cc/paper_files/paper/2023/hash/b63ad8c24354b0e5bcb7aea16490beab-Abstract-Conference.html
limitations: limitations: N/A

Comments: title: Thank you for the comment and here are our response
Look forward to further discussion
Look forward to further discussion
comment: Dear reviewer iwdV,

Thank you for your feedback and we regret to hear that our rebuttal is not to your satisfaction. The following is our response to your further comment. 

As you mentioned "different kinds of architecture beyond CNNs", with all due respect, we want to emphasize that proposing a streamlining technique is only a small part of this paper.  As the title of this paper indicates, we mainly identify the weight norm variance pattern regarding the layer width, **where we provide theoretical analysis in Sec. 3.1 and extensive experimental results on RNN, GNN, Transformer, and CNN in Sec. 3.2.** 

In utilizing the identified pattern, besides the proposed method, we investigate the training dynamic of widely used CNNs with extensive experimental results providing insights such as there are three stages during the training procedure. We have also provided our analysis regarding each stage of training. 

The proposed method and corresponding experiments in Sec.5 are our attempts to provide a simple sample using the identified pattern to guide the network design and further validate the observations and insights provided in this paper. **In believing the novelty of the insights and the new perspective provided in this paper, we agree with the comment that "there is more space" for more effective methods. We sincerely hope our paper can innovate future works and contribute to the community in the attempt to follow the footsteps of many excellent previous works (Best papers) [1, 2] in providing insights and innovating further applications.**

Thank you once again for your valuable time and feedback. We hope the identified pattern and provided insights in this paper could also be taken into consideration, as well as the proposed method.

Best regards,

Authors

[1] Frankle, Jonathan, and Michael Carbin. "The lottery ticket hypothesis: Finding sparse, trainable neural networks." ICLR 2019, Best Paper.

[2] Schaeffer, Rylan, Brando Miranda, and Sanmi Koyejo. "Are emergent abilities of large language models a mirage?." NeurIPS 2023, Best Paper.
I appreciate the efforts of the authors in the rebuttal. The rebuttal addressed some of the concerns. However, I believe there is more space to make the experiment more comprehensive on larger-scale datasets, models, and different kinds of architecture beyond CNNs (as stated in W2). As a result, I will keep my score.
Dear reviewer iwdV

We hope this message finds you well. We appreciate your valuable feedback and have tried our best to address each point of your concerns in our rebuttal. Since the discussion period is approaching its end, we are eager to hear from you, whether our rebuttal has addressed your concerns. Please feel free to comment on our rebuttal if you have further questions or comments. We are more than happy to respond to any further comments. Thank you for your commitment to the review process.

Best regards,

Authors
Dear AC and Reviewers,

We sincerely appreciate your valuable feedback and the efforts you devoted to the reviewing process. Since the discussion is approaching its end, we look forward to your opinion on whether our rebuttal has addressed the concerns and issues in the initial review. 

In the initial review, we are glad to find that the reviews are generally positive. We cherish your valuable feedback and the opportunity for us to have further discussions. We will keep on polishing our paper and are more than happy to respond to further comments.

Best regard,

Authors
Dear Reviewers,

I hope this message finds you well. I would like to take this opportunity to express my gratitude for the valuable feedback and insights you have provided during the review process. Your expertise and dedication are essential to maintaining the high standards of our community.

The authors have devoted considerable time and effort to addressing the concerns and suggestions you raised in your reviews. They have carefully crafted a detailed rebuttal, aiming to clarify any misunderstandings and incorporate your feedback into their manuscript.

I kindly ask that you take a thorough and thoughtful look at the authors' responses. Your assessment of their rebuttal is crucial in determining whether they have satisfactorily addressed the issues and concerns highlighted in your initial review.

Thank you once again for your hard work and commitment to advancing the quality of our scholarly community. Your contributions are greatly appreciated.

Best regards,
AC