Are wider nets better given the same number of parameters?

1 INTRODUCTION Deep neural networks have shown great empirical success in solving a variety of tasks across different application domains . One of the prominent empirical observations about neural nets is that increasing the number of parameters leads to improved performance ( Neyshabur et al. , 2015 ; 2019 ; Hestness et al. , 2017 ; Kaplan et al. , 2020 ) . The consequences of this effect for model optimization and generalization have been explored extensively . In the vast majority of these studies , both empirical and theoretical , the number of parameters is increased by increasing the width of the network ( Neyshabur et al. , 2019 ; Du et al. , 2019 ; Allen-Zhu et al. , 2019 ) . Network width itself on the other hand has been the subject of interest in studies analyzing its effect on the dynamics of neural network optimization , e.g . using Neural Tangent Kernels ( Jacot et al. , 2018 ; Arora et al. , 2019 ) and ∗Work done while an intern at Blueshift . 1Code is available at https : //github.com/google-research/wide-sparse-nets Gaussian Process Kernels ( Wilson et al. , 2016 ; Lee et al. , 2017 ) . All studies we know of suffer from the same fundamental issue : When increasing the width , the number of parameters is being increased as well , and therefore it is not possible to separate the effect of increasing width from the effect of increasing number of parameters . How does each of these factors — width and number of parameters — contribute to the improvement in performance ? We conduct a principled study addressing this question , proposing and testing methods of increasing network width while keeping the number of parameters constant . Suprisingly , we find scenarios under which most of the performance benefits come from increasing the width . 1.1 OUR CONTRIBUTIONS . In this paper we make the following contributions : • We propose three candidate methods ( illustrated in Figure 2 ) for increasing network width while keeping the number of parameters constant . ( a ) Linear bottleneck : Substituting each weight matrix by a product of two weight matrices . This corresponds to limiting the rank of the weight matrix . ( b ) Non-linear bottleneck : Narrowing every other layer and widening the rest . ( c ) Static sparsity : Setting some weights to zero using a mask that is randomly chosen at initialization and remains static throughout training . • We show that performance can be improved by increasing the width , without increas- ing the number of model parameters . We find that test accuracy can be improved using method ( a ) or ( c ) , while method ( b ) only degrades performance . However , we find that ( a ) suffers from another degradation caused by the reparameterization , even before widening the network . Consequently , we focus on the sparsity method ( c ) , as it leads to the best results and is applicable to any network type . • We empirically investigate different ways in which random , static sparsity can be distributed among layers of the network and , based on our findings , propose an algorithm to do this effectively ( Section 2.3 ) . • We demonstrate that the improvement due to widening ( while keeping the number of parameters fixed ) holds across standard image datasets and models . Surprisingly , we obesrve that for ImageNet , increasing the width according to ( c ) leads to almost identical performance as when we allow the number of weights to increase along with the width ( Section 2.3 ) . • To understand the observed effect theoretically , we study a simplified model and show that the improved performance of a wider , sparse network is correlated with a reduced distance between its Gaussian Process kernel and that of an infinitely wide network . We propose that reduced kernel distance may explain the observed effect ( Section 3 ) . 1.2 RELATED WORK . Our work is similar in nature to the body of work studying the role of overparametrization and width . Neyshabur et al . ( 2015 ) observed that increasing the number of hidden units beyond what is necessary to fit the training data leads to improved test performance , and attributed this to the inductive bias of the optimization algorithm . Soltanolkotabi et al . ( 2018 ) ; Neyshabur et al . ( 2019 ) ; Allen-Zhu et al . ( 2019 ) further studied the role of over-parameterization in improving optimization and generalization . Woodworth et al . ( 2020 ) studied the implicit regularization of gradient descent in the over-parameterized setting , and Belkin et al . ( 2019 ) investigated the behavior at interpolation . Furthermore , Lee et al . ( 2017 ) showed that networks at initialization become Gaussian Processes in the large width limit , and Jacot et al . ( 2018 ) showed that infinitely wide networks behave as linear models when trained using gradient flow . Lee et al . ( 2020 ) systematically compared these different theoretical approaches . In all the above works , the number of parameters is increased by increasing the width . However , in this work , we conduct a controlled study of the effect of width by keeping the number of parameters fixed . Perhaps Littwin et al . ( 2020 ) is the closest work to ours , which investigates overparametrization achieved through ensembling as opposed to layer width . Ensembling is achieved by connecting several networks in parallel into one “ collegial ” ensemble . They show that for large ensembles the optimization dynamics simplify and resemble the dynamics of wide models , yet scale much more favorably in terms of number of parameters . However , the method employed there is borrowed from the ResNeXt architecture ( Xie et al. , 2016 ) , which involves altering the overall structure of the network as width is increased . In this work we try to make minimal changes to network structure , in order to isolate the effect of width on network performance . Finally , static sparsity is the basis of the main method we use to increase network width while keeping the number of parameters fixed . There is a large body of work on the topic of sparse neural networks ( Park et al. , 2016 ; Narang et al. , 2017 ; Bellec et al. , 2018 ; Frankle and Carbin , 2018 ; Elsen et al. , 2019 ; Gale et al. , 2019 ; You et al. , 2019 ) , and many studies derive sophisticated approaches to optimize the sparsity pattern ( Lee et al. , 2018 ; Evci et al. , 2019 ; Wang et al. , 2020 ; Tanaka et al. , 2020 ) . In our study , however , sparsity itself is not the subject of interest . In order to minimize its effect in our controlled experiments , the sparsity pattern that we apply is randomly chosen and static . A recent work ( Frankle et al. , 2020 ) demonstrates that a random , fixed sparsity pattern leads to equal performance as the more involved methods for pruning applied at initialization , when the per-layer sparsity distribution is the same . However , we do not explore this direction in our study . 2 EMPIRICAL INVESTIGATION . In this section , we first explain our experimental methodology and then investigate the effectiveness of different approaches to increase width while keeping the number of parmeters fixed . Finally , we discuss the respective roles of width and the number of parameters in improving performance . 2.1 METHODOLOGY . In order to identify how width and number of parameters separately affect performance , we need to decouple these quantities . For a fully-connected layer , width is the number of hidden units , and for a convolutional layer , width corresponds to the number of output channels . Increasing the width of one layer ordinarily entails an increase in the number of weights . Therefore , some adjustment is required to keep the total number of weights constant . This adjustment can be made at the level of layers by reducing some other dimension ( i.e. , by introducing a “ bottleneck ” ) , or at the level of weights by setting some of them to be zero . When choosing a method for our analysis , we take particular care about possible confounding variables , as many aspects of a neural network are intertwined . For example , we prefer to keep the number of non-linear layers in the network fixed , as it has been shown that changing it can significantly affect the experssive power , optimization dynamics and generalizaiton properties of the network . With these constraints in mind , we specify three different methods listed in Section 1.1 . In this section , we discuss these methods in detail , evaluate them , and present the results obtained on image classification tasks . In summary , our approach is as follows : We select a network architecture which specifies layer types and arrangement ( e.g. , MLP with some number of hidden layers , or ResNet-18 ) , set layer widths , and count the number of weights . We refer to this model as the baseline , and derive other models from it by increasing the width while keeping the number of weights constant . In this way , we obtain a family of models that we train using the same training procedure , and compare their test accuracies . For comparison and as a sanity check , we also consider the dense variants of the wider models . We tested a variety of simple models , and observed the same general behavior in the context of our research question . For the discussion in this paper , we focus on two model types : a MLP with one hidden layer , and a ResNet-18 . We chose the ResNet-18 architecture because it is a standard model widely used in practice . Its size is small enough that it allows us to increase the width substantially , yet it has sufficient representational power to obtain a nontrivial accuracy on standard image datasets . When creating a family of ResNet-18 models , we refer to the number of output channels of the first convolutional layer as the width , and do not alter the default width ratios of all following layers . Further details of each experiment and figure are specified in Appendix A . 2.2 BOTTLENECK METHODS . Here we discuss the two bottleneck methods described in the previous section and in Figure 2 . Linear Bottleneck : Substitute each weight matrix W ∈ Rdi×do by W1W2 , where W1 ∈ Rdi×db , W2 ∈ Rdb×do and db ≤ min ( di , do ) .2 Note that if db = min ( di , do ) , then this reparametrization has the exact same expressive power 3 as the original network . Now , one can make the network wider by increasing the number of hidden units ( di and do here ) and reducing the bottleneck size db . This would correspond to imposing a low-rank constraint on the original weight matrix W . Linear bottleneck has been proposed as a way to reduce the number of parameters and improve training speed ( Ba and Caruana , 2014 ; Urban et al. , 2016 ) . The caveat of this method is that , even though the expressive power of the reparametrized network is the same as the original one , the reparameterization changes the gradient descent trajectory and hence can affect the final results . It is therefore not possible to control the implicit regularization caused by this transformation . Note that substituting the weight matrix by a product of two matrices changes the number of parameters of the model . Non-linear Bottleneck : One way to create a non-linear bottleneck is to split each layer in two as described above and add a non-linearity to the first layer . The problem with this approach is that adding a non-linearity changes the expressive power of the model , and hence adds a factor that we can not control . An alternative approach , which is more favorable in our case and which we adopt here , is to modify the layers in pairs . The input dimension of the first layer is increased , while its output dimension ( and consequently the input dimension of the second layer ) is reduced ( db < d ) . Non-linear bottlenecks are widely used in practice , particularly as a way to save parameters in very deep architectures , such as deeper variants of ResNet ( He et al. , 2016 ) and DenseNet ( Huang et al. , 2017 ) . The performance of these methods on CIFAR-10 and CIFAR-100 datasets is demonstrated in Figure 3 . The results indicate that increasing width using the linear bottleneck indeed leads to improved 2In the case of a ResNet-18 model , each convolutional layer is replaced by two convolutional layers with kernel dimensions 3× 3 and 1× 1 , respectively , and with db = di = do before widening . 3Here , expressive power refers to the set of functions that can be represented by the network . accuracy up to a certain width . Moreover , this effect is more pronounced in smaller models that are less overparameterized . This is similar to the improvement gained when increasing the width along with the number of parameters ( without a bottleneck ) : The improvement tends to be diminished when the base network is wider . However , as discussed above , the act of substituting a weight matrix by a product of two weight matrices changes the optimization trajectory which could in turn affect generalization . Indeed , the performance of the original model , indicated by dashed horizontal lines in the Figure , is significantly higher than that of the transformed models . Therefore , even though the width increase at a constant number of weights with the linear bottleneck method improves the result of the most narrow model obtained after the transformation , it typically does not outperform the default ResNet-18 model . Due to lack of control over the effect of inductive bias , we conclude that this choice does not qualify to be used in our controlled experiments . The non-linear bottleneck method does not suffer from the same issues as the linear version in terms of inductive bias of reparametrization . However , as Figure 3 demonstrates , no empirical improvement is obtained by increasing the width , except for the model with 1.8e+05 weights ( and then the improvement is mild ) . Therefore , we conclude that the non-linear bottleneck model does not show a significant enough improvement to be considered as an effective method for our study .

Summary: This paper analyzed the influence of neural network width on the network performances while fixing the total number of parameters. Specifically, the authors introduced several ways to change the model width without increasing the number of parameters, and showed experimentally that for widened networks with a random static mask on weights to keep the number of parameters, increasing the width can improve the performances of the models until the network become very sparse and hard to train. They author theoretically showed that for a not so sparse one-hidden-layer neural network, increasing width decreases the distance to the Gaussian Process kernel corresponding to the infinith-width limit, which partially supports their experimental findings.