Are wider nets better given the same number of parameters?

1 INTRODUCTION Deep neural networks have shown great empirical success in solving a variety of tasks across different application domains . One of the prominent empirical observations about neural nets is that increasing the number of parameters leads to improved performance ( Neyshabur et al. , 2015 ; 2019 ; Hestness et al. , 2017 ; Kaplan et al. , 2020 ) . The consequences of this effect for model optimization and generalization have been explored extensively . In the vast majority of these studies , both empirical and theoretical , the number of parameters is increased by increasing the width of the network ( Neyshabur et al. , 2019 ; Du et al. , 2019 ; Allen-Zhu et al. , 2019 ) . Network width itself on the other hand has been the subject of interest in studies analyzing its effect on the dynamics of neural network optimization , e.g . using Neural Tangent Kernels ( Jacot et al. , 2018 ; Arora et al. , 2019 ) and ∗Work done while an intern at Blueshift . 1Code is available at https : //github.com/google-research/wide-sparse-nets Gaussian Process Kernels ( Wilson et al. , 2016 ; Lee et al. , 2017 ) . All studies we know of suffer from the same fundamental issue : When increasing the width , the number of parameters is being increased as well , and therefore it is not possible to separate the effect of increasing width from the effect of increasing number of parameters . How does each of these factors — width and number of parameters — contribute to the improvement in performance ? We conduct a principled study addressing this question , proposing and testing methods of increasing network width while keeping the number of parameters constant . Suprisingly , we find scenarios under which most of the performance benefits come from increasing the width . 1.1 OUR CONTRIBUTIONS . In this paper we make the following contributions : • We propose three candidate methods ( illustrated in Figure 2 ) for increasing network width while keeping the number of parameters constant . ( a ) Linear bottleneck : Substituting each weight matrix by a product of two weight matrices . This corresponds to limiting the rank of the weight matrix . ( b ) Non-linear bottleneck : Narrowing every other layer and widening the rest . ( c ) Static sparsity : Setting some weights to zero using a mask that is randomly chosen at initialization and remains static throughout training . • We show that performance can be improved by increasing the width , without increas- ing the number of model parameters . We find that test accuracy can be improved using method ( a ) or ( c ) , while method ( b ) only degrades performance . However , we find that ( a ) suffers from another degradation caused by the reparameterization , even before widening the network . Consequently , we focus on the sparsity method ( c ) , as it leads to the best results and is applicable to any network type . • We empirically investigate different ways in which random , static sparsity can be distributed among layers of the network and , based on our findings , propose an algorithm to do this effectively ( Section 2.3 ) . • We demonstrate that the improvement due to widening ( while keeping the number of parameters fixed ) holds across standard image datasets and models . Surprisingly , we obesrve that for ImageNet , increasing the width according to ( c ) leads to almost identical performance as when we allow the number of weights to increase along with the width ( Section 2.3 ) . • To understand the observed effect theoretically , we study a simplified model and show that the improved performance of a wider , sparse network is correlated with a reduced distance between its Gaussian Process kernel and that of an infinitely wide network . We propose that reduced kernel distance may explain the observed effect ( Section 3 ) . 1.2 RELATED WORK . Our work is similar in nature to the body of work studying the role of overparametrization and width . Neyshabur et al . ( 2015 ) observed that increasing the number of hidden units beyond what is necessary to fit the training data leads to improved test performance , and attributed this to the inductive bias of the optimization algorithm . Soltanolkotabi et al . ( 2018 ) ; Neyshabur et al . ( 2019 ) ; Allen-Zhu et al . ( 2019 ) further studied the role of over-parameterization in improving optimization and generalization . Woodworth et al . ( 2020 ) studied the implicit regularization of gradient descent in the over-parameterized setting , and Belkin et al . ( 2019 ) investigated the behavior at interpolation . Furthermore , Lee et al . ( 2017 ) showed that networks at initialization become Gaussian Processes in the large width limit , and Jacot et al . ( 2018 ) showed that infinitely wide networks behave as linear models when trained using gradient flow . Lee et al . ( 2020 ) systematically compared these different theoretical approaches . In all the above works , the number of parameters is increased by increasing the width . However , in this work , we conduct a controlled study of the effect of width by keeping the number of parameters fixed . Perhaps Littwin et al . ( 2020 ) is the closest work to ours , which investigates overparametrization achieved through ensembling as opposed to layer width . Ensembling is achieved by connecting several networks in parallel into one “ collegial ” ensemble . They show that for large ensembles the optimization dynamics simplify and resemble the dynamics of wide models , yet scale much more favorably in terms of number of parameters . However , the method employed there is borrowed from the ResNeXt architecture ( Xie et al. , 2016 ) , which involves altering the overall structure of the network as width is increased . In this work we try to make minimal changes to network structure , in order to isolate the effect of width on network performance . Finally , static sparsity is the basis of the main method we use to increase network width while keeping the number of parameters fixed . There is a large body of work on the topic of sparse neural networks ( Park et al. , 2016 ; Narang et al. , 2017 ; Bellec et al. , 2018 ; Frankle and Carbin , 2018 ; Elsen et al. , 2019 ; Gale et al. , 2019 ; You et al. , 2019 ) , and many studies derive sophisticated approaches to optimize the sparsity pattern ( Lee et al. , 2018 ; Evci et al. , 2019 ; Wang et al. , 2020 ; Tanaka et al. , 2020 ) . In our study , however , sparsity itself is not the subject of interest . In order to minimize its effect in our controlled experiments , the sparsity pattern that we apply is randomly chosen and static . A recent work ( Frankle et al. , 2020 ) demonstrates that a random , fixed sparsity pattern leads to equal performance as the more involved methods for pruning applied at initialization , when the per-layer sparsity distribution is the same . However , we do not explore this direction in our study . 2 EMPIRICAL INVESTIGATION . In this section , we first explain our experimental methodology and then investigate the effectiveness of different approaches to increase width while keeping the number of parmeters fixed . Finally , we discuss the respective roles of width and the number of parameters in improving performance . 2.1 METHODOLOGY . In order to identify how width and number of parameters separately affect performance , we need to decouple these quantities . For a fully-connected layer , width is the number of hidden units , and for a convolutional layer , width corresponds to the number of output channels . Increasing the width of one layer ordinarily entails an increase in the number of weights . Therefore , some adjustment is required to keep the total number of weights constant . This adjustment can be made at the level of layers by reducing some other dimension ( i.e. , by introducing a “ bottleneck ” ) , or at the level of weights by setting some of them to be zero . When choosing a method for our analysis , we take particular care about possible confounding variables , as many aspects of a neural network are intertwined . For example , we prefer to keep the number of non-linear layers in the network fixed , as it has been shown that changing it can significantly affect the experssive power , optimization dynamics and generalizaiton properties of the network . With these constraints in mind , we specify three different methods listed in Section 1.1 . In this section , we discuss these methods in detail , evaluate them , and present the results obtained on image classification tasks . In summary , our approach is as follows : We select a network architecture which specifies layer types and arrangement ( e.g. , MLP with some number of hidden layers , or ResNet-18 ) , set layer widths , and count the number of weights . We refer to this model as the baseline , and derive other models from it by increasing the width while keeping the number of weights constant . In this way , we obtain a family of models that we train using the same training procedure , and compare their test accuracies . For comparison and as a sanity check , we also consider the dense variants of the wider models . We tested a variety of simple models , and observed the same general behavior in the context of our research question . For the discussion in this paper , we focus on two model types : a MLP with one hidden layer , and a ResNet-18 . We chose the ResNet-18 architecture because it is a standard model widely used in practice . Its size is small enough that it allows us to increase the width substantially , yet it has sufficient representational power to obtain a nontrivial accuracy on standard image datasets . When creating a family of ResNet-18 models , we refer to the number of output channels of the first convolutional layer as the width , and do not alter the default width ratios of all following layers . Further details of each experiment and figure are specified in Appendix A . 2.2 BOTTLENECK METHODS . Here we discuss the two bottleneck methods described in the previous section and in Figure 2 . Linear Bottleneck : Substitute each weight matrix W ∈ Rdi×do by W1W2 , where W1 ∈ Rdi×db , W2 ∈ Rdb×do and db ≤ min ( di , do ) .2 Note that if db = min ( di , do ) , then this reparametrization has the exact same expressive power 3 as the original network . Now , one can make the network wider by increasing the number of hidden units ( di and do here ) and reducing the bottleneck size db . This would correspond to imposing a low-rank constraint on the original weight matrix W . Linear bottleneck has been proposed as a way to reduce the number of parameters and improve training speed ( Ba and Caruana , 2014 ; Urban et al. , 2016 ) . The caveat of this method is that , even though the expressive power of the reparametrized network is the same as the original one , the reparameterization changes the gradient descent trajectory and hence can affect the final results . It is therefore not possible to control the implicit regularization caused by this transformation . Note that substituting the weight matrix by a product of two matrices changes the number of parameters of the model . Non-linear Bottleneck : One way to create a non-linear bottleneck is to split each layer in two as described above and add a non-linearity to the first layer . The problem with this approach is that adding a non-linearity changes the expressive power of the model , and hence adds a factor that we can not control . An alternative approach , which is more favorable in our case and which we adopt here , is to modify the layers in pairs . The input dimension of the first layer is increased , while its output dimension ( and consequently the input dimension of the second layer ) is reduced ( db < d ) . Non-linear bottlenecks are widely used in practice , particularly as a way to save parameters in very deep architectures , such as deeper variants of ResNet ( He et al. , 2016 ) and DenseNet ( Huang et al. , 2017 ) . The performance of these methods on CIFAR-10 and CIFAR-100 datasets is demonstrated in Figure 3 . The results indicate that increasing width using the linear bottleneck indeed leads to improved 2In the case of a ResNet-18 model , each convolutional layer is replaced by two convolutional layers with kernel dimensions 3× 3 and 1× 1 , respectively , and with db = di = do before widening . 3Here , expressive power refers to the set of functions that can be represented by the network . accuracy up to a certain width . Moreover , this effect is more pronounced in smaller models that are less overparameterized . This is similar to the improvement gained when increasing the width along with the number of parameters ( without a bottleneck ) : The improvement tends to be diminished when the base network is wider . However , as discussed above , the act of substituting a weight matrix by a product of two weight matrices changes the optimization trajectory which could in turn affect generalization . Indeed , the performance of the original model , indicated by dashed horizontal lines in the Figure , is significantly higher than that of the transformed models . Therefore , even though the width increase at a constant number of weights with the linear bottleneck method improves the result of the most narrow model obtained after the transformation , it typically does not outperform the default ResNet-18 model . Due to lack of control over the effect of inductive bias , we conclude that this choice does not qualify to be used in our controlled experiments . The non-linear bottleneck method does not suffer from the same issues as the linear version in terms of inductive bias of reparametrization . However , as Figure 3 demonstrates , no empirical improvement is obtained by increasing the width , except for the model with 1.8e+05 weights ( and then the improvement is mild ) . Therefore , we conclude that the non-linear bottleneck model does not show a significant enough improvement to be considered as an effective method for our study .

Summary: This manuscript provides an intriguing discussion on the different roles that the width and parameter size could play in a neural network. While these two aspects are traditionally treated as -if not identical- correlated, the authors managed to develop a couple of configurations to decouple and analyze both separately. Especially the wide and sparse approach could be a new way to design neural networks that are supposed to be small and expressive at the same time.