Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript provides an intriguing discussion on the different roles that the width and parameter size could play in a neural network. While these two aspects are traditionally treated as -if not identical- correlated, the authors managed to develop a couple of configurations to decouple and analyze both separately. Especially the wide and sparse approach could be a new way to design neural networks that are supposed to be small and expressive at the same time. 
I'm only concerned about two aspects: i) The section 3 could have been perhaps easier to follow if the work by Jacot et al. 2018 had been briefly recapped. ii) The experiments are all conducted on image dataset. One might wonder whether we could draw the same conclusion for datasets of more general nature. 
review_title: Intriguing discussion which could have been more self-contained

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors analyze the enhancements brought by widening networks with the number of parameters fixed. From the experimental side, they conduct various experiments to compare the methods of widening the networks and demonstrate different ratios of widening different networks on diverse datasets. From the theoretical side, the authors relate the training dynamics of neural networks to kernel-based learning, in  the infinite-width limit. As a consequence, the authors claim that wider networks indeed improve the performance of algorithms under certain conditions.

My personal comments are listed below.
Pros.
1. The idea of widening the neural networks with fixed number of parameters is novel and the authors conducts abundant experiments to support their assertions.
2. The expressions and grammars are excellent.

Cons.
I. Clarity Part.
1. The authors spend much time and space to introduce linear and non-linear bottleneck methods to widening the neural networks with fixing the number of parameters, and, subsequently, prove the 2 methods are not qualified. From my opinion, since the authors have asserted that they try to avoid big adjustments on structure of the networks, the bottleneck-like methods are inherently excluded from the list. Meanwhile, the explanations for sparsity methods seems to be lack.

2. While the authors provide meaningful and comprehensive plots to demonstrate the results of widening neural networks, the explanations and instructions of the plots and pictures are sometimes missing. For example, In figure 5, the trends of test accurarcy to widening factor are not always positive, some go down when the widening factors are big, others seems to be static. How do they happen? I think further explorations can be made.

3. As for the theoretical side, the authors reference the NTK techniques (Jacot et al., 2018) to explain the relationships between the finite-width neual network and the kernel-based learning. However, NTK techniques are designed for infinite-width neural networks, and have constraints when applied on finite-width networks. As the authors assert, the synchronization of optimizations of 2 methods is only conjecture, which means the prove provide here can only be used to support the NTK theorems. Besides, the relationships between NTK and widening neural networks are not detailedly explained.

II. Quality Part
1. Figure 3, (b) has no horizontal baselines for traditional methods.
review_title: I provide my personal opinions on the analyses about the effect brought by widening nets from both experimental and theoretical perspectives.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

This paper analyzed the influence of neural network width on the network performances while fixing the total number of parameters. Specifically, the authors introduced several ways to change the model width without increasing the number of parameters, and showed experimentally that for widened networks with a random static mask on weights to keep the number of parameters, increasing the width can improve the performances of the models until the network become very sparse and hard to train. They author theoretically showed that for a not so sparse one-hidden-layer neural network, increasing width decreases the distance to the Gaussian Process kernel corresponding to the infinith-width limit, which partially supports their experimental findings.

Pros:

1. Understanding the reason behind the generalization performance of neural networks is a very important problem, and the idea of decoupling the effects of network width and the number of parameters on the generalization performance is interesting.

2. This paper is clearly written, well organized, and generally easy to understand. The figures in this paper are clear, and each figure was explained in detail. The methods/experimental settings are explained in detail, and the reasons why the authors use these models/methods are also discussed in the paper. The relationship to prior works has been clearly discussed.

3. The experimental methodologies and theoretical computations appear to be correct.

Cons:

1. The justification for this specific problem analyzed in this paper, i.e., fixing the number of parameters and changing the width, is a bit unclear. The number of parameters can be interpreted as a measure for the expressive power of the model, but this is not a very direct relationship because the expressive power is also related to other factors, e.g., the depth, the width, the choice and ordering of the layers. In the methods proposed in this paper, the authors were either changing the explicit number of parameters (e.g., linear bottlenetwork) or changing the expressive power of the network, which included other (perhaps unwanted) factors in this framework and could potentially influence the results. Details will be discussed in the next point.

2. The definition of "number of parameters" is a bit unclear in this paper, making the problem that the authors want to analyze a bit vague. The authors didn't formally define it, but it is actually one of the core concept in this paper. The authors used three similar terms, i.e., "number of parameters"/"number of tunable parameters", "effective number of parameters", and "expressive power", in this paper. However, these three terms have slightly different meanings, and none of them is defined in this paper. From my understanding, "expressive power" is the set of all functions that can be reprensented by a model, and "number of parameters"/"number of tunable parameters" is a simple parameter count and can be used as a complexity measure of the expressive power, but "effective number of parameters" is kind of confusing. For instance, putting a low-rank constraint in Linear Bottleneck or using a sparse model actually changes the expressive power of the neural network but (claimed by this paper) does not change the "effective number of parameters". Due to the lack of formal definition of these three terms, I think the problem analyzed by this paper is not clear enough, making the points made in this paper unclear.

3. The authors are explaining some of the experimental results in a very intuitive way without enough supporting evidence. For example, the authors said that Linear Bottleneck suffers from some "implicit bias", deleting weights from a sparse layer make optimization "more difficult", and small connectivity can hurt the "trainability" of the model. These terms are not formally defined or explained in detail, making the points made by the authors a little vague.

4. The connection between the theory and experiments in this paper is a bit weak. People have observed that the performances of commonly-used neural networks always outperform their infinith-width limit, i.e., the corresponding infinith-width NTK. Therefore, in practical neural network models, simply decreasing the distance to the infinith-width limit may hurt the performance. Thus, I suspect that the explanation provided by the authors only works for some particular examples like small networks on MNIST.

Recommendation:

Overall, I vote for rejecting this paper. Although the idea of decoupling the influence of width and number of parameters is interesting, the problem analyzed in this paper is vague and a lot of terms used in this paper is not properly defined, making the arguments in this paper unclear.

Supporting arguments for recommendation:

See "Cons".

Questions for the authors:

1. Please address the cons mentioned above.

2. For the experiments, I wonder whether you have done enough fine-tuning for the hyperparameters of each model and used the best test accuracy. Also, have you run each experiment multiple times or only once?

3. I didn't understand why you do not consider the linear/non-linear bottleneck as an effective method. I think the goal is to analyze the effect of width but not to get a significant improvement on the test accuracy.

Minor comments:

1. Typo: Page 5, "MLP", paragraph 3, "This results" -> "This result"

2. Page 5, paragraph 1, which model is "the model with baseline width 8"? Is it the model with the number of parameters 1.8e+5?
review_title: Interesting but a bit vaguely-defined problem

Meta-review: The paper investigates the interesting question whether increasing the width or the number of parameters is 
responsible for improved test accuracy. The paper is very well written and the question is novel and innovative.
From a methodological point of view, the experiments are well conducted, too.
The theoretical part of the paper is somewhat detached from the experimental part and constitutes more of
a heuristic conjecture. In addition, more experiments on a variety of other data sets would have been great.
Ideally, the theoretical section would thus be replaced by such additional experiments, but this is of course not 
an option for a conference reviewing system.
Given the innovative question and well-conducted experiments I think that the pros outweighs the cons,
and for this reason I recommend to accept the paper. Reviewer concerns have been well addressed by the authors in their rebuttal and updated version of the paper.
Decision: Accept (Poster)