Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper investigates the two regimes in the training of overparameterized networks (with small learning rates):
* kernel regime: the tangent kernel doesn't change much during training. The training behavior is then well approximated by a linear model (Taylor expansion at the initialization). This can happen when the weights are initialized to large values.
* rich regime: The kernel regime is turned into a rich regime when the assumptions of kernel regimes aren't met.

Specifically, the paper emphasizes how the scale of initialization controls the transition between the two regimes, which was first pointed out by Chizat & Bach (2018).

My main concern is that it is unclear what unique contributions are made by the paper, as the theoretical results are not more general than that of Chizat & Bach (2018). The contributions are not clearly stated and I can only see the execution of ideas from Chizat & Bach (2018) and applying them to more concrete examples, which leads to analytical results (for linear networks) in Theorem 1/2. This feels rather incremental. 

Some other comments:
* In experiments it was shown that popular initialization schemes are right on the edge of entering the kernel regime, which is very interesting. How does this change with network widths and different architectures?
* It's difficult to see what Figure 2b tells because several notations are undefined. What are $e_1$ and $1_d$?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper analyzes an inductive bias of the gradient flow for diagonal two-or higher-homogeneous models and characterizes a limit point depending on the initialization scale of parameters. Concretely, the paper shows that the gradient flow converges to an interpolator attaining minimum L1- (or L2-norm) when the scale is small (or large). In addition, these analyses are well verified empirically on MNIST and CIFAR-10 datasets.

Quality:
The work is of good quality and is technically sound.

Clarity:
The paper is well organized and easy to read.

Significance:
To explain the generalization ability of powerful machine learning models that can perfectly learn a training dataset, the implicit bias of the optimization methods and models play key roles when explicit regularization is not adopted. For instance, deep neural networks fall into this scenario. I think this paper makes a better contribution in this line of researches. Although, homogeneous models treated in this study is restricted (essentially linear models) and a theory is limited to the continuous gradient flow, these settings are rather common in this context. In [Gunasekar+(2017)], the convergence to the minimum L1-norm solution was shown for a slightly different model when the scale goes to zero. However, in addition to this property, the paper analyzes arbitrary scales of parameters and shows the convergence to the minimum L2-norm solution when the scale goes to infinity for diagonal homogeneous models.
It would be nice if the authors could emphasize the technical difficulty compared to [Gunasekar+(2017)] to strengthen the contribution of the paper.

A few questions:
- Can this analysis be extended to the setting of early stopping? Toward a better explanation of the generalization performance of deep learning, understanding of the inductive bias of the early stopping before convergence is more important.   
- A provided theory is limited to linear models essentially. Is it possible to extend a theory to non-linear models?

-----
Update:
I thank the authors for the response. I am convinced of the difference from [Gunasekar+(2017)] and my review stands. I  would like to keep my score.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: I really appreciated this paper. It discusses a very complex question ("Are we learning in a kernel regime, or in a rich regime where features are identified") by looking at perhaps the simplest model the authors could think of, and then study in detail the model. And how simple it turns out to be: just a linear regression with a twist. All in all, the paper is indeed is a clear demonstration that the differences between"Kernel" regime and one where some actual Learning is done can be demonstrated on simple examples. It is also the simplest model where one can observe  a non-trivial inductive bias and 'implicit regularisation'

I do not have much to say on the paper, except that I fully support publication. 




review_title: Official Blind Review #2

Meta-review: The paper studies how the size of the initialization of neural network weights affects whether the resulting training puts the network in a "kernel regime" or a "rich regime". Using a two-layer model they show, theoretically and practically, the transition between kernel and rich regimes. Further experiments are provided for more complex settings.

The scores of the reviewers were widely spread, with a high score (8) from a low confidence reviewer with a very short review. While the authors responded to the reviewer comments, two of the reviewers (importantly including the one recommending reject) did not further engage.

Overall, the paper studies an important problem, and provides insight into how weight initialization size can affect the final network. Unfortunately, there are many strong submissions to ICLR this year, and the submission in its current state is not yet suitable for publication.
Decision: Reject