Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an approximation of Shallow ReLU networks in the large width regime to study their optimization dynamics. More particularly, they show that for uni-dimensional inputs, this approximation exhibits the 'catapult phase phenomenon', which corresponds to a regime of learning rate larger than the largest step size advocated by the theory on linear dynamics. 
strength_and_weaknesses: ## Strengths:
- This paper tries to tackle a difficult problem which is to understand the non-linear dynamics of learning neural networks. 
- The model they are proposing is simple enough to be analyzed but complex enough to model non-linear learning dynamics that seem to correspond (to a certain degree) to what happens with two-layers ReLU networks. In particular, it exhibits a 'catapult phase' for the step-size.
- The intuition of the main elements of the proofs is well communicated. 
- The experiments back up the theory pretty well.

## Weaknesses:
- My first main complaint would be regarding the critical assumption regarding the fact that the input has to be of dimension 1. This is very restrictive and relatively buried in the paper (see my comments in the section about clarity)
- My second main complaint would be with the proof of the main theorem. (In particular, Theorem 1). What I  thought was a proof sketch was actually supposed to be the complete proof (please let me know if I missed the detailed proof). The statement in the "proof of Theorem 1" (page 6 and 7) are not formal enough to be verified. I give a very good idea of the structure of the actual proof, but it is not enough. We cannot formally have $g(t) = o(\sqrt{m})$ for $t<T_{incr}$ and then $g(T_{incr }) = \Theta(m)$ (even though I understand the intuition).
- The argument that $\sum_{t=0}^{T_{incr}} \Theta(g(t)^2/m) = o(1)$ is not 100% clear to me since $g(t)$ will be close to $\Theta(m)$ when $t$ is close to $T_{incr}$, moreover even if we had $g(t)^2= o(\sqrt{m})$ we would have $\sum_{t=0}^{T_{incr}} \Theta(g(t)^2/m) = o(T_{incr}) = o(\log(m))$.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is very clear and conveys very well the intuitions behind the results. My only concern regarding the clarity is that one critical assumption is relatively "buried" in the paper (It is not mentioned as an "official" assumption and only appears in section 3.2). More precisely, for their main result (Thm 2), the authors assume that the input data is unidimensional ($x_i \in \mathbb{R}$. It is an assumption that is quite restrictive, and that should be mentioned in the abstract, introduction, and related work. In particular, the author claims that their formulation is more general than Lewkowycz et al. [2020] but it seems that Lewkowycz et al. [2020] does not have to restrict their analysis to unidimensional inputs. If it is the case, I think the author should mention it in the introduction/related work. 
Quality: Except for my concern regarding the proofs of the main results of Theorem 1 and 2, I think the quality of this paper is pretty good. 

Novelty: To the extent of my knowledge, the results are novel. 

Reproducibility: I think that the proofs of the main theorems (Thm 1 and 2) lack details and formalism. So I am not sure an independent researcher could actually reprove these results formally on the basis of the proof sketches presented in this paper. 

Minor comment:
- End of page 5 you write that $\lambda(t) = O(1)$ but actually, what you need is $\lambda(t) = \Theta(1)$ (which is what you noted elsewhere in the paper)

summary_of_the_review: This paper provides some exciting contributions. However, I have some concerns regarding the formalism of the proof. The results feel correct to me, but, at this stage, the paper only contains sketches of proof. Moreover, I am slightly uncomfortable with how a critical assumption is presented. This critical assumption could be missed by quickly reading the paper. It should be mentioned earlier in the text, and it should be more visible. 

For all these reasons, I am currently evaluating this paper under the acceptance threshold. However, if the authors address my concern, I am eager to significantly increase my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the learning dynamics of deep neural networks using the quadratic models.  Specifically, the "catapult phase" is identified under this model when the learning rate is high, while it is not for linear models. Numerical simulations are provided to support the theoretical analysis. 




strength_and_weaknesses: Strenghths:

1. It is a solid theoretical work on the learning dynamics of deep neural networks. The adopted quadratic model successfully captures the interesting "catapult phase" from Lewkowycz et al. (2020), which provides insights and thus help understanding of the sophisticated dynamics of NN. 

2. Numerical results support their theoretical analysis. 

Weaknesses:

1. The output dimension d is limited to dimension 1 and it is unclear what happens in the case d>1, and whether similar results still hold, which I think is the biggest weakness, since in high-dimensional case, some counter-intuitive phenomena might happen. 


clarity,_quality,_novelty_and_reproducibility: This paper is well-written. I did not fully check all the details of the proof though they are assumed to be right. ]
Moreover, this paper is novel in firstly analyzing the non-linear wide neural networks  the catapult regime through the perspective of the quadratic approximation
summary_of_the_review: A solid theoretical work on the analysis of the learning dynamic of NN by using the quadratic models, as opposed to previously used linear models. The most interesting result is that, it  successfully captures the interesting "catapult phase" from Lewkowycz et al. (2020), while linear networks fail. This demonstrates the usefulness of the quadratic models in understanding NNs. The only concern is that the current analysis is restricted to the output dimension d=1 and it is not clear whether the results or methods still apply to high-dimensional outputs, or even deeper hidden layers.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present the theoretical analysis of a neural quadratic model as an approximation of a two-layer neural network.
The neural quadratic model exhibits an interesting non-linear aspect of training dynamics that can be seen with two-layer neural networks in practice: The catapult phase.
The catapult phase occurs when the learning rate is high enough to be beyond locally critical but low enough so as to not globally diverge and leads to a jump into a potentially better optimum with lower
neural tangent Kernel norm.
The authors' theoretical analysis makes use of insightful estimation possible due to considering the wide network limit but the theoretically predicted phenomena are relevant in practice and the behavior in training loss and tangent kernel largest eigenvalue are also demonstrated using experiments. 
strength_and_weaknesses: Strengths:
- The paper analyzes an interesting practical phenomenon using powerful theoretical tools that will enable better understanding of training dynamics for neural networks
- Having the main theoretical analysis demonstrated for the single training example case first helps make it more accessible
- The empirical validation in Figure 3 backs up the theory

Weaknesses:
- NQMs appears in the figure caption for Figure 1 (b) which is referenced before the abbreviation (for Neural Quadratic Model) is first introduced in the text
- Does the theoretical analysis in principle extend for squared to cross-entropy loss? (Makes the dynamics less elegant but would still be interesting to know how this could work)
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well written and clearly structured 

Quality: The theoretical analysis is of high quality 

Novelty: The presented analysis seems to be a novel, original and deeply insightful extension of the existing literature on analyzing training dynamics via the neural tangent kernel idea

Reproducability: The focus of the paper is the theory which is elaborated in rich detail in the appendix
summary_of_the_review: I recommend to accept the paper as I perceive it to analyze an interesting practical phenomenon of neural networks training using and extending the toolbox of modern theoretical analysis tools for neural
network training dynamics.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This article analyzes the dynamics of GD for quadratic approximations to sufficiently wide one layer ReLU network in which the network function is replaced by its second order Taylor expansion with respect to the parameters. The main result of this article is that for rather simple data (either one datapoint, or multiple datapoints in 1D), unlike linear approximations, quadratic approximations exhibit the so-called catapult phase. Specifically, for moderately large learning rates the training loss is unimodal, first rapidly growing and only then decaying, and the top eigenvalue of the NTK becomes decreases significantly by the end of training.

UPDATE AFTER REBUTTAL AND READING OTHER REVIEWS
I share the concerns of the other reviewers regarding the rigor of the proof of the main results. Originally I was kind of OK with it because I saw that I could fill in the details myself. But ultimately I when I read the other reviews and looked again, I became of the opinion that I quite like the results but that a paper of this kind should really have a higher standard for providing precise mathematical details in the proofs. So I am revising my assessment of this paper down from an 8 to a 6. 
strength_and_weaknesses: Strengths:

- It is rare to be able to more or less fully analyze any kind of non-linear learning dynamics. That the authors manage to do this, even if it is for somewhat stylized datasets, in a regime of large learning rate is really nice.
- The fact that the top eigenvectors for the NTK and the Hessian provably align in the catapult phase is an interesting and important contribution. Empirically, this is often known to happen but as far as I am aware this is the first time this has been proved to occur in a non-linear model. 

Weaknesses:

- The restriction to very simple datasets, either with one example, or inputs in 1D is quite severe. While the authors write “The assumption of d = 1 is common in the literature and simplifies the analysis for neural networks,” this assumption drastically simplifies (at least as far as I know) the structure of the NTK, as evidenced by Proposition 1. Perhaps the authors could comment on what is expected to occur in higher dimensions.
- To a lesser extent, the restriction to one layer networks is undesirable. Perhaps the authors could comment on what is expected to occur in deeper networks.
- There is a slightly strange apparent inconsistency between the theory developed in this article and the original work on the catapult phase. Specifically, figures 3 and 4 in that article seem to show that for shallow ReLU networks the catapult phase is delineated approximately by taking  the learning rate to be between $2 / \lambda_{max}$ and $12 / \lambda_{max}$. However, the authors here seem to prove that the largest convergent learning rate is $4 / \lambda_{max}$. Of course this is not a direct contradiction: quadratic models are not necessarily high enough fidelity to capture the constants controlling the catapult phase. Nonetheless, it would be nice if the authors could comment on this point. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written. 

Quality: The theoretical results are novel and interesting. 

Originality: The analysis proposed in this article is, as far as I know, the first time the catapult phase has been established beyond linear networks.
summary_of_the_review: This article gives a novel analysis of GD with moderate learning rates in a second order approximation to wide one layer ReLU networks. The main result of this article is that for simple data (either one datapoint, or multiple datapoints in 1D), unlike linear approximations, quadratic approximations exhibit the so-called catapult phase. As mentioned in the strengths section, it is rare to give a more or less complete characterization of GD dynamics in non-linear models at relatively large learning rate. I think the basic observations and techniques in this article will be useful for further developing theory and also that the results themselves are interesting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approximation of Shallow ReLU networks in the large width regime to study their optimization dynamics. More particularly, they show that for uni-dimensional inputs, this approximation exhibits the 'catapult phase phenomenon', which corresponds to a regime of learning rate larger than the largest step size advocated by the theory on linear dynamics.

All the reviewers agreed that this paper makes an interesting contribution to ICLR. They agree it should eventually be published. Yet, despite the high original ratings, reviewer uqyZ convinced the other reviewers in an extensive discussion after the rebuttal that the level of details given in the proof was not sufficient to be able to verify it and was not appropriate for an ICLR publication. They pointed out some original inconsistencies in the proof that were corrected in the revision, but there were still more gaps to fill in the revision. As an example, here is a quote from them from the discussion:
> I spent a decent amount of time on their proof. In my opinion, it is still a sketch of proof. In particular, it is not clear what happens in between their phases (e.g. what happens between $T_1$ and $T_2$, do we still have that $\lambda(t)$  increases, are we sure we can bound the sequence btw $T_1$ and $T_2$? ) ?
>
> I am not saying the result is not correct, but with this amount of details, the only way to verify the result is to spend a large amount of time writing the proof oneself. I do not think it is the task of the readers to do that.


Note that reviewers LYg1 and JdaH, while being positive, had a confidence of 2. The other reviewer Ctfq who had a confidence of 5 decided to downgrade their rating from 8 to 6 after the discussion and they agreed with reviewer uqyZ that the proof should be updated in a revision before resubmitting. The authors are encouraged to make a resubmission after including a more detailed proof.

justification_for_why_not_higher_score: The proofs are not verifiable.
Decision: Reject