Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The submission shows the numerical instabilities in the current training of deep networks (DNN) with SGD and claims that the numerical error is due to floating point arithmetic. 
Then, it shows the condition of the learning rate and the weight decay to perform stable training from the perspective of numerical PDE.

strengths_and_weaknesses: Strengths 
- The submission shows some evidence of numerical instabilities in reproducing the results of DNN training.
- The submission studies a simplified CNN by the numerical PDE analysis and offers some principles for choosing learning rate and weight decay.

Weaknesses
- The current experiment lacks some details, so the non-reproducibility of the training trajectory cannot be strictly guaranteed to come from floating point arithmetic. As we all know, if we perform GPU-based training, too many seeds need to be fixed (involving cuda, cudnn, etc.). Small negligence will lead to inconsistent trajectory and output in the training process. Therefore if it is claimed that the instability comes from the floating-point calculation, it should be ensured that the dynamic trajectories of weights can strictly coincide for several tries when k=1.
- The experimental findings and theoretical results seem to be separated. No matter what the unstable factors are, as long as the conditions given by the theorem are not satisfied, it appears that the optimal trajectory will diverge.
- The authors provide too many linear PDE-related validation experiments. Since the title and intro try to present the contribution from the perspective of DNN training, it is better to show the application of stability conditions in neural network training rather than the numerical simulation of PDE.
- The traditional convergence analysis for the optimizer can also get the conditions related to LR and weight. It's NOT that there is ``no theoretical result related to learning rate'', but it's not as good as the heuristic methods in practical use. Secondly, in practice, it seems that the large LR (violating the stability condition) can still be used to train DNN without the divergence described by the theorem. From this perspective, the conclusion of this submission is not very practical.
questions: See the first point in Weaknesses.
limitations: Not related to the potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors show that there are numerical instabilities in the training of neural networks that inherently result from floating point errors. They empirically observe the phenomenon using a simple perturbation to the learning rate of SGD, which leads to some variations in the test performance. The authors further analyze one-layer CNN in which they can analytically derive an PDE of the gradient flow and use linear analysis to derive CFL conditions for stabilities. 
strengths_and_weaknesses: This paper discovers some interesting empirical observations on the numerical instabilities of training neural networks with SGD. The author presents a theoretical analysis in a simplified setting and implications in choosing principle to choosing learning rates. 

However, while there are variations in test accuracy shown in Table 1, I do not think they are that surprising. We do not know whether there is any other source of non-determinism in the implementation, which might cause the variations. I wonder if it is worth checking with a simple network on toy data using batch gradient descent?

On the theoretical side, the analysis, while revealing some conditions on the learning rate and weight decay, is rather limited to a simple architecture and the optimization requires regularization albeit common.

questions: 1) What does “restrained” mean exactly?

2) Theorem 5.2: What can we say about the PDE with RELU activation?

3) In Section 5, you consider the regularized cross-entropy loss. How about unregularized loss?

limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper models optimization algorithms as PDEs and studies their stability to obtain bounds on the hyperparameters of the optimization algorithms (such as the learning rate). The authors first empirically demonstrate instabilities in training of deep networks induced by floating point approximations, and use that to make the case for choosing the learning rate carefully.
strengths_and_weaknesses: The paper is well supported by theory and experiments, and is easy to follow. Section 3 provides a nice introductory example to the technique that the authors are using. While the paper is technically sound in terms of the technique presented, there is no comparison to other bounds on the learning rate obtained from convex optimization, or BIBO stability of control systems. A more compelling case could be made by either showing that the PDE technique obtains bounds that are in line with previous observations, or that the technique obtains better bounds on the learning rate.

The paper would also be stronger if the authors could extend their approach to other optimization algorithms like Adam/Adagrad, etc.
questions: The authors take a discrete time system (gradient descent optimization), and analyze a continuous analog of it (the gradient flow PDE), and then study ways to discretize the PDE. This is a curious approach - is there an advantage to this approach over studying the stability of the discrete time system in the first place?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper observes that the training dynamics of deep neural networks is unstable in the sense that a tiny perturbation of the updates lead to significant changes in the dynamics (at the level of the trajectory), and also to a change in the final test accuracy (although rather benign). The authors propose to study this phenomenon from the point of view of numerical analysis and carry this program for a simplified model of a one-hidden-layer convnet with a single input and an infinite dimensional conv filter. For this toy model, a linearized analysis shows that there is a range of stable and unstable stepsizes and weight decay regularization. This observation is confirmed numerically on a deeper model.

strengths_and_weaknesses: I enjoyed reading this paper: the presentation of the results is fair and with an appreciated scientific humbleness, the experiments are carried out with care and with a proper control of the various effects that could influence the outcome and the simple mathematical model is insightful.

The main weakness is that it is not clear whether this observation/analysis is an isolated remark or whether will lead to a fruitful line of research which will potentially lead to useful practical recommendations for deep learning (admittedly, this weakness applies to many papers that propose a new point of view). So the main question is whether the conclusions and recommandations remain relevant for more complex architectures (with more that one input, more than one filter, etc). In particular, it is not clear whether it is worth the computational cost (longer training time) to attempt to avoid this instability in practice by taking smaller step-sizes (because the variance on the test error is not that big).
questions: - In Section 5, I would rather call the system an (infinite dimensional) ODE rather than PDE because only a derivative in time is involved (while PDE is usually reserved for equations which involve partial derivatives in different variables).
limitations: N/A
review_title: null

Meta-review: metareview: The paper studies the instabilities of neural networks. Most of the reviewers recommend acceptance. I also think the paper seems interesting.
confidence: Less certain
Decision: Accept