Global Convergence in Neural ODEs: Impact of Activation Functions

Authors: Tianxiang Gao, Siyuan Sun, Hailiang Liu, Hongyang Gao
Primary area: learning theory
Keywords: Neural ODEs, Gradient Descent, Neural Tangent Kernel (NTK)

Abstract:
Neural Ordinary Differential Equations (ODEs) have been successful in various applications due to their continuous nature and parameter-sharing efficiency. However, these unique characteristics also introduce challenges in training, particularly with respect to gradient computation accuracy and convergence analysis. In this paper, we address these challenges by investigating the impact of activation functions. We demonstrate that the properties of activation functions—specifically smoothness and nonlinearity—are critical to the training dynamics. Smooth activation functions guarantee globally unique solutions for both forward and backward ODEs, while sufficient nonlinearity is essential for maintaining the spectral properties of the Neural Tangent Kernel (NTK) during training. Together, these properties enable us to establish the global convergence of Neural ODEs under gradient descent in overparameterized regimes. Our theoretical findings are validated by numerical experiments, which not only support our analysis but also provide practical guidelines for scaling Neural ODEs, potentially leading to faster training and improved performance in real-world applications.

Decision: accept (oral)
Reviewer scores: 8, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730525945029

thread_type:
official_review

<thread_object>
creator: reviewer_hphg

<content>
summary:
This paper studies the convergence of gradient descent for training neural ODEs from the neural tangent kernel perspectives, with experimental results validating the theoretical claims.

strengths:
(1) The paper extends NTK theory from general regression problems with ReLU nets to Neural ODEs with ResNet architectures. The analysis and results are novel for Neural ODEs.

(2) The empirical results validate the theorems.

weaknesses:
(1) Some prior work has extended NTK theory to analyze CNNs (for image classification tasks), PINNs, Transformers, and various gradient-based algorithms beyond vanilla gradient descent. However, related references are missing. 

(2) SGD is more commonly used in practice, making a convergence analysis of SGD particularly valuable and interesting.

(3) The results are specific to ODEs of the form given in Equation (2). Theorem 2 does not hold for PINNs if the underlying PDE is nonlinear. How would Theorem 2 apply if the ODE were not limited to this special case?

(4) This paper claims that the nonlinear and non-polynomial nature of the activation function is crucial for convergence. However, I believe the convergence results (Theorem 3) may also hold with polynomial activation functions, provided that Condition (1) of Assumption 1 is further strengthened to ensure linear independence.

(5) From the title, the topic of the paper is exploring the smoothness of activation functions in neural ODEs. However, after reading your paper, your main contribution is developing the global convergence of GD in training neural ODEs. The title is quite strange...

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645429035

thread_type:
official_review

<thread_object>
creator: reviewer_zcba

<content>
summary:
The authors studied activation functions in the training dynamics of neural ODEs. They showed that the choice of activation function has significant impacts on the dynamic and asymptotic behavior. Additionally, they propose a new mathematical framework to study continuous models from the perspective of approximation theory. Finally, the authors conduct experiments to empirically validate their theoretical results.

strengths:
The paper is well-written. The ideas presented are novel and represent a significant contribution in the context of neural ODEs. The main strength is the theoretical foundation of the work.

weaknesses:
The proofs are generally well-written and clear, but since they are in the appendix, the flow of the paper's main narrative can sometimes be interrupted.

questions:
No questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694589578

thread_type:
official_review

<thread_object>
creator: reviewer_e13d

<content>
summary:
This paper investigates the role of activation functions in the training dynamics of Neural ODEs, particularly focusing on the convergence and stability properties of two key methods: "Optimize-then-discretize" and "Discretize-then-optimize." The authors establish theoretical guarantees for the convergence of both methods under certain conditions on the activation functions, and they provide numerical experiments to support their findings.

strengths:
Theoretical Analysis: The paper offers a rigorous mathematical analysis of the well-posedness of Neural ODE solutions and their gradients, focusing on the impact of activation function smoothness. The authors clearly demonstrate the importance of Lipschitz continuity for both the activation function and its derivative to ensure stable and accurate gradient computation.

NTK Analysis: The paper extends the concept of the Neural Tangent Kernel (NTK) to the continuous setting of Neural ODEs. This is a valuable contribution, as it provides insights into the training dynamics of these models. The authors show that the NTK converges to a deterministic limit under certain conditions, which helps explain the convergence properties of gradient descent in Neural ODE training.

Global Convergence Proof: The authors establish the global convergence of Neural ODEs under gradient descent in the overparameterized regime, under assumptions about the activation function's nonlinearity and non-polynomial nature. This theoretical result provides a strong foundation for understanding the optimization behavior of Neural ODEs.

Empirical Validation: The paper presents several numerical experiments to validate the theoretical findings. These experiments demonstrate the convergence of Neural ODE outputs and gradients to their finite-depth ResNet approximations, the behavior of the NTK, and the convergence properties of Neural ODEs under gradient descent.

weaknesses:
Lack of Demonstration for "Discretize-then-optimize" Convergence Issue: While the paper theoretically analyzes the convergence problem of "Discretize-then-optimize" with non-smooth activation functions, it does not provide a dedicated numerical experiment to showcase this issue. This would be valuable for empirically confirming the theoretical claims and for understanding the practical implications of using non-smooth activations with this method.

The lack of a numerical demonstration of the "Discretize-then-optimize" convergence problem weakens the overall impact of the paper. Without empirical evidence, the theoretical claims regarding the convergence issue remain unsubstantiated, and the practical implications of using non-smooth activation functions with this method remain unclear.

questions:
1.
Could you provide a numerical experiment that specifically highlights the convergence problem of the "Discretize-then-optimize" method when using an activation function that does not meet the Lipschitz continuity requirements? This would strengthen the paper by providing empirical support for your theoretical analysis.

2.
How does the choice of numerical ODE solver in the "Optimize-then-discretize" method impact the accuracy of the gradient computation and the overall training dynamics? Exploring the sensitivity of the method to different solvers would be valuable.

3.
What are the practical implications of your findings for the choice of activation functions in real-world applications of Neural ODEs? Providing guidance on activation function selection based on the desired trade-off between theoretical guarantees and empirical performance would be beneficial.

Addressing these points would significantly improve the paper and provide a more comprehensive understanding of the impact of activation functions on Neural ODE training.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712084942

thread_type:
official_review

<thread_object>
creator: reviewer_c5fc

<content>
summary:
This paper presents several theoretical results about Neural ODEs, built around sufficient regularity of activations:
- It proves that "optimize then discretize" setting, built upon the use of classical ODE solvers, is well-posed under the assumption of Lipschitz-continuous activations (proposition 1).
- It proves that "discretize-then-optimize" setting, built as an Euler scheme discretization of the ODE by growing the depth of a residual neural network, is also well posed in the sense that the solution of the discretization scheme converges to the actual solution of the ODE in the infinite depth limit, if the activations and there derivatives are lipschitz-continuous (proposition 2). It has to be noted that authors gives explicit convergence rates.
- It shows that under the "discretize-then-optimize" setting with fixed depth L, the associated residual network converges in infinite width limit to a fixed kernel with explicit form, under the assumption of Lipschitz continuity of activations (proposition 3).
- It shows that under the assumption of Lipschitz continuity of activations, the infinite depth and width limits commute, implying that the neural ODE converges to a Gaussian process with covariance given as the infinite depth limit of the covariance of proposition 3 (theorem 1).
- Similar results are shown for the convergence of the NTK (proposition 4 and theorem 2)
- In section 5, authors show that the resulting limiting covariance of the gaussian process as well as NTK are Semi-positive definite (lemma 3, proposition 5, corrolary 1), resulting in derivation of explicit convergences rates (theorem 3).
Finally empirical evidences of the theoretical results are presented in section 6.

strengths:
- precise, rigorously demonstrated theoretical results
- clear presentation
- extension of fundamental NTK results to neural ODEs, using classical ODE analytical tools (Picard theorem, Groenwald theorem etc).

weaknesses:
- Typo : the index of the last h in equation 1 should probably be T instead of t
- Figure 1 : a,b,c should rather be in log-log scale to better appreciate the slope comparison with 1/L.

questions:
I don't think it's quite right (or at least I don't understand exactly why, but I'm not a specialist in Neural ODEs) that the title of the paper focuses on the impact of activations and their regularities, whereas most of your results use these as hypotheses for theorem proving. In my opinion, you should assume a more theoretical title (or present more results comparing activations, but it seems to me that this is a slightly different job than what you present, which is amply sufficient from a theoretical point of view).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732008016135

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all reviewers for their thoughtful and constructive comments, which have significantly enhanced the quality of our work. Based on your invaluable feedback, we conducted additional experiments and expanded our discussions to address your suggestions. These updates are primarily included in the appendix, ensuring the core contributions of the paper remain clear while providing deeper insights into the discussed topics.

Specifically, we added the following sections:
- Appendix H.6 (Non-Smooth Activation Functions): This section presents experiments comparing the “Discretize-then-optimize” and “Optimize-then-discretize” frameworks for non-smooth activations, such as ReLU. We provide empirical evidence supporting our theoretical claims about gradient alignment challenges.
- Appendix H.7 (Sensitivity of “Optimize-then-discretize” to ODE Solvers): This section investigates the impact of different numerical solvers on training dynamics and gradient computation accuracy, providing practical guidance for solver selection in Neural ODE applications.
- Appendix I (Discussion on General Dynamic Form in Neural ODEs): Here, we extend the theoretical results to more general dynamic forms, including non-autonomous systems and broader nonlinear mappings, discussing the necessary conditions for the applicability of our results.
- Appendix J (Related Works): We provide a comprehensive review of related works, situating our contributions within the broader context of Neural ODEs and NTK theory. This section acknowledges foundational studies and highlights how our work builds on and extends existing research.

We are grateful for your thoughtful suggestions, which guided us to clarify and enrich our contributions further. These additions not only strengthen the theoretical and empirical aspects of our work but also provide practical insights for the community. Thank you for helping us improve the paper’s quality and impact.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the impact of activation functions on Neural ODE training. It shows that smoothness of the activations are essential for stabilizing training and guaranteeing a unique solution. In addition, the paper extends NTK theory to continuous-layered models, showing a well-defined NTK for Neural ODEs. Crucially, higher nonlinearity in the activation function not only preserves desirable NTK properties, but also accelerates convergence in practice. In essence, the choice of activation function significantly impacts Neural ODE stability and training efficiency.

The paper received positive reviews, all of which appreciated its theoretical rigor, clarity of presentation, and the significance of its findings, resulting in a final rating of 8 across the board. Reviewer C5fC noted a few typos and offered suggestions for improvement. Reviewer E13D highlighted the lack of demonstration for the "discretize-then-optimize" convergence issue, prompting the authors to conduct further experiments. This reviewer also inquired about the sensitivity of the "optimize-then-discretize" method to numerical ODE solvers, to which the authors responded with additional experiments comparing different solvers. Consequently, E13D raised their score. Reviewer Hphg inquired about convergence with polynomial activations and generalization beyond specific ODE forms, while also pointing out some missing references. The authors either provided clarification or revised the manuscript to address these points, leading the reviewer to increase their initial rating to 8.

In agreement with the reviewers, I find the paper to be well-written and the presented results on the effect of activation functions on neural ODE stability and training efficiency to be of interest to the community and recommend accept.