Title: Nyström-Accelerated Primal LS-SVMs: Breaking the $O(an^3)$ Complexity Bottleneck for Scalable ODEs Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Closed-form approximate solution

Nyström-accelerated learning theory

least squares support vector machines (LS-SVMs)

ordinary differential equations (ODEs)
Abstract: A major problem of kernel-based methods (e.g., least squares support vector machines, LS-SVMs) for solving linear/nonlinear ordinary differential equations (ODEs) is the prohibitive $O(an^3)$ ($a=1$ for linear ODEs and 27 for nonlinear ODEs) part of their computational complexity with increasing temporal discretization points $n$. We propose a novel Nyström-accelerated LS-SVMs framework that breaks this bottleneck by reformulating ODEs as primal-space constraints. Specifically, we derive for the first time an explicit Nyström-based mapping and its derivatives from one-dimensional temporal discretization points to a higher $m$-dimensional feature space ($1< m\le n$), enabling the learning process to solve linear/nonlinear equation systems with $m$-dependent complexity. Numerical experiments on sixteen benchmark ODEs demonstrate: 1) $10-6000$ times faster computation than classical LS-SVMs and physics-informed neural networks (PINNs), 2) comparable accuracy to LS-SVMs ($<0.13\%$ relative MAE, RMSE, and $\left \| y-\hat{y}  \right \| _{\infty } $difference)  while maximum surpassing PINNs by 72\% in RMSE, and 3) scalability to $n=10^4$ time steps with $m=50$ features. This work establishes a new paradigm for efficient kernel-based ODEs learning without significantly sacrificing the accuracy of the solution.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on solving ordinary differential equations using kernel methods, with the focus being on the use of Least-Squares SVMs (LS-SVMs). Similar to the classic supervised learning setting, LS-SVMs can struggle on large-scale tasks owing to their cubic scaling. In the ODE setting, the cost of LS-SVMs, scales cubically with $n$, the number of timepoints, making the method prohibitive as the number of timepoints becomes large.\n\nTo address this challenge, the paper adapts the classic Nyström method to their setting, and show that it reduces the cost to $O(m^3)$, where $m$ is the number of landmarks selected. Numerical experiments on several benchmark ODEs, shows the proposed method exhibits good performance relative to the LS-SVM baseline and PINNs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThe main strength of the paper is that the proposed method is a lot faster than the LS-SVMs baseline without compromising performance too much. The method also outperforms the basic PINNs baseline on certain problems.\n\n**Weaknesses**\n\nThe paper has many weaknesses, which I summarize below, that make it inappropriate for publication at Neurips.\n\n*Lack of theoretical support*\n\n    The paper does not perform any theoretical analysis that would help validate the empirical \n    effectiveness of the proposed algorithm.\n    This is a significant weakness in my view, in light of the lack of novelty of the approach (see below), \n    and the fact that most papers that use the Nyström provide strong theoretical guarantees on its performance, i.e. [1], [2], [3], [4].\n\n*Lack of Novelty*\n\n    The Nyström method is a classic approach for reducing the cost of kernel methods, from SVMs to \n    kernel ridge regression. \n    The main contribution of the paper seems to just be applying this technique to the setting of solving ODEs \n    and verifying it works well empirically.\n    As mentioned in my previous point, the work introduces no new theory for this specific application that justifies the effectiveness of the method.  \n    While the use of Nyström might be new in the context of solving ODEs, by itself this is not enough to warrant \n    acceptance at Neurips. \n    Especially since prior works in Neurips such as [5], that use the Nyström method to accelerate SciML come with strong theoretical support.\n\n*Weak baselines* \n\n    The baselines used for empirically evaluating the method are quite weak to me. \n    \n    The baseline LS-SVMs method is from 2012. \n    The main bottleneck of this approach is the need to solve kernel linear systems in Newton's method. \n    Since this method was introduced, the sketching and randomized linear algebra literature have developed efficient \n    methods for solving large kernel systems [6,7,8,9], as well as Newton's method for solving non-linear equations [9]. \n    Rather than solving the system that arises in solving the LS-SVM method using a direct method \n    which naively costs $O(n^3)$, \n    a more reasonable comparison would be to use one of the randomized iterative methods I just referenced to solve the \n    Newton system, which cost at most $O(n^2)$. \n\n    As for the PINNs baseline,  the vanilla PINN is very rarely used in practice, as it is known to exhibit failure modes [8, 9], which has spawned the development of much more effective variants. \n    The empirics would be more convincing if you compared with a more sophisticated PINN architecture, \n    ideally one tailored to solving ODEs or a specific class of ODE.\n\n*Related work*\n\n    The Nyström method has a rich history and has played a significant role \n    in making large-scale kernel computations possible, and the present paper is missing quite a few references. \n    Works 1-5, as well as 8-9 below should all be cited as examples of how the Nyström method has accelerated \n    kernel learning.\n\n**Summary**\n\n    In its present form this submission is not a good fit for Neurips. \n    The submission develops no new theory and the numerical baselines are too weak to demonstrate whether or not the proposed method represents a real practical advance. \n\n    If the authors do not plan to develop more theoretical support for their method, \n    I would recommend instead that they improve the numerical baselines and submit to a \n    computational science/engineering journal as it would be a better fit.\n\n**References**\n\n[1] Rudi, A., Camoriano, R. and Rosasco, L., 2015. Less is more: Nyström computational regularization. Advances in Neural Information Processing Systems, 28.\n\n[2] Rudi, A., Carratino, L. and Rosasco, L., 2017. Falkon: An optimal large scale kernel method. Advances in Neural Information Processing Systems, 30.\n\n[3] Marteau-Ferey, U., Bach, F., & Rudi, A. (2019). Globally convergent newton methods for ill-conditioned generalized self-concordant losses. Advances in Neural Information Processing Systems, 32.\n\n[4] Della Vecchia, A., De Vito, E., Mourtada, J. and Rosasco, L., 2024. The Nyström method for convex loss functions. Journal of Machine Learning Research, 25(360), pp.1-60.\n\n[5] Meanti, G., Chatalic, A., Kostic, V., Novelli, P., Pontil, M., & Rosasco, L. (2023). Estimating Koopman operators with sketching to provably learn large scale dynamical systems. Advances in Neural Information Processing Systems, 36, 77242-77276. \n\n[6] Tu, Stephen, Shivaram Venkataraman, Ashia C. Wilson, Alex Gittens, Michael I. Jordan, and Benjamin Recht. \"Breaking locality accelerates block Gauss-Seidel.\" In International Conference on Machine Learning, pp. 3482-3491. PMLR, 2017.\n\n[7] Gazagnadou, Nidham, Mark Ibrahim, and Robert M. Gower. \"RidgeSketch: a fast sketching based solver for large scale ridge regression.\" SIAM Journal on Matrix Analysis and Applications 43, no. 3 (2022): 1440-1468.\n\n[8] Frangella, Zachary, Joel A. Tropp, and Madeleine Udell. \"Randomized nyström preconditioning.\" SIAM Journal on Matrix Analysis and Applications 44, no. 2 (2023): 718-752.\n\n[9] Díaz, Mateo, Ethan N. Epperly, Zachary Frangella, Joel A. Tropp, and Robert J. Webber. \"Robust, randomized preconditioning for kernel ridge regression.\" arXiv preprint arXiv:2304.12465 (2023).\n\n[10] Yuan, Rui, Alessandro Lazaric, and Robert M. Gower. \"Sketched Newton--Raphson.\" SIAM Journal on Optimization 32, no. 3 (2022): 1555-1583.\n\n[11] Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R. and Mahoney, M.W., 2021. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34, pp.26548-26560.\n\n[12] Rathore, Pratik, Weimu Lei, Zachary Frangella, Lu Lu, and Madeleine Udell. \"Challenges in training PINNs: A loss landscape perspective.\" arXiv preprint arXiv:2402.01868 (2024).\n\nQUESTIONS:\nI provided suggestions for improvement in my reply under strengths and weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Nyström-accelerated LS-SVM framework that reduces the $\\mathcal{O}(n^3)$ complexity of kernel-based ODE solvers to $\\mathcal{O}(m^3)$ using explicit primal-space formulations. The method enables efficient handling of linear/nonlinear ODEs through closed-form feature mapping and derivative computations. Experiments on fifteen benchmark problems show substantial speedups over LS-SVMs and PINNs with comparable or better accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. By introducing Nyström approximation in the primal space, the method effectively reduces the computational cost of LS-SVM-based ODE solvers from $\\mathcal{O}(n^3)$ to $\\mathcal{O}(m^3)$, improving scalability for large time steps and high-resolution settings.\n2. The paper derives explicit expressions for the feature mapping $\\varphi(t)$ and its derivatives, enabling model training without relying on the kernel trick.\n3. The paper is well-structured and clearly written, and the open-sourced code facilitates reproducibility and follow-up research.\n\nWeaknesses:\n1. The method employs simple uniform random sampling to select landmarks, which may lead to suboptimal approximation quality in highly nonlinear ODEs. The choice of sampling strategy could impact robustness and generalization.\n2. While the paper systematically integrates Nyström approximation into LS-SVM-based ODE solvers, it lacks theoretical analysis on subspace stability and solver convergence in nonlinear settings.\n3. The PINN baseline used in comparison adopts a shallow architecture with identity activation, without benchmarking against more recent improvements in the PINN literature.\n4. The experiments mainly focus on first/second-order ODEs, with no discussion of higher-order or real-world systems, limiting the evaluation scope.\n5. Mathematical notation could be further clarified, and visual explanations (e.g., diagrams) are insufficient. Specific concerns are outlined in the Questions section.\n\nQUESTIONS:\n1. The use of uniform random sampling for Nyström landmarks may limit approximation quality in nonlinear ODEs. Have you considered evaluating more informed strategies (e.g., leverage score sampling), or include an ablation study to show how sampling affects stability and performance.\n2. The iterative optimization solver is critical in nonlinear ODEs, yet the paper provides no theoretical discussion on its convergence or stability behavior. A brief analysis of convergence would help build confidence in the solver’s reliability.\n3. The PINN used for comparison is based on a shallow architecture with identity activation, which may not reflect the performance of recent PINN variants. It would be helpful if the authors could justify the architectural choice and consider recent variants, especially those designed to reduce computation costs.\n4. The experiments focus primarily on first-/second-order ordinary differential equations, without evaluating higher-order systems or real-world dynamics. A discussion on the potential generalizability of the method to more complex systems would be valuable. Adding even one case study on a higher-order or real-world problem would significantly enhance the paper's applicability.\n5. Adding a schematic pipeline diagram and more visualization of the results would improve clarity. In addition, the notation could be made more consistent throughout the paper, for example, symbols like $\\varphi_k^p $ and $f_k$​ appear in both bold and non-bold forms across different equations and lines (e.g., Eq.(13), Eq.(15) , Eq.(16) ,Eq.(17), Eq.(18) and line 177), which may confuse readers.\n6. While the paper adopts an RBF kernel, it does not explain why this choice was made or how it compares with other kernels. It is suggested that author includes an empirical comparison with alternative kernels, along with a basic sensitivity analysis for RBF kernel parameters.\n7. Since the method involves iterative optimization, it would be important to clarify whether the results in Appendix A.7 and A.8 are based on single runs or averages over multiple trials. The definition and measurement procedure for the reported \"Time\" metric should also be clearly described.\nIf the authors address these concerns or provide convincing clarification, I would be open to revisiting my final evaluation.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for solving linear and nonlinear ordinary differential equations (ODEs) based on kernel-based approaches. Although kernel-based strategies have been employed to solve ODEs, the computational complexity of such solvers scales as $O((n+p)^3)$ where $n$ is the number of discretization points and $p$ is the order of the ODE. This makes them impractical for settings where the simulation is long-term or has a fine-grained temporal resolution. To address this challenge, authors propose a new Nyström-accelerated LS-SVMs framework that uses the Nyström method to reduce the computational complexity to $O((m+p)^3)$ where $m<<n$ is the number of subsampled landmark points. The authors test the new approach empirically and report huge speed-ups compared to traditional LS-SVMs and PINNs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Important problem - quick ODE solving can have a huge impact on many areas\n- Clearly-written\n- Empirically validated\n- Huge reported speed-ups on the tested datasets with no or insignificant increase in error\n\nWeaknesses:\n- My main weakness concerns lack of comparison with standard approaches such as Runge-Kutta schemes. Is the approach also quicker or more accurate than those? Are there settings where kernel-based methods need to be used because those schemes are too inaccurate or computationally intensive? The paper mentions that \"these methods struggle with high-dimensional systems, noisy data assimilation, and adaptive mesh refinement\", but I could not find any reference or empirical result that would show it. \n- I think the technical novelty could be better emphasized. Was it challenging to apply Nyström method to LS-SVMs for ODE learning, or is it a straightforward combination of previous ideas? \n- Minor:\n\t- Typo in line 225. I think it should be \"second-order\".\n\nQUESTIONS:\n- How does this approach compare to Runge-Kutta schemes?\n- What are some of the technical challenges of applying the Nyström method to this setting?\n\nIf those two points are clarified, I'll be happy to consider increasing my score. Currently, it is not yet sufficiently clear to me how impactful and novel the proposed method is.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an efficient method to solve ordinary differential equations by using Nystrom approximation. There are several machine learning based methods for solving ODEs, and least square support vector machine (LS-SVM) is one of most successful among these ML methods. However, LS-SVM reformulate the ODE problem to a optimization problem, which involves solving a matrix of dimension n x n, where n is the number of discrete time steps. This incurs a O(n^3) time complexity, which is prohibitive for long time ODE solution/simulation. The authors use Nystrom method, which is a low-rank matrix approximation to reduce the time complexity to O(m^3), where m<<n is the number of subsampled points. This leads to 10-3000 times faster computation with little compromise on accuracy. Experimental results are provided to support the authors’ claim.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\nThe motivation for the authors work is explained well, detailed experimental results are provided. \n\nWeakness:\n1. The problem of computational complexity needs to be discussed in more detail. Currently, it is unclear where the number of equations n and O(n^3) complexity comes from. My understanding is that due to finite differences applied on n discrete points in time, there are n variables, which all need to be solved for, along with the p initial/boundary conditions. If this is the case, add a small description about this for readers unfamiliar with ODEs and numeric solvers.\n2. For a paper with Nystrom in the title, there could be more details about the Nystrom methods in the main manuscript. The authors at present put the details in the supplementary, however it is difficult to understand the methodology without an intuitive/conceptual idea about the Nystrom method. Maybe give small overview in a few sentences in section 3, before 3.1\n3. In equation 15, what are f_k, f_p?\n4. Is there a missing summation in equation 16? Summation over i maybe\n5. The authors can think about rearranging their experimental results. As the major claim of the paper is computational efficiency, charts and tables in the main manuscript should support that claim. Currently I think those are in the supplementary. From figure 1,2 the authors can select a subset of ODEs to show that the proposed method achieves comparable accuracy, while using the rest of the space to show evidence for 10-3000 times computation efficiency. \n6. The figures legends at present need to be updated/enlarged, it is almost impossible to read at present, figure 1,2. Figure captions should also be updated, highlight the key takeaway in the caption, for example: The results show that the proposed method achieved comparable accuracy with baseline methods …\n\nQUESTIONS:\nI am curious why the proposed method is particular for ODEs. Reducing the O(n^3) complexity will be super helpful for many applications, what makes the proposed method only specific for solving ODEs? Or maybe its not, in that case could you clarify where your method will be a good fit in addition to solving ODEs?\n\nLIMITATIONS:\nMentioned in strength and weakness section", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}