Title: SING: SDE Inference via Natural Gradients
Venue: NeurIPS 2025
Accepted: yes
Keywords: latent SDEs

dynamical systems

variational inference

natural gradient
Abstract: Latent stochastic differential equation (SDE) models are important tools for the unsupervised discovery of dynamical systems from data, with applications ranging from engineering to neuroscience. In these complex domains, exact posterior inference of the latent state path is typically intractable, motivating the use of approximate methods such as variational inference (VI). However, existing VI methods for inference in latent SDEs often suffer from slow convergence and numerical instability. Here, we propose SDE Inference via Natural Gradients (SING), a method that leverages natural gradient VI to efficiently exploit the underlying geometry of the model and variational posterior. SING enables fast and reliable inference in latent SDE models by approximating intractable integrals and parallelizing computations in time. We provide theoretical guarantees that SING will approximately optimize the intractable, continuous-time objective of interest. Moreover, we demonstrate that better state inference enables more accurate estimation of nonlinear drift functions using, for example, Gaussian process SDE models. SING outperforms prior methods in state inference and drift estimation on a variety of datasets, including a challenging application to modeling neural dynamics in freely behaving animals. Altogether, our results illustrate the potential of SING as a tool for accurate inference in complex dynamical systems, especially those characterized by limited prior knowledge and non-conjugate structure.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SDE Inference via Natural Gradients (SING) as a method that leverages natural gradient VI to improve inference in latent SDE models. The paper also describes a method for parallelisation, further improving efficiency and extends the approach to GP-SDEs. The paper provides extensive theoretical support and proofs as well as solid experimental support via validation on simulated and real-world data.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper provides a solid contribution to the field by introducing SING as a way to enable fast and reliable inference in latent SDE models. The claims are supported by solid proofs. The parallel approach yields convincing improvements. The experimental validation clearly demonstrate its benefits compared to methods like VDP. The paper is very well written in general.\n\nThere are a couple of points that warrant attention. \n\nThe diffusion term Sigma is assumed given. However, it can also be considered a model parameter. Can the theory be extended to incorporate fitting of Sigma? If yes, I suggest to incorporate this. If no, please explain why and explain why this is not an issue when fitting models on real-world data where the process noise is unknown.\n\nThe authors state that the method is closest to that of Verma et al. However, comparisons are with VDP. While the authors clearly describe the differences in the appendix I would have expected an explicit comparison in terms of empirical results. Please add these comparisons to the plots or show this using a targeted comparison in the appendix or make more explicit why such a comparison is irrelevant.\n\nQUESTIONS:\nIn related work, the authors should also describe alternative approaches to latent SDE estimation based on flow matching. See e.g. https://arxiv.org/abs/2502.02472. As these methods significantly improve on VI it would be important to contrast against these. Ideally the authors show competitive performance with these alternative approaches that have been introduced to address the limitations of VI.\n\nLIMITATIONS:\nThe assumption that the diffusion term is known can be acknowledged as a limitation if this cannot be addressed in the current setup.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a method for estimating latent stochastic differential equations (SDEs) through a natural gradient estimator. The natural gradient provides gradient updates that adhere to the geometry of the KL divergence. The model is assumed to have observations of a known link function applied to an affine transformation of a latent SDE, and the objective is to recover the affine transformation as well as the posterior latent distribution. The authors consider a family of linear SDEs for the variational distribution, which provides Gaussian time marginals and is amenable for use with natural gradient variational inference. The authors test the proposed method on a number of different experiments and illustrate the performance of the natural gradient estimator versus baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe authors provide a bound on the ELBO under a discrete setting to the continuous time setting that has the same order of convergence as the Euler Maruyama method. \n\nThe method empirically displays good performance on a variety of tasks. A number of comparisons are particularly impressive, for example the Adam vs NGVI plots are very informative.\n\nThe derivation of this formulation is neat and a slightly different take than many other papers along the lines of latent SDEs.\n\n**Weaknesses:** \n\nThe authors mention a variety of related work but do not consider comparisons to them. It would be interesting to see how these methods compare in both computation speed and memory usage. \n\nIt would be useful to get more context as to where using the natural gradient is more appropriate versus using standard gradient descent with an ELBO with a more flexible variational distribution. \n\nThe variational distribution being a linear SDE to be a major limitation especially since the volatility needs to be constrained quite heavily. On that note, the main techniques involve making the variational distribution Guassian and applying NGVI is not entirely original.\n\nThis field has a lot of related work and has been well studied from a variety of different angles, so it would be good to get more context on the contributions made in the paper compared with prior work. While this work focuses on the natural gradient variational inference under a linear SDE variational distribution for discovering SDE parameters, others have considered adjacent strategies. Just as an example of some related work missing: \n\nLatent SDEs on Homogeneous Spaces, NeurIPS 2023\n\nNeural Structure Learning with Stochastic Differential Equations, ICLR 2024\n\nIdentifying Latent Stochastic Differential Equations, IEEE Transactions on Signal Processing 2021\n\nLearning interpretable dynamics of stochastic complex systems from experimental data, Nature Communications 2024\n\nComputing high-dimensional invariant distributions from noisy data, Journal of Computational Physics 2023\n\nQUESTIONS:\nHow limiting is the variational distribution being a linear SDE? For example, since we approximate objects of the form $g(Cx+d)$, this would imply a limitation on the volatility, for example, of the process of $y$.\n\nIs computing the Jacobian in the ELBO expensive? Can the authors describe some more details on this. \n\nIs there a way to estimate the parameters of $g$ using this method? \n\nFor other SDEs (for example, GBM) where the volatility is a function of state, are there other ways to extend the method to these cases? The GBM case has log normal marginal distributions, so I assume something similar could work by reformulating in terms of the parameters of a log normal, but is this generally extensible? \n\nMany of the experiments suggest that the true drift can be recovered but I was wondering if a more concrete theoretical statement could be made regarding this? Along the lines of the previous questions, how does the constrained variational distribution affect this?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to use natural gradient VI for inference in latent SDEs, proposing an algorithm SING where a variational family of linear SDEs is prescribed over the latent dynamics, and the parameters of the drift and link function are learnable. The authors provide several numerical exploits to speed up inference with parallelization and provide illustrative and quantitative examples of their method in simulated and real data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is well written and compelling. It provides a pedagogical introduction to Bayesian inference problems for SDEs, natural gradient VI, etc. \n\nThe formulation of SING is rigorous and the use of a linear SDE variational family clearly leads to a statistically efficient and elegant model. This is evident from the mathematical derivations as well as the numerical experiments. \n\nThe extension to SING-GP is a nice addition and in my view it significantly broadens the relevance of the work, since in many applications the underlying class of drift functions would not be known.\n\nThe numerical results presented are polished and clearly illustrate the favourable behavior of the proposed algorithm. \n\nWeaknesses:\n\nIn all the examples shown, the latent dynamics are two-dimensional. The authors mention that this is a limitation of their method. I understand that this arises from the need to compute expectations in dimension $d$. The authors mention they use Gaussian quadrature, but it would be good to have an idea of what would happen if Monte Carlo estimation were used instead. The estimates would be unbiased, so some commentary on whether this stochastic approach is likely to work would be nice. \nIt is furthermore unclear from the related work section whether the baseline methods would also have the same scalability issues to higher dimensional problems.\n\nQUESTIONS:\n- I am concerned with the linear family for the posterior drift, since this would preclude dynamics that for instance exhibit multiple modes. Similarly, it seems to me that if the prior SDE allow for bistable behavior (e.g. bifurcations in the trajectories), using a linear variational family cannot model this. \n\n- On the other hand, the prior family on the drift could admit bifurcations for some values of $\\theta$. How does this reconcile with the linear variational family for the latent SDE? In particular, should one expect any connection between (1) the latent linear SDE learned by NGVI and (2) the learned vector field?\n\n- The authors mention several related works that use more general families of processes for inference of trajectories. Presumably these are more computationally intensive or less accurate overall. However, are there some scenarios (such as bifurcations) where more flexible model classes may help? Some discussion of this would be valuable I think. \n\n- Can anything be said in the case where the diffusion coefficient has some dependency on the state $x$? For instance, SDEs of the Cox-Ingersoll-Ross type. Such systems may have applications in some biophysics domains, e.g. [1]\n\n- As I understand it, SING infers a (variational) posterior over latent trajectories, but MAP estimate for the parameters $\\Theta$. Is extension to the fully Bayesian setting straightforward, or would it require new techniques?\n\n- For the question on the dimensionality of the latent dynamics, I think it would be valuable (and potentially not too much work) for some empirical results comparing the use of Gaussian quadrature to Monte Carlo for estimation of expectations. It seems to me that the most glaring limitation in the present work is that all examples are 2D; at least in the synthetic examples it would be good to see some results for moderate dimensions (2-5 dims). \n\n\n[1] Frishman A, Ronceray P. Learning force fields from stochastic trajectories. Physical Review X. 2020 Apr 1;10(2):021009.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method for inferring (latent) stochastic differential equations based on variational inference of natural parameters of probability densities represented in terms of families of natural parameters.\n\n\nThe work builds on the natural-gradient formulation of variational inference and extends existing literature in two directions. First, it provides a derivation for the closed-form natural-gradient updates for a latent SDEs discretised with the Euler–Maruyama scheme, exploiting the Gaussian structure of the discretised state marginals. Second, it introduces a parallelisation scheme for converting natural-to-mean parameters\nvia associative scans that reduce the computational cost from $\\mathcal{𝒪}(D^3 T)$ to $\\mathcal{𝒪}(D^3 log T)$ enabling thereby learning of long sequences.\n\nThe authors prove that the evidence lower bound (ELBO) computed on a finite grid converges to its continuous-time counterpart linking thereby the discrete-time and continuous-time objectives.\n\nThe authors validate their approach empirically on two synthetic datasets: one created with a latent linear dynamical system with Gaussian observations, and one with latent dynamics from a Van der Pol oscillator coupled with Poisson observations. In both settings, the proposed algorithm converges more quickly and recovers the latent dynamics more accurately than existing variational and filtering baselines. The authors further compare the approach against the variational diffusion process of Archambeau et al.. They further demonstrate their method on calcium imaging data for on modeling neural dynamics during aggression. \n\n \n\nWhile natural-gradient variational inference for SDEs is not entirely new, see for example  Verma et al. (2024), the present work differentiates itself through the parallelisation acceleration and the convergence analysis, both detailed in the appendix.\n\nI find the paper well fit for the NeurIPS audience.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper tackles a challenging problem, that of inference of latent stochastic systems, where the posterior inference is intractable for general systems.\n- The authors provide rigorous derivations.\n- They provide empirical evidence on both conjugate and non-conjugate models, as well as on synthetic and experimental datasets. \n\n\n**Weaknesses:**\n\n- The authors demonstrate the results relying only on measurements up to 10 neurons. While impressive I wonder how the performance and computational demands of the method would scale if the considered neurons were on the order of hundreds.\n- Variational inference with natural gradients has been introduced in previous work.\n\nQUESTIONS:\n- How would the method scale computationally with increasing latent dimension and/or increasing observation dimension (number of observed neurons)?\n\n- For the SING-GP, how do you select inducing points? How sensitive is the performance of the method to the number and the method of selection of inducing points?\n\n- Is there a reason you do not compare the synthetic datasets with the VDP-GP?\n\nLIMITATIONS:\nthe authors discuss the limitations of their approach in discussion and in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}