Title: Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs
Venue: NeurIPS 2023
Accepted: yes
Keywords: variational inference

differential equations

dynamical systems

neural ordinary differential equations

latent stochastic differential equations
Abstract: We consider the problem of inferring latent stochastic differential equations (SDEs) with a time and memory cost that scales independently with the amount of data, the total length of the time series, and the stiffness of the approximate differential equations. This is in stark contrast to typical methods for inferring latent differential equations which, despite their constant memory cost, have a time complexity that is heavily dependent on the stiffness of the approximate differential equation. We achieve this computational advancement by removing the need to solve differential equations when approximating gradients using a novel amortization strategy coupled with a recently derived reparametrization of expectations under linear SDEs. We show that, in practice, this allows us to achieve similar performance to methods based on adjoint sensitivities with more than an order of magnitude fewer evaluations of the model in training.

Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs

Abstract

We consider the problem of inferring latent stochastic differential equations (SDEs) with a time and memory cost that scales independently with the amount of data, the total length of the time series, and the stiffness of the approximate differential equations. This is in stark contrast to typical methods for inferring latent differential equations which, despite their constant memory cost, have a time complexity that is heavily dependent on the stiffness of the approximate differential equation. We achieve this computational advancement by removing the need to solve differential equations when approximating gradients using a novel amortization strategy coupled with a recently derived reparametrization of expectations under linear SDEs. We show that, in practice, this allows us to achieve similar performance to methods based on adjoint sensitivities with more than an order of magnitude fewer evaluations of the model in training.

1 Introduction

Recent years have seen the rise of continuous time models for dynamical system modeling [1]. As compared to traditional autoregressive style models [2], continuous time models are useful because they can deal with non-evenly spaced observations, they enable multilevel/hierarchical and adaptive prediction schemes, and because physics is (mostly) done in continuous time. For example, recent developments in inferring continuous time differential equations from data has been met by a flurry of work in endowing models with physics informed priors [3, 4, 5].

Despite their advantages, continuous time models remain significantly more computationally challenging to train than their autoregressive counterparts due to their reliance on adjoint methods for estimating gradients. Adjoint methods introduce a significant computational burden in training because they require solving a pair of initial value problems to estimate gradients. Solving such initial value problems as a part of an iterative optimization procedure is computationally demanding for the following reasons:

(i) Gradient based updates to models of differential equations can cause them to become extremely stiff. This will have the effect of causing the cost per iteration to explode mid-optimization.

(ii) With the exception of parareal methods [6], differential equation solvers are fundamentally iterative sequential methods. This makes them poorly suited to being parallelized on modern parallel computing hardware.

In accordance with such challenges a number of methods have been introduced to speed up training of continuous time models including regularizing dynamics [7, 8] and replacing ordinary differential equation (ODE) solvers with integration methods where possible [9]. Despite the computational advancements brought about by such innovations, continuous time models remain expensive to train in comparison to discrete time models.

Prasanth B. Nair

In addition to these computational challenges, it is well-known that adjoint methods suffer from stability issues when approximating gradients of time averaged quantities over long time intervals for chaotic systems [10].

In the current work, we present a memory and time efficient method for inferring nonlinear, latent stochastic differential equations (SDEs) from high-dimensional time-series datasets. In contrast to standard approaches for inferring latent differential equations that rely on adjoint sensitivities [1, 11, 12], our approach removes the requirement of solving differential equations entirely. We accomplish this advancement by coupling a novel amortization strategy with a recently derived reparametrization for expectations under Markov Gaussian processes [13]. We show that our approach can be used to approximate gradients of the evidence lower bound used to train latent SDEs with a time and memory cost that is independent of the amount of data, the length of the time series, and the stiffness of the approximate differential equations. The asymptotic complexity for our approach is compared to well-known methods from the literature in Table 1. We note that our method has a constant cost that is chosen by the user. Moreover, we will show that our method is embarrassingly parallel (i.e. all evaluations of the model can be performed in parallel over each iteration) whereas, we reiterate, differential equation solvers are iterative sequential methods.

The applications of our method span various standard generative modeling tasks, such as autoencoding, denoising, inpainting, and super-resolution [15], particularly tailored for high-dimensional time-series. Crucially, the computational efficiency of our approach not only enables the allocation of more computational resources towards hyperparameter tuning but also democratizes access to state-of-the-art methods by making them feasible to train on lower performance hardware.

In the next section we provide a description of the theory underpinning our work with the main result of a stochastic, unbiased estimate for gradients appearing in Lemma 1. In Section 4 we provide a number of numerical studies including learning latent neural SDEs from video and performance benchmarking on a motion capture dataset. Notably we show that we are able to achieve comparable performance to methods based on adjoints with more than one order of magnitude fewer evaluations of the model in training (Section 4.1). We also demonstrate that our approach does not suffer from the numerical instabilities that plague adjoint methods for long time-series with chaotic systems (Section 4.2). Finally, we close with a discussion of the limitations of our approach as well as some suggestions for future work. All code is available at github.com/coursekevin/arlatentsde.

2 Method

2.1 Problem description

Consider a time-series dataset D = { ( x i , t i ) } N i =1 , where t i ∈ R is the time stamp associated with the observation, x i ∈ R D . For example x i may be an indirect observation of some underlying dynamical system, a video-frame, or a snapshot of a spatio-temporal field. We assume that each observation was generated via the following process: first a latent trajectory, z ( t ) , is sampled from a general,

nonlinear SDE with time-dependent diffusion,

with initial condition p θ ( z 0 ) and then each x i is generated from the conditional distribution p θ ( x | z ( t i )) where z ( t i ) is the realization of the latent trajectory at time stamp t i . Here f θ : R × R d → R d is called the drift function, L θ : R → R d × d is called the dispersion matrix, and β denotes Brownian motion with diffusion matrix Σ . We will model both the drift function of the SDE and the conditional distribution using neural networks in this work. This combination of SDE and conditional distribution define the generative model that we wish to infer.

Like in the standard generative modeling setting, the 'true' parameters defining the latent SDE and the conditional likelihood, as well as the specific realization of the latent state, remain unknown. In addition, the posterior over the latent state at a point in time, p θ ( z ( t ) | D ) , is intractable. Our objectives are two-fold, we wish to: (i) infer a likely setting for the parameters, θ , that is wellaligned with the observed data and (ii) infer a parametric model for the posterior over the latent state at a particular point in time as a function of a small window of the observations, for example q ϕ ( z ( t ) | x i , x i +1 , . . . , x i + M ) ≈ p θ ( z ( t ) | D ) for t ∈ [ t i , t i + M ] and 1 ≤ M << N . This latent variable model is depicted in Figure 1. In this work we will tackle this problem statement using the machinery of stochastic variational inference [16].

x

i

z

(

t

)

θ

ϕ

N

Before proceeding with an explanation of our approach, it is worthwhile taking a moment to consider why assuming the latent state is a realization of a SDE, as opposed to an ordinary differential equation (ODE) with random initial conditions [1, 11, 17], merits the additional mathematical machinery. First, there is a long history in using SDEs in the physical sciences to model systems (even deterministic systems) that lack predictability due to super-sensitivity to parameters and or initial conditions [18]. One reason for the preference towards this modeling paradigm is that deterministic ODEs with random initial conditions only permit uncertainty to enter in their initial condition. This means that we are assuming that all randomness which may affect the latent trajectory at all future points in time are precisely encoded in the uncertainty in the initial condition. In contrast, SDEs make the less restrictive assumption that uncertainty accumulates over time.

2.2 Evidence lower bound

As is the case with standard latent variable models, training proceeds by specifying approximate posteriors over latent variables (in this case the latent state, z ( t ) ) and then minimizing the KullbackLeibler (KL) divergence between the approximate and true posterior. An added complication in our specific case comes from the fact that the latent state is defined by a stochastic process rather than a random-vector as is more typical. While it is possible to choose more general approximate posteriors [12], in this work we make the choice to approximate the posterior over the latent state as a Markov Gaussian Process (MGP) defined by the linear SDE,

with Gaussian initial condition, z ( t 0 ) ∼ N ( m 0 , S 0 ) where m 0 ∈ R d and S 0 ∈ R d × d is a symmetric positive definite. Here A ϕ : R → R d × d and b ϕ : R → R d are symmetric matrix and vector-valued functions of time, respectively. Before proceeding, it is worth emphasizing that this choice of approximate posterior is not as restrictive as it may appear since it is only used to approximate the posterior (i.e. the latent state given the observations p ( z ( t ) | D ) for t ∈ [ t 1 , t N ] ). When forecasting

beyond the data window, we will use the nonlinear SDE in (1). In this way, the approximate posterior primarily serves as a useful intermediary in training the generative model rather than the end-product.

For our purposes, MGPs are useful because the marginal distribution q ϕ ( z ( t )) = N ( m ϕ ( t ) , S ϕ ( t )) (i.e. the distribution of the latent state at time t ) is a Gaussian whose mean and covariance are given by the solution to the following set of ODEs [19],

where m (0) = m 0 and S (0) = S 0 .

Using the reparametrization trick from [13], the evidence lower bound (ELBO) can be written as,

where

cates the Kronecker sum, vec : R d × d → R d 2 maps a matrix into a vector by stacking columns, and vec -1 : R d 2 → R d × d converts a vector into a matrix such that vec -1 ( vec ( B )) = B ∀ B ∈ R d × d . See Appendices A and B for a detailed derivation. We note that computing the residual, r θ,ϕ , scales linearly in the dimension of the state so long as S ϕ ( t ) and C θ ( t ) are diagonal. To ensure our approach remains scalable, we will make this assumption throughout the remainder of this work. In this case, we have

Often we may wish to place additional priors onto a subset of the generative model parameters, θ , and infer their posterior using stochastic variational inference as well. In this case we add the KL-divergence between the approximate posterior and prior onto the ELBO, KL ( q ϕ ( θ ) || p ( θ )) and rewrite all expectations with respect to q ϕ ( z ( t )) and q ϕ ( θ ) ; details are provided in Appendix C.

Remark 1: Despite having defined the approximate posterior in terms of an SDE with parameters A ϕ and b ϕ , the ELBO only depends on the mean and covariance of the process at a particular point in time, m ϕ and S ϕ . For this reason we can parametrize m ϕ and S ϕ directly while implicitly optimizing with respect to A ϕ and b ϕ . In addition, we can efficiently compute ˙ m ϕ and ˙ S ϕ using automatic differentiation.

Remark 2: Despite the fact that the prior and approximate posterior are SDEs, all expectations in the ELBO are taken with respect to normal distributions . Moreover, in contrast to the approach in [20, 21] there are no differential equality constraints - instead we have been left with an integral over the window of observations.

Taken together, these observations allow us to infer the parameters of the generative model (a nonlinear, latent SDE with additive diffusion (1)), without the use of a forward solver.

2.3 Amortization strategy

The implicit assumption in the ELBO in (4) is that the state mean and covariance will be approximated over the entire window of observations. This can pose a serious computational bottleneck with long or complicated time-series. In this section we propose a novel amortization strategy that will allow us to effectively eliminate this cost by requiring that we only approximate the posterior over short partitions of total the data time-window at once.

Rather than attempting to compute and store the posterior over the entire latent trajectory, we will instead construct an approximation to the posterior over a small window of observations as a function of those observations. Consider a reindexing of the dataset by splitting it into N/M non-overlapping partitions where 1 ≤ M << N ,

In the case that N is not evenly divisible by M we allow the final split to contain less elements. We approximate the latent state over each partition using only the M observations in each partition, q ϕ ( z ( t ) | x ( j ) 1 , x ( j ) 2 , . . . , x ( j ) M ) ≈ p ( z ( t ) | D ) for t ∈ [ t ( j ) 1 , t ( j +1) 1 ] . This can be interpreted as a probabilistic encoder over the time interval of the partition of observations. Letting t N/M +1 1 ≡ t N/M M , the ELBO can be compactly rewritten as, L ( θ, ϕ ) = ∑ N/M j =1 L ( j ) ( θ, ϕ ) , where

An additional advantage of this amortization strategy is that it allows our approach to scale to multiple trajectories without an increase to the overall computational cost. If there are multiple trajectories, we can reindex each trajectory independently and subsequently sum all sub loss functions.

To reiterate, the probabilistic encoder is a function which takes in M observations from a particular partition along with a time stamp, t , and outputs a mean vector and covariance matrix as an estimate for the latent state at that particular time. In principle, any function which can transform a batch of snapshots and a time stamp into a mean and covariance could be used as an encoder in our work. In our implementation, we use deep neural networks to encode each x ( j ) i using i ∈ I where I ⊂ [ x ( j ) 1 , x ( j ) 2 , . . . , x ( j ) M ] contains some temporal neighbours of x i into a single latent vector. This approach yields a set of latent vectors associated with each observation in the partition h i for i = 1 , 2 , . . . , M . We then interpolate between each latent vector using a deep kernel based architecture to construct the posterior approximation for any time stamp in the partition; see Appendix D for details. We emphasize this is one choice of encoding architecture that we found convenient, it is straightforward to incorporate an autoregressive or transformer based encoder in our methodology [22]

An important consideration is the selection of the partition parameter, M . In practice, M should be large enough so that the assumption of a linear SDE for the approximate posterior is appropriate (i.e. we have enough observations in a partition so that the assumption of a Gaussian process over the latent state is reasonable). For example, as we will see in an upcoming numerical study, in the context of inferring latent SDEs from videos, we will need to choose M to be large enough so that we can reasonably infer both the position and velocity of the object in the video.

2.4 Reparametrization trick

While the previous sections have demonstrated how to eliminate the need for a differential equation solver by replacing the initial value problem with an integral, in this section we show how the reparametrization trick can be combined with the previously described amortization strategy to construct unbiased gradient approximations for the ELBO with a time and memory cost that scales independently with the amount of data, the length of the time series, and the stiffness of the approximation to the differential equations. Consider a reparametrization to the latent state of the form z ( t ) = T ( t, ϵ, ϕ ) where ϵ ∼ p ( ϵ ) so that z ( t ) ∼ q ϕ ( z ( t ) | x ( j ) 1 , x ( j ) 2 , . . . , x ( j ) M ) . We can rewrite the second term in the evidence lower bound as,

where p ( t ) is a uniform distribution, U ( t ( j ) 1 , t ( j +1) 1 ) and p ( ϵ ) ∼ N (0 , I ) is a Gaussian. With this rearrangement, we can derive the main result of this work.

Lemma 1. An unbiased approximation of the gradient of the evidence lower bound, denoted as ∇ θ,ϕ L ( θ, ϕ ) , with an O ( R ) time and memory cost can be formulated as follows:

where each t ( k ) ∼ U ( t ( j ) 1 , t ( j +1) 1 ) and each ϵ ( k ) ∼ N (0 , I ) .

The proof follows by applying the standard reparametrization trick [15] to estimating gradients of the amortized objective in (7).

Remark 1: In practice we found choosing R ∼ 100 worked well for the problems we considered. Note that in terms of elapsed time, 100 evaluations of this objective, which can be computed in parallel, is far cheaper than 100 evaluations of the SDE forward model evaluated as a part of an iterative sequential SDE solver. Moreover we found that adaptive stepping schemes required far more evaluations of the SDE forward model than our stochastic approach (see Section 4.1).

Remark 2: In the case that evaluations of the SDE drift term were relatively cheap compared to decoder evaluations (for example in the case the dimension of the latent state is much smaller than the dimension of the data), we found it useful to increase the number of samples used to approximate the integral over time without increasing the number of samples from the variational posterior. To do so, we made use of a nested Monte Carlo scheme to approximate the second term in the ELBO,

where, again, each ϵ ( k ) ∼ N (0 , I ) and each t ( k, 1) , t ( k, 2) , . . . , t ( k,S ) ∼ U ( t ( j ) 1 , t ( j +1) 1 ) . In addition, because the integral over time is one-dimensional we used stratified sampling to draw from U ( t ( j ) 1 , t ( j +1) 1 ) in order to further reduce the variance in the integral over time. In this case we often found we could choose R ∼ 10 and S ∼ 10 . To be clear, (10) is simply a method for variance reduction that we found to be useful; it is not a necessary component for our approach.

3 Limitations & Related Work

Summary of assumptions. In the previous sections we introduced an ELBO which, when maximized, leaves us with a generative model in the form of a nonlinear, latent SDE with time-dependent diffusion and an approximation to the latent state over the time-window of observations in the form of a Gaussian process. To reiterate, we only assume that the approximating posterior, i.e. the distribution over the latent state given a batch of observations, is a Gaussian process; this is an assumption that is commonly made in the context of nonlinear state estimation, for example [23, 24]. When making predictions, we sample from the nonlinear SDE which characterizes the generative model (1).

Stochastic adjoint sensitivities. Li et al. [12] proposed the stochastic adjoint sensitivity method, enabling the inference of latent SDEs using a wide range of approximate posteriors over the latent state. In our work we choose to approximate the posterior over the latent state using a MGP which enables us to eliminate the requirement of solving any differential equations entirely; as we have discussed extensively this choice enables dramatic computational savings. A limitation of our approach as compared to the stochastic adjoint sensitivities method is that our method should only be used to approximate the posterior over the latent state when it is approximately a MGP. Intuitively, this limitation is akin to the limitations of mean-field stochastic variational inference as compared

to stochastic variational inference with an expressive approximate posterior such as normalizing flows [25]. From our practical experience working on a range of test cases, this has not been a limiting factor. It is worth reiterating that this limitation applies only to the approximate posterior over the time window of observations; the predictive posterior can be a complex distribution defined by a nonlinear SDE with a Gausian initial condition.

In addition, the stochastic adjoint sensitivity method allows for state dependent diffusion processes whereas our approach only allows for a time dependent diffusion process. In cases where a state dependent diffusion process is deemed necessary, our approach could be used to provide a good initial guess for the parameters of the drift function. It remains a topic of future work to determine if this limitation is mitigated by the fact that we are learning latent SDEs rather than SDEs in the original data space. Across the range of test cases we considered, we have not encountered a problem for which the assumption of a time-dependent diffusion matrix was limiting.

Latent neural ODEs. Chen et al. [1], Rubanova et al. [11], and Toth et al. [17] presented latent ordinary differential equations (ODEs) as generative models for high-dimensional temporal data. These approaches have two main limitations: (i) they encode all uncertainty in the ODE's initial condition and (ii) they rely on adjoint sensitivities, necessitating the solution of a sequence of initial value problems during optimization. As was discussed previously, SDEs provide a more natural modeling paradigm for estimating uncertainty, naturally capturing our intuition that uncertainty should accumulate over time [18]. Moreover, to reiterate, our work avoids solving differential equations entirely by relying on unbiased approximations of a one-dimensional integral instead; as we will show, this can result in a dramatic decrease in the number of required function evaluations in training as compared to methods based on adjoints. Moreover, we will show that our approach avoids the numerical instabilities of adjoint methods when they are used to approximate gradients of time averaged quantities over long time intervals for chaotic systems. It is worth mentioning that gradients computed by backpropagation of a forward solver are not consistent with the adjoint ODE in general [26] so we do not consider comparisons to such approaches here.

Weak form methods. Methods for inferring continuous time models of dynamical systems using the weak form of the differential equations were introduced in the context of learning ODEs with linear dependence on the parameters [27, 28]. More recently these methods were adapted for training neural ODEs more quickly than adjoint methods for time-series prediction problems [5]. These methods share some similarities to the present approach in how they achieve a computational speed-up - both methods transform the problem of solving differential equations into a problem of integration. In contrast to the present approach, these methods only allow for one to learn an ODE in the data coordinates (i.e. they do not allow for one to infer an autoencoder and a set of differential equations simultaneously). Moreover, these methods rely on a biased estimate for the weak form residual which will fail when observations become too widely spaced. In contrast, in the present approach, we rely on unbiased approximations to the evidence lower bound. Finally, these methods require the specification of a carefully designed test-space [29] - a consideration not required by our approach.

4 Numerical Studies

In this section we provide a number of numerical studies to demonstrate the utility of our approach. In the first study, we show that our approach can be used to train neural SDEs using far fewer evaluations of the model than adjoint methods. In the second study, we consider the problem of parameter tuning for a chaotic system over long time intervals. We show that our approach does not suffer from the numerical instabilities which are known to cause issues with adjoint methods on problems such as these. Finally we close this section with two practical test cases: the first demonstrating competitive performance on a motion capture benchmark and the second showing how our approach can be applied to learn neural SDEs from video. An additional numerical study exploring the effect of the nested Monte Carlo parameter, S , is provided in Appendix H. Details on computing resources are provided Appendix F. All code required to reproduce results and figures is provided at github.com/coursekevin/arlatentsde.

2

3

4

5

log

NFE

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

log

10

Validation RMSE

ARCTA

NODE, tol=10

-

2

NODE, tol=10

-

4

NODE, tol=10

-

6

35

40

45

50

55

60

65

Time

Test

Train

10

4.1 Orders of magnitude fewer function evaluations in training

In this numerical study we consider the task of building a predictive model from noisy observations of a predator-prey system. We simulated the Lotka-Volterra equations for 50 seconds collecting data at a frequency of 10 Hz. Observations were corrupted by Gaussian noise with a standard deviation of 0 . 01 . Validation data was collected over the time inverval [50 , 65] seconds. We then attempt to build a predictive model from the data using a neural ODE (NODE) and our method, amortized reparametrization for continuous time auto-encoding (ARCTA), with the same model for the ODE and drift function respectively. To make comparisons with the NODE fair, we set the decoder to be the identity function. We assume the diffusion matrix is constant and place a log-normal prior on its diagonal elements. We approximate the posterior over these elements using a log-normal variational posterior. Details on the architecture and hyperparameters are provided in Appendix G.1. For this experiment, as well as subsequent experiments, we made use of the Adam optimizer [30].

We considered three different tolerances on the NODE adaptive stepping scheme. We trained our model as well as the NODEs using 10 different random seeds while recording the validation RMSE and the number of evaluations of the model. Looking to Figure 2, we see that our approach required more than an order of magnitude fewer evaluations of the model to achieve a similar RMSE on the validation set. This remains true even when the tolerance of the ODE solver is reduced such that the validation RMSE is substantially higher than our approach.

4.2 Numerical instabilities of adjoints

It is well-known that adjoint based methods produce prohibitively large gradients for long time averaged quantities of chaotic systems [10] and accordingly methods, such as least squares shadowing [31], have been introduced to address such concerns. In this section we reproduce this phenomena on a simple parameter tuning problem and show that our approach does not suffer these same issues.

Given the parametric form of the chaotic Lorenz equations,

along with an initial guess for the parameters, σ 0 , ρ 0 , and β 0 , our goal is to tune the value of parameters such that they align with the observed data.

For this experiment we collect data at a frequency of 200Hz and corrupt observations by Gaussian noise with a covariance of 1 . We generate five independent datasets over the time intervals [0 , 1] , [0 , 10] , [0 , 50] , and [0 , 100] . For each dataset we generated an initial guess for the parameters by sampling from a Gaussian whose mean is the true value of the parameters and standard deviation is 20 %of the mean. For the adjoint methods we report the ℓ 2 -norm of the gradient with respect to the parameters at the initial guess. For our method (ARCTA) we optimize for 2000 iterations (which tended to be enough iterations to successfully converge to a reasonable solution) and report the average gradient across all iterations. Details on hyperparameters and our architecture design are provided in Appendix G.2. Results are summarized in Figure 3. While adjoints expectedly provide prohibitively large gradients as the length of the time series is increased, our approach remains numerically stable.

4.3 Motion capture benchmark

1.0

10.0

50.0

100.0

T

(

s

)

10

0

10

5

10

10

10

15

10

20

10

25

10

30

10

35

10

40

10

45

log

10

||∇

θ

ARCTA

Adjoints

-

25

0

25

x

0

-

25

0

25

x

1

0

5

10

15

20

25

30

t

(

s

)

0

25

x

2

In this experiment we consider the motion capture dataset from [32]. The dataset consists of 16 training, 3 validation, and 4 independent test sequences of a subject walking. Each sequence consists of 300 time-series observations with a dimension of 50 . We made use of the preprocessed data from [34]. Like previous approaches tackling this dataset, we chose a latent dimension of 6 . We assume a Gaussian observation likelihood. We place a log-normal prior on the diagonal elements of the diffusion matrix and the noise on the observations. We approximate the posterior of the diffusion matrix and observation noise covariance using a log-normal approximate posterior. Details on our architecture design and hyperparameter selection are provided in Appendix G.3.

For our approach, we train 10 models and report their average performance on the test set due to the extremely limited number (4) of independent test sequences. Looking to Figure 4, we see that our approach provided competitive performance on this challenging dataset. This result, in combination with those presented previously demonstrating we require fewer function evaluations for similar

forecasting accuracy and improved gradient stability for chaotic systems, make clear the utility of the present work. It is possible to achieve state-of-the-art performance at a significantly reduced computational cost as compared to adjoint based methods.

4.4 Neural SDEs from video

0

5

10

15

20

25

30

t

(

s

)

-

2

0

2

z

(

t

)

In this experiment we attempt to learn a latent SDE from video. We generated 32 × 32 black and white frames of a nonlinear pendulum as it evolves for 30 seconds collecting data at a frequency of 15 Hz. We transform the 1024 dimensional state down to two dimensions using a convolutional architecture. Details on the hyperparameters and architecture are provided in Appendix G.4. This problem is similar to the problem considered in [3] except the dynamical system we consider is nonlinear. In this prior work, the authors were forced to regularize the latent space so that one set of coordinates resembles a generalized velocity. In the present work, no such regularization is required.

We assume a Bernoulli likelihood on the pixels. Like in previous numerical studies we place a lognormal prior on the diagonals of the diffusion term and approximate the posterior using a log-normal variational distribution. After training we generate a forecast that is visualized in Figure 5. We see that we were successfully able to build a generative model for this simple video. This result demonstrates the broad applicability of the present approach to standard generative modeling tasks.

5 Conclusions

Here we have presented a method for constructing unbiased approximations to gradients of the evidence lower bound used to train latent stochastic differential equations with a time and memory cost that scales independently with the amount of data, the length of the time-series, and the stiffness of the model for the latent differential equations. We achieve this result by trading off the numerical precision of adaptive differential equation solvers with Monte-Carlo approximations to expectations using a novel amortization strategy and a recently derived change of variables for expectations under Markov Gaussian processes [13].

We have demonstrated the efficacy of our approach in learning latent SDEs across a range of test problems. In particular we showed that our approach can reduce the number of function evaluations as compared to adjoint methods by more than one order of magnitude in training while avoiding the numerical instabilities of adjoint methods for long time series generated from chaotic systems. In addition, we showed that our approach can be used for generative modeling of a simple video.

In the immediate future, there is significant room for future work in applying variance reduction schemes to the expectation over time to further reduce the total number of required function evaluations. There are also opportunities to explore the utility of the proposed approach for generative modeling on more realistic problems. Finally, there are opportunities to apply our work in the context of implicit densities [35].

Acknowledgments and Disclosure of Funding

This research is funded by a grant from NSERC.

References

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations. In Advances in neural information processing systems , pages 6571-6583, 2018.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997. Publisher: MIT press.

Sam Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. In Advances in Neural Information Processing Systems , pages 15353-15363, 2019.

Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian Neural Networks. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations , 2020.

Kevin Course, Trefor Evans, and Prasanth B. Nair. Weak Form Generalized Hamiltonian Learning. In Advances in Neural Information Processing Systems , volume 33, pages 18716-18726, 2020.

Emmanuel Lorin. Derivation and analysis of parallel-in-time neural ordinary differential equations. Annals of Mathematics and Artificial Intelligence , 88:1035-1059, 2020. Publisher: Springer.

Jacob Kelly, Jesse Bettencourt, Matthew J Johnson, and David K Duvenaud. Learning differential equations that are easy to solve. Advances in Neural Information Processing Systems , 33:4370-4380, 2020.

Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ODE: the world of Jacobian and kinetic regularization. In International conference on machine learning , pages 3154-3164, 2020.

Patrick Kidger, Ricky T. Q. Chen, and Terry J Lyons. "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5443-5452, July 2021.

Daniel J Lea, Myles R Allen, and Thomas WN Haine. Sensitivity analysis of the climate of a chaotic system. Tellus A: Dynamic Meteorology and Oceanography , 52(5):523-532, 2000. Publisher: Taylor & Francis.

Yulia Rubanova, Tian Qi Chen, and David K Duvenaud. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. In Advances in Neural Information Processing Systems , pages 53215331, 2019.

Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable Gradients for Stochastic Differential Equations. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 3870-3882, August 2020.

Kevin Course and Prasanth B. Nair. State estimation of a physical system with unknown governing equations. Nature , 622(7982):261-267, October 2023.

Mike Giles and Paul Glasserman. Smoking adjoints: Fast monte carlo greeks. Risk , 19(1):88-92, 2006.

Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations , 2014.

Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research , 14(1):1303-1347, 2013. Publisher: JMLR. org.

Peter Toth, Danilo J. Rezende, Andrew Jaegle, Sébastien Racanière, Aleksandar Botev, and Irina Higgins. Hamiltonian Generative Networks. In International Conference on Learning Representations , 2020.

James Glimm and David Sharp. Stochastic Partial Differential Equations: Selected Applications In Continuum Physics. In Stochastic Partial Differential Equations: Six Perspectives, Mathematical Surveys and Monographs , pages 3-44. American Mathematical Society, 1997.

Simo Särkkä and Arno Solin. Applied stochastic differential equations , volume 10. Cambridge University Press, 2019.

Cedric Archambeau, Dan Cornford, Manfred Opper, and John Shawe-Taylor. Gaussian process approximations of stochastic differential equations. In Gaussian Processes in Practice , pages 1-16. PMLR, 2007.

Cédric Archambeau, Manfred Opper, Yuan Shen, Dan Cornford, and John Shawe-taylor. Variational Inference for Diffusion Processes. In Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Timothy D. Barfoot. State estimation for robotics a matrix Lie group approach . Cambridge University Press, Cambridge, United Kingdom, 2017. ISBN 978-1-107-15939-6.

Timothy D Barfoot, James R Forbes, and David J Yoon. Exactly sparse Gaussian variational inference with application to derivative-free batch nonlinear state estimation. The International Journal of Robotics Research , 39(13):1473-1502, 2020. Publisher: SAGE Publications Sage UK: London, England.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015.

Mihai Alexe and Adrian Sandu. On the discrete adjoints of adaptive time stepping algorithms. Journal of computational and applied mathematics , 233(4):1005-1020, 2009. ISSN 0377-0427. doi: 10.1016/j.cam. 2009.08.109. Place: Kidlington Publisher: Elsevier B.V.

Hayden Schaeffer and Scott G. McCalla. Sparse model selection via integral terms. Physical Review E , 96 (2):023302, August 2017. doi: 10.1103/PhysRevE.96.023302.

Yannis Pantazis and Ioannis Tsamardinos. A unified approach for sparse dynamical system inference from temporal measurements. Bioinformatics , 35(18):3387-3396, September 2019. ISSN 1367-4803. doi: 10.1093/bioinformatics/btz065.

Daniel A Messenger and David M Bortz. Weak SINDy: Galerkin-based data-driven model selection. Multiscale Modeling & Simulation , 19(3):1474-1497, 2021. Publisher: SIAM.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Qiqi Wang, Rui Hu, and Patrick Blonigan. Least squares shadowing sensitivity analysis of chaotic limit cycle oscillations. Journal of Computational Physics , 267:210-224, 2014. Publisher: Elsevier.

Zhe Gan, Chunyuan Li, Ricardo Henao, David E Carlson, and Lawrence Carin. Deep temporal sigmoid belief networks for sequence modeling. Advances in Neural Information Processing Systems , 28, 2015.

Markus Heinonen, Cagatay Yildiz, Henrik Mannerström, Jukka Intosalmi, and Harri Lähdesmäki. Learning unknown ODE models with Gaussian processes. In International Conference on Machine Learning , pages 1959-1968. PMLR, 2018.

Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. ODE2VAE: Deep generative second order ODEs with Bayesian neural networks. Advances in Neural Information Processing Systems , 32, 2019.

Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as infinite-dimensional gans. In International Conference on Machine Learning , pages 5453-5463. PMLR, 2021.

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics , pages 370-378. PMLR, 2016.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Appendices

Amortized Reparametrization: Efficient and Scalable Variational Inference for Latent SDEs

A Expectations under linear SDEs

In this section we re-derive a result from [13] regarding how to rewrite expectations under linear stochastic differential equations. As we will explain in greater detail in Appendix B, it is this result that allowed [13] to rewrite the ELBO entirely in terms of quantities that do not require differential equation solvers.

Theorem 1. Consider the density, q ( z ( t )) , of the solution to a linear SDE dz = ( -A ( t ) z + b ( t )) dt + L ( t ) dβ with initial condition z 0 ∼ N ( m 0 , S 0 ) , where A : R → R d × d is symmetric, b : R → R d , L : R → R d × d , and β indicates Brownian motion with diffusion matrix Σ . Then the expected value of a bounded functional, f , satisfies

where B ( t ) = vec -1 (( S ( t ) ⊕ S ( t )) -1 vec ( L ( t )Σ L ( t ) T -˙ S ( t ))) and m ( t ) and S ( t ) indicate the mean and covariance, respectively, of the SDE solution at time t .

Proof. The solution of a linear SDE defines a Markov Gaussian process with marginal statistics, q ( z ( t )) ∼ N ( m ( t ) , S ( t )) , given by the solution to the ODEs,

with initial condition m (0) = m 0 , S (0) = S 0 [19]. Noticing that equation (16) defines a set of matrix Lyapunov equations in terms of A ( t ) allows us to express A ( t ) as a function of S ( t ) as follows,

where ⊕ is called the Kronecker sum and is defined as S ⊕ S = I ⊗ S + S ⊗ I and ⊗ indicates the standard Kronecker product. Letting B ( t ) = A ( t ) be the expression for A ( t ) written in terms of S ( t ) , we can rearrange Equation (15) to solve for b ( t ) as,

Substituting the expressions for A ( t ) and b ( t ) into Equation (14) yields the desired result.

Remark In the case that S ( t ) and L ( t )Σ L ( t ) T are diagonal as we assume throughout this work, Equation 17 simplifies to,

B Evidence lower bound derivation

Given a dataset of observations, D = { ( x i , t i ) } N i =1 , a generative likelihood, p θ ( x | z ( t i )) a prior SDE, dz = f θ ( z ( t ) , t ) dt + L θ ( t ) dβ , and an approximation to the posterior of the latent state, dz = ( -A ϕ ( t ) z ( t ) + b ϕ ( t )) dt + L θ ( t ) dβ , it is possible to derive the ELBO [20, 12],

where C θ ( t ) = ( L θ ( t )Σ L T θ ( t )) -1 . The second term in (20) contains an integral over time of terms of the form which matches that of the expectation in (14). From this observation, Course and Nair [13] applied the identity in (14) to derive the reparametrized ELBO,

where,

To reiterate what was mentioned in the main body of the present work, the advantage of this reparametrized ELBO is that all expectations are now taken with respect to normal distributions this has effectively eliminated the requirement of a differential equation solver. Unfortunately this approach requires storing the entire estimate for the latent state - making it scale poorly for long time series with complex dynamics. After amortizing as is suggested in the main body of the present work we arrive at the final ELBO,

Again, as is discussed at length in the main body of the present work, the advantage of such an amortization strategy is that it is possible to construct an unbiased approximation to the gradients of this ELBO that scales independently with the amount of the data, the length of the time series, and the stiffness of the underlying differential equations.

C Evidence lower bound with priors on generative parameters

In many circumstances, it will be advantageous to place priors on a subset of the parameters, θ . Using the tools of stochastic variational inference, we can infer the posterior on these variables with only a marginal increase to the overall computational cost and with no impact to the asymptotic computational complexity.

For example, in all numerical studies we placed a log-normal prior on the diagonal of the diffusion process term C -1 θ ( t ) and approximated the posterior using a log-normal approximate posterior, see Appendix E for details. While not necessary, we found that doing so helped to stabilize training in the examples we considered. The particular choice of prior-posterior pair was made to ensure that the Kullback-Leibler (KL) divergence between the approximate posterior and the prior could be written in closed form; see Appendix E. Given a particular choice of prior, p ( θ ) , along with an approximate posterior, q ϕ ( θ ) , we can amend the previously derived ELBO as,

where D KL ( q || p ) indicates the KL divergence between q and p .

Consider a reparametrization to the latent state as z ( t ) = T ( t, ϵ, ϕ ) where ϵ ∼ p ( ϵ ) = ⇒ z ( t ) ∼ q ϕ ( z ( t ) | x ( j ) 1 , x ( j ) 2 , . . . , x ( j ) M ) . Also consider a reparametrization to the approximate posterior for the

generative variables, θ = V ( ν, ϕ ) where ν ∼ p ( ν ) = ⇒ θ ∼ q ϕ ( θ ) . Then, building on Lemma 1, we can arrive at an unbiased estimate for the gradient of the modified ELBO which inherits all the properties discussed with the original approximation:

D Detailed description of recognition network

This section details the deep kernel based encoder we used in all numerical studies. We found this particular encoding architecture to be useful for our purposes because it is interpretable and stable in training. With this being said, any encoder which can transform a batch of observations down to a reduced dimension latent state can be used in combination with Lemma 1 to arrive at an unbiased estimate for the gradient of the ELBO which retains all the properties discussed in the main body of this work.

Given a dataset, D = { ( t i , x i ) } N i =1 , recall that the first step in our amortization strategy is to split the dataset into N/M non-overlapping partitions,

Recall that we would like to approximate the latent state over each partition using only the M observations in each partition, q ϕ ( z ( t ) | x ( j ) 1 , x ( j ) 2 , . . . , x ( j ) M ) ≈ p ( z ( t ) | D ) for t ∈ [ t ( j ) 1 , t ( j +1) 1 ] . Going forward we will drop writing the superscript as we will be only working on a single partition, ( t 1 , x 1 ) , ( t 2 , x 2 ) , . . . ( t M , x M ) . The user first selects how many snapshots into the future they would like to use to estimate the latent state at the present time, K ; in our own studies we found choosing K ∈ [1 , 5] worked well for the problems we considered.

Given some encoding network, ENC ϕ , we compute:

where h i ∈ R 2 d with d < D . We will describe the particular architecture design for ENC ϕ in the context of each numerical study in Appendix G. We note that for our approach, it is often important to use at least a small number of neighbours (i.e. K > 0 ) when estimating the latent state because we are limited to approximating MGPs over the latent state.

To explain this point more clearly, let us consider the example of inferring a latent SDE using video of a pendulum as we did in Section 4.4. If we choose a single frame near the center, we have no way of knowing from that frame alone if the pendulum is currently swinging left or right. In other words, if we were to build an encoder which takes in one single frame, the encoder should predict that the posterior given that single frame is multimodal. As our approach only allows for one to approximate the latent state using MGPs, this is not an option. Allowing the encoder to take in a few frames at a time remedies this issue. We also note that previous works for inferring latent differential equations made this same choice [1, 11, 12, 17].

Recall that we need to approximate the latent state at any time over the window of observations in the partition, t ∈ [ t ( j ) 1 , t ( j +1) 1 ] . To accomplish this we effectively interpolate between encodings using a deep kernel [36]. Letting,

where H ∈ R M × 2 d and t node = [ t i , t 1 , . . . , t M ] , we construct the encoder for the mean and diagonal of the covariance over the latent state as,

Here we note that the right hand side of (28) is a row vector of length 2 d . We use the notation [ m ϕ ( t ) T log S ϕ ( t ) T ] to indicate that m ϕ ( t ) is given by the first d elements of this vector and log S ϕ ( t ) is given by the next d elements. Here k ϕ is a so-called deep kernel and k ϕ ( t, t node ) ∈ R M and k ϕ ( t node , t node ) ∈ R M × M . In addition σ n ∈ R + is tuned as a part of the optimization procedure. While many options for the base kernel are possible, we made use of a squared exponential kernel,

where σ f , ℓ ∈ R + are positive constants tuned as a part of the optimization procedure and DK ϕ is a neural network whose architecture we will describe in the context of the numerical studies. While the base kernel is stationary, the neural networks allow for the encoder to infer non-stationary relationships [36]. It is worth noting that without this deep-kernel our approach struggled to achieve good validation accuracy on the datasets we considered.

Advantages of this encoder design are that it can easily take in varying amounts of data, it is interpretable because [ m ϕ ( t i ) T log S ϕ ( t i ) T ] ≈ h T i , and it is cheap to compute so long as M is relatively small because evaluations of ENC ϕ can be performed in parallel. Particular choices for ENC ϕ and DK ϕ are described in context in Appendix G.

E Approximate posterior on diffusion term

This section summarizes the log-normal parameterization we used to approximate the posterior over the diagonal drift function terms in the main body of the paper. First, we note that in the loss function we only require access to the product, C θ ( t ) -1 = L θ ( t )Σ L θ ( t ) T so, rather than parametrizing L θ on its own, we parametrize C -1 θ .

Specifically we parametrize C θ ( t ) -1 = diag ( θ ) , where θ ∈ R d . The prior is defined as,

We parametrize the approximate posterior as,

where µ i and σ i are the variational parameters. The KL divergence between the posterior and prior is given by,

F Computing resources

Experiments were performed on an Ubuntu server with a dual E5-2680 v3 with a total of 24 cores, 128GB of RAM, and an NVIDIA GeForce RTX 4090 GPU. The majority of our code is written in PyTorch [37]. For benchmarking we made use of torchdiffeq [1], torchsde [12, 35], and pytorch_lightning . All code is available at github.com/coursekevin/arlatentsde.

G Details on numerical studies

This section contains more details on the numerical studies including the specific architecture design and hyperparameter selection for each experiment. For each experiment we design an encoder

consisting of two neural networks, ENC ϕ ( x i , . . . , x i + K ) and DK ϕ ( t ) (see Appendix D), a decoder DEC θ ( z ( t )) , and a model for the SDE consisting of a drift, f θ ( t, z ) , and dispersion matrix, L θ ( t ) . For all numerical studies we place a log-normal prior on the dispersion matrix and assume that the approximate posterior is constant in time, see Appendix E. For all experiments we gradually increased the value of the KL-divergence (both the KL-divergence due to the SDE prior and the KL-divergence on the dispersion matrix parameters) from 0 to 1 using a linear schedule.

G.1 Orders of magnitude fewer function evaluations

In this section we provide a more detailed description of the numerical study in Section 4.1. As a reminder, we tasked a neural ODE (NODE) and our approach with building a predictive model for the Lotka-Volterra system given a dataset of time-series observations. The Lotka-Volterra equations are a system of nonlinear ODEs usually written as,

In our experiment we chose α = 2 / 3 , β = 4 / 3 , and δ = γ = 1 . We also assumed that there was some small amount of Brownian noise given by Σ = diag (10 -3 , 10 -3 ) . Using the initial condition x = 0 . 9 and y = 0 . 2 , we draw a sample from the system using the default adaptive stepping scheme in torchsde from time 0 to 65 with an initial step size of 0 . 1 and an absolute and relative tolerance of 10 -5 . We then evaluate the solution at a frequency of 50 Hz and added Gaussian noise with a standard deviation of 0 . 01 . We use the first 50 seconds for training and reserve the remaining 15 seconds for validation.

Both the NODE and our approach use the same model for the gradient field and drift function respectively, see Figure 6a. The encoder and deep kernel architecture are provided in Figures 6b and 6c respectively. As mentioned in the main body of the paper, we set the decoder to be the identity function so as to force our model to learn the dynamics in the original coordinates. We selected a Gaussian likelihood with a constant standard deviation of 0 . 01 .

In terms of hyperparameters we set the schedule on the KL-divergence to increase from 0 to 1 over 1000 iterations. We choose a learning rate of 10 -3 with exponential learning rate decay where the learning rate was decayed lr = γlr every iteration with γ = exp(log(0 . 9) / 1000) (i.e. the effective rate of learning rate decay is lr = 0 . 9 lr every 1000 iterations.). We used the nested Monte-Carlo approximation described in Equation (10) with R = 1 , S = 10 , and M = 256 . In terms of kernel parameters, we initialized ℓ = 10 -2 , σ f = 1 , and σ n = 10 -5 . In terms of the diffusion term, set µ i = σ i = 10 -5 and ˜ µ i = ˜ σ i = 1 .

G.2 Adjoint instabilities experiment

In this section, we provide some additional details of the numerical study described in Section 4.2. Recall the parametric model for the Lorenz system in equations (11-13). As a reminder, given time-series dataset of observations, our goal was to infer the value of the parameters, σ , β , ρ , which were likely to have generated the data starting from an initial guess: θ 0 = [ σ 0 , β 0 , ρ 0 ] . The true value of the parameters was chosen as θ ∗ = [10 , 8 / 3 , 28] . For all experiments we used the initial condition [8 , -2 , 36 . 05] as was suggested in [10]. We generated data by solving the differential equation using scipy 's RK4(5) initial value problem solver with a relative and absolute tolerance of 10 -6 and 10 -8 respectively. We generated data at a frequency of 200 Hz over the time intervals [0 , 1] , [0 , 10] , [0 , 50] , and [0 , 100] . For each time interval we generated 5 datasets by adding independent Gaussian noise with a variance of 1 to the data.

To arrive at an initial guess we sample from the distribution θ 0 ∼ N ( θ ∗ , (0 . 2 θ ∗ ) 2 ) . For each time series length we tasked our approach with inferring the true value of the parameters given 5 different guesses for the initial condition (i.e. one guess / dataset). The reported gradients for our approach are given by the average ℓ 2 -norm of the gradient of the ELBO with respect to the parameters σ , β , and ρ after optimizing for 2000 iterations. For the adjoint method, we report the gradient of the function:

x

FC-64

ReLU

FC-64

ReLU

FC-64

ReLU

FC-2

drift function

(a)

f

θ

(

t, x

)

x

i

FC-32

ReLU

FC-32

ReLU

FC-4

m

(

t

i

)

log

S

(

t

i

)

(b) ENC

ϕ

(

x

i

)

t

FC-32

ReLU

FC-32

ReLU

FC-1

nonstationary transform

(c) DK

ϕ

(

t

)

x

i

FC-32

ReLU

FC-32

ReLU

FC-6

m

(

t

i

)

log

S

(

t

i

)

(a) ENC

ϕ

(

x

i

)

t

FC-32

tanh

FC-1

nonstationary transform

(b) DK

ϕ

(

t

)

at the starting iteration. Note we cannot provide average gradients over the entire optimization procedure for the adjoint based method because the initial gradients are too large when the time interval was [0 , 50] or [0 , 100] . The error bars are given by one standard deviation from the mean.

A description of the encoder architecture is provided in Figure 7. As was the case in the previous experiment, we set the decoder to be the identity function and placed a log-normal prior on the diffusion term.

In terms of hyperparameters, we choose M = 128 , R = 10 , and S = 100 . We selected a learning rate of 0 . 1 and decayed the learning rate lr = γlr every iteration with γ = exp(log(0 . 9) / 1000) . We linearly increased the value of the KL-divergence from 0 to 1 over the course of 100 iterations. In terms of kernel parameters we initialized σ f = 1 , ℓ = 10 -2 , and σ n = 10 -5 . For the diffusion term, we set µ i = σ i = 10 -5 and ˜ µ i = ˜ σ i = 10 -5 .

G.3 MOCAP experiment

For this experiment we use the preprocessed dataset provided by [34].

Like in previous examples, we place a log-normal prior on the diffusion term. Like previous works making use of the benchmark, we assume a Gaussian likelihood. We place a log-normal prior on the variance of the likelihood. A description of the architecture is provided in Figure 8. Note that the architecture we chose is very similar to the architecture used by [34, 12].

In terms of hyperparameters, we chose a batch size of 512. We used a linear schedule to update the weighting on the KL-divergence from 0 and 1 over the course of 200 iterations. We make use of the nested Monte-Carlo scheme in equation (10) with R = 10 and S = 10 . We chose a learning rate of 0 . 01 and decayed the learning rate each iteration according to the schedule lr = γlr with γ = exp(log(0 . 9) / 1000) . In terms of kernel parameters, we initialize σ n = 10 -5 , ℓ = 10 -2 , and σ f = 1 . In terms of the prior on the diffusion term we initialize µ i = σ i = 10 -5 and set ˜ µ i = ˜ σ i = 1 . In terms of the prior on the variance of the observations, we initialize µ i = σ i = 10 -2 and set ˜ µ i = ˜ σ i = 1 . We considered both Softplus [12] and tanh [34] nonlinearities and found that tanh nonlinearities provided improved validation performance. We train for 100 epochs testing validation accuracy every 10 epochs. We report the average test accuracy after training 10 models from different random seeds.

Previous studies tended to report mean-squared-error as, MSE ± ERROR. We report RMSE ± NEWERRORso that error units are consistent with the units of the original dataset. To convert MSE to RMSE we used a first-order Taylor-series approximation,

G.4 Neural SDE from video

In this section, we provide a more detailed description on the numerical study described in Section 4.4. We generated data by simulating a nonlinear pendulum with the equations,

for 30 seconds while sampling the state at a frequency of 15Hz. The architecture we used for this experiment is provided in Figure 9.

In terms of hyperparameters, we chose a batch size of 128. We gradually increased the weighting of the KL-divergence term using a linear schedule over the course of 1000 iterations. We used the nested Monte-Carlo method suggested in Equation 10 and set R = 20 and S = 10 . We chose a learning rate of 0 . 001 and decayed the learning rate each iteration according to the schedule lr = γlr with γ = exp(log(0 . 9) / 1000) . In terms of kernel parameters, we initialized σ f = 1 , σ n = 10 -5 , and ℓ = 10 -2 . We placed a log-normal prior on the diffusion term and approximated the posterior using log-normal variational distribution. With regards to the prior on the diffusion term we initialize µ i = σ i = 0 . 1 and set ˜ µ i = ˜ σ i = 10 -5 .

x

i

x

i

+1

x

i

+2

FC-30

tanh

FC-30

tanh

FC-15

tanh

FC-6

h

i

(a) ENC

ϕ

(

x

i

)

z

(

t

)

FC-15

tanh

FC-30

tanh

FC-30

tanh

FC-50

µ

(

t

)

(b) DEC

ϕ

(

t

)

t

FC-32

tanh

FC-1

nonstationary transform

(c) DK

ϕ

(

t

)

x

FC-30

tanh

FC-6

drift

(d)

f

θ

(

z, t

)

x

i

x

i

+1

. . .

x

i

+5

Conv-16-3

BN-ReLU

Pool-Max-2

Conv-32-3

BN-ReLU

Pool-Max-2

FC-128

BN-ReLU

FC-4

h

i

(a) ENC

ϕ

(

x

i

)

z

(

t

)

FC-(

64

×

32

×

32

)

ReLU

FC-(

64

×

32

×

32

)

ReLU

ConvT-32-4

BN-ReLU

ConvT-1-4

logit

(b) DEC

ϕ

(

t

)

t

FC-32

ReLU

FC-32

ReLU

FC-1

nonstationary transform

(c) DK

ϕ

(

t

)

x

FC-64

tanh

FC-64

tanh

FC-2

drift

(d)

f

θ

(

z, t

)

H Numerical study on the effect of Monte-Carlo parameters

In this study we investigate the effect of varying the number of nested Monte Carlo samples on the rate of validation RMSE convergence. To do so, we consider the problem of building a predictive model for a four-dimensional predator-prey system. The system consists of two predators and two prey where there is a competitive dynamic between the predators. The equations governing the dynamics of the system are,

where α i is the grow rate of the prey x i , β i is the rate that predator y 1 is consuming prey x i , γ i is the rate that predator y 2 is consuming prey x i , k i is the carrying capacity for prey x i , δ i is the death rate of predator y i , ϵ i is the conversion rate for predator y i from x 1 , ξ i is the conversion rate for predator y i from prey x 2 , and ν i represents the competitive effects on y i caused by the other predator.

We simulated the system for 300 units of time collecting data at a frequency of 10Hz. We assume a Gaussian noise with a standard deviation of 10 -2 . For all experiments we used the following hyperparameters: a batch size of 256 , 1000 warmup iterations, and a learning rate of 10 -3 . In terms of kernel parameters, we initialized σ f = 1 , ℓ = 10 -2 , and σ n = 10 -5 . For the diffusion term we set µ i = σ i = 10 -5 and ˜ µ i = ˜ σ i = 1 . The architecture description for the neural networks used in this example are provided in Figure 11.

Results are summarized in Figure 10 below. On this problem we find we are able to achieve a reasonable validation accuracy for S = 10 , S = 50 , and S = 100 ; however, it is challenging to know beforehand what gradient variance will be acceptable for a particular data set. We see that increasing S increases the total number of function evaluations. Note that the total number of parallel evaluations of the forward model in all cases remains constant.

0

500

1000

1500

2000

Iteration

-

1

.

4

-

1

.

2

-

1

.

0

-

0

.

8

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

log

10

Validation RMSE

R=1, S = 10

R=1, S = 50

R=1, S = 100

0

50000

100000

150000

200000

NFE

-

1

.

4

-

1

.

2

-

1

.

0

-

0

.

8

-

0

.

6

-

0

.

4

-

0

.

2

0

.

0

log

10

Validation RMSE

R=1, S = 10

R=1, S = 50

R=1, S = 100

x

i

FC-32

ReLU

FC-32

ReLU

FC-4

m

(

t

i

)

log

S

(

t

i

)

(a) ENC

ϕ

(

x

i

)

t

FC-32

ReLU

FC-32

ReLU

FC-1

nonstationary transform

(b) DK

ϕ

(

t

)

x

FC-64

ReLU

FC-64

ReLU

FC-2

drift function

(c)

f

θ

(

z, t

)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop a reparameterization scheme for affine SDEs which is advantageous in memory cost and time. Specifically, the authors consider splitting the computation into a series of different chunks such that each can be effectively parallelized and integrated individually by taking the expectation with respect to a uniform random variable over the time interval. This is possible since the SDE is linear and a mean and covariance matrix provide the full description of the data at all time marginals and individual trajectories are not needed. The points are then combined using an interpolation technique that is based upon deep kernels. The authors finally consider experiments to highlight the utility of the method. They consider synthetic and a real experiment on time series data that compares the number of function evaluations and the accuracy of the methods.\n\nSTRENGTHS:\nThe idea of transforming the sequential computations into parallelizable expectations is appealing and the numerical results suggest this approach provides a significant computational speed up. This is a useful extension beyond the usual sequential nature of ODE which cannot be parallelized and could be particularly useful and impactful in certain cases. Additionally, empirical results suggest that the modifications provide increased stability. The authors also show the utility of the method on a real dataset where the performance is superior compared to most of the baselines. Finally, the paper is fairly well written and the presentation is intuitive.\n\nWEAKNESSES:\nThe method only applies to SDEs where the marginal descriptions are known, as the authors make an assumption that the evolution of the latent space is given by a Markov Gaussian process. However, this is not the case for most SDEs, which makes the algorithm limited in scope. From the paper, it is not clear where such an assumption would hold that the latent distribution should be Gaussian everywhere. This question was not studied, and I think that's the major weakness of the paper. While the computations are easier to perform, it is mainly due to the simpler model. SDEs are particularly useful when sampling from a complicated distribution, but in this case the latent distribution Gaussian for all time marginals. Maybe to strengthen this, it would be helpful to show the types of stochastic processes in the observable space that can be modeled (e.g. by applying Ito's lemma to the latent process). This would the provide a more concrete statement on the expressiveness of the model.\n\nThe experimental evaluation is somewhat limited since the evaluation is only on two synthetic datasets and one real dataset. I would be curious to see how the other methods compare on the pendulum dataset. Additionally, I think some more synthetic datasets would be useful to describe the utility/use of the method, particularly on non-affine SDEs and seeing the performance in approximation. It would be helpful to see a comparison to backpropagating through the solver rather than the adjoint method, which the authors mention is numerically unstable. Understanding how the behavior of backpropagation through the solver (since this is possibly the most common method when not using a fine step size or memory constraints are not a concern) compares to the proposed method. Finally, the authors mention in line 184 the linear assumption of the SDE is not a limiting factor, but the experiments do not seem to be extensive enough to validate such a claim. \n\nOther types of methods that are related and should be mentioned are [1, 2]. These methods consider similar strategies with [1] considering a gaussian process in the latent space and [2] considering a linear SDE approximating the latent evolution (using some of the same results mentioned in this paper based on casting linear SDEs in terms of their moments).\n\n[1] Fortuin et al 2020 AISTATS, GP-VAE: Deep Probabilistic Time Series Imputation\n[2] Duncker et al 2019 ICML, Learning Interpretable continuous-time models of latent stochastic dynamical systems\n\nQUESTIONS:\nHow does the interpolation between the windows affect the quality of the sample path? This point was not discussed thoroughly and it seems like it might have impact on the sample quality.\n\nWere there any comparisons to non-adjoint based back-propagation through the solver? For example, using just an Euler solver, how does the method compare with backpropagation through that? \n\nLine 147: “While the previous sections have demonstrated how to replace a differential equation with an integral …” Can the authors please expand upon this? When solving a differential equation, the integral is approximated as a sum, as the authors have done. Is this comment due to the fact that you are not using an adaptive solver but you are using an expectation to compute the integral? Or is it more that you do not require the value of $z(t)$ at some later time, only the expectation?\n\nIs there an ablation on changing the window size/number of windows? How does this parameter affect the performance?\n\nWhy is the drift parameterized as a neural network of z on line 519? I thought the drift was linear in z.\n\nIs there a way to verify that the linear SDE assumption is a valid one? Are there some equivalence classes one can derive for the case studied here and a possibly more general class of SDEs?\n\nLIMITATIONS:\nThe authors provide a discussion on the issues associated with the linear SDE assumption.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method for identifying latent stochastic models from discrete time observations.\nThe authors perform\nunbiased approximations to gradients of the\nevidence lower bound to identify parameters and estimate the state for latent stochastic differential equations. \nThy propose to combine a recently introduced (by the same authors - provided anonymous paper) reparametrisation of the ELBO\nin terms of time dependent linear SDEs and amortisation to reduce the computational demands.\nThe resulting approach does not require numerical integration of the approximate SDE or the ODEs defining the evolution of the central moments (as required by previous approaches), and seems to perform on par or better than existing frameworks. \n\nAlthough the method is limited by their assumptions of linear covariances and additive noise, the merging of the amortisation with the reparametrisation of the ELBO seems a rather interesting contribution. The performance of the method on the numerical experiments seem to be on par with existing approaches. However in most numerical examples the employed noise in the system was negligible when considering the dynamic range of the employed systems.\n\nSTRENGTHS:\n- The authors eliminate the need for numerically integrating the approximate differential equations by replacing the initial value problem with an integral resulting in reduced computational demands.\n\n- The approach avoids numerical instabilities often encountered in adjoint methods.\n\n- the proposed solution has a time and memory cost that depends only on the number of parallel evaluations of the ELBO, i.e., scales independently with the amount of data, the length of the time series, and the stiffness of the approximation to the differential equations.\n\nWEAKNESSES:\n- The approximate process is a linear SDE, and thus cannot probably capture effectively nonlinear dynamics that may result in multimodal transition densities.\n\n\n- To my understanding the proposed framework requires a diagonal covariance matrix (but I am happy to be corrected if this is not the case) for the diffusion process, as well as a diagonal time dependent covariance for the approximate process. How would you tackle problems with interactions in the diffusion component with your framework?\n\n\n\n- I consider the diffusion term for the experiment of the Lorenz system substantially small ($\\sigma = 10^{-5}$) considering the state space volume the state of the system spans.\n\nQUESTIONS:\n- How do you select the partitions of the temporal axis and how does this selection influence the obtained results?\n\n- In line 498 in the supplement you mention that the encoder design is interpretable. Can you explain what you mean with this?\n\n- Could you please explicitly mention in the appendix what observation model was employed in each of the numerical experiments?\n\n- The time complexity of the algorithm depends on the number of evaluations $R$. Can you provide a systematic evaluation of how the number of evaluations $R$ influences the performance of the framework?\n\n- What kind of issues do emerge in your framework if you don't split the observations into chunks?\n\n- How does the method perform for decreasing sampling frequency?\n\n- Does the method require the entire state of the system to be accessible to the observation model, or could also a lower dimensional projection of the latent state work?\n\nLIMITATIONS:\n- The proposed framework is limited only to systems with diagonal covariances.\n\n- As far as I understand, they employed only a Gaussian observation model in all experiments.\n\n---\n\n[1] Duncker, Lea, et al. \"Learning interpretable continuous-time models of latent stochastic dynamical systems.\" International Conference on Machine Learning. PMLR, 2019.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new inference method for latent SDE model. Due to the continuous nature of SDE, inference methods for latent SDE models are usually expensive and the cost would grow as the dataset goes larger or longer. Existing methods also would require differential equation solvers, which makes them unfriendly to GPU parallelization. The approach proposed in this work uses an amortized Markov Gaussian process as a variational posterior for inference over the latent SDE. The proposed variational objective allows for factorization over time steps and therefore enables alleviates the need to iterate over all time steps. Under such a framework, the cost at each iteration is only dependent on the number of Monte Carlo samples utilized for gradient estimation. The authors then empirically evaluate the efficiency of their method in a wide range of tasks, in which the proposed method shows much better performance in terms of computational cost, numerical stability, and convergence compared with existing methods.\n\nSTRENGTHS:\n- The paper is well-motivated and the proposed approach, to the best of my knowledge, is very sensible and well supported by empirical evaluations.\n\n- The authors provide a thorough discussion of existing methods and include them in the experiments as baseline methods.\n\n- The paper is well written and the proposed method is presented clearly.\n\nWEAKNESSES:\nSome technical details seem to be discussed too briefly in the main text, for example, the implementation of the encoder and the use of deep kernel for \"interpolate between latent vectors\". I can hardly understand the meaning without referring to the appendix. In Remark 2, the authors mention \"make use of a nested Monte Carlo scheme along with stratified sampling to reduce the number of decoder evaluations and the variance of gradients.\" but the meaning of stratified sampling and nested Monte Carlo in this context is not discussed.\n\nQUESTIONS:\nI am not familiar with the literature of latent SDE but I wonder whether the proposed variational posterior is more biased than existing approaches. Assuming computational cost is not an issue, would the baseline approaches be more favorable to the proposed method?\n\nLIMITATIONS:\nNothing I can think of other than the limitation discussed by the authors at line 182.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present work proposes a time and memory efficient method to perform inference in a directed probabilistic model in the presence of a latent stochastic process with intractable posterior (path) distribution. Learning is performed using a variational Auto-Encoding approach, in which an approximate latent posterior process is selected from the class of Markov-Gaussian processes that are solutions to linear non-autonomous (Itô) SDEs. An evidence lower bound objective (ELBO) is derived that removes the need for numerical solvers when approximating gradients. Included experiments on toy and  real world time series data approve similar performances compared to methods based on adjoint sensitivities but with two-orders of magnitude fewer evaluations and with a time and memory cost that scales independently with the amount of data, the total length of the time series and the stiffness of the approximate differential equations.\n\nSTRENGTHS:\nI agree, a huge drawback of those existing models is the dependency on stochastic optimization methods that require backpropagading gradients through the numerical (stochastic) differential equation solver and hence rely on adjoint methods.\nIt is the attempt itself tackling the road blocks by combining a variety of ideas that rely on model simplification, that I particularly appreciate. Especially under the lens of Bayesian inference it is recommended to question the need for unbounded expressivity in case of learning latent dynamics with continuous time models. This consideration is interesting on its own and adds progress to the community, e.g., regarding the applicability on low performance hardware.\n\nThe authors propose improvement in time and memory cost by combining the following three major strategies: (1) simplifying the function class available for learning latent dynamics while reducing the expressiveness of the latent (SDE) models used, (2) exploiting the nature of sequential data such as time series that allow divide-and-conquer strategies such as the analysis of small sliding observation windows, and (3) introducing an optimized objective (i.e. an ELBO) by exploiting the reparameterization trick within the model used.\n\nThis method builds on a novel combination of well-known techniques, and is original to the best of my knowledge. Further, the paper is well positioned to existing work, the content is well organized and, with a few exceptions, has no spelling or grammatical flaws. Empirical evidence is accessed through experiments on toy data as well as established benchmark data from real world applications; however, I would classify them small-scale.\n\nWEAKNESSES:\nIt is not the value of the approach I am concerning here, but the shortcomings related to the clarity of the submission and the technical soundness which I will address with examples in the following.\n\n1. (2.1 Problem description)\n   - I am missing a clear definition of the observation model, e.g., cf [Vrettas et al. 2015; Variational mean-field algorithm for efficient inference in large systems of stochastic differential\nequations ; Section II.A].\n   - Considering time $t_i \\in \\mathbb{R}$ instead of $\\mathbb{R}_{+}$ is not wrong but unusal.\n   - Minor ambiguities make understanding difficult in total: $dz$ vs $dz(t)$ and $\\beta$ vs $\\beta(t)$; the definition of $\\theta$ is missing.\n   - Is there any motivations for the stochastic variational inference approach?\n   - I would $q_\\phi(z(t)| x_i,\\dots,x_{i+M}) \\approx p_{\\theta}(z(t)|\\mathcal{D})$ as an local approximate posterior with respect to the time variable; are there necessary and/or sufficient conditions for an observed stochastic process that guarantee the validity of this approximation?\n2. (2.2 Evidence lower bound)\n   - (l. 96f) The following statement: \"An added complication in our specific case comes from the fact that the latent state is defined by a stochastic process rather than a random-vector as is more typical.\" is poorly worded. You probably intend to say that, typically, stochastic variational inference considers vector-valued random variables to define latent states. But your approach requires function valued random variables, i.e., stochastic processes. What are those complications expected here?\n   - It would be advantageous in combination with Eq. (1) to mention that the time dependency of the drift coefficient is necessary to account for a possible non-stationary observation process $x(t)$ that is driven by the latent stochastic dynamics of $z(t)$; cf. [32; Section 3].\n   - (Eq. (3)): To derive this result, the authors refer to unpublished work without stating the authors, cf. [14], but they included the reference in the supplementary material submitted. Notwithstanding the fact that I cannot call this good practice, this result is based on known facts and the derivation could have been included in the supplementary material of the present work, for example. Besides that, the derivation in [14] only concerns the case of a constant dispersion matrix $\\Sigma$ with respect to time $t$ whereas $L(t)$, e.g., in Eq. (1) depends on time.\n   - (Eq. (4)): Section A of the supplementary material indicates that this result follows by applying properties concerning a Lyapunov equation and by using the Kronecker sum. Following the arguments myself raised difficulties; in my opinion some given identities only hold in case of $A(t)$ is symmetric, i.e., $A(t) = A\\top(t)$. Last but not least titling the result Theorem overstates its contribution.\n   - (Remark 2): I would like to see the form of the prior SDE. In the setting of stochastic variational inference it is important to well design the dispersion matrix between prior and posterior SDE in order to guarantee that the KL divergence is finite in value.\n   - Honestly I still don't get how the authors include the fact, that $m_\\varphi$ and $\\dot{m}_\\varphi$ as well as \n    $S_\\varphi$ and $\\dot S_\\varphi$ are related through an ODE? Because all four are included into $r_\\theta, _\\varphi$ and e.g., Section G.1 only shows architecture design for $m$ and $S$.\n3. (Experiments)\n    - (Figure 2): Can you quantify \"a similar validation accuracy\"? Does the right plot only contain the ARCTA probabilistic prediction? Section F.1 does not contain the promised information, did you mean Section G.1?\n    - (Section 4.2; l. 259ff.): The evaluation protocol is unclear to me; why are there multiple adjoint methods included and do you only optimize ARCTA for 2000 iterations? Can the quality be evaluated using error measures, e.g. MSE? A very similar experiment was presented in [13] proposing an adjoint method for SDEs. The results are promising. Can you please take these results into account?\n    - (Section 4.3): From my point of view, the motion capture data set under consideration is quite delicate, since I know the shortcomings very well from my own experiments with it in the past. In my opinion it is the overall size of time series contained (16 training, 3 validation and 4 testing) that frequently causes severe overfitting. So I am still bothered with severe instabilities when reproducing the results from [30 & 13]. Therefore, I also view the results presented here with caution. There are 4 times 50 = 200 predictive posterior outputs by the model, evaluated on the test dataset; given four test time series you only provide three single dimension, that is strange. Secondly, I would like to know why the evaluation of the results is carried out in relation to the RMSE, requiring all benchmark results to be translated.\n\nFinally, I would like to reiterate my appreciation for the authors' interest in expanding the field of learning latent SDEs in the setting of stochastic variational inference. I'm sure that by working on the deficiencies mentioned, the work will gain in value and will deliver the desired contribution to the community in a new round.\n\nQUESTIONS:\n- (l. 32f.) \"... replacing ordinary differential equation (ODE) solvers with integration methods ...\" But don't those numerical ODE solver build on integration methods?\n- In terms of the total number of time series samples included, the experiments dealt with are probably rather small-scale. To really highlight the added value of the proposed method, I would like to see large-scale experiments like the PhysioNet 2012 experiment in [12].\n- Minor typos:\n  - (l. 28) parareal -> parallel\n  - (l. 54/67) time-series -> time series\n  - (l. 66) Description -> description\n\nLIMITATIONS:\nThe authors address the limitations of the approach regarding the models expressivity compared to existing stochastic adjoint sensitivity methods in Chapter 3 of their work.\nI would appreciate some remarks on limitations concerning the recognition module, i.e., the encoder. As stated in Section D, data sparsity is only accepted to a certain extent and cf. (l 495f.) \"It is worth noting that without this deep-kernel our approach struggled to achieve good validation accuracy on the datasets we considered\". This suggests that the correct choice of the encoder is crucial and, on the other hand, raises doubts about the contribution of the latent stochastic dynamic. Is the proposed approach limited to interpolation and time series classification tasks or may we solve also tackle extrapolation problems?\nNo evidence was found that the authors addressed any potential negative societal impact. Please make the effort to include this mandatory information in your work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}