A Gaussian approximation for stochastic non-linear dynamical processes with annihilation

We consider a stochastic nonlinear dynamical process with annihilation of particles. This process can be viewed as the continuous time version of the extended Kalman filter/smoother. It also plays an important role in stochastic optimal control theory. We derive a Gaussian approximation for this process. With the use of the path integral formalism we derive Euler-Lagrange equations for the mode. Furthermore, we derive a linear noise approximation to estimate the size of the fluctuations around the mode, and estimates of the partition function, based on the mode and Gaussian corrections. Numerical experiments confirm the validity of the approximation method. In addition, they show that the Gaussian correction provides a significant improvement of the estimate of the partition function. 1 A stochastic dynamical process with annihilation We consider the following stochastic simulation of particles with nonlinear dynamics, additive Gaussian noise and stochastic annihilation of particles, dx = b(x, t)dt + dξ x = x + dx, with probability 1 − V (x, t)dt x = †, with probability V (x, t)dt (1) where dξ is the noise process with strength ν, (i.e., dξ = N(0, νdt)), and † denotes that the particle is taken out of the simulation. (Note that, for convenience, we restrict to 1-D throughout this paper. Results are easily generalized to higher dimensional spaces.). We initialize at t0, x0, and we can describe the simulation by the unnormalized density ρ(x, t|x 0, t0). The density evolves in time according to ∂tρ(x, t|x0, t0) = −∂x(b(x, t)ρ(x, t|x0, t0)) + 1 2 ν∂ xρ(x, t|x0, t0) − V (x, t)ρ(x, t|x0, t0). (2) If we would set V = 0 in this equation, it would reduce to the Fokker-Planck equation, modeling a process of drift and diffusion, due to the terms with b(x, t) and ν respectively. These terms conserve the total probability of the density. The extra term with the potential V makes that probability is not conserved (since particles are annihilated at a rate V (x, t)dt). 1.1 Motivation from machine learning Equation (2) has a number of relations with the subject of machine learning. For instance, the process (2) can be viewed as a continuous time limit of the conventional discrete time non-linear dynamical systems with Gaussian noise (see e.g. [1]), i.e., it is a continuous time version of the extended Kalman filter in which drift and diffusion can be thought of as predictor, and the annihilation potential V (x, t) as corrector. With prior ρ̃(x0, t0) and corrections up to time tf the smoothed estimate of the state distribution at time t0 < t ≤ tf is given by ρ̃(x, t) = 1 Z(V ) ∫ ρ(y, tf |x, t)dy ∫ ρ(x, t|x0, t0)ρ̃(x0, t0)dx0 . (3) The process also has a central role in a recent stochastic optimal control theory [2, 3]. Under certain conditions, the optimal cost-to-go J(x0, t0) in a control problem with horizon time tf can be shown to be proportional to minus the log partition function of ρ, i.e., J(x0, t0) = −λ log ∫ ρ(y, tf |x, t)dy = −λ log Z(x, t) . (4) If the partition function – and hence J – is known, the control u simply follows by its gradient. The problem is that the computation of this partition function is intractible, and therefore approximations are needed. 2 The Gaussian approximation Our goal is to approximate the density of particles at given final time t f . To do this, we will consider the density of particles that survive the process up to the final time. At each intermediate time, we will approximate this density by a Gaussian shaped density, which is described by its mode and width σ. The mode as a function of time is the trajectory from initial time to final time that has maximum probability. The width σ(t) as a function of time describes a band of typical fluctuations around the optimal trajectory. 2.1 The path integral formulation In order to find the optimal trajectory, we will write the density ρ(y, t f |x0, t0), obeying Eq. 2, as a path integral. In order to make this construct, we first consider the transition density from a state x to a state y with an infinitesimal time step ∆t ρ(y, t + ∆t|x, t) ∝ exp ( − [ (y − x − b(x, t)∆t)2 2ν∆t + V (x, t)∆t ]) . (5) This could be seen as a factor in an extended Kalman filter that is the combination of a predictor term modelled by the stochastic state transition y = x + b∆t + dξ and a corrector term modelled by V (x, t)∆t. The transition probability for a finite time step is the product of n of these infinitesimal steps, tf = tn = t0 + n∆t, y = xn, ρ(xn, tn|x0, t0) ∝ ∫ n−1 ∏ i=1 dxi exp ( −∆t n−1 ∑ i=0 [ 1 2 ( xi+1 − xi ∆t − b(xi, ti) )2 + V (xi+1, ti+1) ]) . (6) In the limit of∆t → 0, the sum in the exponent becomes an integral: ∆t ∑n−1 i=0 → ∫ tf t dτ , and the repeated integral at discrete times ∫ ∏n−1 i=1 dxi becomes an integral over paths that start at x and end at y, denoted as ∫ [dx]x (or simply ∫ [dx] later in the paper). Thus we write ρ(y, tf |x0, t0) = ∫ [dx]x exp (−S[x]) (7) S[x] ≡ S(x(t0 → tf )) = ∫ tf t0 ( (ẋ(τ) − b(x(τ), τ)) 2ν + V (x(τ), τ) ) dτ (8) with x(t0 → tf ) a path with x(τ = t0) = x0, x(τ = tf ) = y, and ẋ(τ) = dx(τ) dτ . The functional S[x] is called the action associated with the path x. 2.2 Euler-Lagrange equations In this subsection, we aim to find the mode of the process. This is the path x(t 0 → tf ), starting at given x0 and ending at arbitrary y, that minimizes the action S. This optimal path x opt can be found by variational calculus. We write the action in a slightly more general (and more conventional) way