Title: STABILIZING NEURAL ODE NETWORKS WITH STOCHASTICITY

Abstract: Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

1. INTRODUCTION:

Residual neural networks (ResNet) (He et al., 2016) are composed of multiple residual blocks transforming the hidden states according to:

EQUATION

where h n is the input to the n-th layer and f (h n ; w n ) is a non-linear function parameterized by w n .

Recently, a continuous approximation to the ResNet architecture has been proposed (Chen et al., 2018) , where the evolution of the hidden state h t can be described as a dynamic system obeying the equation:

EQUATION

where f (h τ , τ ; w) is the continuous form of the nonlinear function f (h n ; w n ); h s and h t are hidden states at two different time s = t. A standard ODE solver can be used to solve all the hidden states and final states (output from the neural network), starting from an initial state (input to the neural network). The continuous neural network described in (2) exhibits several advantages over its discrete counterpart described in (1), in terms of parameter efficiency, memory efficiency, explicit control of the numerical error of final output, etc.

One missing component in the current Neural ODE network is the various regularization mechanisms commonly employed in discrete neural networks. These regularization techniques have been demonstrated to be crucial in reducing generalization errors, and in improving the robustness of neural networks to adversarial attacks. Many of these regularization techniques are based on stochastic noise injection. For instance, dropout (Srivastava et al., 2014) is widely adopted to prevent overfitting; injecting Gaussian random noise during the forward propagation is effective in improving generalization (Bishop, 1995; An, 1996) as well as robustness to adversarial attacks (Liu et al., 2018; Lecuyer et al., 2018) . However, these regularization methods in discrete neural networks are not directly applicable to Neural ODE network, because Neural ODE network is a deterministic system.

Our work attempts to incorporate the above-mentioned stochastic noise injection based regularization mechanisms to the current Neural ODE network, to improve the generalization ability and the robustness of the network. We propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which models stochastic noise injection by stochastic differential equations (SDE) . In this new framework, we can employ existing techniques from the stability theory of SDE to study the robustness of neural networks. Our results provide theoretical insights to understanding why introducing stochasticity during neural network training and testing leads to improved robustness against adversarial attacks. Furthermore, we demonstrate that, by incorporating the noise injection regularization mechanism to the continuous neural network, we can reduce overfitting and achieve lower generalization error. For instance, on the CIFAR-10 dataset, we observe that the new Neural SDE can improve the test accuracy of the Neural ODE from 81.63% to 84.55%, with other factors unchanged. Our contributions can be summarized as follows: Notations: Throughout this paper, we use h ∈ R n to denote the hidden states in a neural network, where h 0 = x is the input (also called initial condition) and y is the label. The residual block with parameters w ∈ R d can be written as a nonlinear transform f (h τ , τ ; w). We assume the integration is always taken from

0 to T . B t ∈ R m is m-dimensional Brownian motion. G(h τ , τ ; v) ∈ R n×m is the diffusion matrix parameterized by v.

Unless stated explicitly, we use • to represent 2 -norm for vector and Frobenius norm for matrix.

2. RELATED WORK:

Our work is inspired by the success of the recent Neural ODE network, and we seek to improve the generalization and robustness of Neural ODE, by adding regularization mechanisms crucial to the success of discrete networks. Regularization mechanisms such as dropout cannot be easily incorporated in the Neural ODE due to its deterministic nature.

Neural ODE The basic idea of Neural ODE is discussed in the previous section, here we briefly review relevant literature. The idea of formulating ResNet as a dynamic system was discussed in (E, 2017) . A framework was proposed to link existing deep architectures with discretized numerical ODE solvers (Lu et al., 2018) , and was shown to be parameter efficient. These networks adopt layer-wise architecture -each layer is parameterized by different independent weights. The Neural ODE model (Chen et al., 2018) computes hidden states in a different way: it directly models the dynamics of hidden states by an ODE solver, with the dynamics parameterized by a shared model. A memory efficient approach to compute the gradients by adjoint methods was developed, making it possible to train large, multi-scale generative networks (Ardizzone et al., 2018; Grathwohl et al., 2018) . Our work can be regarded as an extension of this framework, with the purpose of incorporating a variety of noise-injection based regularization mechanisms. Stochastic differential equation in the context of neural network has been studied before, focusing either on understanding how dropout shapes the loss landscape (Sun et al., 2018) , or on using stochastic differential equation as a universal function approximation tool to learn the solution of high dimensional PDEs (Raissi, 2018) . Instead, our work tries to explain why adding random noise boosts the stability of deep neural networks, and demonstrates the improved generalization and robustness. Noisy Neural Networks Adding random noise to different layers is a technique commonly employed in training neural networks. Dropout (Srivastava et al., 2014) randomly disables some neurons to avoid overfitting, which can be viewed as multiplying hidden states with Bernoulli random variables. Stochastic depth neural network (Huang et al., 2016) randomly drops some residual blocks of residual neural network during training time. Another successful regularization for ResNet is Shake-Shake regularization (Gastaldi, 2017) , which sets a binary random variable to randomly switch between two residual blocks during training. More recently, dropblock (Ghiasi et al., 2018) was designed specifically for convolutional layers: unlike dropout, it drops some continuous regions rather than sparse points to hidden states. All of the above regularization techniques are proposed to improve generalization performance. One common characteristic of them is that they fix the network during testing time. There is another line of research that focuses on improving robustness to perturbations/adversarial attacks by noise injection. Among them, random self-ensemble (Liu et al., 2018; Lecuyer et al., 2018) adds Gaussian noise to hidden states during both training and testing time.

In training time, it works as a regularizer to prevent overfitting; in testing time, the random noise is also helpful, which will be explained in this paper. Very recently, there are several concurrent works on Neural SDE (Jia & Benson, 2019; Tzen & Raginsky, 2019; Wang et al., 2019) By comparing the simulations under σ = 0 and σ = 2.8, we see adding noise to the system can be an effective way to control x t . Average over multiple runs is used to cancel out the volatility during the early stage. It is noteworthy that here we employ the multiplicative noise, where the deviation term scales proportional to x t .

In this section, we first introduce our proposed Neural SDE to improve the robustness of Neural ODE. Informally speaking, Neural SDE can be viewed as using randomness as a drop-in augmentation for Neural ODE, and it can include some widely used randomization layers such as dropout and Gaussian noise layer (Liu et al., 2018) . However, solving Neural SDE is non-trivial, we derive the gradients of loss over model weights. Finally we theoretically analyze the stability conditions of Neural SDE.

Before delving into the multi-dimensional SDE, let's first look at a 1-d toy example to see how SDE can solve the instability issue of ODE. Suppose we have a simple SDE, dx t = x t dt + σx t dB t with B t be the standard Brownian motion. We provide a numerical simulation in Figure 1 for x t with different σ.

When we set σ = 0, SDE becomes ODE dx t = x t dt and x t = c 0 e t where c 0 is an integration constant. If c 0 = 0 we can see that x t → ±∞. Furthermore, a small perturbation in x t will be amplified through t. This clearly shows instability of ODE. On the other hand, if we instead make σ > √ 2 (the system is SDE), we have

x t = c 0 exp (1 -σ 2 /2)t + σB t a.s. → 0. ℓ(𝑧 $ , 𝑐 $ ) ℎ ) ℎ ) = ℎ + + -𝑓 ℎ / , 𝑡; 𝑤 𝑑𝑡 ) + + -𝐺 ℎ / , 𝑡; 𝑣 𝑑𝑊 / ) + ℎ + 𝑋 +

Conv2d:

Figure 2: Our model architecture (more details can be found in appendix). The initial value of SDE is the output of a convolutional layer, and the value at time T is passed to a linear classifier after average pooling.

The toy example in Figure 1 reveals that the behavior of solution paths can change significantly after adding a stochastic term. This example is inspiring because we can control the impact of perturbations on the output by adding a stochastic term to neural networks.

Figure 2 shows a sample Neural SDE model architecture, and it is the one used in the experiment. It consists of three parts, the first part is a single convolution block, followed by a Neural SDE network (we will explain the detail of Neural SDE in Section 3.1) and lastly the linear classifier. We put most of the trainable parameters into the second part (Neural SDE), whereas the first/third parts are mainly for increasing/reducing the dimension as desired. Recall that both Neural ODE and SDE are dimension preserving.

3.1. MODELING RANDOMNESS IN NEURAL NETWORKS:

In the Neural ODE system (2), a slightly perturbed input state will be amplified in deep layers (as shown in Figure 1 ) which makes the system unstable to input perturbation and prone to overfitting. Randomness is an important component in discrete networks (e.g., dropout for regularization) to tackle this issue, however to our knowledge, there is no existing work concerning adding randomness in the continuous neural networks for regularization purpose. And it is non-trivial to encode randomness in continuous neural networks, such as Neural ODE, as we need to consider how to add randomness so that to guarantee the robustness, and how to solve the continuous system efficiently. To solve these challenges, motivated by (Lu et al., 2018; Sun et al., 2018) , we propose to add a single diffusion term into Neural ODE as:

EQUATION

where B t is the standard Brownian motion (Øksendal, 2003) 

EQUATION

where Σ(t) is a diagonal matrix and its diagonal elements control the variance of the noise added to hidden states. This can be viewed as a continuous approximation of noise injection techniques in discrete neural network. For example, the discrete version of the additive noise can be written as

h n+1 = h n + f (h n ; w n ) + Σ n z n , with Σ n = σ n I, z n i.i.d.

∼ N (0, 1),

which injects Gaussian noise after each residual block. It has been shown that injecting small Gaussian noise can be viewed as a regularization in neural networks (Bishop, 1995; An, 1996) . Furthermore, (Liu et al., 2018; Lecuyer et al., 2018) recently showed that adding a slightly larger noise in one or all residual blocks can improve the adversarial robustness of neural networks. We will provide the stability analysis of (25) in Section 3.3, which provides a theoretical explanation towards the robustness of Neural SDE. Dropout: Our framework can also model the dropout layer which randomly disables some neurons in the residual blocks. Let us see how to unify dropout under our Neural SDE framework. First we notice that in the discrete case

EQUATION

where

γ n i.i.d.

∼ B(1, p) and indicates the Hadamard product. Note that we divide γ n by p in (6) to maintain the same expectation. Furthermore, we have

γ n p -I = 1 -p p • p 1 -p γ n p -I ≈ 1 -p p z n , z n i.i.d.

∼ N (0, 1).

The boxed part above is approximated by standard normal distribution (by matching the first and second order moment). The final SDE with dropout can be obtained by combining ( 6) with ( 7)

EQUATION

Others: Lu et al. (2018) includes some other stochastic layers that can be formulated under Neural SDE framework, including shake-shake regularization (Gastaldi, 2017) and stochastic depth (Huang et al., 2016) . Both of them are used as regularization techniques that work very similar to dropout. 

10 1 10 2 10 3 10 4 Network depth t 2 -9 2 -8 2 -7 2 -6 2 -5 2 -4

Discritization error:

Figure 3 : Left: we compare the propagation time between Neural ODE, ODE with adjoint, and SDE. We can see that the running time increases proportionally with network depth and there is no significant overhead in neural SDE. Right: We compute the error of SDE solver caused by discretization in Euler schemes, measured by the relative error in h t , i.e. ε = h T -ĥT h T and h T is the ground-truth (computed with a very fine grid), ĥT is computed with coarse grid ∆t ∈ [1.0 × 10 -4 , 1.0 × 10 -1 ] (note that network depth t = T /∆T ).

3.2. PROPAGATION THOUGH SDE SOLVERS:

The overall implementation of training neural network containing SDE solver is similar to Neural ODE (Chen et al., 2018) and it is stated in Algorithm 1. We can see from the algorithm that for forward propagation we apply some standard SDE solvers such as Euler-Maruyama (Kloeden & Platen, 2013), Milstein (Milshtein, 1975) or higher order Runge-Kutta method, but for backward propagation we can simply rely on the automatic gradient. In practice, we find that the most straightforward autograd Given initial state h 0 , integral range [0, T ].

3:

h T = SDE Solve f (h t , t; w), G(h t , t; v), [0, T ] .

Call a black-box SDE solver 4:

Calculate loss L = (h T ).

5::

Calculate gradient ∂L ∂w and ∂L ∂v with autograd.

6::

Update network parameters w and v.

method works efficiently in our experiments, see Figure 3 (left). Furthermore, we find that the discretization error is small enough to perform classification task even for larger grid size in SDE solver (Line 3 in Algorithm 1) as shown in Figure 3 (right). More details can be found in Appendix C.

Note that instead of using automatic gradient computation, we can also calculates the gradients by adjoint method, which first transforms the SDE system in (25) into a deterministic PDE through Feynman-Kac formula (Cattiaux & Mesnager, 2002) . Due to the space limit we defer it to appendix E. Since a simple implementation with auto-differentiation is already achieving competitive run time and approximation with Neural ODE as shown in Figure 3 , we will use that for the experiments throughout this paper.

3.3. ROBUSTNESS OF NEURAL SDE:

In this section, we theoretically analyze the stability of Neural SDE, showing that the randomness term can indeed improve the robustness of the model against small input perturbation. This also explains why noise injection can improve the robustness in discrete networks, which has been observed in literature (Liu et al., 2018; Lecuyer et al., 2018) . First we need to show the existence and uniqueness of solution to (25), we pose following assumptions on drift f and diffusion G. Assumption 1. f and G are at most linear, i.e. f (x, t)

+ G(x, t) ≤ c 1 (1 + x ) for c 1 > 0, ∀x ∈ R n and t ∈ R + . Assumption 2. f and G are c 2 -Lipschitz: f (x, t) -f (y, t) + G(x, t) -G(y, t) ≤ c 2 x -y for c 2 > 0, ∀x, y ∈ R n and t ∈ R + .

Based on the above assumptions, we can show that the SDE (25) has a unique solution (Øksendal, 2003) . We remark that assumption on f is quite natural and is also enforced on the original Neural ODE model (Chen et al., 2018) ; as to diffusion matrix G, we have seen that for dropout, Gaussian noise injection and other random models, both assumptions are automatically satisfied as long as f possesses the same regularities.

We analyze the dynamics of perturbation. Our analysis applies not only to the Neural SDE model but also to Neural ODE model, by setting the diffusion term G to zero. First of all, we consider initializing our Neural SDE (25) at two slightly different values h 0 and h e 0 = h 0 + ε 0 , where ε 0 is the perturbation for h 0 with ε 0 ≤ δ. So, under the new perturbed initialization h e 0 , the hidden state at time t follows the same SDE in (25),

EQUATION

where B t is Brownian motions for the SDE associated with initialization h e 0 . Then it is natural to analyze how the perturbation ε t = h e th t evolves in the long run. Subtracting ( 25) from ( 9)

dε t = f (h e t , t; w) -f (h t , t; w) dt + G(h e t , t; v) -G(h t , t; v) dB t = f ∆ (ε t , t; w) dt + G ∆ (ε t , t; v) dB t .

(10)

Here we made an implicit assumption that the Brownian motions B t and B t have the same sample path for both initialization h 0 and h e 0 , i.e. B t = B t w.p.1. In other words, we focus on the difference of two random processes h t and h e t driven by the same underlying Brownian motion. So it is valid to subtract the diffusion terms.

An important property of ( 10) is that it admits a trivial solution ε t ≡ 0, ∀t ∈ R + and w ∈ R d . We show that both the drift (f ) and diffusion (G) are zero under this solution:

EQUATION

The implication of zero solution is clear: for a neural network, if we do not perturb the input data, then the output will never change. However, the solution ε t = 0 can be highly unstable, in the sense that for an arbitrarily small perturbation ε 0 = 0 at initialization, the change of output ε T can be arbitrarily bad. On the other hand, as shown below, by choosing the diffusion term G properly, we can always control ε t within a small range.

In general, we cannot get the closed form solution to a multidimensional SDE but we can still analyze the asymptotic stability through the dynamics f and G. This is an extension of Lyapunov stability theory to a stochastic system. First we define the notion of stability in the stochastic case. Let (Ω, F, P ) be a complete probability space with filtration {F t } t≥0 and B t be an m-dimensional Brownian motion defined in the probability space, we consider the SDE in (10) with initial value ε 0

EQUATION

For simplicity we dropped the dependency on parameters w and v. We further assume f ∆ : R n × R + → R n and G ∆ : R n × R + → R n×m are both Borel measurable. We can show that if assumptions (1) and ( 2) hold for f and G, then they hold for f ∆ and G ∆ as well (see Lemma A.2 in Appendix), and we know the SDE ( 12) allows a unique solution ε t . We have the following Lynapunov stability results from (Mao, 2007) . Definition 3.1 (Lyapunov stability of SDE). The solution ε t = 0 of (12):

A. is stochastically stable if for any α ∈ (0, 1) and r > 0, there exists a δ = δ(α, r) > 0 such that Pr{ ε t < r for all t ≥ 0} ≥ 1α whenever ε 0 ≤ δ. Moreover, if for any α ∈ (0, 1), there exists a δ = δ(α) > 0 such that Pr{lim t→∞ ε t = 0} ≥ 1α whenever ε 0 ≤ δ, it is said to be stochastically asymptotically stable; B. is almost surely exponentially stable if lim sup t→∞ 1 t log ε t < 0 a.s.1 for all ε 0 ∈ R n .

Note that for part A in Definition 3.1, it is hard to quantify how well the stability is and how fast the solution reaches equilibrium. In addition, under assumptions (1, 2), we have a straightforward result Pr{ε t = 0 for all t ≥ 0} = 1 whenever ε 0 = 0 as shown in Appendix (see Lemma A.3) . That is, almost all the sample paths starting from a non-zero initialization can never reach zero due to Brownian motion. On the contrary, the almost sure exponentially stability result implies that almost all the sample paths of the solution will be close to zero exponentially fast. One important result regarding to stability of this system is Mao (2007) , deferred to Theorem A.1 in Appendix. We now consider a special case, when the noise is multiplicative G(h t , t) = σ • h t and m = 1. The corresponding SDE of perturbation ε t = h e th t has the following form

EQUATION

)

Note that for the deterministic case of ( 13) by setting σ ≡ 0, the solution may not be stable in certain cases (see Figure 1 ). Whereas for general cases when σ > 0, following corollary claims that by setting σ properly, we will achieve an (almost surely) exponentially stable system.

Corollary 3.0.1. For (13), if f (h t , t; w) is L-Lipschtiz continuous w.r.t. h t , then (13) has a unique solution with the property lim sup

t→∞ 1 t log ε t ≤ -( σ 2 2 -L) almost surely for any ε 0 ∈ R n .

In particular, if σ2 > 2L, the solution ε t = 0 is almost surely exponentially stable.

4. EXPERIMENTAL RESULTS:

In this section we show the effectiveness of our Neural SDE framework in terms of generalization, non-adversarial robustness and adversarial robustness. We use the SDE model architecture illustrated in Figure 2 during the experiment. Throughout our experiments, we set f (•) to be a neural network with several convolution blocks. As to G(•) we have the following choices:

• Neural ODE, this can be done by dropping the diffusion term G(h t , t; v) = 0.

• Additive noise, when the diffusion term is independent of h t , here we simply set it to be diagonal G(h t , t; v) = σ t I. • Multiplicative noise, when the diffusion term is proportional to h t , or G(h t , t; w) = σ t h t .

• Dropout noise, when the diffusion term is proportional to the drift term f (h t , t; w), i.e.

G(h t , t; v) = σ t diag{f (h t , t; w)}.

Note the last three are our proposed Neural SDE with different types of randomness as explained in Section 3.1. For more experimental details, such as neural network architecture of f (•) and the solver for Neural SDE, please refer to Appendix B. Note that we use the same architecture for both ODE and SDE, so the comparisons are fair.

4.1. GENERALIZATION PERFORMANCE:

In the first experiment, we show small noise helps generalization. However, note that our noise injection is different from randomness layer in the discrete case, for instance, dropout layer adds Bernoulli noise at training time but not testing time; whereas our Neural SDE model keeps randomness at testing time and takes the average prediction of multiple forward propagations.

As for datasets, we choose CIFAR-10, STL-10 and Tiny-ImageNet 2 to include various sizes and number of classes. The experimental results are shown in Table 1 . We see that for all datasets, Neural SDE consistently outperforms ODE, and the reason is that adding moderate noise to the models at training time can act as a regularizer and thus improves testing accuracy. Based upon that, if we further keep testing time noise and ensemble the outputs, we will obtain even better results. 

4.2. IMPROVED NON-ADVERSARIAL ROBUSTNESS:

In this experiment, we aim at evaluating the robustness of models under non-adversarial corruptions following the idea of (Hendrycks & Dietterich, 2019) . The corrupted datasets contain tens of defects in photography including motion blur, Gaussian noise, fog etc. For each noise type, we run Neural ODE and Neural SDE with dropout noise, and gather the testing accuracy. The final results are reported by mean accuracy (mAcc) in Next, we consider the performance of Neural SDE models under adversarial perturbation. Clearly, this scenario is strictly harder than the previous case: by design, the adversarial perturbations are guaranteed to be the worst case within a small neighborhood (ignoring the suboptimality of optimization algorithms) crafted through constrained loss maximization procedure, so it represents the worst case performance. In our experiment, we adopt multi-step ∞ -PGD attack (Madry et al., 2017) , although other strong white-box attacks such as C&W (Carlini & Wagner, 2017) In this experiment, we take a look at the perturbation ε t = h e th t at any time t. Recall the 1-d toy example in Figure 1 , we observe that the perturbation at time t can be well suppressed by adding a strong diffusion term, which is also confirmed by theorem. However, it is still questionable whether the same phenomenon also exists in deep neural network since we cannot add very large noise to the network during training or testing time. If the noise is too large, it will also remove all useful features. Thus it becomes important to make sure that this will not happen to our models. To this end, we first sample an input x from CIFAR-10 and gather all the hidden states h t at time t = [0, ∆t, 2∆t, . . . , N ∆t]. Then we perform regular PGD attack (Madry et al., 2017) to find the perturbation δ x such that x adv = x + δ x is an adversarial image, and feed the new data x adv into network again so we get h e t at the same time stamps as h t . Finally we plot the error ε t = h e th t w.r.t. time t (also called "network depth"), shown in Figure 5 . We can observe that by adding a diffusion term (dropout-style noise), the error accumulates much slower than the ordinary Neural ODE model.

5. CONCLUSION:

To conclude, we introduce the Neural SDE model which can stabilize the prediction of Neural ODE by injecting stochastic noise. Our model can achieve better generalization and improve the robustness to both adversarial and non-adversarial noises.

We include the omitted proofs, experiment details and performance analysis here.

APPENDIX A OMITTED THEOREMS AND PROOFS:

Theorem A.1. (MAO, 2007) If there exists a non-negative real valued function V (ε, t) defined on R n × R + that has continuous partial derivatives

EQUATION

In particular, if c 3 ≥ 2c 2 , the solution ε t ≡ 0 is almost surely exponentially stable.

We present the proofs of theorems on stability of SDE. The proofs are adapted from (Mao, 2007) . We start with two crucial lemmas.

Lemma A.2. If f , G satisfy Assumption (2), then f ∆ , G ∆ satisfy Assumption (1,2).

Proof. By Assumption (2) on f , G, we can obtain that for any ε

, ε ∈ R n , t ≥ 0 f ∆ (ε, t) + G ∆ (ε, t) ≤ c 2 ε ≤ c 2 (1 + ε ), f ∆ (ε, t) -f ∆ ( ε, t) + G ∆ (ε, t) -G ∆ ( ε, t) ≤ c 2 ε -ε .

This guarantees the uniqueness of the solution of (12).

Lemma A.3. For (12), whenever ε 0 = 0, Pr{ε t = 0 for all t ≥ 0} = 1.

Proof. We prove it by contradiction. Let τ = inf{t ≥ 0 : ε t = 0}. Then if it is not true, there exists some ε 0 = 0 such that Pr{τ < ∞} > 0. Therefore, we can find sufficiently large constant T > 0 and θ > 1 such that Pr(A) := Pr{τ < T and |ε t | ≤ θ -1, ∀ 0 ≤ t ≤ τ } > 0. By Assumption 2 on f and G, there exists a positive constant K θ such that

EQUATION

Let V (ε, t) = -1 . Then, for any 0 ≤ ε ≤ θ and 0 ≤ t ≤ T , we have

EQUATION

where the first inequality comes from Cauchy-Schwartz and the last one comes from (15). For any δ ∈ (0, ε 0 ), we define the stopping time τ δ := inf{t ≥ 0 :

EQUATION

)

Since τ δ ≤ T and ε τ δ = δ for any ω ∈ A, then ( 17) implies

EQUATION

Thus, Pr(A) ≤ δ ε 0 -1 e K θ (1+K θ )T . Letting δ → 0, we obtain Pr(A) = 0, which leads to a contradiction. Table 3 : Model hyper-parameters. We follow the parameter convention in PyTorch (Paszke et al., 2017) . "GAP" means global average pooling Zhou et al. (2016) . 

EQUATION

Note that w are the parameters of our neural network. 

Under smoothness assumptions on f (•) and G(•) in equation ( 25), from Feynman-Kac formula (Cattiaux & Mesnager, 2002) , u(h t , t) satisfies:

EQUATION

with the boundary condition (final condition):

EQUATION

Our objective is to find w that minimize J(h 0 , w) = E(L(h T )|h 0 ; w) = u(h 0 , 0; w)

with u(h, t; w) satisfies the PDE system specified in equation( 27).

This optimization problem can be solve with the adjoint state method, with the forward state solved from the PDE and boundary condition defined in equation( 27). It can be derived that the adjoint state a(h, t) and the gradient ∂J ∂w can be solved by the following equations: The adjoint state satisfies the following system:

∂a ∂t + Σ d i=1 ∂ ∂h i [f i (h, t; w)a] - 1 2 Σ d i,j=1

∂ 2 ∂h i ∂h j [[GG ] i,j a] = 0 (30) with the boundary condition (initial condition):

a(h, 0) = δ(h 0 ) (31)

:

"a.s." is the abbreviation for "almost surely".

Downloaded from https://tiny-imagenet.herokuapp.com/

Figures and Tables:

Figure1: Toy example. By comparing the simulations under σ = 0 and σ = 2.8, we see adding noise to the system can be an effective way to control x t . Average over multiple runs is used to cancel out the volatility during the early stage. It is noteworthy that here we employ the multiplicative noise, where the deviation term scales proportional to x t .

Forward and backward propagation of Neural SDE 1: procedure TRAINING-PROCESS Do forward & backward propagation 2:

Figure4: Comparing the robustness against 2 -norm constrained adversarial perturbations, on CIFAR-10 (left), STL-10 (middle) and Tiny-ImageNet (right) data. We evaluate testing accuracy with three models, namely Neural ODE, Neural SDE with multiplicative noise and dropout noise. Next, we consider the performance of Neural SDE models under adversarial perturbation. Clearly, this scenario is strictly harder than the previous case: by design, the adversarial perturbations are guaranteed to be the worst case within a small neighborhood (ignoring the suboptimality of optimization algorithms) crafted through constrained loss maximization procedure, so it represents the worst case performance. In our experiment, we adopt multi-step ∞ -PGD attack(Madry et al., 2017), although other strong white-box attacks such as C&W(Carlini & Wagner, 2017) are also suitable. The experimental results are shown in Figure 4. As we can see Neural SDE with either multiplicative noise or dropout noise are more resistant to adversarial attack than Neural ODE, and dropout noise outperforms multiplicative noise.

Figure 7: Running time comparison (forward propagation) between Neural SDE, Neural ODE and Neural ODE -adjoint. The curves are largely overlapped, meaning all methods have running time proportional to network depth.

We propose a new Stochastic Differential Equation (SDE) framework to incorporate randomness in continuous neural networks. The proposed random noise injection can be used as a drop-in component in any continuous neural networks. Our Neural SDE framework can model various types of noises widely used for regularization purpose in discrete networks, such as dropout (Bernoulli type) and Gaussian noise. • We carry out a theoretical analysis of the stability conditions of the Neural SDE network, to prove that the randomness introduced in the Neural SDE network can stabilize the dynamical system, which helps improve the robustness and generalization ability of the neural network. • We verify by numerical experiments that stochastic noise injection in the SDE network can successfully regularize the continuous neural network models, and the proposed Neural SDE network achieves better robustness and improves generalization performance.

Table content: <table/>

Evaluating the model generalization under different choices of diffusion matrix G(h t , t; v) introduced above. For the last three noise types, we search a suitable parameter σ t for each so that the diffusion matrix G properly regularizes the model. TTN means testing time noise. Model size is counted by #parameters.

Table content: <table><tr><td>Data</td><td>Model size</td><td/><td colspan="2">Accuracy@1 -w/o TTN</td><td/><td/><td colspan="2">Accuracy@1 -w/ TTN</td><td/></tr><tr><td/><td/><td colspan="8">ODE Additive Multiplicative Dropout ODE Additive Multiplicative Dropout</td></tr><tr><td>CIFAR-10</td><td>115 K</td><td>81.63</td><td>83.65</td><td>83.26</td><td>83.60</td><td>-</td><td>83.89</td><td>83.76</td><td>84.55</td></tr><tr><td>STL-10</td><td>2.44 M</td><td>58.03</td><td>61.23</td><td>60.54</td><td>61.26</td><td>-</td><td>62.11</td><td>62.58</td><td>62.13</td></tr><tr><td>Tiny-ImageNet</td><td>2.49 M</td><td>45.19</td><td>45.25</td><td>46.94</td><td>47.04</td><td>-</td><td>45.39</td><td>46.65</td><td>47.81</td></tr></table>

Table 2 by changing the level of corruption. Both models are trained on completely clean data, which means the corrupted images are not visible to them during the training stage, nor could they augment the training set with the same types of corruptions. From the table, we can see that Neural SDE performs better than Neural ODE in 8 out of 10 cases. For the rest two, both ODE and SDE are performing very close. This shows that our proposed Neural SDE can improve the robustness of Neural ODE under non-adversarial corrupted data. Testing accuracy results under different levels of non-adversarial perturbations.

Table content: <table><tr><td>Data</td><td>Noise type</td><td colspan="5">mild corrupt ← Accuracy → severe corrupt Level 1 Level 2 Level 3 Level 4 Level 5</td></tr><tr><td/><td>ODE</td><td>75.89</td><td>70.59</td><td>66.52</td><td>60.91</td><td>53.02</td></tr><tr><td>CIFAR10-C  †</td><td>Dropout</td><td>77.02</td><td>71.58</td><td>67.21</td><td>61.61</td><td>53.81</td></tr><tr><td/><td>Dropout+TTN</td><td>79.07</td><td>73.98</td><td>69.74</td><td>64.19</td><td>55.99</td></tr><tr><td/><td>ODE</td><td>23.01</td><td>19.18</td><td>15.20</td><td>12.20</td><td>9.88</td></tr><tr><td>TinyImageNet-C  †</td><td>Dropout</td><td>22.85</td><td>18.94</td><td>14.64</td><td>11.54</td><td>9.09</td></tr><tr><td/><td>Dropout+TTN</td><td>23.84</td><td>19.89</td><td>15.28</td><td>12.08</td><td>9.44</td></tr><tr><td colspan="6">† Downloaded from https://github.com/hendrycks/robustness</td><td/></tr><tr><td colspan="3">4.3 IMPROVED ADVERSARIAL ROBUSTNESS</td><td/><td/><td/><td/></tr></table>

are also suitable. The experimental results are shown in Figure 4. As we can see Neural SDE with either multiplicative noise or dropout noise are more resistant to adversarial attack than Neural ODE, and dropout noise outperforms multiplicative noise.

Table content: <table><tr><td>4.4 VISUALIZING THE PERTURBATIONS OF HIDDEN STATES</td><td/><td/><td/><td/><td/></tr><tr><td colspan="2">0.100 0.125 ∥ht∥ ∥h δ t -ht∥</td><td>ODE SDE</td><td/><td/><td/></tr><tr><td>Relative perturb</td><td>0.000 0.025 0.050 0.075</td><td/><td/><td/><td/></tr><tr><td/><td>0.0</td><td>0.2</td><td>0.4</td><td>0.6</td><td>0.8</td><td>1.0</td></tr><tr><td/><td/><td colspan="3">Network depth t</td><td/></tr><tr><td colspan="7">Figure 5: Comparing the perturbations</td></tr><tr><td colspan="7">of hidden states, ε t , on both ODE and</td></tr><tr><td colspan="7">SDE (we choose dropout-style noise).</td></tr></table>

Our new loss function for the stochastic neural network is the expectation of the original loss function (e.g. softmax) L acting on the end time state h T (counterpart to the last layer of discrete neural network), conditioned on initial state h 0 , or intermediate state h t . Let's denote this expectation as u(h t , t)

Table content: <table/>

Peer review (iclr2020):
This paper proposes a new framework, named Neural Stochastic Differential Equation (Neural SDE), for regularizing and stabilizing Neural Ordinary Differential Equation (Neural ODE). For this reason authors consider SDE by adding stochastic term to ODE. They propose three alternatives for stochastic term and provide experimental comparison between all of them and pure Neural ODE. Results of these experiments demonstrate that Neural SDE outperforms Neural ODE. Overall, proposed concept is quite reasonable but supplied experiments have methodological issues. Therefore, I suppose this work is not a sufficient contribution to neural continuous models and could be rejected. 1. There are several controversial questions, which do not allow me to put a higher score: The information provided to set up the experiment is not enough to reproduce. Learning rate, optimizer, batch size and other parameters are missing. I have conducted experiments for CIFAR10 dataset for baseline Neural ODE model using authors’ network architecture, standard data augmentation and those parameters - optimizer: sgd, learning rate: 0.1, momentum: 0.9, batch size: 128, integration method: euler, amount of steps: 10, with learning rate decay by 10 times on 60, 100, 140 epoch. With this parameters I am able to achieve 85.40% accuracy, while best result for Neural SDE model is 84.55%. Given this, it is very difficult to draw any conclusions about the effectiveness of Neural SDE. 2. Supplied experiments are various and shows that Neural SDE achieves better test accuracy and is more robust to adversarial and non-adversarial perturbations to the input. Nevertheless, all of the experiments are conducted with relatively small models, which achieve low test quality. One cannot guarantee that obtained results will remain the same for large models. Therefore, judgement about significance of these results gets clouded. Moreover, I suppose this issue to be important because authors state their model as a way to regularize Neural ODE. That means, that the initial model has to be large enough to be able to overfit. 3. The main advantage of Neural ODE is memory efficiency of the training procedure, which is achieved by adjoint method. Authors suggest their framework as regularizing technique to Neural ODE, but they conduct experiments without adjoint method. Therefore, it is hard to make any conclusions about advantages of proposed model in comparison with others. In order to demonstrate that Neural SDE saves memory efficiency feature authors should conduct experiments with Neural SDE with adjoint if this is possible. For making this paper better I would suggest to conduct experiments with stronger baseline networks. Currently accuracies for baseline networks are very low. Also I would like to make some notes which can help to clarify text of the paper: In the end of the second paragraph of introduction authors say that regularization methods are not applicable to Neural ODE, since Neural ODE is deterministic system. But ordinary neural networks are also deterministic, so this idea is not clear. In the part 3.3 ROBUSTNESS OF NEURAL SDE authors say that assumption on function f and G quite natural and is enforced in the original Neural ODE paper. As I could find there is no first assumption about at most linearity in original paper. Maybe authors can add some additional argumentation about this assumption. Experiment in section 4.1 demonstrates that Neural SDE achieves better test accuracy than Neural ODE. Given this, authors state that noise injection acts like a regularization and therefore Neural SDE achieves better generalization performance. However, it would be more obvious for readers if train accuracies were also provided. Page 6, Definition 3.1 B: it is not clear with respect to which parameter supremum is taken. In the main body of the paper there are a lot of references to formula 25 (from appendix), but almost the same formula 3 is in the main text. Questions for authors: Do you have some intuition why Dropout without TTN works worse than ODE for TinyImageNet in experiment session 4.2? And why different methods work differently for different datasets in experiment session 4.3? What is the size of ensemble for TTN results? Do you search noise parameter by the best validation or test accuracy?