Neural Stochastic Control
Authors: Jingdong Zhang
Qunxi Zhu
Wei Lin
Venue: NeurIPS 2022 Accept (2022)
Keywords: Stochastic control
differential equations
neural networks

Abstract: Control problems are always challenging since they arise from the real-world systems where stochasticity and randomness are of ubiquitous presence.  This naturally and urgently calls for developing efficient neural control policies for stabilizing not only the deterministic equations but the stochastic systems as well.  Here, in order to meet this paramount call, we propose two types of controllers, viz., the exponential stabilizer (ES) based on the stochastic Lyapunov theory and the asymptotic stabilizer (AS) based on the stochastic asymptotic stability theory.  The ES can render the controlled systems exponentially convergent but it requires a long computational time; conversely, the AS makes the training much faster but it can only assure the asymptotic (not the exponential) attractiveness of the control targets. These two stochastic controllers thus are complementary in applications. We also investigate rigorously the linear control in both convergence time and energy cost and numerically compare it with the proposed controllers in these terms.  More significantly, we use several representative physical systems to illustrate the usefulness of the proposed controllers in stabilization of dynamical systems.

Neural Stochastic Control
Jingdong Zhang' Qunxi Zhu
Wei Lin 123,,,55+
School of Mathematical Sciences, SCMS, SCAM, and CCSB, Fudan University
Research Institute Intelligent Complex Systems and MOE Frontiers Center for Brain Science, Fudan University
3 Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University
State Key Laboratory of Medical Neurobiology, Institutes of Brain Science, Fudan University Shanghai Artificial Intelligence Laboratory
{zhangjd20, qxzhu16, wlin]@fudan edu. cn
Abstract
Control problems are always challenging since they arise from the real-world systems where stochasticity and randomness are of ubiquitous presence. This naturally and urgently calls for developing efficient neural control policies for stabilizing not only the deterministic equations but the stochastic systems as well. Here, in order to meet this paramount call, we propose two types of controllers, viz., the exponential stabilizer (ES) based on the stochastic Lyapunov theory and the asymptotic stabilizer (AS) based on the stochastic asymptotic stability theory. The ES can render the controlled systems exponentially convergent but requires long computational time; conversely, the AS makes the training much faster but it can only assure the asymptotic (not the exponential) attractiveness of the control targets. These two stochastic controllers thus are complementary in applications. We also investigate rigorously the linear controller and the proposed neural stochastic controllers in both convergence time and energy cost and numerically compare them in these two indexes. More significantly, we use several representative physical systems to illustrate the usefulness of the proposed controllers in stabilization of dynamical systems.
1 Introduction
In the field of controlling dynamical systems, one of the major missions is to find efficient control policies for stabilizing ordinary differential equations (ODEs) targeted equilibriums. The policies for stabilizing linear polynomial dynamical systems have been fully developed using the standard Lyapunov stability theory, e.g., the linear quadratic regulator (LQR) Khalil (2002) and the sum- of-squares (SOS) polynomials through the semi-definite planning (SDP) Parrilo (2000). As for stabilizing more general and nonlinear dynamical systems, linearization technique around the targeted states is often utilized and thus the existing control policies are effective in the vicinity of the targeted states (Sastry& & Isidori, 1989) but likely lose efficacy in the region far away from those states. Moreover, in real applications, the explicit forms of the controlled nonlinear systems often partially or completely unknown, so very difficult to directly design controllers only using the Lyapunov stability theory. To overcome these difficulties, designing the controllers via training neural networks (NNs) become one the mainstream approaches in the community cybernetics (Polycarpou, 1996). Recent outstanding developments using NNs include enlarging the safe region
"To whom correspondence should be addressed: Q.Z. and W.L,
https://faculty.fudae edu. wat.e///nndduu.//w/////h CN/inde.hhm
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(Richards et al. 2018), learning the stable dynamics (Takeishi & Kawahara 2021), and constructing the Lyapunov function and the control function simultancously (Chang al. 2019). In Kolter Manek (2019) a projected NN has been constructed to directely learn stable dynamical system and fits the observed time series data well, but it did not focus on learning a control policy to stabilize the original dynamics. All these existing developments are formulated only for deterministic systems but inapplicable directly to the dynamical systems described by stochastic differential equations (SDEs), requiring us to include the stochasticity appropriately into the use of neural controls to different types of dynamical systems.
The stability theory for stochastic systems has been systematically developed in the past sev- eral decades. Representative contributions in
Original system
Orbit
Initial state
dato)
foo gix)d
Equilibiium
the literature include the Lyapunov-like stability theory for SDEs Mao (2077), stabilization of unstable states in ODEs only using noise per- turbations Mao (1994b), and the stability in- duced by randomly switching structures Guo al. (2018). Generally, for any SDEs governed by dæ = dt + ((()g(())))) (B,, control policies as (uy, Mgy are introduced, which transforms the original equations into the controlled system dæ [x() + (x)]]d + [g(x) Appropriate forms of control policies are able to steer the controlled system the equilibriums that are unstable in the original SDEs. Tradi- tional control methods focus on designing deter- ministic control y and regard noise as negative part. Innovatively, we treat noise as a beneficial part and design stochastic control n achieve the stabilization.
dxtt)
fooddt (gix) ucx)) dB,
Controlled system
Controlled
Fast
Exponential Stabiliser
High
Contipotatioall cost
OPI
Asympyutic Stabilizer
ug (z((z) (x))) JdB,
Lows Max fial ylal
Slow
Low
Figure Sketches the two frameworks of neural stochastic controller. Both the ES and AS find control function with fully connected feedforward NN (FNN).
In this article, we articulate two frameworks neural stochastic control which can complement each other in terms of convergence rate and the computational time of training NNs. Additionally, analytically investigate the convergence time and the energy cost for the classic linear control and the proposed neural stochastic control and numerically compare them. We further extend frameworks to model-free case with existing data reconstruction methods. The major contributions of this article multi-folded, including:
designing two frameworks of neural stochastic control, viz., the ES and the AS, and present- ing their advantages in the stochastic control, providing theoretical estimation for ES/AS and classic linear control in terms of convergence time and the energy cost,
computing the convergence time and the energy cost of particular stochastic neural control, demonstrating the efficacy of the proposed stochastic neural control in important control problems arising from representative physical systems. and we make our code available at bettp:///githn wttchuu..ooomaecc hommmjinnddoongzzie wzhagggg-hhhhagga
1.1 Related Works
Lyapunov Method in Machine Learning The recent work Chang et al. (019) proposed an NN framework of learning the Lyapunov function and the linear control function simultancously for stabilizing ODEs. comparison, we select several specific types of NNs which have typical properties of the I Lyapunov function. For instance, we use the input convex neural network (ICNN)
Amos et al. (2017). constructing positively definite convex function as a neural Lyapunov function (Kolter & Manek 2019 Takeishi & Kawaharal 2021), and construct the NN in quadratic form
(Richards et al. 2018 Gallieri et al. 20119) for linear or sublinear systems where the SDP method is
often used to find the SOS-type Lyapunov function (Henrion & Garulli 2005 Jarvis-Wloszek et al.
2003 Parrilo 2000).
Stochastic Stability Theory of SDEs Stochastic stability theory for SDEs have been system-
atically and fruitfully achieved in the past several decades (Kushner) 1967, Arnold 2007] Mao
1991 1994a). The positive effects of stochasticity have also been cultivated in control fields Maa

---

et al. 2002 Deng et al. 2008 Caraballo et al. 2003 Appleby et al. 2008 Mao et al. 2007). These, therefore, motivate us develop only neural stochastic control stabilize different sorts dynamical systems in this article. More stochastic stability theory for different kinds of systems are included in Appleby et al. (2006); Appleby 2003); Caraballo Robinson (2004); Wang Zhu (211)).
Preliminaries
To begin with, consider the SDE which is written in general form as:
dx=(t) (()())) G( 2 0, æ=0) E R
(I)
where F Rd - Rd is the drift function, and G: Rd -> Rd is the diffusion function with Rdxr, a space of d x matrices with real entries, and B, E IR", r-dimensional (r-D) Brownian motion. Without loss of generality, we set F(O)= = 0 and G( = 0 so that Zo = 0 is zero solution Eq. ①
Notations. Denote by || the L -oom for any given vector in R Denote by the absolute value of scalar number the modulus length of complex number number. For A (a:y), matrix of dimension r, denote by ||All >i E' oig the Frobenius norm.
Assumption 2.1 (Locally Lipschitzian Continuity) For every integer n 2 1, there is a number Kn 0 such that
FF(x) F S K yll. [[G(x) - G(y)lle S K2I[ - yyl.
for any æ, y E Rd with ||a||| V |/ll S n. Definition 2.1 (Derivative Operator) Define the differential operator L associated with Eq. ① by
0²
F((x))
+
Σ [G(x)) (æ)) Or,0s, i,j=1
Oz:
Definition 2.2 (Exponential Stability) The zero solution of Eq. T is said to be almost surely expo-
nentially stable, if lim sup, 1 t log [|= æo))l < 0 a.s. for all Dy E Rdd. Here and throughout, a.s. stands for the abbreviation of almost surely.
Then, the following Lyapunov stability theorem will be used in the establishment of our main results.
Theorem 2.2 Mao (2007) Suppose that Assumptions h.lds. Suppose further that there exist > 0, > 0, C2 R and 2 0, such that (i) [ 1 < V(x). (ii) (x) S c2 V (x), and (iii) VVV (2 (x)((x)) 2 (x) for all x # 0 and
function V E C"RR
R with V(0) O, constants p
t 0. Then,
lim sup log (= to æo)ll C3 2cc
1
lim tutx t
a.s..
2p
In particular, c3 202 > 0, the zero solution of Eg. ① is exponentially stable almost surely.
The following asymptotic theorem also will be used in the establishment of our main results. Theorem 2.3 Appleby et al.) (2008) Suppose that Assumptions 2.7 holds. Suppose further
| G(x) for any M > and there exists number E (0, 1) such that
()(( [| (2 a) G(z) < 0, Va Rd.
(3)
Then, the unique and global solution of Eq. satisfies lim (tt, mp) 0 a.s., and we call this property as asymptotic attractiveness.
Designing Stable Stochastic Controller
Here, we assume that the zero solution of the following SDE:
dx = f fxx (tt) + g(æ)dB,
(4)
unstable. Note that, for any nontrivial targeted equilibrium x", direct transformation y # xr make the zero solution the equilibrium of the transformed system. Thus, mission is to

---

stabilize the zero solution only. As such, we are to use the NNs to design the control u R - Rdxr with u()) and apply Eq.
dæ f())dt + [g(=)) u(x))dBB
(5)
Since integrated with d. the controlled system (5) we regard it as a stochastic controller. In what follows, two frameworks of neural stochastic control, the exponential stabilizer (ES) and the asymptotic stabilizer (AS), are articulated, respectively, Sections 3.1 and 3.2 All these control policies are intuitively depicted in Figure]
3.1 Exponential Stabilizer
Once we find the Lyapunov function V and the neural controller u, making the controlled system (5) meet all the conditions assumed in Theorem 2.2 the equilibrium 0 can be exponentially stabilized. To this end, we first provide two different types of functions for constructing V, which actually could be complementary in applications. Then, we design the explicit forms of control function and loss function.
ICNN V Function. We use the ICNN (Amos al., 2017) represent the candidate Lyapunov function V. This guarantees V as convex function with respect to the input æ. order further make V as true Lyapunov function, we use the following form:
#1 do (Wox bu), 2i+1 0 (U,2)
Wa b.),
((x) = Zk. = 1...,, , ,k-1,
(6)
(xx) Tk+i ( ((F())))) + +
as introduced in Manek & Kolter 2020). Here, W,, by are real-valued weights, U2 are positive weights, 0; are convex, monotonically non non-decreasing activation functions in the i-th layer, is small positive constant, and FF is a continuously differentiable and invertible function. In our framework, we require V E (Rd.) R.) according to Definition 2.1 however, each activation function in Manek & Kolter 2020) is C' only. Thus, we modify the original function as:
((1)
0, (2dr3 x¹ )/23 x d/2.
if S 0,
(xx)
if x < d, otherwise
(7)
Figure The smoothed ReLU or-))
which not only approximates the typical ReLU activation but also becomes continuously differentiable to the second order (see Figure 2
Quadratic V Function. For any input æ and with L-nh(-)) as the activation functions, where the parameter vector. To meet the condition used in Definition 2. we cannot use the ReLU, non-smooth function, as the activation function. Hence, we use the candidate Lyapunov function as:
Rd, let Vo E Rd be multilayered feedforward NN of the
V(x) x T [87 + Va(x:) Vo(=)) x,
(8)
which was introduced in Gallieri et al. (2019). Here, is small positive constant.
Control Function. We introduce multi-layer feedforward NN (FNN), denoted by NN(x) E R". to design the controller u. Since we require u(0) O, we set u(x) & NN(x)) NN(O) or
(xx) A diag(z))N(()) with = d. Here, diag(æ) (x) is a diagonal matrix with its i-th diagonal element as
As reported in Chang et al. (2019), single-layer NN without the bias constants in its arguments, which degenerates as linear control. could sufficiently take effect in the stabilization of many deterministic systems. However, this is NOT always the case for achieving the stabilization of highly nonlineur systems or even SDEs. The following proposition with F provides an example, where neither the classic linear controller nor the stochastic linear controller can stabilize the unstable equilibrium in a particular SDE. The proof of this proposition is included in Appendix A.1
Remark 3.1

---

Proposition 3.2 Consider the following 1-D SDE:
dx(()= x() log (x(t) + ulx(())dBB
(9)
with a solution x² O. Then, for u(æ) kæ with any lo and zo * 0. x² 0 is neither
exponentially stable nor of globally asymptotic attractiveness almost surely. For u(x) = 2x² xx
of globally asymptotic attractiveness. For u(x) O, the determinisfic system cannot be stabilized by any classic linear controller.
Loss Function. When the learning procedure updates the parameters in the NNs such that the constructed V and u with the coefficient functions, J and Su - 9 + u. in the controlled
(00
1013
120
10
50
system 55 meet all the conditions assumed in
Time 2.0
Theorem 2.2 the exponential stability of the controlled system is assured. Thus, we demand a suitable loss function to evaluate the likelihood that those conditions are satisfied. First, from Theorem 2.2 it follows that V(x) 2 e||a=||² for all x E Rd, Thus, Conditions (ii)-(iii) together with
Figure 3: (a) u(x) = kx. (b) x(x)) = 2x
C3 202 > Theorem 2.2 equivalently become
(V((2) ((((=)]T[((ee (u())))
sup (V b> 2.
inf my10 V(x)² 2 b sup
(10)
2#0 V(z)
These conditions further imply that
vv)))) (x)) 9())))
(a V(x)
2 b> 2, #0.
(11)
V(x)²
With these reduced conditions, we design the ES loss function for the controlled system (5) as follows. Definition 3.1 (ES loss) Consider a candidate Lyapunov function V and controller u for the controlled system 5 Then, the ES loss is defined as
Ln,,, u) = Enzp max (0, b LV(x) (VV(x) (V ]}
V(x)
V(x)²
where the state variable æ obeys the distribution μ. In practice, we consider the following empirical loss function:
LN...((O. u) = Σ max (o. b (v:)) (VV(x;) (xx)))) (
1 N
(12)
N 1=1
V(x;)
V(x,)²
where { are sampled from the distribution μu (u(2)) and is some closed domain in Rd
For convenience, we summarize the developed framework in Algorithm
I Here, b is hyper-
parameter that can be adjusted as required by solving specific problem.
Remark 3.3 Section 5 we show numerically that the conditions reduced in (T) are sufficiently effective for designing the ES loss. Actually, it is not necessary to design the loss function using the conditions in (T0).
Now, for controlling any nonlinear ODEs or SDEs, we design the ES according to Algorithm [ As such, using the ES framework not only stabilize those unstable equilibriums (constant states) of the given systems, but also can stabilize those unstable oscillators, e.g., the limit cycle. This is because the solution corresponding to the oscillator can be regarded zero solution of the controlled system after appropriate transformations are implemented.
Another point needs attention. During the construction of V in (6), =||=|| the L L2-regularization, is used to guarantee the positive definiteness of V. However, often in the application of the Lyapunov stability theory, the form of ||=|| is not always a suitable candidate for the Lyapunov function. It may restrict the generalizability of using our framework, so needs necessary adjustments. The following example illustrates this point.
5

---

Example 3,4 Consider 2-D SDE as follows:
{ [d (dzi(t) (t) = =t)ddtt
(dr2(1) =
(t))dB
In Appendix A.2 the zero solution of this system is validated to be exponentially stable almost surely: however, 2||==||= 10 for any k IR cannot be usefud auxiliary function to identify the exact stability of the zero solution.
To be candid, using the current framework takes longer time for training and constructing the neural Lyapunov function. In the next subsection, we thus establish an alternative control framework that can reduce the training time.
3.2 Asymptotic Stabilizer
Here, in light of Theorem 2.3 we are to establish the second framework, the AS, for stabilizing the unstable equilibrium of system (5). This framework only makes the equilibrium asymptotically attractive almost surely. Its control function is designed in the same way as the one used in the ES framework, whereas the loss function differently designed.
Definition 3.2 (AS loss) Utilization of the notations used Definition with the controller u is defined as:
3.1 the loss function for the
controlled system 5
Li uu)
E Ex
max (0, (a 2a Iu (=x(=)]]==
| (2 ((x)) + (1)x()))))))
Akin to Definition 3.1 we set the empirical loss function as:
1 (N) = N Σ max (0, (a 2]][<x] 9u (x;))1 + |ax:|/2((x, f(xx)) + (x;) (x))] (13)
Here, a is an adjustable parameter, which is related to the convergence time and the energy cost using the controller u. We show in Appendix AA.8 the influence of selecting different a. For convenience, we summarize the AS framework in Algorithm
Convergence Time and Energy Cost
The convergence time and the energy cost are the crucial factors to measure the quality of a controller Yan et al. 2012 Li et al. 2017; Sun et al. (017). In this section, we provide a comparative study between the traditional stochastic linear control and the ES/AS, the above-articulated neural stochastic control.
To this end, first present theorem the estimations of the convergence time and the energy cost for the stochastic linear control on a general SDE.
Theorem 4.1 Consider the SDE with stochastic linear controller as:
dæ = f(x)dt + u(w)dBr, x00) 20 E I
(14)
where (x. f(x)) I L|[ and u(x)= = kæ with [k] > V2L. Then, for e < ||00l|| we have
2 2log (//=)l//c)
E[T.] TT
22
2L
k22 [ 2
22(k2 + 22) log (|
-1].
E(T,,
<
exp
1.2 +
2T.
22
where, for a sufficiently small and denote the energy cost by
> we denote the stopping time by Te A inf(t > 0: ||m((]]] c)
T.AT.
[[ul(2dt) = EE/"" ||
E(Te, Te) 2 E

---

The proof of this theorem is provided in Appendix A.3.1
We further consider the for NN controller (x)). In general, the u(x:) is Lipschitz continuous with Lipsheitz constant ki under suitable activation function such as ReLU in NN (Fazlyab et al.
2019; Pauli al. (021). Then have the following upper bound estimation of the convergence
time and the energy cost for ES and AS, whose proofs are provided in Appendix [A.3.2] A.3.3
Theorem 4.2 (Estimation for ES) For ES stabilizer u(x) in (14) with (x, f(x)) s L| E
|| under the same notations and conditions in Theorem 2.2 with C3 2c² > 0, we have
2 log (V(z))/11))
B|Te] T.
c3 202
Ru 21 x 1
2(ku 2 + 2L) log (V(zo) (ca))
(Ter Tc)
exp
kiu + 2L
C3 20g
Theorem 4.3 (Estimation for AS) For 14 with (x, f(x)) I L L| E Toll under
same notations and conditions in Theorem 2.3 the left term in
3
further satisfies
|a [|=]|= (2(x, [(x)) +
||(u(a))l+) (2 a)il (²))
6g
0, then for
NN controller (xx) with Lipschizt constant ku, we have
2 (1120) EM
[ S
E(Te, TT)
hu 2|ool
exp
2(kku + 2L) (1 E²)
-1
kin + 2L
..
Based the theoretical results for ES and AS, can further analyze the effects of hyperparameters b, a and neural network structures on the convergence time and energy cost in the control process. There are some interesting phenomena such the monotonicity of I' along a for AS change with the relative relationships of ||>p|| and E, this inspires us to select suitable according to the specific problems. We leave more discussions in Appendix [A.3.4
Now, we numerically compare the performances of the linear controller u(x) = ko and the AS on the convergence time and the energy cost of the controls applied to system (14) with specific configurations (see Figure
4). We numerically find that u(x) = kr can efficiently stabilize the fix k 6.0, and compare the
equilibrium for k Kg 5.6. Without loss of generality, corresponding performances. As clearly shown in Figure 4 the AS outperforms u(x) kr from both perspectives, the convergence speed and the energy cost. In the simulations, the energy cost E (Te, (T) defined above is computed in a finite-time duration as E A T.). where T < oo is selected be appropriately large. We leave more results of the comparison study in Appendix A.5.5
(a)
15
(b) 200
(c)
log(1 +x)
Linor co
10
0
6
8
10
0.25
0.5
0
0.075
0.15
Time
Time
Figure 4: The performances of system 14 with specific configurations: f(x) log(1 + .). (a) m(x) = kr. k = 0.2 j.j = 1, 50. plot log(1 + x(1)) against k. (b) Linear controller with 6.0, E(To.1, 1) = 38, 388. (c) AS control with E(To. 1) = 1438
Experiments
In this section, we demonstrate the efficacy of the above-articulated frameworks of stochastic neural control, the ES and the AS, on several representative physical systems. We also compare these two frameworks, highlighting their advantages and weaknesses. The detailed configurations for these experiments are included in Appendix A.5 Additional illustrative experiments are included in Appendix A.6
5.1 Harmonic Linear Oscillator
7

---

First, consider the harmonic linear oscillator ÿ + 2,8 + way = O, where w is the natural fre- quency and 3 >> is the damping coefficient rep- resenting the strength of the external force on the vibrator (Dekker 1981). Although this system is exponentially stable, the system with stochastic perturbations ji+ (2
Original system
Controlled system
10-
10
1.0
2.0
3.0
-0.
2.1
2.5
28
3.0
2.1
2.6
23
30
20
20
(w++(ww++4)) = 0 be- comes unstable even if [E (t)] = [t] = (Amoll et al. 1983). Now, we apply the non- linear ES(+ ES(+ICNN) the linear ES(+Quadratic), and the nonlinear AS, respectively, to stabiliz- ing this unstable dynamics (A.5.I) with w² = 1, 3 = 0.5, C1 3, and (2 2.15, the results are shown in Figure [5] Indeed, we find that the two nonlinear stochastic neural controls more robust than the linear control, and that the ES(+ICNN), rather than the AS, makes the controlled system more stable.
R
1.0
2.0
3.0
1.0
2.0
3.0
Time
Time
Figure 5: The solid lines are obtained through averaging the 20 sampled trajectories, while the shaded areas stand for the variance regions.
In Table T we compute the training time (Tt) used for the loss function converging to 0, the
Table 1: Performance on Harmonic Linear Oscilla-
tor
number of iterations (Ni), the distance (Di) be-
tween the trajectory and the targeted equilibrium at time T 4, and the convergence time (Ct) when the distance between the trajectory and the targeted equilibrium is less than 0.05. The re- sults are obtained through averaging the corresponding quantities produced by 20 randomly-sampled trajectories and the detailed training configurations are shown in Appendix A.5.I
Tt
Ni
Di
Ct
ES ES(NICNN)
276.385s
121
1e-9
0.459
ES(+Quarratcc)
78.071s
107
0.049
3.083
AS
4.839s
184
0.027
2.027
We further provide a comparison study of our newly proposed ES(+ICNN) with HDSCLF in (Sarkar et al. 2020), BALSA in (Fan et al. 2020) and classic LQR controller in controlling the harmonic linear oscillator. Both HDSCLF and BALSA are based on the Quadratic Program (P)), and they seek the control policy dynami- Figure 6: Comparison with existing methods. cally for each state in the control process. By contrast, our proposed learning control policy is directly used in the control process. Hence, method is more efficient in the practical control problems. The results are shown in Figure. [] see more details in Appendix A.5.I. As can be seen that our learning control methods outperforms all others.
NCSCL
22
0.2
0.4
0.2
0.4
Time
Time
5.2 Stuart-Landau Equations
In this subsection, we show that our frameworks are beneficial to realizing the control and the synchronization of complex networks. To this end, we consider the single Stuart-Landau oscillator which is governed by the following complex-valued ODE:
= = + n4 zZ) ZEC.
(15)
This equation is paradigmatic model undergoing the so-called Andronov bifurcation (Kuznetsov) 2013): Stability of the equilibrium changes and the limit cycle emerges as the parameter passes some critical value. In what follows, we consider two cases based on system [5)
Case We set 9 = 25, 7 = 1, and μ = 1, so that system (5)) has a stable equilibrium p = and an unstable limit cycle 5. Here, 7 = x + iy = pell Now, the AS steers the dynamics to the unstable limit cycle, as successfully shown in Figure 7 The trajectories (the left column) and the phase orbits (the right column) of system (5). initiated from 30 randomly-selected initial states, without control (the upper panels) and with control of the AS (the lower panels). The initial values inside (resp., outside) the limit cycle p = 5 are indicated by the blue (resp.. purple) pentagrams.
Case Next, we consider the synchronization problem of the coupled Stuart- equations. Successful deterministic methods have been systematically developed for realizing synchronization, including the adaptive control with time delay (ellvanovv et al. (2012) and the open-loop temporal network controller Zhang & Strogatz 2021)) These methods majorly depend on the technique
8

---

Wishout Control
With Willout
0.1
Wish Control
Time
Time
Figure 7: The trajectories (the left column) and the phase orbits (the right column) of system (1)).
The dynamics of the first (resp., second) component of the coupled oscillators are shown in the panels in the left (resp., middle) column. The dynamics in the phase space (the right column).
Figure 8:
of linearization in the vicinity of the synchronization manifold. Here, we show how to apply our framework to achieving the synchronization in the coupled system. We set the corresponding Laplace matrix L (Ligk) as >%h Ljk- = O, which guarantees the synchronization manifold is an invariant manifold of the coupled system (Pecora & Carroll, 1998). Specifically, we select as n = 20, a 0.01, Ci = 11.8, C2 = 4, and Ljs = òÿā - -. where dHs is the Kronecker function. Then, we apply the AS to this system and realize the stabilization of the synchronous manifold, as shown in Figure g
5.3 Data-Driven Pinning Control for Cell Fate Dynamics
Indeed, our frameworks can be extended to the model-free version via combination with exist- ing data reconstruction method. To be concrete, we show that our framework can combine with
State variables
Neural ODEs (NODEs) (Chen et al.) 2018) to learn the control policy from time series data for the Cell Fate system (Sun et al.) [017:: Laslo jet al., 2006), which describes the interaction be- tween two suppressors during cellular differentiation for neutrophil and macrophage cell fate choices. The system = fxx) x (51, rg. has three steady states: P1,2,3, where P2.3 correspond to different cell fates and are stable and P1 represents critical expression level connecting the two fates and is unstable. The network structure of this 6-D system is a treemap, where one root node I can stabilize itself under the original dynamic. Hence, we choose root node 22 with maximum our degree and add pinning control on it to stabilize the system to unstable state Pt. The original trajectory that converges to P3 (left) and the controlled trajectory that converges P1 (right) are shown in Figure 17 The original trajectory is used to train the NODE to reconstruct the vector field f. then we use the sample of f as training data to learn our stochastic pinning control. We provide experimental details in Appendix [.5
2
0.0
2.5
Time
Time
Figure 9: Pinning control for cell fate dynamics.
In addition to the above controlled systems, we include the other illustrative examples, the controlled inverted pendulum, reservoir computing and the controlled Lorenz system, in Appendix A.5
6 Conclusion and Future Works
In this article, we have proposed two frameworks of neural stochastic control for stabilizing different types of dynamical systems, including the SDEs. We have shown that the neural stochastic control outperforms the classic stochastic linear control in both the convergence time and the energy cost for typical systems. More importantly, using several representative physical systems, we have demonstrated the advantages of our frameworks and showed part of their weaknesses possibly emergent in real applications. Also, we present some limitations of the proposed frameworks in Appendix A.9 Moreover, we suggest several directions for further investigations: (i) acceleration of the training process of the ES, (ii) the basin stability of the neural stochastic control (Menck et al.) 2013), (iii) the trade-off between the deterministic controller ug using the NNs and the stochastic controller u using the NNs, (iv) the safe learning in Robotic control with small disturbances (Berkenkamp et al. 2017). and (v) the design of the purely data-driven stochastic neural control.

---

7 Acknowledgments
We thank the anonymous reviewers for their valuable and constructive comments that helped us to improve the work. Q.Z is supported by the Shanghai Postdoctoral Excellence Program (No, 2021091) and by the STCSM (Nos. 2151 1100200 and 2 1407300). W.L. is supported by the National Natural Science Foundation of China (No. 11925 103) and by the STCSM (Nos. 1951 22JC1402500 22JC1401402, and 2021SHZDZZX0133,
References
Amos, B., Xu, L., and Kolter, Z. Input convex neural networks. In International Conference on Machine Learning, pp. 146-155. PMLR, 2017.
Anderson, C. W. Learning to control an inverted pendulum using neural networks. IEEE Control Systems Magazine, 9(3):31-37, 1989.
Appleby, A. Stabilisation of functional differential equations by noise. Systems Control Letters, 2003.
Appleby, A., Mao, X., and Rodkina, A. On stochastic stabilization difference equations. Discrete & Continuous Dynamical Systems, 15(3):843, 2006.
Appleby, J. A., Mao, X., and Rodkina, Stabilization and destabilization of nonlinear differential equations by noise. IEEE Transactions on Automatic Control, 53(3):683-66 691, 2008.
Arnold, L. Stochastic Differential Equations: Theory and Applications. Stochastic Differential Equations: Theory and Applications, 2007.
Arnold, L., Crauel, H., and Wihstutz, V. Stabilization of linear systems by noise. SIAM Journal Control and Optimization, 2133):451,461 1983.
Berkenkamp, F., Turchetta, M. Schoellig, A. P., and Krause, A. Safe model-basee reinforcement learning with stability guarantees. arXiv preprint 1rXiv:1755.0853 2017.
Caraballo, T. and Robinson. J. C. Stabilisation of linear pdes by stratonovich noise. Systems & Control Letters, 53(1):41-50, 2004.
Caraballo, T., Garrido-Atienza, M. J., and Real, J. Stochastic stabilization of differential systems with general decay rate. Systems & Control Letters, 48(5):397-406, 2003.
Chang, Y.C.., Roohi, N., and Gao, S. Neural Iyapunov control. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3245-3254, 2019.
Chen, R. T.. Rubanova, Y.. Bettencourt. J., and Duvenaud, D. Neural ordinary differential equations. arXiv preprint arXiv:1806. 100.006666. 2018
Dekker, H. Classical and quantum mechanics of the damped harmonic oscillator. Physics Reports, 80( 1):1-110, 1981.
Deng, F., Luo, Q., Mao, X., and Pang. S. Noise suppresses or expresses exponential growth. Systems & Control Letters, 57(3):262-211 270, 2008.
Fan, D. D., Nguyen, J., Thakker, R., Alatur, N., Agha-mohammadh A.-a., and Theodorou, E. A. Bayesian learning- based adaptive control for safety critical systems. In 2020 IEEE international conference on roboties and automation (ICRA), pp.
IEFE, 2020.
Fazlyab, M., Robey, A., Hassani, H., Morari, M., and Pappas, G. Efficient and accurate estimation of lipschitz constants for deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Gallieri, M., Salehian, S. M., Toklu, N. E., Quaglino, A., Masci, J., Koutník, J., and Gomez, Safe interactive model-based learning, arXiv preprint arXiv: 2019.
10

---

Guo, Y., Lin, W., and Chen, G. Stability of switched systems on randomly switching durations with random interaction matrices. IEEE Transactions on Automatic Control, 63(1):21-36, 2018.
He, K., Zhang. X., Ren, S., and Sun, Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Henrion, D. and Garulli, A. Positive Polynomials in Control, volume 312. Springer Science & Business Media, 2005.
Huang, S.-J and Huang, C.-L. Control of an inverted pendulum using grey prediction model. IEEE Transactions Industry Applications, 36(2):452-4455) 2000.
Jaeger, H. The "echo state" approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148(34):13, 2001.
Jaeger, H. Echo state network. Scholarpedia, 2(9):2330, 2007.
Jaeger, H. and Haas, H. Harnessing nonlincarity: Predicting chaotic systems and saving energy in wireless communication. Science, 304(5667):78-80) 80, 2004.
Jarvis- Wloszek, Z., Feeley, R., Tan, W., Sun, K., and Packard, A. Some controls applications of sum of squares programming. In 42nd IEEE International Conference Decision and Control (IEEE Cat. No. 03CH37475), volume 5, pp. 4676-468 IEEE, 2003.
Khalil, H. K. Nonlinear systems third edition. Patience Hall, 115, 2002.
Kolter, J. Z. and Manek, G. Learning stable deep dynamics models. Advances in Neural Information Processing Systems, 1111112--11366 2019.
Kushner, H. Stochastic stability and control. Technical report, Brown Univ Providence RI, 1967.
Kuznetsov, Y. A. Elements of applied bifurcation theory, volume 112, Springer Science & Business Media, 2013.
Laslo, P., Spooner, J., Warmflash, A., Lancki, D. W., Lee, H.-J., Sciammas, R., Gantner, B. N., Dinner, A. R., and Singh, H. Multilineage transcriptional priming and determination of alternate hematopoietic cell fates. Cell, 126(4):755-766, 2006.
Leong, Y. P., Horowitz, M. B., and Burdick, J. W. Linearly solvable stochastic control Yyapunov functions. SIAM Journal on Control and Optimization, 1066):31 2016.
Li, A., Cornelius, P., Liu, Y.-Y., Wang, L., and Barabási, A.-L. The fundamental advantages of temporal networks. Science, 1042 2017.
Liptser, R. and Shiryayev, A. N. Theory of martingales, volume 49, Springer Science & Business Media, 2012.
Lukoševičius, M. and Jaeger, H. Reservoir computing approaches to recurrent neural network training. Computer Science Review, 3(3):127-149, 2009.
Ma. H., Ho, D. W., Lai, Y.-C., and Lin, W. Detection meeting control: Unstable steady states in high- nigh-dimensional nonlinear dynamical systems. Physical Review 92(4):042902, 2015.
Manek, G. and Kolter, J. Z. Learning stable deep dynamics models. arXiv preprint arXiv:2001.06/ 2020.
Mao, X. Stability of Stochastic Differential Equations with Respect to Semimartingales: Longman, 1991.
Mao, Exponential Stability of Stochastic Differential Equations. Marcel Dekker, 1994a.
Mao, X. Stochastic stabilization and destabilization. Systems Control Letters, 23(4):279-290, 1994b.
Mao, X. Stochastic differential equations and applications. Elsevier, 2007.

---

Mao, X., Marion, G., and Renshaw, E. Environmental brownian noise suppresses explosions in population dynamics. Stochastic Processes and Their Applications, 97(1):95-110, 2002.
Mao, X., Yin, G. G., and Yuan, C. Stabilization and destabilization hybrid systems stochastic differential equations. Automatica, 43(2):264 273, 2007.
Menck, P. J., Heitzig, J., Marwan, N., and Kurths, J. How basin stability complements the linear- stability paradigm. Nature Physics, 9(2):89 2013.
Parrilo, P. A. Structured Semidefinite Programs and Semialgebraic Geometry Methods in Robustness and Optimization. California Institute of Technology, 2000.
Pauli, P., Koch, A., Berberich, J., Kohler, P., and Allgöwer, Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters, 6:121-126, 2021.
Pecora, L. M. and Carroll, T. L. Master stability functions for synchronized coupled systems. Physical Review Letters, 80(10)22109, 1998.
Polycarpou, M. Stable adaptive neural control scheme for nonlinear systems. IEEE Transactions on Automatic Control, 41(3):447-451, 1996. doi: 10. 1 10999.4466444
Richards, S. M., Berkenkamp. F., and Krause, A. The Iyapunov neural network: Adaptive stability
certification for safe learning of dynamical systems. Conference on Robot Learning, pp. 466-476. PMLR, 2018.
Sarkar, M., Ghose, D., and Theodorou, E. A. High-relative degree stochastic control Iyapunov and barrier functions. arXiv preprint arXiv:2004. 200.000000000056,, 2020.
Sastry, S. and Isidori, A. Adaptive control of linearizable systems. IEEE Transactions on Automatic Control, 34(1 1):1 123-1131. 1989. doi: 111009994744
Selivanov, A. A., Lehnert, J., Dahms, T., Hövel, P., Fradkov, A. L., and Schöll, Adaptive synchronization in delay-coupled networks of stuart-landau oscillators. Physical Review E, 85(1): 016201, 2012.
Sparrow, C. The Lorenz Equations: Bifurcations, Chaos, and Strange Attractors, volume 41. Springer Science & Business Media, 2012.
Sun, Y.-Z., Leng, S.-Y., Lai, Y.-C., Grebogi, C., and Lin, W. Closed-loop control of complex networks:
A trade-off between time and energy. Physical Review Letters, 1 (000:9999300) 2017.
Takeishi, N. and Kawahara, Y. Learning dynamics models with stable invariant sets. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9782-9790, 2021.
Tan, W. and Packard, A. Searching for control Iyapunov functions using sums of squares programming. sibi, 1(1), 2004.
Wang, B. and Zhu, Q. Stability analysis of markov switched stochastic differential equations with both stable and unstable subsystems. Systems Control Letters, 105:55-61, 2017.
Xie, C., Wu, Y., Maaten, L. v. d., Yuille, A. L., and He, K. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 501-509, 2019.
Yan, G., Ren, J., Lai, Y.-.., Lai, C.-H., and Li, B. Controlling complex networks: How much energy is needed? Physical Review Letters, 108(21):218703, 2012.
Yan, H., Du, J.. Tan, V. Y., and Feng, On robustness of neural ordinary differential equations. arXiv preprint arXiv=7910.055/3 2019.
Yang, S.-K., Chen, C.-L., and Yau, H.-T. Control of chaos in lorenz system. Chaos, Solitons & Fractals, 13(4):767-780, 2002.
Yang, T., Yang, L..B., and Yang, C.-M. Impulsive control of lorenz system. Physica Nonlinear Phenomena, 110(1-2):1-22) 1997.
12

---

Zhang, Y. and Strogatz, S. H. Designing temporal networks that synchronize under resource constraints. Nature Communications, 12(1):1-8, 2021.
Zhu, Q. Ma, H., and Lin, W. Detecting unstable periodic orbits based only on time series: When adaptive delayed feedback control meets reservoir computing. Chaos: An Interdisciplinary Journal of Nonlinear Science, 29(9):093125, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See the supplementary material.
3. If you experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the supplementary material.
(c) Did you report error bars (c.g., with respect to the random seed after running experi-
ments multiple times)? (Yes] See the Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] The repos we used do not mention the license.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "iq68fX1jewc", "reviewer": "Reviewer_Eycg", "summary": "Prior work on approximating a Lyapunov-like function as a neural network focused on deterministic stabilization where the controls affect the system deterministically. For a system, for which the stochastic noise process can have a non-linear impact on the system, the gains on the noise in itself can be controlled to stabilize or de-stabilize the system. Generating such gain controls requires developing a Lyapunov-like function that satisfies certain criteria. This paper extends previous work that models such functions as a neural network. The hard constraints are converted into a soft loss function and minimized using gradient descent on the neural network. The novelty is in extending deep Lyapunov functions to stochastic stabilization where the noise process itself can be fed back with suitable gains to stabilize the system.", "questions": "For the harmonic linear oscillator, how would an optimal deterministic controller (u=K_{lqr}x) perform compared to the stochastic counterparts. Maybe, previous work showed that neural lyapunov controllers perform better than LQR.\n\nIt would be better if the authors can present the preliminaries in section 2 more elaborately to enable deeper understanding. \n\nIs the derivative operator an extension of the Lie-derivative operator? The derivative operator here is related to Ito’s lemma from stochastic DDE literature.\n\nSimilar to SOS (sum of squares) methods, are there methods that don’t use deep learning to generate V. Is it possible to compare to these? Also, [A] uses deep RL for stochastic control. How does your work qualitatively compare to that?\n\nPlease try to make the figures more readable.\n\n[A] Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications - Bachouch  et al\n", "limitations": "I don't see any potential negative social impact.\n\nThe general technique of learning lyapunov-like functions is limited to lower-dimensional systems as the methods require sampling to enforce the constraints.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The experiments on Stuart-Landau equations, Lorenz oscillator are new contributions and provide strong evidence for the paper’s hypotheses. Instead of convergence to an equilibrium point, the controller is optimized to ensure convergence to a manifold of constant phase for synchronization. This can have real-world applications.\n\n\nLyapunov stability theorems have been studied in non-linear dynamical systems for a long period of time. This paper uses existing theory to bring new problems to the fore. Original theoretical contributions are interesting but not as strong a contribution as they are applicable to the linear controller only. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UtzjKvCJJ7R", "reviewer": "Reviewer_a1wg", "summary": "This paper gives an iterative procedure for computing model-based controllers that stabilize the system with noise. In particular, the control law enters the equations in the form $u(x_t)dB_t$, where $B_t$ denotes Brownian motion. The controllers are designed based on two sufficient conditions for stochastic stability, one requiring a Lyapunov function and the other working directly on the terms of the SDE. These sufficient conditions are used to derive loss functions for optimization-based search algorithms. For the Lyapunov method, both the Lyapunov function candidate and the control law are parameterized by neural  networks. The controller is computed by optimizing the resulting loss with respect to the network parameters over a randomly chosen set of states. In the other approach, no Lyapunov function is needed, and so only the control parameters are optimized. The method is tested in a variety of examples. ", "questions": "From what I can tell, the notation $\\mathbf{u}$ is overloaded to denote simultaneously the control strategy $u(x)$ and the neural network parameters of the control strategy. Is this correct? If it is, it is a bit confusing and an alternative notation to denote the function and the parameters is advisable.", "limitations": "They listed this as N/A, and I agree with the assessment.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper shows how controllers can be designed to solve some interesting and non-standard control design problems. Namely, controlling the system via noise is theoretically established, but substantially less studied than noise-free control. The results show promising performance for a variety of nonlinear stabilization tasks. \n\nEdit: The newer results based on learning the dynamics and partial control show promise. I would still like better comparisons, but this merits a score increase in my view.\n\nWeaknesses:\n\nIt is unclear if the methods in this paper have any advantages over well-known stabilization methods. So, while the methods outperform simple methods that inject noise for control, it is unclear if the method has any benefit over more standard methods that do not inject noise. \n\nMore explanation on the comments above: The methodologies are model based. Furthermore, in all examples where the dynamics, (5), are written explicitly, they are fully actuated. In other words, the inputs directly affect all states. In all cases, very simple feedback-linearization schemes can stabilize these systems. Presumably, other schemes can stabilize them as well. \n\nFirst we consider the case that the only noise is actually due to the controller. This is  the case in the inverted pendulum and the Stuart-Landau systems. This is presumably also the case for the coupled Stuart-Landau oscillators, the echo state network, and the Lorenz system. However, (5) does appear to be written explicitly for these systems, so this is just a guess. Anyways, when the system is fully actuated and the model is fully known, and the system has no intrinsic noise (i.e. no term $g(x)dB$), we could alternatively write the system as \n$$\n\\dot x = f(x) + u(x).\n$$\na simple controller of form \n$$\nu(x) = -f(x) - x\n$$\nSure, we could make the problem harder by injecting noise, but in the fully actuated case, this will do.\n\nIn the simple harmonic oscillator problem, the system has the form:\n$$\ndx = Ax + (g(x)+u)dB,\n$$\nbut again it is assumed that the system is fully actuated. Furthermore, $A$ is Hurwitz.  So, inn this case, $g(x)$ could be cancelled by the controller leaving a stable system. \n\nA more minor weakness is that the theoretical results are completely restricted to model-based stochastic control, and so the learning component is not very strong.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ALfroQiUWB", "reviewer": "Reviewer_7JKN", "summary": "This paper deals with stochastic problems and is in alignment with previous neural network studies inspired by the Lyapunov stability theory. In this paper, the stochastic controller is defined as a neural network that stabilizes the \"zero\" solution and is applied $dB_t$ in the SDE. To efficiently obtain these solutions, the authors propose two types of controllers for neural networks in stochastic control systems: the (1) exponential stabilizer and the (2) asymptotic stabilizer. The first one actually requires a neral Lyapunov function, so it can be quite accurate with nonlinear systems but computing the loss function takes a relatively long time. The second one does not require actual function but guarantees almost-sure convergence, and it updates fast in reality. This paper shows that the proposed model outperforms the classical stochastic linear control in both time and cost for systems.", "questions": "1. (about Theorem 4.1) In the theorem, only the result of a stochastic linear controller ($u(x) = kx$) is presented. Are there any theoretical results on the convergence of the nonlinear ES/AS controllers?\n2. (about the AS loss) Could you explain more about the theoretical background of the AS loss function? Is the AS stabilizer \"safe\" to use for complex physical systems?\n3. (about the ES stabilier) How much model complexity can the training of ES(+ICNN) model hold in a reasonable time?", "limitations": "From the authors' description, I can imagine a situation that AS loss is later discovered by the authors and applied in this work mainly for its low computational costs. Therefore, it is worth questioning whether the AS loss (or the notion of asymptotic attractiveness) is appropriately presented in this paper titled \"Neural Stochastic Control.\" First, I think overall dedicated technical descriptions for this loss (Section 2.3) are too short. As far as I understand, the only reasonable description for AS controller is Theorem 2.3. I have a concern about whether Theorem 2.3 is tightly coupled with other theoretical results such as Theorem 2.2 or whether it is a way of approximation that can simplify the aforementioned framework. Therefore, the lack of justifications for AS loss is one limitation of this work.\n\nAnother limitation is that the ES stabilizer seems to be performant, but it might take a considerable amount of time to be trained. Therefore, I have another concern that this two-fold approach is actually applicable, or only AS is the reasonable algorithm that can successfully train a controller in a limited time.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is overall a solid paper that retains theoretical and empirical novelties for challenging control problems. The technical framework can be considered novel and general since it is dealing with stochastic problems. This paper provides some convergence analyses and experimental results on several physical systems. I think this work contributes to stochastic control studies that conventionally have used linear controllers to solve such problems. Meanwhile, I also would like to point out I have a few concerns regarding the assumptions on stabilizers and experimental results. I hope to see responses to these concerns from the authors.\n\n\n### Strengths\n\n* Major\n  * This work presents trainable nonlinear controllers for stochastic control problems.\n  * The paper offers two distinct stabilizers which can operate with different characteristics of convergence rates and computational costs.\n  * The manuscript is clearly written; I think overall analyses are technically sound.\n\n* Minor\n  * The experiments cover various physical environments. General readers can grasp how each stabilizer works through the experiments.\n  * I think the techniques shown in the exponential stabilizer show a promising direction for designing physics-informed neural networks.\n\n### Weaknesses\n\n* Major\n  * Theoretical analyses might work in ideal situations, but they seem to not fully cover addressed claims.\n  * If I understand correctly, I think controlling mechanism $\\mathrm{d} B_t$ is somewhat limited. For example, instead of $\\mathrm{d}x = f(x)\\mathrm{d}t + [g(x) + u(x)]\\mathrm{d}B_t$ why not use\n $\\mathrm{d}x = [f(x) + u_1(x)]\\mathrm{d}t + [g(x) + u_2(x)] \\mathrm{d}B_t$ for efficient stabilization?\n  * I could not find meaningful comparison studies with other approaches in the experiment section. The framework proposed in this work is expected to replace classical linear controllers. Therefore, it is very important to provide comparison studies with these controllers (e.g., the classical LQR techniques). The toy experiments in Figure 4 have such results for a simple system, but it seems the performance of $u(x) = kx$ is missing for other experiments. I think this work needs a revision since comparison experiments with previous methods should be included.\n* Minor\n  * I think the title \"Neural Stochastic Control\" does not clearly explain the methodology in this paper. Maybe this has to be more specific for representing two stabilizers (ES and AS). I believe the title has to be changed to indicate that this work is for the stochastic Lyapunov theory and the stochastic asymptotic stability theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "41GDe2pYS_r", "reviewer": "Reviewer_qG7C", "summary": "Update: I've checked out authors' response and other reviewers' comment, and my rating stays unchanged.\n\nIn this paper, the authors proposed two frameworks of neural stochastic control, i.e., the ES (exponential stabilizer) and the AS (asymptotic stabilizer), and presenting their advantages in the stochastic control. The convergence time and the energy cost of particular stochastic neural control were computed, and the efficacy of the proposed stochastic neural control were demonstrated in control problems arising from multiple physical systems. Compared with traditional control methods in the existing literature, one major novelty the authors claimed is that, the authors treat noise as a beneficial part and design stochastic control u_g to achieve the stabilization.", "questions": "1. Some of the numerical performance comparison with baseline methods could be improved.  For example,  in page 7, when numerically comparing the performance of the proposed method with traditional linear controller, it was claimed that \"Without loss of generality, we fix k = 6.0,...\", and only describe the numerical performance for k=6.0. I'd expect the value of k to have quite some impacts on the numerical performance (convergence speed and energy cost here) of the traditional linear controller, and I'm not quite convinced by the argument of \"fixing k=6.0\" is \"without loss of generality\", in terms of \"numerical performance comparison\" (instead of stability). Did you conduct comprehensive experiments for many different values of k and confirm that their numerical performance stay similar to each other, and always much worse than the proposed new neural stochastic control method here? To be more convincing for such numerical comparison and advantage claiming, it would be helpful for the authors to conduct such more comprehensive performance investigations and at least share a brief summary about the results in the paper (and share configs to make sure the summarized results/(claimed benefits) could be reproduced using the shared codes).", "limitations": "The authors have adequately discussed the limitations of the proposed framework in the manuscript and suggested a couple of promising future research directions for further improvements.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The problem under study is interesting and important, and the obtained results on neural stochastic control seem to be novel and valuable, which would be good contribution to the research community in this area.\n\n2. The paper presentation is good and clear, and I found the manuscript easy to follow. The concepts and main results are described/summarized clearly with clean proofs provided, and both the advantages and limitations of the proposed methodology are well explained.\n\n3. The numerical experiments on multiple representative physical systems are helpful, and the advantages of the proposed method is reasonably convincingly demonstrated in these numerical examples.\n\n4. The discussed limitations seem to be reasonable & helpful, and suggested future research directions look promising.\n\n5. Overall speaking, in my opinion, this seems to be a solid paper that is acceptable to NIPS and could benefit the research community in this research area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept