SYNC: SAFETY-AWARE NEURAL CONTROL FOR STABILIZING STOCHASTIC DELAY-DIFFERENTIAL EQUATIONS
Authors: Jingdong Zhang
Qunxi Zhu
Wei Yang
Wei Lin
Venue: ICLR 2023 poster (2023)
Keywords: Stochastic delay-differential equations
safety guarantee
stochastic stabilization
neural networks

Abstract: Stabilization of the systems described by \textit{stochastic delay}-differential equations (SDDEs) under preset conditions is a challenging task in the control community. Here, to achieve this task, we leverage neural networks to learn control policies using the information of the controlled systems in some prescribed regions.  Specifically, two learned control policies, i.e., the neural deterministic controller (NDC) and the neural stochastic controller (NSC), work effectively in the learning procedures that rely on, respectively, the well-known LaSalle-type theorem and the newly-established theorem for guaranteeing the stochastic stability in SDDEs. We theoretically investigate the performance of the proposed controllers in terms of convergence time and energy cost.  More practically and significantly, we improve our learned control policies through considering the situation where the controlled trajectories only evolve in some specific safety set. {\color{black}  The practical validity of such control policies restricted in safety set is attributed to the theory that we further develop for safety and stability guarantees in SDDEs using the stochastic control barrier function and the spatial discretization}. We call this control as SYNC (\textbf{S}afet\textbf{Y}-aware \textbf{N}eural \textbf{C}ontrol).   The efficacy of all the articulated control policies, including the SYNC, is demonstrated systematically by using representative control problems.

Published as a conference paper at ICLR 2023
SYNC: SAFETY-AWARE NEURAL CONTROL FOR STA- BILIZING STOCHASTIC DELAY-DIFFERENTO EQUA-
TIONS
Jingdong Zhang Qunxi Zhu Wei Yang?."", Wei Lin 123,44
School of Mathematical Sciences, SCMS, SCAM, and CCSB, Fudan University,
Shanghai 200433, China
2 Research Institute of Intelligent Complex Systems, Fudan University, Shanghai 200433, China
Shanghai Artificial Intelligence Laboratory, China
4 MOE Frontiers Center for Brain Science and State Key Laboratory of Medical Neurobiology, Fudan University, Shanghai 200032, China
(zhangjd20, qxzhu16 6, yangwei, , wlin) (fudan.edu...
ABSTRACT
Stabilization of the systems described by stochastic delay-differential equations (SDDEs) under preset conditions is a challenging task in the control community. Here, achieve this task, we leverage neural networks to learn control policies using the information of the controlled systems in some prescribed regions. Specif- ically, two learned control policies, i.e., the neural deterministic controller (NDC) and the neural stochastic controller (NSC), work effectively in the learning pro- cedures that rely on, respectively, the well-known LaSalle-type theorem and the theorem for guaranteeing the stochastic stability in SDDEs. We theoretically investigate the performance of the proposed controllers in terms of convergence time and energy cost. More practically and significantly, we improve our learned control policies through considering the situation where the controlled trajectories only evolve in some specific safety set. The practical validity of such control policies restricted in safety set is attributed to the theory that we further develop for safety and stability guarantees in SDDEs using the stochastic con- trol barrier function and the spatial discretization. We call this control as SYNC (SafetY-an -aware Neural Control). The efficacy of all the articulated control policies, including the SYNC, is demonstrated systematically by using representative control problems.
INTRODUCTION
Stochastic delay-differential equations (SDDEs) (Mao 1996 Lin & He 2005 Sun & Cao 2007
Guo et al. 2016) have been widely applied to characterize the complex dynamical behavior emergent in real-world systems with dependence on the current state, the past state, and the noise. Efficiently controlling these systems is a long-standing and crucial problem, with the consequent emphasis being placed on the design of control policies and analysis of stability in SDDEs. Traditional control methods in stochastic settings have been fully developed in the convex optimization frameworks using the control Lyapunov stability theory, e.g. the quadratic programming (QP) (Fan al.. 2020 Sarkar et al., (222)). These methods cannot provide the analytical form of feedback controllers and own high computational cost, requiring solving QP problems at each iteration step. To overcome these difficulties, utilizing neural networks (NNs) to automatically design controllers becomes one of the mainstream approaches in recent years (Zhang et al.) [022 Chang et al. (2019). However, existing machine-learning learning-based based methods either focus on controlling systems without time-delay or aim at learning the control Lyapunov function instead of the control policy (Khansari-Zadeh
"To whom correspondence should be addressed: Q.Z., W.Y. and. W.I.,
http::/faculty. fudan.edu www.cn/wwin//hhh CN/indee.nnim

---

Published as a conference paper at ICLR 2023
Billard 2014). All these, therefore, motivate to design neural controllers for general nonlinear SDDES.
The safety verification of controlled systems plays an important role in many branches cy- bernetics and industry. For example, with the safety verification, one can reduce a significant economic burden and loss of life (Ames al. 2016, Wang et al. 2016). In particular, the dom- inant framework for safety control in stochastic settings is the use of stochastic control barrier function (SCBF) (Clark) 2019: 202T, Santoyo et al. 2021). The core idea of designing can- didate SCBF that its value tends explode as the system's state leaves the safe region, im- plying safety guarantee as long as one could design controller such that the SCBF is always finite within the controlled time duration. Un- fortunately, the existing theories of SCBF either require a lot of inequality constraints or are lim- ited in handling systems without any time delay.
Initial position
Target position
X. Uncontrolled orbit
Safe region
dx(() f(x(t), x(t D) (endddegg((.....neeeeee g(x(t),x(t
r), d)
Uyde
Ugd
Tiiined
Section 3: NDC
Section 4: NSC
[Theorem 2.2,
(Theorem 4.1,
LaSalle-type
s astic stabilization)
stabilizatioon
(ot no
Section SYNC
(Theorem 5.3, safety guarantee)
Section 6: Upper bounds of SYNC's convergence time and energy cost (Theorem 6.1)
Untrained
In this paper, we utilize neural networks (NNs) learn control policies for SDDEs based on the corresponding stability theories. Addition- ally, we develop simplified SCBF theory for SDDEs and then use it to construct the neu- ral controller with safety guarantee, named SYNC. All these control policies are intuitively depicted in Figure The major contributions of this paper include:
NDC
NSC
Figure I: Overall work flow. Sketches of SYNC. Both the NDC and NSC stabilize the SDDEs to the target unstable equilibrium x". The safety- controlled state trajectories are restricted in the safe region.
designing a novel and practical framework of neural deterministic control based on the existing LaSalle-Tyl Type stability theory.
proposing a simplified stability theorem and designing the second novel neural stochastic control framework that benefit from noise according this theorem,
establishing an SCBF theory for SDDEs as well as a theory of safety guarantee and stability guarantee using neural network settings,
providing theoretical estimation for the proposed neural controller in terms of convergence time and energy cost based on the developed theory of safety and stability guarantees, and
demonstrating the efficacy of the proposed neural control methods through numerical comparisons with the typical existing control methods several representative physical systems.
PRELIMINARIES
To begin with, we consider the SDDE in a general form of
(I) dæ(t) = ((((((((())))))))) xtt 7), + (t) 1 0, r 0, x(t) E RRd,
where ax)) = E(t) E Cr₂ (1-7, 0]: Rd) is the initial function, the drift term F Rd x Rdd R. - Rd and the diffusion term G R2d Rd R, + Rdxr are Borel-measurable functions, and B, is standard r-dimensional (r-D) Brownian motion defined on probability space (Ω, F, (Fi IP) with a filtration (Feho2 satisfying the regular conditions. Without loss of generality, we assume that F(O, 0, 0 and G(O, 0, t) 0. This assumption guarantees that the zero solution æ(t) = 0 with t t 0 is an equilibrium of Eq. (T. Additionally, the following notations and assumptions are used throughout the paper.
2

---

Published as a conference paper at ICLR 2023
Assumption 2.1 Assume that Eq. 1 has unique solution æ(t, E) t 2 for any & Crr ([-T, 0]; 0j; Rd) and that, for every integer n 2 1, there is a number Kn > 0 such that
[1 y. 1) (z, y, t)]lr I Kn,
for any (a. y. E X R4 with |x| v l - where |I denotes the L--norm and || || denotes the Frobenius norm, i.e. [[G(æ, y, 1) Σ Ej- Giy(a, y,t)²
Definition 2.1 (Derivative Operator) Define the differential operator & associated with Eq. 7 by
0 dt
0 dx:
C
Σ Fl y.
+
[G(z, y, t) y y. t)ls dz,0x;
1,j=1
According the above definition of the derivative operator, an operation of on the function V C (RR x R4;:R) yields:
F(x, y, t(a,t))F(a, I [G (x, I) ()G y, y. (2)
(æ y, Vi(ae, 1) + VV(x, ()
Here, Ve, VV and HV represent, respectively, the time derivative, the gradient, and the Hessian matrix of V. Notably, the following LaSalle-type stability theorem will be crucial to the establishment of our partial results.
Theorem 2.2 MMa
2002 Suppose that Assumptions 2.1 holds. Assumes there are functions V
CR x R4; R.).
(R4: IR4), and w1.
C(X;R,, such that LV (æx y. t) S (t)
wi (a:) + w2( W1 (x) ) 2 (2(x), and | 1 infogesoo V(x, = oc. Here, X C R2d is the
state space. Then, Ker(w1 - w2) + and lim dist( (t, (t, E), Ker(w] w2)) a.s., where
Ker(wi W2) & {x wi w1(x)= wz(x)) 0}, dist(rr, K) & infyek || - for a S Rd. and
a.s. stands for the abbreviation of almost surely.
Problem Statement
We assume that the zero solution of the following SDDE:
(t) ((x, t)dt g(x, x(t r),
(3)
is unstable, i.c. lim4-+00 æ(t; () # 0 some set of positive measures. We aim to stabilize the zero solution using control based on neural networks (NNs). In other words, our goal is to leverage the NNs to design appropriate controller (u Ug) with (0, 0, t) (0 0, 0 such that the controlled system
dæ [f (a(()))), ai(t r), 1)]dt [g + Uy (x) art -))
(4)
is steered to the zero solution. We call v, Rd x Rdd x R + Rdd as deterministic control while we call ug : TRd x Rd x R4 Rd as stochastic control, since they are integrated with dt and dB,, respectively. The major difficulty of this problem comes from the non-Markovian property of SDDEs. As such, we cannot apply the Markov decision process (MDP)-based methods, such the reinforcement learning, control SDDEs. The majority of existing works prefer learn deterministic control and often regard the noise as a negative ingredient that may destroy the natural dynamics of f. what follows, we not only show that the deterministic control can achieve stabilization in probability sense, but also that claborately-designa the same stabilization. This, therefore, yields two frameworks, viz., the neural deterministic control (Section 3 and the neural stochastic control (Section 4). We make all our code and data available at https: http///gtttuu.eeee com/ http://oooooiminniiiianaaa Saaerdddoonghhagg//SSS/ chang/SYNG
stochastic control can make
NEURAL DETERMINISTIC CONTROL
In this section, we propose the neural deterministic controller (NDC) based on the Theorem 2.2 to stabilize system 33.. Heuristically, we construct the neural network form auxiliary functions and control functions, and integrate the sufficient conditions in the theorem into the loss function to find the neural controller that satisfies the expected conditions. However, the NDC can neither be used to find stochastic controllers nor rigorously satisfy the expected stability conditions. These problems will be addressed in Section 4nnd

---

Published as a conference paper at ICLR 2023
3.1 METHOD: LEARNING CONTROL AND AUXILIARY FUNCTIONS
The core idea of our method is base on using Theorem 2.2 that is, once we construct the auxiliary functions V, 7: Wi, W2 and the neural controller meet all the conditions assumed in Theorem [2. for the controlled system (4) the solution æ(t; {) converges to the Ker(u
W2). In particular, if we
set [1r - w2) {0). the unstable zero solution of the control-free system (3) can be stabilized.
To this end, we first provide appropriate constructions of NNs to learn these candidate functions.
Thus, we design the explicit form of the loss function in the learning step.
Auxiliary Function We employ multi- layer feedforward neural network, denoted by NN(:; to design all the functions. Precisely, 0₂ is the parameter vector of the positive function V(x, 1; 01), and the Tn term ||==| is added guarantee - infoge<co V( 1; 01) xc, that is
V( 1v0 = NN(x, 1;0v)² + =||=??? 8>0 0.
(5)
In our framework, it requires V E CR.1(R44 x R.). We therefore use a C2 activation function for an NN, such as the hyperbolic tangent function, Tanh(.). We further discuss the impact of the L2 In order to design an integrable positive function y() with the NN, we use an activation function with most linear growth such ReLU and multiply exponential decay factor to the output of the NN, that is
term in Appendix(A.
(t: 0-) = exp(-ct) IN(1;0,)² c>
(6)
For simplicity, we design wax kw) = NN(x;; )² as a positive function. Additionally, we set
W2 = w, W1 = w + p(z), 2 O, Ker(p) = {0}-
Deterministic Control Function We first consider the deterministic control, i.e. u (uy, y, 0). To guarantee the same zero solution of the control system (3) and the controlled system 4 the NDC uy IRd Rd IR+ -> Rdd should satisfy u = (0, 0, f) O. One feasible way to meet such a condition is to set uy (2, y, () NN(æ, y, L; 0f) NN(O, O, t; 0y) (1, y, diag(r) NN y, (x,y,4;03) (;)
Here, diag(a)) is diagonal matrix with x; as its i-th diagonal element.
Loss Function Once the learned functions V.y; W1, u2 and u with the coefficient functions, fu & f + and g. in the controlled system (4), meet all the conditions assumed in Theorem 2.2 the stability of zero solution is naturally assured. To achieve this, we demand a suitable loss function to evaluate the likelihood that those conditions satisfied. It can be from our construction that the only condition needed to be satisfied is LV 2v. y, t) I 7(1) wi (æ) + (2 Hence, we define LaSalle's loss function for the controlled system 4 as follows.
Definition 3.1 (LaSalle's 's Loss) Consider the above parameterized candidate functions V,y, W1, W2 and a controller u, for the controlled system Then, LaSalle's loss defined as
1
LN.s.c. 0, Ow. 01)
max (0, LV v., (tt)) + wx(xx) wy wv)))
(8)
N
1=1
where (a; yi, 4 are sampled from some distribution μ on Rd x Rd R,.
In summary, the developed NDC framework shown in Algorithm I Appendix A.3.
The proposed framework can be easily applied to the autonomous SDDE: dæ((t) f(x, æt(t - -d)) + g( xx! - BT)),B, In particular, one can simply consider the autonomous auxiliary function V and the control function, and set 7(1) 0. For sample distribution p(l)), here we select the uniform distribution on sufficiently large and closed region Ω as used in (Han et al. 2016 Chang et al. 2019), and we include further analyses for the impact of p in Appendix
Remark 3.1
3.2
NUMERICAL AND ANALYTICAL INVESTIGATIONS
Recent works controlling time-delayed systems mainly focus elabo- rately designing the analytical form of control to satisfy the conditions in the LaSalle- Type Theorem
Comparison Studies
2.2 (Lin & He 2005 Xu al. 2014), or simultaneously designing control and the Lyapunov function
4

---

Published as a conference paper at ICLR 2023
drive
-1
response
NDC
y1
y1 V/2
y>
drive
drive
LC
LC
y;
y, y:
-7
-7
0.1
0.25
0.5
0.1 0
0.25
0.5
(a)
(b)
(c)
(d)
Figure 3: (a) The original driving-responn model, (b) the controlled orbits under LC and NDC, (c) the time trajectory of y/2 with autonomous noise, and (d) the nonautonomous noise. The solid lines are obtained through averaging the 10 sampled trajectories, while the shaded areas stand for the standard errors.
to satisfy the conditions based on the Lyapunov theory (Yu & Cao 2007). It should be noted that all these methods require delicate design of functions for specific dynamics, and thus are limited in practical application for controlling general time-delayed systems. However, our neural method lever- ages NNs to automatically learn the control policies, and can be applied in any kind of time-delayed systems with stochastic settings. In Figure
3 we numerically compare the NDC and a baseline, the linear control (LC) proposed in (Lin He 2005), on noised driving-responss Chua's circuit. Here, Chua's circuit is three-dimensional autonomous dynamical system with unique nonlinear element, producing typical chaotic dynamics [Matsumoto] 1984). In the simulation, we show that the NDC can find the neural control for the response system y (V1 U2, y3) with the autonomous and even the nonautonomous time-delay noise. Actually, the nonautonomous time-delay noise was not considered in (Lin & He 2005). The simulation configurations described in Appendix A.3.4
Failure in Finding Stochastic Control As we can see that the NDC performs well, a natural idea is utilize the noise part to achieve the stabilization of the SDDE (3). To ex- plore this idea, we adopt the same NN of uy. design ug NN(x. y. f: f; and train its parameters 0g with LaSalle's loss ( However, in Figure we show that the loss can- Figure 2: Training loss for 1-D SDDE. not converge to zero in controlling simple 1-D toy system via the stochastic controller ug: dr(() = [x((t) + x(t - T)]dt + [a[t -T) + un (x((),, - 7); O,)]dB. Actually, this phe- nomenon can be analytically explained. Notice that 0g arises in loss function
Lassatte 1979
0.10
0.05
0.00
1000
Training Epoch
quadratic term ((0,) T [u, Vu,) according Eq. 2 the sign of this term depends the convexity of V. i.c. the maximum eigenvaluc's sign of HV. Nevertheless, the positive function V with |im Vzx t) = implies ((0,) 2 for most of time. Hence, when we minimize 1(0,) 0 in the training procedure, the ideal case 1(0,)) = 0 is equivalent to un O. This indicates that are unable to learn stochastic controller under LaSalle's loss (8) satisfying the sufficient conditions
assumed in Theorem 2.2
4 NEURAL STOCHASTIC CONTROL
To find the neural stochastic controller (NSC), we provide the following theoretical result on stabi- lization of general stochastic functional differential equations (SFDEs) with the proof provided in Appendix 1.1.4 Since the failure of NDC in the stochastic control case comes from the positive number contributed by the diffusion term, we aim at constructing stability condition such that the part related to the diffusion term can be negative. We further explain the Theorem (.1 in Appendix A.I.4
Theorem 4.1 (Stochastic Stabilization) Consider the SFDE dæ(()) = F(,,, t) + G( tt)B(t). with F, G being locally Lipschitzian functions, F F(0, t) O, and G(O, O. For every M > assume that min (O)]]= [|x,, (0) G( ()ll > 0. If there exists a number o E (0, 1) such that
[0 (2(, Fw,,, 1)) + G( 1l:) -(2 -2-aa) (2)) G(æ,, t)1 20,
(9)
for s; E C([-T, x), where (s) = æ(1 s) for E [-T,0] and X is the state space. Then, the solution of the SFDE satisfies lim; x (t: {) 0 f.r. any & (I-([-,, 0]: () Rdd.
formulated in a very general rype, including the SDDE dar(1) (1x (1 TT)... x1 T t)dt + G(æ((1), 2x(1 TT) , ( rq),t)dD with
Remark 4.2 The SFDE in Theorem
5

---

Published as a conference paper at ICLR 2023
T1 <
(.. This indicates that our framework can be generalized stabilize the SDDEs with multiple delays and even more general SFDEs as well.
<
In light of Theorem 4. we establish a more general framework for learning a neural controller of system (4 with the form u (u Myy) designed in the same NN architecture as the one used in the NDC framework. We focus on stochastic control with u = 0 and provide more control combinations in Appendix [A.3.3] whereas the loss function differently designed as follows.
Definition 4.1 (Asymptotic Loss) Utilize the notations set in Definition 3.1 and Iu = g + ug. The loss function for the controlled system {
with the controller u is defined as:
L.
max (0, (ax 2) 2) 9u gulu, Mes 4:]l + || (2 [w, w1,4)) + (1y Wes (1)))
(10)
where 0 (0). 0g). Akin to Definition 3.1 we use the empirical loss function for training.
Here, a is am adjustable parameter, which related to the convergence rate and the control energy. We further discuss the design of the asymptotic loss in Appendix [A.2.2] the role of a in Appendix [A.4.1] We summarize the framework in A Algorithm in Appendix A.3. And further compare the computational complexity in Appendix A.3.2
(a.pendixAA..221 and numerically investigate
4.1 EXPERIMENTS OF THE COMBINATION METHODS
Table 1: Results on kinematic bicycle model.
We compare neural control methods noise- perturbed kinematic bicycle model for car-like vehi- cles (Rajamani 201T) in terms of the convergence time and the energy cost, which are two important indexes measure the quality of a controller ( (Yan) et al. 2012 Li et al., 2017 Sun et al., (017). To
Tt
E0,301
Nd
E[Ta.co1]
NDC NSC
1028, 81s
102.17
6.3e-4
59.80s
62.10
4.0e-7
0.29
QP
0.016
5
quantify energy cost the control process, we first denote by T. A inf 0 || = }
the stopping time and then by & - E [ (1|wy]] + [12) df the energy cost. We approximate
this expectation value by the empirical value 1 Carlo sampling. We show the results in Figure 4 and Table T well. Table inncludes the training time (Tt), empirical energy cost En o0.0 nearest distance (Nd) between the bicycle and target position, and empirical expectation E B[ for different methods. We include more experimental details in Appendix A.3.5 We can see that the ranking of the comprehensive performance is NSC > NDC >
1= E 1|2 || 13) dt through the Monte
N
QP. This means that we can really benefit from introducing noise in the control protocol. This is reasonable because, when we regard the energy cost randomness is more likely to lead this functional to the shortest path, akin to the common case where the stochastic gradient descent outperforms the full-batch gradient descent. We show the NSC can enlarge the region of attraction of the 100-D gene regulatory networks in Appendix A.4.2
objective function for minimization, the
Uncontrollable Fluctuation The neural stochastic method we propose outperforms the control methods including the deterministic control. However, the method can uncontrollable fluctua- tion due to the stochasticity. In practice, we always want to bound this perturbation owing to physical and engineering restrictions in the real world. We tackle this safety guarantee problem in Section
Without contori
NDC
NSC
state ables
2
Target position (0,0)
X
Figure 4: (Left) A schematic diagram of the kinematic bicycle model. (Right) Time trajectories of the state variables x, y of the kinematic bicycle under different control cases. The solid lines are obtained through averaging the 10 sampled trajectories, while the shaded areas stand for the standard errors.

---

Published as a conference paper at ICLR 2023
5 SAFETY GUARANTEE FOR SDDES
In this section, we study the safety and stability guarantees for the SYNC framework. Based on the stochastic control barrier functions, we establish an analytical result on the safety guarantee problem for SDDEs, which guarantees that the process æ(t; 5) satisfies the safety constraint, i.c.,
( -1())
Safety region (GO
art:: E) E int(C)) for all & with the initial value [(0) E
hO0 <D
iC)(C). Here, C {ær h(x) Z 0} is a compact set
and the local Lipschitz function h: IRd + I. is called a stochastic control barrier function (SCBF). Inspired by
Figure 5: Diagram of the safety guarantee.
We check the safety condition on discretiza- tion points with mesh f. functions can be conditions for NN
(Lechner et al. 2022). prove that the safety and stability guaranteed through a stronger condition on finite samples. the results in Appendix A.
We include the analytical proof's for all
Definition 5.1 continuous function a (-b, (-o0, +00) is said be of an extended class-K. function for some b > 0 if it is strictly increasing and (0) = 0.
Baseline We extend the recent results on stochastic control barrier functions SDEs Clark 2019 to the SDDEs and summarize the results in Proposition 5.1 With this proposition and Theorem 2.2 the traditional deterministic control methods based on the Quadratic Program (QP) in (Fan et al. 2020. Sarkar al. (020) can be applied to test on the SDDEs. We use this QP method as the baseline and the specific algorithm shown Appendix I We also take the classic MPC method as the baseline.
Let the function B: Rd - R be locally Lipschitz and miice-differentaalle on iC), there exist three extended class-K functions 01.2,3(2) (x) such that [a (hr(h()))] < B(x)) s [a2 (h(x))]-1 and CB(x, y,t) S. ar3(h(z)) for the SDDE in 11 Then, ((x(t) E int(()) = 1.for all t, provided with (0) E iC).
Proposition 5.1
natural idea is integrate Proposition 5. into our proposed neural control framework, but the main drawback in the usage of this proposition is that (xx)) is unbounded on C, lacking Lipschitz continuity. This drawback makes it impossible to fulfill the expected conditions only through numerical verification on finite samples. To conquer the difficulty, we propose the following theorem for safety guarantee, which, we believe, is significant promotion of the existing barrier function theory.
Theorem 5.2 For the SDDE specified in 4. where F and G satisfy locally Lipschitz condition and locally linear growth condition, if there exists an extended class-K function Mz) such that Ch 2 o h for x D. where represents the function composition, D is compact and C c D. Then, the solution satisfies F(x(t::) E int(C)) = for any & E Cy. (I-,00] Rd) with 5(0) E int(C).
Discretization and Safety Guarantee. Based on the Theorem 5.2 we can construct neural candidate Class-K. function \ and combine it with the NDC and NSC to learn safe controller, where the candidate À is required to satisfy the condition assumed in Theorem 5.2 However, the main difficulty is to guarantee the condition for every point æ E D, since, in practice, we can basically guarantee this condition on a finite number of training data D with D being discretization of D. Surprisingly, the following theorem suggests that we only need check slightly stronger condition on a finite number of states D in order to establish the safety guarantee on the whole D.
Theorem 5.3 Let M = M(F, G, h, X, D) be the maximum of the Lipschitz constants of Lh and A o h on D. Also, let r be the mesh size D. D. for each x E D, there exists ā E D such that || Ø|12 r. Suppose there exists non-negative constant ó < Mr such that
-Ch Xoh 4Mr S 6, Vx D.
(11)
Then, À satisfies the safety condition specified in Theorem
5.2
Remark 5.4 Here, the non non-negative regarded as the tolerance error in the training stage. So, practically, terminate the training until the safety loss is smaller than Mr.
7

---

Published as a conference paper at ICLR 2023
(a)
(b) Without control
(c)
Baseline
(d)
NSC
(e) NSC+
2
MPC
0
0
0
0.2 0.4
0.2
0.4
0.2 0.4
0.2
0.4
Figure 6: Schematic diagram of inverted pendulum task (a). The 0 component of the original system (b), under baseline control (c), under NSC (d), and under our proposed safe control (e). The solid lines are obtained through averaging the sampled trajectories, while the shaded areas stand for the standard errors.
Construct Neural Networks with Bounded Lipschitz Constant. We can define the loss function for safety in the manner of the left-hand side in (11). However, M depends on the Lipschitz constants of the NN functions and u. which probably makes it complex and difficult to train the loss function. To simplify the loss function, we construct the NNs with bounded Lipschitz constants for À and Specifically, we add the spectral normalization for the neural control function to constrain Lipschitz constant lower than 1 (Miyato et al., 2018; Yoshida & Miyato
We apply the monotonic NNs ge, (s)ds, where qo, (-). the To constrain the Lipschitz constant of λ we modify the integral formula as Xo, (x) Jo min Lo, (s), Ms)ds, where M, is predefined hyperparameter. Thus, the Lipschitz constant of Ae smaller than M,, Therefore, can calculate M from the considered functions and My. Other Lipschitz regularization methods can well.
2017).
to construct the candidate extended class-K function as Ao.
2
output of the NNs, is definitely positive (Wehenkel & Louppe
2019
be applied in our framework (Gouk et al. 2021 Liu
2022
SYNC Algorithm: We define the loss function for the safety guarantee of the controlled system as follows, (the specific algorithms are summarized in Algorithm andd
LBM,, (0, 00.
Σ
max {0, CCh(x y) - Xe, (h(x)) + 4Mr}.
(12)
D
(x
We add this loss to equation and equation I0 respectively, to separately train the NDC and NSC. To obtain the safety guarantee, terminate the training process once LD,M, (0, 02) less than Mr.
From Safety Guarantee to Stability Guarantee. Akin to the safety guarantee, we provide the stability guarantee for the candidate neural control functions satisfying the condition in Theorems 2.2 and[4.1 However, both theorems require their conditions to be valid for every point æ E X' C R while, in practice, it impossible obtain finite discretization or a bounded Lipschitz constant on the unbounded X. Ingeniously, this difficulty can be conquered with the help of the safety guarantee since the safety condition restricts X' C where results on stability guarantee for NDC and NSC in a similar manner as that in Theorem 5.3 summarize all these results in Appendix A.
compact. As such, can establish theoretical
We thus
We test the proposed safe control method to suppress the fluctuations emergent in the control process on the task of controlling noise pesturrer inverted pendulum with time-delay. This control task is standard nonlinear control problem for testing different control methods (Anderson, 1989 Huang & Huang 2000). We apply the safe control method to steer the system to the upright position without rotating semi-circle, i.e. 10) < The results are shown in Figure 6 and the experimental details are provided in Appendix AA..6 It observed that the safe control method significantly outperforms the baseline and the stochastic control method in terms of stabilization and safety guarantee.
THEORETICAL RESULTS FOR NDC AND NSC
We have mentioned the stopping time and the energy cost in section 4.1 and numerically compare the proposed neural controllers with these indexes. These two indexes are the classic factors to measure the performance of the controller (Sun et al. (017). From the construction in Section we circumscribe the Lipschitz constant k, of the control function. Based on the safety and stability guarantee, the neural controller thus satisfies the conditions assumed in Theorems 2.2 and 4.1
8

---

Published as a conference paper at ICLR 2023
Then, have the following theoretical results and include their proofs in Appendix Theorem 6.1 (Estimation for NDC) Consider the SDDE with NDC controller as
A.1.9
(1) - (f(ax, -)))
7) + ((( -))dBB x<0)) 2o E R
where (zf fa, y)ll V | y) (τ, 8) - [[] - + 11y y (.). Assume that the
controlled system satisfies the conditions assumed in Theorem 2.2 and Remark 3.1 with Ker(wi
W2) O. Denote by do2 inf(t
||x(t]] e} the stopping time and by E(N,, =
BI [[ xxs - ds]
the corresponding energy cost in the control process with e
| Thus, using the same notations in Theorem 2.2 we have
Blm.] S. V(zo) - = (ax + f", (2(()))
minjall2 (an (a) wy()))
E(n. TT) 1 2(122 L+ k.)
exp (1(L2 + kT) -1 + (s)dss
where Co = | + (2I2 + L + km) -_, Ess) and & C(-r,0) is the initial data.
We provide the similar theoretical results for NSC in Appendix A.1.10
7 RELATED WORKS
The early endeavors to develop the stability theory for SDDEs were attributed (Mao 1999: 2002) inspired by LaSalle's theory (LaSalle) 1968). The subsequent developments have been systematically and fruitfully achieved in the last twenty years in the control community Appleby ) Song et al.) (2014); Liu et al. (2016); Zhu (2018); Peng et al. (202T). These works reveal the positive effect of multiplicative noise to the stochastic dynamics with delays, and motivate us to develop only neural stochastic control stabilize dynamical systems.
Stability Theory of SDDEs.
Finding Stabilization Controller. Traditional control methods focus on transforming control criteria, such as the control Lyapunov functions (CLFs), into the QP (Fan et al., 2020 |Sarkar et al. 2020) or the semi-definite planning (SDP) problems (Henrion & Garulli 2005 Jarvis Wloszek et al. 2003 Parrilo (200) to find optimal control iteratively. These methods have high computational complexity since they cannot give the closed form of the control. Hence, machine-learning- based control methods have been introduced to improve the generalization and efficiency of the original convex optimal problems (Khansari-Zadel & Billard 2014: [Ravanbakhsh & Sankaranarayanan 2019; Gurriet et al., 2018). However, all the existing learning methods consider dynamics without time-delay (Wagener al. 2019 Williams et al. 2018 Chang et al. 2019 Zhang et al. 2022).
Theory and Application of Control Barrier Function The barrier function method has been extensively researched in the problem of safety verification of controlled dynamics (Prajna Jad babaie 2004 Jankovic 2018; Prajna et al., [2004] [Clark] 22019; 2021). Existing works for constructing barrier functions applications typically based on quadratic programming (Ames et al.) (014 2016):
Khojasteh et al. 2020 Fan et al. (2020). Machine learning methods have also been introduced in safe
control fields RRobey et al. 2020 Dean al. 2020 Taylor et al. 2020).
DISCUSSION
We heuristically design two kinds of neural controllers for SDDEs based on the classic LaSalle-type stabilization theory and the newly proposed stochastic stabilization theorem. To assure the controlled trajectories can stay the safety region, we cultivate the safety guarantee theorem through the SCBF and the discretization techniques. Since the state space of the controlled SDDEs with safety guarantee bounded by the compact safety region, we can similarly deduce the stability guarantee theorem for neural controllers through spatial discretization. Furthermore, we theoretically and numerically investigate the neural controllers' performance in terms of convergence time and energy cost. The proposed neural controllers with safety and stability guarantee are summarized SYNC, which significantly simplifies the process of control design and has extensive potential in different control fields, such as financial engineering (Zhou & Li 2000

---

Published as a conference paper at ICLR 2023
9 ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable and constructive comments that helped us to improve the work. Q.Z is supported by the China Postdoctoral Science Foundation (No. 2022M720817), by the Shanghai Postdoctoral Excellence Program (No. 2021091), ), and by the STCSM (Nos. 2151 1100200 and
W.Y. supported by the STCSM (Nos. 2151 215111001 1100200, 22ZR 1407300 and 220z1200502). W.L. is supported by the National Natural Science Founda- tion of China (No. 11925103) and by the STCSM (Nos. 22JC1402500, 22JC1401402, and 2021SH2D22X0133)
REFERENCES
Uri Alon. An introduction to systems biology: design principles of biological circuits. Chapman and Hall/CRC, 2006.
Aaron D Ames, Jessy Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs with application to adaptive cruise control. In 53rd IEEE Conference Decision and Control, pp. 6271-6278. IEEE, 2014.
Aaron Ames, Xiangru Xu, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions Automatic Control, 62(8): 3861-3876, 2016.
Charles W Anderson. Learning to control an inverted pendulum using neural networks. IEEE Control Systems Magazine, 933)311-37, 1989.
John AD Appleby. Stabilisation of functional differential equations by noise. Systems & Control Letters, 2003.
Eduardo F Camacho and Carlos Bordons Alba. Model predictive control. Springer science & business media, 2013.
Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Iyapunov control. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3245-3254, 2019.
Andrew Clark. Control barrier functions for complete and incomplete information stochastic systems. In 2019 American Control Conference (ACC), pp. 2928-2935. IEEE, 2019.
Andrew Clark. Control barrier functions for stochastic systems. Automatica, 130:109688, 2021.
Sarah Dean, Andrew Taylor, Ryan K Cosner, Benjamin Recht, and Aaron Ames. Guaranteeing safety of learned perception modules via measuremeent-rouust control barrier functions. arXiv preprint arXiv:2010. 16001, 2020.
David D Fan, Jennifer Nguyen, Rohan Thakker, Nikhilesh Alatur, Ali-akbar
and
Evangelos A Theodorou. Bayesian learning-based adaptive control for safety critical systems. In 2020 IEEE international conference on robotics and automation (ICRA), pp. 4093-4099, IEEE, 2020.
Marco Gallieri, Seyed Sina Mirrazavi Salehian, Nihat Engin Toklu, Alessio Quaglino, Jonathan Masci, Jan Koutník, and Faustino Gomez. Safe interactive model-based learning. arXiv preprint arXiv:1911.06556 2019.
Henry Gouk, Fibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 10(2):393 2021.
Qian Guo, Xuerong Mao, and Rongxian Yue. Almost sure exponential stability of stochastic differential delay equations. SIAM Journal on Control and Optimization, 54(4):1919-1933, 2016.
Thomas Gurriet, Andrew Singletary, Jacob Reher, Laurent Ciarletta, Eric Feron, and Aaron Ames. Towards framework for realizable safety critical control through active set invariance. In 2018 ACM/IEE 9th International Conference on Cyber-Physich Systems (ICCPS), pp. 98-106. IEEE, 2018.
10

---

Published as a conference paper at ICLR 2023
Jiequn Han et al. Deep learning approximation for stochastic control problems. arXiv preprint arXiv:1611.07422. 2016.
Didier Henrion and Andrea Garulli. Positive Polynomials in Control, volume 312. Springer Science & Business Media, 2005.
Shiuh-Jer Huang and Chien-Lo Huang. Control of an inverted pendulum using grey prediction model. IEEE Transactions Industry Applications, 36(2):452-458, 2000.
Mrdjan Jankovic. Control barrier functions for constrained control of linear systems with input delay. In 2018 annual American control conference (ACC), pp. 3316-3321. IEEE, 2018.
Zachary Jarvis- Woosee,, Ryan Feeley, Weehong Tan, Kunpeng Sun, and Andrew Packard. Some controls applications of sum of squares programming. In 42nd IEFE International Conference on Decision and Control (IEEE Cat. No. 03CH37475), volume 5, pp. 4676-4681 IEEE, 2003.
loannis Karatzas and Steven Shreve. Brownian motion and stochastic calculus, volume 113, Springer Science & Business Media, 2012.
S Mohammad Khansari-Zadee and Aude Billard. Learning control Iyapunov function to ensure stability of dynamical system-based robot reaching motions. Robotics and Autonomous Systems, 62(6):752-765, 2014.
Mohammad Javad Khojasteh, Vikas Dhiman, Massimo Franceschetti, and Nikolay Atanasov. Prob- abilistic safety constraints for learned high relative degree system dynamics. In Learning for Dynamics and Control, pp. 781-792. PMLR, 2020.
J Zico Kolter and Gaurav Manek. Learning stable deep dynamics models. Advances in Neural Information Processing Systems, 11111128-11366 2019.
Joseph P LaSalle. Stability theory for ordinary differential equations. Journal of Differential equations, 4(1):57-65, 1968.
Mathias Lechner, Dorde Žikelić, Krishnendu Chatterjee, and Thomas A Henzinger. Stability verifi- cation in stochastic control systems via neural network supermartingales. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7326-7336, 2022.
Aming Li, Sean Cornelius, Y-Y Liu, Long Wang, and A-L Barabási. The fundamental advantages of temporal networks. Science, 1586666)):042220444 2017.
Wei Lin and Yangbo He. Complete synchronization of the noise-perturbed chua's circuits. Chaos: An Interdisciplinary Journal of Nonlinear Science, 15(2):023705, 2005.
Hsueh-T Ti Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, and Or Litany. Learning smooth neural functions via lipschitz regularization. arXiv preprint 2tt.::2222.03445... 2022.
Liang Liu, Shen Yin, Lixian Zhang, Xunyuan Yin, and Huaicheng Yan. Improved results on asymptotic stabilization for stochastic nonlinear time-delay systems with application to chemical reactor system. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47(1):195-22) 204, 2016.
X Mao. Robustness of exponential stability of stochastic differential delay equations. IEEE Transac- tions on Automatic Control, 41(3):442-444 447, 1996.
Xuerong Mao. Lasalle-type theorems for stochastic differential delay equations. Journal of mathe- matical analysis and applications, 236(2):350 1999.
Xuerong Mao. A note on the lasalle-type theorems for stochastic differential delay equations. Journal of mathematical analysis and applications, 268(1):12--142, 2002.
Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
Takashi Matsumoto. A chaotic attractor from chua's circuit. IEEE Transactions on Circuits and Systems,
1984.
11

---

Published as a conference paper at ICLR 2023
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Pablo A Parrilo. Structured Semidefinite Programs and Semialgebraic Geometry Methods in Robust- ness and Optimization. California Institute of Technology, 2000)
Dongxue Peng, Xiaodi Li, R Rakkiyappan, and Yanhui Ding. Stabilization of stochastic delayed systems: Event-triggered impulsive control. Applied Mathematics and Computation, 401:126054, 2021.
Stephen Prajna and Ali Jadbabaie. Safety verification of bybrid systems using barrier certificates. In International Workshop on Hybrid Systems: Computation and Control, pp- 477-492. Springer, 2004.
Stephen Prajna, Ali Jadbabaie, and George Pappas. Stochastic safety verification using barrier certifi- cates. In 2004 43rd IEEE conference on decision and control (CDC)IEEF Cat. No. 04CH37601), volume 1, pp. 929-934, IEEE, 2004.
Rajesh Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
Hadi Ravanbakhsh and Sriram Sankaranarayanan. Learning control Iyapunov functions from coun- terexamples and demonstrations. Autonomous Robots, 43(2):275-300 2019.
Alexander Robey, Haimin Hu, Lars Lindemann, Hanwen Zhang, Dimos V Dimarogonas, Stephen Tu, and Nikolai Matni. Learning control barrier functions from expert demonstrations. In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3717-3724. IEEE, 2020.
Cesar Santoyo, Maxence Dutreix, and Samuel Coogan. A barrier function approach finite-time stochastic system verification and control. Automatica, 125:109439, 2021.
Meenakshi Sarkar, Debasish Ghose, and Evangelos Theodorou. High-relative degree stochastic control іyаpиnov and barrier functions. arXiv preprint arXiv:2004.0388. 2020.
AN Shiryaev. Theory of martingales, 1989.
Bo Song, Ju H Park, Zheng-Guang Wu, and Xuchao Li. New results on delay-dependent stability analysis and stabilization for stochastic time-delay systems. International Journal of Robust and Nonlinear Control, 24(16):2546-2555 2014.
Yong-Zheng Sun, Yang Leng, Ying-Cheng Lai, Celso Grebogi, and Wei Lin. Closed-loop control of complex networks: A trade-off between time and energy. Physical Review Letters, 119(19): 198301, 2017.
Yonghui Sun and Jinde Cao. Adaptive synchronization between two different noise-perturbed chaotic systems with fully unknown parameters. Physica A: statistical mechanics and its applications, 376:253-265, 2007.
Andrew Taylor, Andrew Singletary, Yisong Yue, and Aaron Ames. Learning for safety-critiall control with control barrier functions. In Learning for Dynamics and Control, pp. 708-717. PMLR, 2020.
Nolan Wagener, Ching-An Cheng, Jacob Sacks, and Byron Boots. An online learning approach to model predictive control. arXiv preprint arXiv: /902.08967 2019.
Li Wang, Aaron D Ames, and Magnus Egerstedt. Multi-objective compositions for collision-free connectivity maintenance teams mobile robots. 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 2659-2664 IEEE, 2016.
Duncan Watts and Steven H Strogatz. Collective dynamics of 'small-world' networks, nature, 393 (6684):440 442, 1998.
Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. model predictive control: Theory and applications to autonomous driv- ing. IEEE Transactions Robotics, 34(6):1603-1122 2018.
Yuhua Xu, Yuling Wang, Wuneng Zhou, and Jian's an Fang. Stochastic complex networks synchronize to the limit set with adaptive controller and adaptive delay. Mathematical Methods in the Applied Sciences, 37(15):2200-2299 2014.
Gang Yan, Jie Ren, Ling-Cheng Lai, Choy-Heng Lai, and Baowen Li. Controlling complex networks:
How much energy is needed? Physical Review Letters, 108(21)211803. 2012.
Xue Ying. An overview of overfitting and its solutions. In Journal of Physics: Conference Series, volume 1168, pp. 022022. IOP Publishing, 2019.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint arXiv:1705. 10941, 2017.
Wenwu and Jinde Cao. Adaptive synchronization and lag synchronization uncertain dynamical
system with time delay based on parameter identification. Physica A: Statistical Mechanics and its Applications, (75(2):467748) 2007.
Jingdong Zhang, Qunxi Zhu, and Wei Lin. Neural stochastic control. arXiv:2209.07244 07.4000 2022.
arXiv preprint
Xun Yu Zhou and Duan Li. Continuous-time mean-variance portfolio selection: A stochastic lq framework. Applied Mathematics and Optimization, 12(1))19-33, 2000.
Quanxin Zhu. Stabilization of stochastic nonlinear delay systems with exogenous disturbances and the event-triggered feedback control. IEEE Transactions on Automatic Control, 6449)3764-3771, 2018.
13


Reviews:
[{"review_id": "z5ut69HL2E", "reviewer": "Reviewer_Tymv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a safety-aware control approach, based on neural networks, for stabilizing stochastic delay differential equations (SDDEs). In particular, the authors construct a controller with two components - a deterministic control $u_f$ tied to the time-evolution of the system, and a stochastic control $u_g$ tied to the Brownian motion of the system. The authors both build off existing theorems and derive new theorems to characterize conditions under which the controllers will stabilize the system. They then construct their neural networks to satisfy some of the assumptions within these theorems, as well as constructing loss functions that incentivize satisfaction of the rest. The authors demonstrate the superiority of their method compared to baselines on small-scale experiments. ", "summary_of_the_review": "The authors provide a seemingly novel, interesting framework for constructing neural network controllers for SDDEs. However, the claim that this framework \"guarantees\" safety is seemingly an overclaim, and the submission needs to be significantly improved in terms of its clarity and accessibility for the ICLR audience. In addition, the authors should further clarify where their problem formulation in Equation (4) comes from, and further justify it if it is a novel problem formulation.", "strength_and_weaknesses": "Strengths: \n* The amount of theoretical and empirical work represented within this submission is seemingly immense. The authors derive new stability theorems, and structure neural network controllers that play well with the assumptions associated with both existing theorems and these new theorems. The authors also run numerous experiments to demonstrate the efficacy of their framework.\n\nWeaknesses:\n* While the authors claim they obtain a safety guarantee through their neural network control framework, as far as I can tell, this is not the case. In particular, while some assumptions within the theorems they build upon are satisfied through the structure of their neural network framework, other assumptions and preconditions are only _incentivized_ (not guaranteed) to be satisfied through the design of loss functions. As such, I don't think the authors can actually claim that their controller \"guarantees\" safety.\n* In part because of the immense quantity of the work in the submission, the paper as written is rather inaccessible for an ICLR audience and is not self-contained. In particular, the authors could spend much more time describing the intuition behind their approach, providing algorithm diagrams and flow charts, etc. in the main text to really impart the big ideas; right now, the main body of the paper is hard to follow without this clearly articulated conceptual framework. The experimental results are also not self-contained within the main paper. \n* The mathematical notation is not adequately defined, given that some of it is not standard or usual for the ICLR audience.\n\nQuestions:\n* Is the problem statement, and particularly the formulation in Equation (4), standard - or is it fully newly proposed by the authors? If the latter, what are the strengths and limitations of the proposed formulation itself (in addition to the solution technique)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "KU_jiemHJLN", "reviewer": "Reviewer_HvjV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method for generating stable and safe policies for stochastic delay-differential equations. Specifically, the paper achieves this by learning a LaSalle's type stability certificate as an MLP and extending it to a stochastic control barrier function.", "summary_of_the_review": "Overall interesting work simplifying existing approaches model and adding time-delay to the model. Contributions compared to existing methods may be incremental and experimental evaluation is limited to simpler control tasks.", "strength_and_weaknesses": "# Strengths:\n- The paper relies on minimal assumptions (twice continuously differentiable systems and policies).\n- The paper introduces a method for an important problem (stable and safe control in stochastic systems)\n\n# Weaknesses:\nGenerally, I feel the writing of the paper could be a bit clearer. For instance, the function in Eq (5) looks quite similar to the neural network learned in Chang et al. 2019.\nMoreover, the failure case of the NDC is not entirely clear to me. How exactly does the limit of x at infinity cause the limitation of the NDC? \nWhat is the role of the L2 regularization in the method and the flaw of NDC specifically?\nOverall, the structure of the paper is good though.\n\nThe contributions seem rather incremental. In particular, the difference to Chang et al. 2019 seems to be only in the time-delay of the model. The main novelty of the paper compared to stochastic control barrier functions is mentioned to be its simplicity. Both arguments seem rather minor advancements.\n\nThe scale of the experiment seems rather limited. Particularly, the bicycle model and the inverted pendulum task are both use cases that much simpler linear or LQR control policies can potentially solve.\nThis raises the question of what bottlenecks the method.\n\nHow does the LaSalle's type loss ensure the fulfillment of the conditions of Theorem 2.2 for all (x,y), especially when the function V is expressed by an MLP and trained by gradient descent on random samples?\n\nTo obtain safety guarantees, the paper uses an input convex neural network. I don't see how this is then more expressive than a QP-based approach to synthesizing control barrier functions.\nCan you elaborate?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5063TVMpne", "reviewer": "Reviewer_cu1i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors studied the stabilization problem of the systems described by stochastic delay-differential equations, and the main contribution is the designed framework of neural deterministic and stochastic control with the analysis of stability and safety. Experiment results are given to compare the performance of  the proposed NSC/NDC method and baseline methods.", "summary_of_the_review": "This paper proposed a novel framework for neural deterministic and stochastic control with the analysis of stability and safety, for systems described by stochastic delay-differential equations. The empirical cases studies are helpful to demonstrate the effectiveness of the proposed method. \n\nHowever, my main concern about the current version manuscript is that, I'm not convinced by the authors' claim that the proposed NSC and NDC methods \"guaranteeing the stochastic stability in SDDEs\", and I think the proposed methods are heuristic in nature and hard to have rigorous stability or safety guarantees for the closed-loop system. Some more clarifications/elaborations about the theoretical theorems (5.1, 5.2, etc.) for the closed-loop controlled systems would be needed to justify the authors' claim about the theoretical stability guarantee of the closed-loop system.\n\nIf the authors can't rigorously prove theoretical stability guarantee of the closed-loop system, the proposed method and the empirical case studies are still somewhat valuable, though the contribution of the paper would be less significant.  In that case, it would be a boarder-line paper (slightly below acceptance threshold), and at the very least, the over-claim about the theoretical stability guarantee needs to be corrected/clarified.", "strength_and_weaknesses": "Strength:\n\n[1] The problem of control of nonlinear stochastic delay-differential equations is very interesting and also extremely challenging\n\n[2] The proposed methodology and main results were presented/explained reasonably clear, and overall speaking I found the manuscript friendly to readers.\n\n[3] The proposed framework is novel, and the provided case studies are helpful to demonstrate the advantages and efficacy of the proposed neural-based method in comparison with the typical/traditional existing control methods.\n\nWeakness:\n\n[1] It was claimed in the paper abstract that the proposed NSC and NDC methods \"guaranteeing the stochastic stability in SDDEs\". However, based on my understanding, such proposed methods for training neural network Lyapunov functions for control synthesis of nonlinear systems are heuristic in nature, and extremely hard to obtain rigorous stability or safety guarantees for the closed-loop system. For the main theoretical results in Theorem 5.1, it was mentioned that \"Assume the controlled system satisfies the similar conditions in Theorem 2.2 and Remark 3.1 with Ker(w1 − w2) = 0\", but it was not clear to me how the authors guarantee that the closed-loop control system would indeed satisfy all the conditions in Theorem 2.2. Similar for other main theoretical results (e.g., Theorem 5.2). Simply adding stability conditions and safety conditions to the loss function of the NNs is not sufficient to guarantee the closed-loop controlled system would be stabilized, since there are a lot of different things affect the NN training and the NN training might even possibly diverge, and there is no much rigorous theoretical guarantee for such NN method. So I'm not convinced by the authors' claim about the proposed methods  \"guaranteeing the stochastic stability in SDDEs\" from the current version manuscript. More clarifications/elaborations from the authors are required to justify such claims. This is the biggest limitation/concern in my mind for the current version manuscript.\n\n[2] It is not comprehensively discussed/explained when to use NSC and when to NDC. For the specific case study in Section 4.1, I saw a brief statement \"We can see that the ranking of the comprehensive performance is that NSC is greater than NDC than QP\". But is such conclusion generic for all cases, or it is just an observation for this special case study? I.e., is that using NSC rather than NDC always the recommended method by the authors (when there are randomness there), and what are the pros and cons? It would be helpful for the authors to elaborate more and share more comprehensive discussions for generic cases.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster