Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees
Authors: Wenqi Cui
Yan Jiang
Baosen Zhang
Yuanyuan Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Control
Stability
Tracking
Passivity
Neural network-based controllers
Power systems

Abstract: We study the optimal control of multiple-input and multiple-output dynamical systems via the design of neural network-based controllers with  stability and output tracking guarantees. While neural network-based nonlinear controllers have shown superior performance in various applications, their lack of provable guarantees has restricted their adoption in high-stake real-world applications. This paper bridges the gap between neural network-based controllers and the need for stabilization guarantees. Using equilibrium-independent passivity, a property present in a wide range of physical systems, we propose neural Proportional-Integral (PI) controllers that have provable guarantees of stability and zero steady-state output tracking error. The key structure is the strict monotonicity on proportional and integral terms, which is parameterized as gradients of strictly convex neural networks (SCNN). We construct SCNN with tunable softplus-$\beta$ activations, which yields universal approximation capability and is also useful in incorporating communication constraints. In addition, the SCNNs serve as Lyapunov functions, giving us end-to-end performance guarantees.  Experiments on traffic and power networks demonstrate that the proposed approach improves both transient and steady-state performances, while unstructured neural networks lead to unstable behaviors.

Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees
Wenqi Cui Yan Jiang' Baosen Zhang' Yuanyuan Shi? 'University of Washington, WA 98195 University of California San Diego, CA 92093 wenqi cui Cuw. edu jiangyan@ur edu zhangbao0uv edu yyshißeng ucsd. edu
Abstract
We study the optimal control of multiple-input and multiple-output dynamical systems via the design of neural network-based controllers with stability and output tracking guarantees. While neural network-bart baaser nonlinear controllers have shown superior performance in various applications, their lack of provable guarantees has restricted their adoption in high-stake real- world applications. This paper bridges the gap between neural network-basse controllers and the need for stabilization guarantees. Using equilbriumninndeendee range of physical systems, we propose neural inporttinnalIntegrre (PI) controllers that have provable guarantees of stability and zero steady-state output tracking error. The key structure the strict monotonicity on proportional and integral terms, which is parameterized as gradients of strictly convex neural networks (SCNN). We construct SCNN with tunable softplus-3 activations, which yields universal approx- imation capability and is also useful in incorporating communication constraints. In addition, the SCNNs serve as Lyapunov functions, providing end-to-end stability and tracking guarantees. Experiments on traffic and power networks demonstrate that the proposed approach improves both transient and steady-state performances, while unstructured neural networks lead to unstable behaviors.
passivity, a property present in a wide
Introduction
Learning-based methods have the potential to solve difficult problems in control and have received significant attention from both the machine learning and control communities. common problem in many applications is to design controllers that-with as little control effort as possible-stabilize system at a prescribed output. canonical example is the LQR problem and its variants, which finds optimal linear controllers for linear systems [1, 2].
For nonlinear systems, the problem is considerably harder. Neither of the two control goals, system stabilization and output tracking at the steady state, is easy to achieve. Their combination, optimizing controller costs while guarantecing stability and output tracking the steady state, is more challenging. For example, vehicles in a platoon should stay in formation (stability) while cruising at the desired speed (output tracking) [3]. The optimization is then done over the of all controllers that achieves both of these goals, e.g., reaching a platoon while consuming the least amount of fuel. Another pertinent application is the stability of electric grids with high amounts of renewables. Unlike systems with conventional generators, the power electronic interfaces of the inverters need to be actively controlled such that the grid synchronizes the same frequency (output tracking), the minimum operational costs. Currently, learning becomes popular tool to parameterize nonlinear controllers as neural networks and train them to optimize performance, but provable guarantees on stability and steady-state error for these controllers are nontrivial to obtain [4-7].
Many real-world applications have multiple (possibly a continuous set of) equilibria that may be of interest. For example, a group of vehicles may need to cruise at different speeds. Moreover, the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

internal states may not be directly accessible and sometimes there are communication limit. These constraints make it difficult to enforce stability by middle steps including learning Lyapunov function and learning model of the system (for details, see related work in Section 1.1). Therefore, we seek to achieve "end-to-end" guarantees: the neural network-basee controller should guarantee stability and steady-state error by constructions, for range of possible tracking points. There are some works that showed a class of monotone controllers can provide stability guarantees for varying equilibria [8-10], but they rely on tailor-made Lyapunov functions that are found in specific applications. Especially, these results are limited to single-input and single-output (SISO) control. In practice, however, lots of complex systems need controllers that are multiple-inpe and multiple-output (MIMO), sometimes with high input and output dimensions.
Moreover, current learning- based approaches typically focus on optimizing transient performance (the cost and speed at which a system reaches an equilibrium), often overlooking the steady-state behavior (cost at the equilibrium state) [11-13]. In classical control terminology, these controller is proportional. Steady-state requirements are difficult to incorporate since training can only occur over finite horizon. To enforce output tracking at the steady-state, integral term is commonly needed to drive the system outputs to the desired value at the steady state [14-17]. This reasoning underlies (PI) controllers in practical applications [18-20].
the widespread use of linear Proportional-Inteeen However, linear parameterization inherently limits the controllers' degrees of freedom, potentially leading suboptimal performance. This work addresses the following question: Can learn nonlinear controllers that guarantee transient stability and zero steady-state error for MIMO systems?
Contributions. Clearly, it is not possible to design a controller for all nonlinear systems and the answer to the question above depends on the class of systems under consideration. This paper focuses on systems that satisfy the equilibrium independent passivity (EIP) [21, 22], which is present in many critical societal-scale systems including transportation [23], power systems [24], and communication network [25]. This abstraction allows us to design generalized controllers without considering the detailed physical system dynamics. We propose a structured Neural-P1 controller that has provable guarantees of stability and zero steady-state ouiput tracking error. The key structure we use is strictly monotone functions with vector-valued inputs and outputs. Experiments on traffic and power systems demonstrate that Neural-Pi PI control can reduce the transient cost by at least 30% compared to the best linear controllers and maintain stability guarantees. Unstructured neural networks, on the other hand, lead to unstable behaviors. We summarize our major contributions as follows.
1) We propose a generalized PI structure for neural network-based controllers in MIMO systems, with proportional and integral terms being strictly monotone functions constructed through the gradient of strictly convex neural networks (SCNN). We construct SCNN with a tunable softplus- activation function and prove their universal approximation capability in Theorem 1.
2) For a multi-agent system with an underlying communication graph, we show how to restrict the controllers respect the communication constraints through the composition of SCNNs.
3) Using EIP and the monotone functions structured by SCNNs, we design a generalized Lyapunov function that works for range of equilibria. This provides end-to-end guarantees on asymptotic stability and zero steady-state output tracking error proved in Theorem 2. The structured neural networks can be trained for transient optimization without jeopardizing the guarantees, establishing a framework for coordinating transient optimization and steady-state requirement.
1.1 Related works
Learning-based control with stability guarantecs. Recently, there has been a large interest in enforcing stability in learning- controller [26-30]. Many works add soft penalties on the violation of stability conditions in the cost function, but it is nontrivial to certify stability for all the possible initial states in compact set [30]. Some recent works learn a Lyapunov function and use it to certify stability through a satisfiability modulo theories (SMT) solver [26-29]. But this approach is difficult to scale to high-dimensional systems and the learned Lyapunov function only works for single equilibrium. For control-affine systems, the work in [31] designs feedback linearizing policy with integral control to guarantee stabilizing to a range of equilibria. But it requires that all the state are accessible and many systems are not control affine. Our proposed controller guarantees stability for set of equilibria by construction, and only needs access to the outputs (not the full states).
Optimizing long-term behavior. To regulate long-term behavior, existing works train neural networks using cost function defined over a long time horizon [32, 33]. However, it is difficult

---

to quantify how long the trajectory is enough to reach the steady state, thus enforcing steady-state tracking performance by adding a long-term cost is challenging. Our proposed controller enforces steady-state tracking by construction, instead of relying on training.
Algorithm to tune MIMO PI controller. Classical Proportional Integral (PI) control structures widely used in industrial applications. while tuning PI controller parameters in MIMO systems is tedious and relies on heuristic rules [34, 35]. Learning-basee methods become popular to tune PI parameters, although still restricted to linear control gains [34, 35J. Our contribution is the more generalized PI control and the MIMO neural network for parameterization.
Monotone neural network. Even though scalar- monotone functions have been studied in
the learning community [36, 37], their generalization to the vector- vector-valued case has not. Numerous
papers studied vector-valued functions that are monotone in every input, that is, ((x) S q(xx) if S x' [38-41]. This is only small subelass of vector-valued monotone functions that defined as functions satisfying (q(x) (ax')) T (æ= x') 2 0. In this paper, we construct the general class of monotone functions (often called monotone operators) [42] that satisfy the inner product inequality.
2 Background and Preliminaries
We consider dynamic system described by:
(1) with state xxt) E R", control action (t)) E R", and output v() E R". We also sometimes omit the time index for simplicity'. In many practical applications, not all of the internal states are directly accessible. Hence, we consider control actions that follow output-feedbatt control laws of the form u = g(y), which is a static function of the output y and not the state æ.
Φ = f(x, u), y = h(x),
A state x² such that f(x", _u") = On. y" h(="), u² g(y") is called an equilibrium since the system stops changing at this state. Throughout this paper, the superseript indicates the equilibrium values of variables.
Definition (Local asymptotic stability [16), Definition 4.1). The system (1) is asymptoticalll stable
around an equilibrium x' if. Ve 0, 35 such that ||=[(0) x=1 || < 5 ensures [|= x= < e,
Vt 2 0, and 38' > 0 such that [0x) - x || < 8' ensures lim [|x(() - x" = O.
One of the main tools prove asymptotic stability is the Lyapunov's direct method [27, 16)
Lemma (Iyapunov functions and asymptotic stability [16). Theorem 4.1). Consider the system (1)
with an equilibrium x² E X C RR.. Suppose there exists a continuously differentiable function
V X + R that satisfies the following conditions: 1) V(=") = 0, V(x) > 0, Va E Xxxx"; 2)
V()) (V=V(a)) & < Va E X with the equality holds when x x=. Then the system is asymptotically stable around x²
The key challenge to using Lyapunov theory is in constructing such a function and verifying the satisfaction of the conditions in Lemma 1. An important part of this paper is to show how to systematically use neural networks to construct Lyapunov functions for a class of nonlinear systems.
Besides stability, we are often interested in achieving a specific equilibrium such that the output converges a prescribed setpoint.
Definition 2 (Output tracking to g). The dynamical system (1) is said to track a setpoint ÿ if lim; (t) ÿ.
For output tracking, the dimension of the input u should be at least the dimension of the output y. otherwise there is not enough degrees of freedom in the input to track all the outputs. If the dimensions of the input are strictly larger than the output, it is always possible to define "dummy" outputs (e.g., by appending zeros) to match the input dimension. Therefore, it is common to assume the same input and output dimensions [22]. In the remainder of this paper, we make the following assumption.
Assumption 1 (Identical input and output dimension). The input u and the output y have the same dimension, namely, ut) and yt) are both vectors in Rmm
'hrooughout this paper, vectors are denoted in lower case bold and matrices are denoted in upper case bold, unless otherwise specified. Vectors of all ones and zeros are denoted as 1n, On E R." respectivelly
3

---

(a) Control Goals
(b) Physical Systems
Transient
Steady State
Target
Stable (Def.1)
position
Tracking(Def.2)
Ditur bences
Time
Exp1: Inverted pendulum
Exp 2: A vehicle platoon
Transient
Guaranteed
Optimization
by Design
End-to-End Stability 1 Output Tracking
Physics (EIP)
Training
Neural
(c) Structured Neural Pl Controller Construction
Network
Stabilizing Structures
Strictly convex
Parameterize
Strictly
Proportional
Trained
NN
Vag 0)
functions
monotone
(x;; 0)
Integral
(Def. 3)
Figure (a) The controller aims to improve transient performances after disturbances while guarantecing stabilization to the desired steady-state state output y. (b) Two examples of physical systems with output tracking tasks. (c) We provide end-to-end stability and output tracking guarantees by enforcing stabilizing Pf structure in the design of neural networks. The key structure is strictly monotone functions, which are parameterized by the gradient of SCNNs. (d) transient performance optimized by training the neural networks.
We will show that strictly monotone functions play an important role guaranteeing stability and output tracking in range of systems. Using neural networks for constructing monotone functions from R. to IR has been well-studied in [36, 37J. However, since we consider MIMO systems this paper, we use generalized notion of monotonicity for vector-valued functions as defined in [42]. Definition 3 (Strictly monotone function [42]). continuous function q R + R is strictly monotone D R if (q(n) q()) (n 2 Vn, & (D, with the equality holds if and only =
In this paper, will show way to construct (strictly) monotone functions using the gradient of (strictly) convex neural networks.
Problem Statement
3.1 Transient and steady-state requirements
We aim to optimize the controller in u to improve the transient response after disturbances while guaranteeing steady-state performance, as shown in Figure ((a). By steady-state performance, mean the system should settle at the desired setpoint y. By transient performance, we want the system quickly reaching the steady state with small control efforts.
The inverted pendulum in Figure 1(b) serves as good motivating example. The input is the force on the pendulum. The objectives include 1) the angle y should reach setpoint (e.g.. 59) and stays there; 2) minimizing the control cost and deviation of y from ÿ before reaching the steady state. These objectives common in many real-world applications, such vehicle platooning (Fig. 1(b)) and power system control our experiments.
Transient performance optimization. During the transient period, our goal is to quickly stabilize the system to the desired setpoint ÿ. while trading off with using large control efforts in u. Let (()) be the cost function of the system. The transient optimization problem up to time T is,
min ((y(t) - ÿ. u()))
(2a)
s.t. dynamics in (1) t f(x, u). y= h()),
(2b)
stability and output tracking in Definition 1-2 : lim y(t) 7
(2c)
Note that (2a) we sum up to time T. which should be long enough to cover the transient period. Since require tracking in (2c), the exact value of T is not critical in the optimization problem. In
Including other forms of differentiable cost functions do change the analysis.

---

practice, the system dynamics (1) (and sometimes even the cost function) can be highly nonlinear and nonconvex. Therefore, the current state-of-the-ee is learn u by parameterizing it as a neural network and training it by minimizing the cost in (2a) [43, 44]. But the key challenge with applying these neural network-based controllers is guaranteeing stability and output tracking [45, 26, 30]. Even if the learned policy may appear "stable" during training, it is not necessarily stable during testing. This can be observed in both the vehicle and power system experiments in Section 6.
Stability for a range of tracking points. We emphasize that the setpoint may vary in practice (e.g., the setpoint velocity of vehicles may change), and thus the controller is required to guarantee stability and output tracking to range of equilibria corresponding to the setpoints. This is difficult to achieve through existing works that enforce stability by learning a Lyapunov function [26, 27, 30], since the learned Lyapunov function is for a single equilibrium (normally setup a O,)). These methods also require that all the states æ are observed, which may not be achieved practice.
Communication requirement. A typical constraint in a large system is limits on communications. For example, vehicles group may only be able to measure the output of their neighbors (line-of- sight) or nodes in a power system may only have real-time communication from subset of other nodes. Therefore, the control action u, may be constrained be a function of subset of the outputs y. We show later in Subsection 4.3 about how these communication constraints can be naturally accommodated in our controller design.
3.2 Bridging controller design and stability through passivity analysis
As shown in Figure (a)), optimizing transient and steady-state performances are two problems two different scales. Therefore, coordinating them is a central challenge.
End-to-end performance guarantees. We propose to overcome this challenge through a structured controller design that provides end-to-end stability and tracking guarantees, as shown in Figure I(c-d). "End-to-end" means that the guarantees are provided by construction, and do depend on how the controller is trained. Thus, the neural networks can be trained to optimize the transient performance without jeopardizing the stability and steady-state guarantees. In particular, the construction works for range of equilibria and can conveniently incorporate communication characteristics.
Instead of working on specific systems individually, we seek find family of physical systems that allows us to derive generalized controller design. It turns out that the notion of equilibrium independent passivity (EIP) [21, 22] provides concise and useful abstraction of physical systems for stability analysis. This paper thus focuses on systems satisfying the following assumptions:
Passivity [21]). The system (1) is strictly equilibrium- independent passive (EIP), which satisfies: (i) there exists a nonempty set UI C CR" such that for every equilibrium u² U", there is a unique x" E R" such that f(x", u²) On, and (ii) there exists a positive definite storage function (I, x²) and a positive scalar such that
Assumption 2 (Equilibrium-Indeeeddee
S xx)
=0 =0 and $xx - ply 1²
(y ---3" (u uu)
(3)
The EIP property in Assumption 2 has been found in a large class of physical systems, including transportation [23], power systems [24, 46), robotics [47], communication [25], and others. We will conduct experiments on vehicle platoons and power systems to show how EIP can be validated.
The condition (3) of EIP systems provides
generalized approach to construct Lyapumov functions without knowing the specifies of the dynamics f-).. In turn, we are able to find the right structure for the controllers. In Section 4, we show what these structures are for PI controllers and how to enforce such structures in the design of neural networks. Then Section 5 formally demonstrates the theoretical guarantees and the training procedure for transient optimization.
4 Structured Neural-PI Control
In this section, we construct controllers with a proportional and an integral term, which are both vector-valued strictly monotone functions parameterized by the gradient of strictly convex functions. Then, we present a neural network architecture that is strictly convex by construction and can conveniently incorporate communication constraints.
5

---

4.1 Generalized PI control
To realize output tracking, we introduce an integral variable - y y. Intuitively, a is the accumulated tracking error and will remain unchanged at the steady-state when y y. On this basis, we consider a generalized Pl controller u p y) + r()). The first component the proportional term, where p y) is function of the tracking error desired output y. The second component is the integral term r(s) as a function of the integral of historical tracking errors. Intuitively, the proportional term drives y close to y. and the integral term ensures the tracking error equals zero at the steady state.
y between the current output y and
Generalized PI Controller. Compactly, the control law is
u
P
+
(4a)
I' Preportional control
Integral control
y.
(4b)
Remark 1. The above controller can be envisioned as nonlinear generalization of widely adopted linear Proportional-Intery
(PI) controllers, where p'y y) Kp(ÿ - y). r(s) := K with Kp and K being the proportional and integral coefficients (scalar for SISO control). Linear PI control laws are almost always used in existing works [14-16], but the performance of linear PI controllers can be poor for large-scale nonlinear systems[6, 48J.
To achieve provable stability guarantees with output tracking, we need to construct Lyapunov function that works for the closed- system formed by (1) and (4). Therefore, we further design structures in the proportional function p() and the integral function r(-) that be utilized in Lyapunov stability analysis. The structures are strictly monotone functions, which generalizes conventional linear functions. In the next subsection, monotone functions. In section 5.1, we will show how the PI controllers structured with monotone functions provide end-to-end stability and output tracking guarantees.
will show how parameterize strictly
4.2 Strictly monotone function through gradients of strictly convex neural networks
It is not trivial to parameterize strictly monotone functions since this is an function space. Scalar-valued monotone functions mapping from IR to [36, 37] has been proposed, but it is difficult to extend these designs MIMO systems.
In this paper, we propose a new parameterization of strictly monotone functions by leveraging the fact in Proposition I that the gradient of strictly convex function is strictly monotone.
Proposition (Gradients of strictly convex functions). Let g IR'' + I be strictly convex function.. then (V,, (nn(()) (E)(E)) (n {) 2 Vn, E RT with equality holds if and only if n E. Namely, Vg: R + Rm is a strictly monotone increasing function..
Hence, the strictly monotone property of p(-) and r(-) can be guaranteed by design if they are the gradient of a strictly convex function, shown by Figure 2. The next proposition shows how to construct a strictly convex neural network (SCNN).
SCNN
9
0g 0)
O2
0g
OK
dz
dz
W:
Strictly increasing
Proposition (Strictly convex neural networks). Consider function g(z; 0) T + IR pa- rameterized by k-layer, fully connected neural network with the input= z E Rm The outpul o of layer l O....,k - represented
Figure 2: Strictly monotone function constructed by strictly convex neural networks (SCNN).
and the function g(2; 0) is
0 = (W(O)
or +W( z + ln). g(2: 0) Ok
wooo 1: buck-1 } are trainable weights and biases, and o; are non-linear activation functions. The function g(z: 0) is strictly convex in z provided that all W W/oo positive, w(()) is nonzero, and all functions df are strictly convex and increasing.
where Ou, (=(0)==00 -= Work-1
1:k-1

---

Full Communication
(b) Decentralized
[c] Partial
Communication
%
Σ gr (May 9/.
(
v, Σ d (28)
Zvg
SCNN
SCNN
g
SCNN
2py
SCNN
V2
SCNN
Zus
SCNN
Zy
SCNN
Figure 3: Communication embedded controller. Take m = 4 where 2 = (21, 23, 24) and P network p Veg(P) (z; r(r) g( as an example. (a) Global communication, where {(1, 2,3, 3, 4)) and p can be a function of the full z. (b) Decentralized, V = ((I).... (4) and ps is a function of 21, Vi. (c) Partial communication, V = {(1, 2), (2, 3, 4)}, where there exists communication within (1, 2) and (2, 3. 4).
The construction of SCNN follows the general structure of input-convex neural networks [49, 50], where the conditions on activation functions and weights are modified to achieve strictly convexity. The proof follows from the fact that positive sums of strictly convex functions are also strictly convex and that the composition of a strictly convex and convex increasing function is also strictly convex.
The next theorem demonstrates the universal approximation property of SCNN in Proposition 2. Theorem 1 (Universal approximation Let Z be compact domain in R.th and g(z) E - IR be Lipschitz continuous and strictly convex function. For any > O, there exists function gzz; 0) Z + R constructed by (5) where the activation function is the softplus-B function
of
Sofvhuus (x) = B
log(1 + cmn)
(6)
such that (g(z)- Oz(;0]] for all Z E Z.
The proof is given in Appendix A.1. We sketch the proof as follows. We leverage the results in [49, 50] that the structure (5) with ReLU activation a universal approximation for any convex function. However, ReLU activations are not strictly convex, and thus we design the Softplus-B activation. By showing that the structure (5) with Softplus-3 activation can approximate neural networks with the ReLU activations arbitrarily closely when is sufficiently large, we then prove that the structure (5) with Softplus-3 can universally approximate any strictly convex function.
Notably, 8 in the activation function is a tunable parameter. Hence, we write down the SCNN (5) with activation function being Softplus-3 function in (6) as g(z; 0, 3), where is an extra trainable parameter.
Thus, parameterize the structured Neural-PI control law = p(ū y) r(s) in (4)
ply y) V V.g (()) ,()))
)
rss) V=g (z; ,())))))))) )=-.
This way, p()) and r(.) by construction are strictly monotone. In particular, the convex functions (z; (()), g(P)), (z; )()) play a vital role in constructing a generalized Lyapunov function and showing the satisfaction of the Lyapunov condition Subsection 5. This subsequently
provides end-to-end stability guarantees that do not dependent on the training algorithm.
4.3 Embedding Communication Constraints
For some large-scale physical systems, not all of the control inputs have access all of the output measurements at real-time. Therefore, some u,'s 's cannot be a function of the full vector y and because of this lack of global communication, as elaborated in Subsection 3.1.
Constructing the controllers as the gradient of convex functions provides us with a convenient way to embed these communication constraints. Let V be the set of indexes with communications. Take m = 4 where = (21, #2, Æ3, 24) and the proportional term p (P) (2; (PP), as an example. Figure 3(a) shows the case with global communication. where = 211,2,3, 3,4) and p can be function of the full Figure 3(b) shows the case without communication where
7

---

V= [(1).... (4)) and pj can only be function of 2 for all i. Figure 3(c) shows the case V
(2.3), 4) where there exists communication within the indexes (1. 2) and within (2, 3, 4). By
defining SCNN g( ; ,))(() for each group in V; E V. the gradient V =zg 0 o ( will only be function of zn, and thus satisfying the commutation constraints. Therefore, we construct the functions in (7) as
V (P) z 0 ( Vx
Σ
g
(ze, 00 0
(8)
BYEV
}. and 0 p 0Thh are parameters of the where ((P) := (0) 0 y/ 3 = (8P) S 1p/) SCNNs for group v, within the communication network.
The function (z; ((;
is also constructed in a similar way as g ( O (PP) in (8) to incorporate the communication constraints V. Strict convexity still holds since a sum of strictly convex functions is also strictly convex.
5 Training with End-to-End Guarantees
5.1 End-to-end stability and output-tracking guarantees
The convex function (s; , ()) constructed from SCNNs can be utilized to construct Lya- punov function for proving stability using the Bregman distance defined in the following Lemma [51]: Lemma 2 (Bregman distance). The Bregman distance associated with the convex function
(s; 0(T), ()()) is defined by
= (s: (() which is positive definite with equality holds if and only if = s"
B(s, s"
,0)) (s";
- (s-s"),
The Bregman distance allows us construct Lyapunov functions without specifying the equilibrium s" Combining the storage function in Assumption 2 and B(s, s" 0()) ())) in Lemma 2, next theorem shows that the Neural-PI controller stabilizes the system to the desired output ÿ.
Theorem 2. Suppose Assumption 2 holds and the input follows the structured PI control law u = p - y) + r(s) where p()) and r(-) are constructed as the gradients of strictly corvex neural networks in (7). If the system (1) has a feasible equilibräum, then the system is locally asymptotically stable around it. In particular, the steady-state outputs track the setpoint y. namely y" = y.
Theorem 2 shows that Neural-Pl control has provable guarantees on stability and zero steady-stat (7). Detailed proof could be found in Appendix A.2 and we sketch the proof below. At an equilibrium, the right side of (4b) equals to zero gives y ÿ. We show that an equilibrium is asymptotically stable by constructing a Lyapunov function V(z, s) S(x, x²) + (1s, s";
output tracking error from the structures in
Since r(s) = (. 0(1), , 3()) by construction in (7), the time derivative of the Bregman distance term is B(s s" 0 = (r(s) r (s*)) (- y²)) )). Combining with the fact Š (x, x²)= S -p lly yI1l I1 + (3 y¹) (u (u=u") (from the EIP assump- tion), and u = p'ÿ - y) + r(s). we can conclude that the Lyapumov stability condition holds.
Remark 2. The satisfaction of the Lyapunov condition dues not depend on the specifics of f(-) (as long as is EIP), making the stability certification robust to parameter changes for systems satisfying Assumption 2. We will demonstrate this in the experiment
system control.
5.2 Training to improve transient performances
The Neural-PI controller can be trained by most model- or model-free algorithms, since the stability and output-tracking are guaranteed by construction. Fig visualizes the detailed construction and the computation graph in the dynamical system in (1). The trainable parameters θ = (ggP),tr)) o(.) 0(0,B(1) ( are contained in p(-) and r(-) functions, where both are parameterized as the gradients of SCNNs. u V, yy (P) y; O(P), ,(P)))(P)) + (xg 0(1), ((4)))()) serves as control signal in the system defined in (1) that evolves through time. The loss function is defined as
T
Loss( = Σ ((v(() - D. ul
t=0
8

---

where J.) is the transient cost function defined in (2a). The parameters O are optimized via gradient descent to minimize this loss function (9).
PPO network
p(-++))
#00) defined
ocfined (40)
delled
SCNN 1
SCNN
)(()
PC)
r(+ network
y(1)
p( network
r() network
P()
((-)
network
r(-) network
F(S)
SCNN
u(O)
R(L)
M(T)
SCNN
System (1) step
System at s=
System s
Figure 4: The computation graph for training the Neural- PI controllers.
6 Experiments
We end the paper with case studies demonstrating the effectiveness of the proposed Neural-PI control two large-scale systems: vehicle platooning and power system frequency control. Detailed problem formulation, verification of assumptions, simulation setting, and results are provided in Appendix B. and B.2 in the supplementary material. All experiments are run with an NVIDIA Tesla T4 GPU with 16GB memory. Code is available at this link.
6.1 Vehicle platooning
Experiment setup. We conduct experiments the vehicle platooning problem in Figure 1(b) with 20 vehicles (m = 20), where E I is the control signal to adjust the velocities of vehicles and the output 1m is their actual velocities. The state is (C,y). where E 1 is the relative position of vehicles. The dynamic model is Č Ty. ÿ 6v- X + p(u EDE ()), where R. p. D, E, I are constant matrices with their physical meaning given in Appendix B.1.1. The vector X E Rm reflects the default velocity of vehicles. In Appendix B.1.2, We verify that this system is EIP (i.e., satisfying Assumption 2) using the storage function S (xx x*) - ( - setpoint velocity quickly with acceptable control effort. We train for 400 epochs, where each epoch trains with the loss (9) averaged on 300 trajectories, and each trajectory evolves 6s from random initial velocities.
y²) ETTEE, The objective is for the vehicles to reach the same
Controller performance. We compare the performance 1) Neural-PI: the learned structured Neural-PI controllers parametrized by (7) with three layers and 20 neurons in each hidden layer. 2) DenseNN: Dense neural networks (5) the same as Neural-PI with unconstrained weights. Both of them have communication constriants. 3) Linear-PI: linear PI control where p(ğ y) Kp(ğ y). r()) == (*) with Kp and K, being the trainable proportional and integral coefficients. Figure 5() shows the transient and steady-state costs on 100 testing trajectories starting from randomly generated initial states. Neural-PI attains much lower costs even though the weights in DenseNN are not constrained. Compared with Linear-PI, Neural-PI achieves similar steady-state cost (since it retains all the stability guarantees of classical linear PI control), but reduces the transient cost by approximately 30%. Given ÿ 5m/s, Figure 5(b) and 5(c) show the dynamics of selected nodes under DenseNN and Neural-PI, respectively. All the outputs track under Neural-P1 but they are unstable under DenseNN (even though the training cost was not prohibitively high). In particular, DenseNN appears to be stable until about 10s, but states blows up quickly after that. Therefore, enforcing stabilizing structures is essential.
6.2 Power systems frequency control
Experiment Setup. The second experiment is the power system frequency control on the IEEE 39-bus New England system [52], where u E R. the control signal to adjust the power injection from generators and the output y E I is the rotating speed (i.e., frequency) of generators. The objective to stabilize generators at the required frequency ÿ 60Hz the steady state while minimizing the transient control cost. The state is x (8, y), where e 1 is the rotating angle of

---

transient
steady
Nomallized
1.0
speed
U
Speen
node2
nosel
node2
model
0.5
noce3
noser
noce15
noce10
2
nodc3
moded
node10
mude15
no0e5
noce18
node5
0.0
Neural-PI
Linear-PI
DenseNN
10
timets)
timets)
Transient and steady-state cost
(b) Velocities under DenseNN (c) Velocities under Neural-Pl
Figure 5: (a) The average transient cost and stcady-state with error bar 100 testing trajectories starting from randomly generated initial states. (b) dynamics DenseNN. (c) dynamics Neural-PL.
generators. The dynamics of the system is å = Ty. Mý = -Dy - D) d u Ebsin E 8), where M, D b, E, I are constant matrices with their physical meaning given in Appendix B.2.1. The vector the net load the system. Appendix B.2.2, We verify that this system is EIP (i.e., satisfying Assumption 2) using the storage function S (æz, x²) = ( y²) Myy y²)
n b(cos( E 8) cos( E 8))) (En E' T (6 8))). Below we show the impact of communication constraints on the performance of Neural-PI control. More simulation results can be found in Appendix B.2.3 to demonstrate 1) Neural-PI control is robust to parameter changes, disturbances and noises. 2) Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN.
Impact of communication constraints. Most systems do not have fully connected real-time communication capabilities, the controller needs to respect the communication constraints and show the flexibility of Neural-PI control under different communication structures. We compare the performance of Neural-PI controller where 1) all the nodes can communicate 2) half of the nodes can communicate and 3) none of the nodes can communicate (thus the controller decentralized), respectively. The transient and steady-state costs illustrated in Figure 6(a). Neural-PI-Full achieves the lowest transient and steady- cost. Notably, the steady-state cost significantly decreases with increased communication capability. The reason that communication serves to better allocated control efforts such that they can maintain output tracking with smaller control costs. The frequency dynamics shown in Figure 6(b)-(d), where under all settings Neural-PI controllers stabilize to the setpoint (60Hz) and converges the fastest under full communication.
Nomallized Cos
0.3
transient
60.00
60.00
Frequencyy
60.00
0.2
Frequency
Frequency
59.98
59.98
59,98
0.1
59.96
59.96
59.96
0.0
Nesurat-P-Fulll
Neural-Pi Pi-Half
Neural-Pi Pi- Decenralized
59.94
59.94
59.94
time(s)
time(s)
timets)
(a) Transient and steady cost Figure 6: (a) average transient and steady-state cost with emrrr on testing trajectories Neural- PI controller with different communication constraints. The dynamics of Neural-PP PI where (b) all nodes can communicate, (c) half of nodes can communicate, (d) none nodes can communicate.
(b) Full Comm
(c) Half Comm
(d) Decentralized
Conclusions
This paper proposes structured Neural-PI controllers for multiple-input multiple-output dynamic systems. We parameterize the P and terms as the gradients strictly convex neural networks. For wide range of dynamic systems, we show that this controller structure provides end-to-end stability and zero output tracking error guarantees. Experiments demonstrate that the Neural-PI control law retains all the stability guarantees of classical linear PI control, but achieves much lower transient cost. Unstructured neural networks, however, lead to unstable behavior and much higher costs. The theoretic guarantees of Neural-PI control also significantly reduce the amount of data required to train well. Since classical PI control is widely utilized in real-world applications, we expect that the controllers can be transferred to real-world scenarios. Potential barriers to the application in real-world scenarios include the verification of EIP when a storage function difficult to find and provable guarantees the robustness noises. These are all important future directions for us.
10

---

Acknowledgments and Disclosure of Funding
The authors are partially supported by the NSF grants ECCS-1930603 1942326, 2200692, and 2153937.
References
[1] Maryam Fazel, Rong Ge, Sham Kakade, and Mchran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. International Conference on Machine Learning, pages 1467-1476. PMLR, 2018.
[2] Yingying Li, Tianpeng Zhang, Subhro Das, Jeff Shamma, and Na Li. Safe adaptive learning for linear quadratic regulators with constraints. Technical report, Technical report, 2023.
[3] Samuel Coogan and Murat Arcak. A dissipativity approach safety verification for intercon- nected systems. IEEE Transactions on Automatic Control, 60(6): 2014.
[4] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
[5] Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi- multi-agent reinforcement learn- ing for networked systems with average reward. Advances in Neural Information Processing Systems, 33:2074 2086, 2020.
[6] Jianhong Wang. Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. Advances in Neural Information Processing Systems, 34:3271-3282 2021.
[7] Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. In International Conference on Learning Representations, 2019.
[8] Wenqi Cui, Yan Jiang, and Baosen Zhang. Reinforcement learning for optimal primary frequency control: Lyapunov approach. IEEE Transactions on Power Systems, 2022.
[9] Yuanyuan Shi, Guannan Qu, Steven Low, Anima Anandkumar, and Adam Wierman, Stability constrained reinforcement learning for real-time voltage control. American Control Conference (ACC), 2022.
[10] Jie Feng, Yuanyuan Shi, Guannan Qu, Steven H Low, Anima Anandkumar, and Adam Wierman. Stability constrained reinforcement learning for real-time voltage control in distribution systems. IEEE Transactions on Control of Network Systems, 2023.
[11] Charles Dawson, Sieun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural Iyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics, 2023.
[12] Guannan Qu, Yuanyuan Shi, Sahin Lale, Anima Anandkumar, and Adam Wierman. Stable online control of linear time-varying systems. In Learning for Dynamics and Control, pages 742-753. PMLR, 2021.
[13] Christian Fiedler, Carsten W Scherer, and Sebastian Trimpe. Learning-enhanced robust con- troller synthesis with rigorous statistical and control-theoretic guarantees. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 5122-5129, IEEE, 2021.
[14] Changhong Zhao, Enrique Mallada, and Florian Dörfler. Distributed frequency control for stability and economic dispatch in power networks. In Proc. of American Control Conference, pages 2359-2364, July 2015. doi: 10.1 10.1009//C..2115..7771000
[15] Martin Andreasson, Dimos V Dimarogonas, Henrik Sandberg, and Karl Henrik Johansson. Distributed control of networked dynamical systems: Static feedback, integral action and consensus. IEEE Transactions Automatic 59()) 2014.

---

[16] Hassan K Khalil. Nonlinear control, volume 406. Pearson New York, 2015.
[17] Yan Jiang, Wenqi Cui, Baosen Zhang, and Jorge Cortés, Stable reinforcement learning for optimal frequency control: A distributed averaging- integral approach. IEEE Open Journal of Control Systems, 1:194 209, 2022.
[18] Sönke Thomsen, Nils Hoffmann, and Friedrich Wilhelm Fuchs. Pi control, pi-based state space control, and model-based predictive control for drive systems with elastically coupled loads comparative study. IEEE transactions on industrial electronics, 3868):3447-3657, 2010.
[19] Jian Wang, Xin Xu, Daxue Liu, Zhenping Sun, and Qingyang Chen. Self-learning cruise control using kernel-based least squares policy iteration. IEEE Transactions Control Systems Technology, 22(3): 1078-1087 2013.
[20] Deany Putri Aulia, Alfiyah Shaldzabila Yustin, Amien Marzuq Hilman, Aulia Rahma Annisa, and Eng Wahyu Kunto Wibowo. Fuzzy gain scheduling for cascaded pi-control for de motor. In 2021 IEEE Conference on Energy Conversion (CENCON), pages 158-162. IEEE, 2021.
[21] Murat Arcak, Chris Meissen, and Andrew Packard. Networks of dissipative systems: composi- tional certification of stability, performance, and safety. Springer, 2016.
[22] George H Hines, Murat Arcak, and Andrew K Packard. Equilibrium-indeeeedenn passivity: A new definition and numerical certification. Automatica, 47(9): 1949-196999 2011.
[23] Mathias Bürger, Daniel Zelazo, and Frank Allgöwer. Duality and network theory in passivity- based cooperative control. Automatica, 50(8):0051-2061 2014.
[24] Wenqi Cui and Baosen Zhang. Equilibriummindependeen stability analysis for distribution systems with lossy transmission lines. IEEE Control Systems Letters, 6:3349-3354, 2022.
[25] John W Simpson-Porew
gquilibrium-independdee dissipativity with quadratic supply rates. IEEE Transactions on Automatic Control, 64(4):1440-1455 2018.
[26] Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Yуаршпоv control. Advances in neural information processing systems, 32, 2019.
[27] Ruikun Zhou, Thanin Quartz, Hans De Sterek, and Jun Liu. Neural Iyapunov control of unknown nonlinear systems with stability guarantees. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: wwwppppevviieeeeeeeeeeeeeeeeeeee
[28] Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barrier functions. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1724-1735. PMLR, 08-11 Nov 2022. URL /stts://prroeedeee
mlr hatps///11//aww//222ea/ html.
[29] Qiuhua Huang, Renke Huang, Weituo Hao, Jie Tan, Rui Fan, and Zhenyu Huang. Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions on Smart Grid, 2019.
[30] Wanxin Jin, Zhaoran Wang. Zhuoran Yang, and Shaoshuai Mou. Neural certificates for safe control policies. arXiv preprint 2rXiv:2006.08465. 2020.
[31] Armin Lederer, Jonas Umlauft, and Sandra Hirche. Uniform error bounds for gaussian process regression with application to safe control. Advances in Neural Information Processing Systems, 32, 2019.
[32] Andreas Doemr Christian Daniel, Duy Seuyen-Tuong. Alonso Marco, Stefan Schaal, Toussaint Mare, and Sebastian Trimpe. Optimizing long-term predictions for model-based policy search. In Conference on Robot Learning, pages 227-238. PMLR, 2017.
[33] Nathan Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, and Roberto Calandra. Learning accurate long-term dynamics for model-based reinforcement learning. In 202/ 60th IEEE Conference Decision and Control (CDC), pages 2880-2887. IEEE, 2021.
12

---

[34] Andreas Doemr Duy Nguyen- Model-based policy search for automatic tuning of multivariate pid controllers. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 5295-5301. IEEF, 2017.
Seuyen-Tuong, Tuong, Alonso Marco, Stefan Schaal, and Sebastian Trimpe.
[35] Ignacio Carlucho, Mariano De Paula, and Gerardo G Acosta. An adaptive deep reinforcement learning approach for mimo pid control of mobile robots. ISA Iransactions, 102:280-294, 2020.
[36] Joseph Sill. Monotonic networks. In Proceedings of the 10th International Conference Neural Information Processing Systems, pages 661-667, 1997.
[37] Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. Advances in neural information processing syslems, 32, 2019.
[38] Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. Advances in Neural Information Processing Systems, 33:15427-15438, 2020.
[39] Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions Neural Networks, 21(6))906 91766699777111 2010.
[40] Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, and Guy Van den Broeck. Counterexample-guided learning of monotonic neural networks. Advances in Neural Informa- tion Processing Systems, 33: 1938-11948, 2020.
[41] Hong Zhang and Zhen Zhang. Feedforward networks with monotone constraints, In 1JCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH36339), volume 3, pages 1820-1823, IEEE, 1999.
[42] Ernest K Ryu and Stephen Boyd. Primer monotone operator methods. Appl. comput. math, 15(1):3-43, 2016.
[43] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566. IEEE, 2018.
[44] Lucian Buçoniu, Tim de Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews Control, 46:8-28, 2018.
[45] Priya Donti, Melrose Roderick, Mahyar Fazlyab, and Zico Kolter. Enforcing robust control guarantees within neural network policies. In International Conference on Learning Representations, 2020.
[46] Pulkit Nahata, Raffaele Soloperto, Michele Tucci, Andrea Martinelli, and Giancarlo Ferrari- Trecate. A passd approach to voltage stabilization in de microgrids with zip loads. Automatica, 113:108770, 2020.
[47] Chris Meissen, Kristian Klausen, Murat Arcak, Thor I Fossen, and Andrew Packard. Passivity- based formation control for uavs with a suspended load. IFAA-PapersOnLinee, 50(1):13150- 13155, 2017.
[48] Wei He, Hejia Gao, Chen Zhou, Chenguang Yang, and Zhijun Li. Reinforcement learning control of a flexible two-link manipulator: an experimental investigation. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(12):7326-7336 2020.
[49] Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, pages 146-155. PMLR, 2017.
[50] Yize Chen, Yuanyuan Shi, and Baosen Zhang. Optimal control via neural networks: A convex approach. In International Conference Learning Representations, 2019.
[51] Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
13

---

[52] T Athay, Robin Podmore, and Sudhir Virmani. A practical method for the direct analysis of transient stability. IEEE Transactions Power Apparatus and Systems, PAS-98(2):573-5844 1979.
[53] Ján Drgona, Aaron Tuor, and Draguna Vrabie. Learning constrained adaptive differentiable predictive control policies with guarantees. arXiv preprint arXi::200.111184 2020.
[54] Erik Weitenberg, Claudio De Persis, and Nima Monshizadeh. Exponential convergence under distributed averaging integral frequency control. Automatica, 98:103-115 113, Dec. 2018.
[55] Erik Weitenberg. Yan Jiang, Changhong Zhao, Enrique Mallada, Claudio De Persis, and Florian Dörfler. Robust decentralized secondary frequency control in power systems: Merits and tradcoft's IEEE Transactions on Automatic Control, 64(10):3967-398,, Oct. 2019.
[56] Florian Dörfler and Sergio Grammatico. father-and-broadean frequency control in power systems. Automatica, 79:296-305, May 2017. ISSN 0005-1098. doi: http:://doi.org/1.1111 automatica.2017.02..00
[57] Peter Sauer, Mangalore A Pai, and Joe Chow. Power system dynamics and stability: with synchrophasor measurement and power system toolbox. John Wiley & Sons, 2017.
14

---

A Proof
A.1 Proof of Theorem
We leverage the results in [49, 50] that the structure (5) with ReLU activation is universal approx- imation for any convex function. However, ReLU activations are not strictly convex, and thus we design the Softplus-3 activation. By showing that the structure (5) with Softplus-B activation can approximate neural networks with the ReLU activations arbitrarily closely when 3 is sufficiently large, then prove that the structure (5) with Softplus-3 can universally approximate any strictly convex function.
To prepare for the proof of Theorem 1, we first derive the following Lemma about the difference between ReLU activations and Softplus-B activation.
Lemma 3. Consider the ReI.U activation of RaLU (a) = max(æ; 0). For all x E RR and A > we
have 0 < (o (x + A) of oRotl (x))l(z)) A log(2).
Proof. Note that
(0, Sohplus (x + + -0,
(x))
(o) Softplus/ (x + A) of
Softphus (x)) + (0, Softplus (x) - of
Rel.U Rel.(()))
we prove the lemma by deriving the bound for (o) Sofplass (ar + A) o Sompluss Solipluss (x) - of offoll(x)) (x)) as follows
(xx) and
(i) Since do (engmpens (x) = @0 (1 + coa) E (0, then 0 <
0 Saliplus3
(x + A) 0,
Solipuus (x:) <
du
A for A (ii) Explicitly represent of
Softplus (x) - [Raal(z)) yields
S Sohpluss (x) - (R(LU(x)) = { log(1 ) -x, x2 0
log(1 + e
x <<0
Thus,
d 01
Softplus/ (x) - - (()o)((()))
1/ c 80
x 2
dx
/11 + ex
I <<< 0
and therefore 0i (x) (xcLU(()) <
Soliptus
01
Somptus (0) of ())
3 log(2). Note
that 1 logg(1 (1g(1=1) + eB2) > log(e x. then 0
0, Sortpluss (z) ofaLL (xx) - $/4 log(2).
Combining (i) and (ii), 0 < (o Sompluss (x+ A) of (BOL((x))
< A + log(2).
The proof of Theorem is given below.
Proof. Previous works [49, 50) have shown that the structure (5) with ReLU activation is a universal approximation for any convex function. Hence, for any (()) + there exists g(z; 0 Rel.U
constructed by (5) where the activation function is ReLU and satisfying 19(2; Rel.U =(x)] de. Note that
ReLu q(2)), [g( (2: θ) Softplus/3 q(x) I [g(z; 0) Sorpllss g((;; 0) ReLU 19(2;
then suffices to prove 19(2; 0) Softplus q(x)] < c by showing the existence of g(z; 0) Softllus such that [z( 0 Somples
g(; 0) ReLU| < J for all 2.
Next, we compute the difference of the structure (5) with softplus-B and ReLU activations by tracing
through the first layer to the last layer, under the same weights { (witk_, W(O) Wio-11 DU.k-1].
15

---

The difference between the output of the first layer in g(z; 0) Sonpluss and g(2; 0) R)RuUUU is
1 - of 01
Somplus
ReLL
Sompluss
(w/+) + bn) - ROLU (w/-) 2 + bo)
(10)
which yields C Somtluug8
oRoLU- - 1 log(2)1 by Lemma 3.
01
The difference of the second layer is
3
ReLU
3
[W[o) 01 Soliptus3
+ W/(1z = + 62) - =2
ReLU (wro0) oRoLU + W/ = + bz)
22
2
2
SoAplus/8
(w/10] (11
Samplus/ - + o + W22 x + bz)
ReLu
02
ReLU
(w( of + w 2 b2)
RaLU
2
(11)
Since all the element of (((0)
(0) is positive, we have 0 < w/(0) Saltplus9 - ) S log(2) (
Applying Lemma element-wise yields 0 S 02
Sofplus
o W log (0) 1 + 1 log(2)1.
R
Similarily = 0g
8
ReL.U
- 3 log(2) w log (2 1+ log By induction
0 S
0441
Softluss8
0111 2 log(2)
ReLU
1+
(12)
Hence,
h- k-1
0 S g(; 0) Sollpuus8 - -g(2;0) O)
ReLU
<
1
log(2)
1+
I
(13)
3
where I1 w/ = ((()) ( ... W:(o)
Let B 2 log(2) (1 (E w)(')) then 0 S g(z; Solphes3 g[(;; 0) Rel.U I fec We
complete the proof using
[g(2; 0) Softplus3
{z)) S 19(2; 0) Sohplus ()L: + [g(z; (g(2;)) R):
a(())
< 2 2
1 1
+ c == c.
A.2
Proof of Theorem 2
Proof. At an equilibrium, the right side of (4b) equals to zero gives y = y and the corresponding set of equlibrium& & = {ar", s" If("", -r(**))== = 0, y, h(x*) y" J. We construct a Lyapunov
function to prove that if there is a feasible equilibrium in E, then the system is locally asymptotically stable around it.
We construct a Lyapunov function using the storage function in Assumption 2 and the Bregman distance in Lemma 2
V(x, s) = S(x, x²) + B(s, s" 0 ,3
(14)
where the functions by construction satisfy S(x, x²) 2 0, B(s, s" 00. 2 with equality holds only when x= and s s" respectively. Hence, V(z, s) wt.. is a well-defined function that is positive definite and equals to zero at the equilibrium.
To prepare for the calculation of the time derivative of the Lyapunov function, we start by calculating the time derivative of the function ((s, s (() ((1))
B(s s 0() (()) (v (1)
(s: (s: (1) ,8())) (s" 000)
(15)
! (r(s) - r (8*)) T (-(g - -))),
16

---

where ! follows from (s: ,07) (s) and á = (- 7) = (- (y y²))
The time derivative of the I Lyapunov function is
(x, 8)(w =5(x, x' + (v) (-
1 2 p 1yy 1²1l + (y - y
- yy)) (16)
ply r²ll + (y - 5))
y - 1²1l y"11
where ① follows from the strict EIP property and equations derived in (15). The equality @ is derived by plugging in the controller design in (4a) where u = p - y) + r(s) and u²" p(-y + D) + r(s"). The inequality + uses strictly monotone property of p(-).
Therefore, Vz s) S with equality only holds the equilibrium. By Lyapunov stability theory in Proposition 1, the system is locally asymptotically stable around the equilibrium.
B Experiments
We demonstrate the effectiveness of the proposed neural-PI control in two dynamical systems: vehicle platooning and power system frequency control. All experiments are run with an NVIDIA Tesla T4 GPU with 16GB memory, For completeness, the figures highlighted in Section 6 are also shown below with more thorough discussions. Code is available at this link.
B.1
Vehicle platooning
B.1.1 Problem statement
The first experiment is the vehicle platoon control in [3, 23] with vehicles, where E Rm is the control signal to adjust the velocities of vehicles and the output E Imm is their actual velocities. The state is æ (C, y), where Ç E R" is the relative position of vehicles with ((0) 1 1m(11) (namely, the vehicles are not in the same position at the time step 0). The dynamic model is
Ty,
(17)
ÿ (- - 29) + à +p(u-EDE
where A diag(Ri,... .mm) ) P diag(p1, .Pm) E Rmxm m constant diagonal matrices and K; > 0, Pi > 0 for all i 1.... m. The vector X = (....... .20) E R reflects the default velocity of vehicles. The matrix E Rmxe is the incidence matrix that indicates the neighbouring relations for pairs of neighbouring vehicles and satisfy ker (T [m(1m). The
matrix r := Im 1m extracts the relative velocities of vehicles by ry. The diagonal matrix
D e mmxm represents the sensitivity to the relative distance of neighbouring vehicles.
B.1.2 Verification of Assumption 2
We start by verifying the uniqueness of xr for any u² E Rmm At the equilibrium, the right side of (17) equals zero gives
) O. and Ty" 0m-
(-18' 29) x (()--))(((-EE
(18)
For a given u² E R", suppose there exists x (C., vn) and o = (4) vy), x; * xú such that (18) holds. Plugging in (18) gives
(v 4b) + PEDE T (5% 4. Om
(19a)
T 4 0m-
(19b)
Note that PE = E. Left multiplying (19a) with (EDET (5 - -G)) yields (EDE (C*
())) PEDE (5% (3) 0. which holds if and only if EDE (S (5) Om since p > 0.
17

---

Hence, (yā vτ4 = PEDE (Ca- - -%) = Om and therefore y. = VE. Note that ker (EDE")
Im((Im) and [m] 1 I.)) thus (S% (3) 1 Im(())). Hence, EDE (S% (t) Om if and only if C = Ç%- Theorefore, for every equilibrium u' E "R"", there is a unique x = (S" y IR" such that f(x", =²) On.
Let the storage function be S x²) (y y²) T²)TT= T p v²) 25 EDE Ç. Then
TEDE
$ (x, (y- y X - < TDET
(y yn) p-1 (- x + à {u u- E-ED <)) + + EDE
1 12112 + (y y³) T (u =²)
(min Pj 1) 1/ y 1/ (g) y²¹
T (u -u)
where ! follows from (- - X e n --DE 5") 5²) Om and E y² = E Ty" = 0. by definition of equilibrium. The inequality 2 follows from ò > 0. Therefore, the dynamics in (17) satisfy conditions in Assumption 2.
B.1.3 Simulation and Visualization
Simulation and training setup We adopt the model setup and parameters in [3, 23). The number of vehicles is v = 20. The sensitivity parameter is K; = for all vehicles. The parameters X and p; are randomly generated by X uniform|5, and p; ~ uni f form 1, 2). respectively. We train for 400 epochs, where each epoch trains with 300 trajectories with initial velocities (0) ~ uniform[5,00 At 0.02s and for K 300
and initial integral variable (0) O. The stepsize in time is steps in trajectory (Namely, each trajectory evolves 6s).
We implement control law in u to realize specific output agreement y and reduce the transient cost. The transient cost is set be [ u) Σ (ky(t 1/1 + é| (I where è diag(C1, same as Jyy u), such that neural networks are optimized to reduce transient cost through training. The neural PI controller can be trained by most model- -bssel or model-free algorithms, and we use the model- framework in [8, 53] by embedding the system dynamic model in the computation graph shown in Figure 4 and training Neural-P1 by gradient descent through J Jyy u..
Cm) with C; uniform(0... (0.0022 0.075J. The loss function in training is set to be the
Controller performances. We compare the performance of 1) Neural-PI: the learned structured Neural-PI controllers parametrized by (7) with three layers and 20 neurons in each hidden layer. The neural networks are updated using Adam with learning rate initializes at 0.05 and decays every 50 steps with base of 0.7. 2) DenseNN-PI: The proportional and integral terms are parameterized by dense neural networks (5) with three layers, 20 neurons in each hidden layer, and unconstrained weights. The neural networks are updated using Adam with learning rate initializes at 0.035 and decays every 50 steps with a base of 0.7. 3) Linear-Pl: linear PI control where p y) == Kr(ū-u) y). r(s) := Ks(s) with K p and K2 being the trainable proportional and integral coefficients. The coefficients are updated using Adam with learning rate initializes at 0.03 and decays every 50 steps with base of 0.7. All of them have no communication constraints. All of the controllers are trained using 5 random seeds. The training time is shown in Table 1.
Table 1: Training time for vehicle platoon control
Method
Average Training time (s)
Standard Deviation (s)
Neural-PI
5232.36
30.55
DenseNN-PI
3567.01
16.28
Linear-PI
1836.93
10.09
The average batch loss during epochs of training with 5 seeds is shown in Figure 7(a). All of the three methods converge, with the Neural-PI achieves the lowest cost. Figure 7(b) shows the transient
18

---

Neural-PI Linear-PI
Normallized Cost
transient
DenseNN-PR
1.5
steady
Loss
1.0
10:
0.5
100
200
300
400
(h) Average transient and steady DenseNN
Neural-PI
Linear-Pl
Epochs
(a) Training Loss
Figure 7: (a) The average batch loss during epochs of training with 5 seeds. All converge, with the Neural Pl achieving the lowest cost. (b) The average transient and steady-state cost with error bar 100 testing trajectories starting from randomly generated initial states. Neural-PI achieves transient cost much lower than others. DenseNN without structured design has hoth high costs in transient and steady-state performances.
and steady-state
100 testing trajectories starting from randomly generated initial states. The steady-state cost is Cyy, u) (15) - øll + cl]x(11))13. where we use the variables at the time t 15s since the dynamics approximately enter the steady state after & 15s as we will show later in simulation. Neural-PI PI and Linear-Pi PI have the lowest steady-state cost, and the output reaches y as guaranteed by Theorem Neural-PI also achieves transient cost that much lower than others. By contrast, DenseNN-PI without structured design has both high costs in transient and steady-state state performances.
speed y
5
Acction
node2
node?
0
4-
node3
node10
node4
node15
3
nodeS
node18
0
10
15
5
10
15
time(s)
time(s)
(a) Neural-P1 PI
6
speed y
Acction
10
node?
node
node3
node10
node15
nodes
node18
10
15
5
10
15
time(s)
time(s)
(b) Linear-PI
6
speed y
Acction
node2
node9
4
node3
node10
node15
node5
node18
-10
5
10
15
time(s)
time(s)
(c) DenseNN-PI
Figure 8: Dynamics of velocity y and control action with y 5m/s. (a) Neural-PI stabilizes z quickly. (b)
Linear-P achieves output tracking with high control effort. (c) DenseNN leads unstable behavior.
19

---

Given Y = 5m/s, Figure 8 shows the dynamics of velocity y and control action on 8 nodes under the three methods, As guaranteed by Theorem 2. Neural-PI Figure 8(a) and Linear-PI in Figure 8(b) reaches the required speed 5m/s. However, Linear-PI has slower convergence and much larger control efforts compared with Neural-PI. Even though DenseNN-PI achieves finite loss both in training and testing, Figure 8(c) actually exhibits unstable behaviors. In particular, DenseNN-PI appears to stable until about 1Os, but states blows up quickly after that. Therefore, enforcing stabilizing structures is essential.
B.2
Power systems frequency control
B.2.1
Problem statement
The second experiment is the power system frequency control on the IEEE 39-bus New England system [52] shown Figure 9, where E I=' is the control signal to adjust the power injection from generators and the output y Ihm is the rotating speed (i.e., frequency) of generators. The objective to stabilize generators at the required frequency ý 60Hz at the steady state while minimizing the transient control cost. The state is (Ó. y), where d E I is the rotating angle of generators in the center-of-inerti coordinates with 8(0) 1 [m(mm [54]. The model of power systems reflects the transmission of electricity from generators to loads through power transmission lines and is represented as follows:
å = Ty,
(20)
Mÿ Dly 1) d + Eb E B 8),
where M- diag(M1,.... Mm)) D = (I. Dm) with M, > and Dj > 0 being the inertia and damping constant of generator j. respectively. The vector d is the load of the system. The matrix E E Rm is the incidence matrix corresponding to the topology of the power network with transmission lines and satisfying ker(ET) = Im(1m)). The matrix Im L 1m1 extracts the relative rotating speed of generators by ry. The diagonal matrix = diag(b), ... ,b) E Rexe with by > 0 being the susceptance of the j-th transmission line.
Bas
Bus
0us
Bux
Dus
Dus 12
Dus
S..21
8ue
Bus
Dus 19
Rax
8ax
0m
Figure 9: IEEE 39-bus test system [52]
adopt a common assumption in literature that the power system operates with satisfying H = { E 8], E (-m/2, vj 1,. e}. where E 8) is the angle difference between the generators in head and tail of the j-th transmission line [55-57]. This range is sufficiently large to include almost all practical scenarios [55-57].
20

---

B.2.2 Verification of Assumption 2
At the equilibrium, the right side of (17) equals zero gives
-Dyy y) d + u' Eb sin E 8*) Om and Ty" Om
(21)
We start by verifying the uniqueness of x for any u" E ll where (21) has feasible solution such that ô E H. For a given u" E U, suppose there exists x = (0% yo) and wij (Øg, vτ). x o xg such that (21) holds. Plugging in (21) gives
D vb) + Eb (sin(E 6%) - -sin( (E' (E)))) = Om
(22a)
r(y.- v = Om-
(22b)
Note that TE E. Left multiplying (22a) with (Eb (sin(EE 8;) sin( " TD yields
(Eb (sin E 5,) ------ -sin(E' ((m)) 1------- D-'(EE (sin E 8%) - sin (E 6;))) = 0, which holds if and only if (Eb (sin (E 6;) Ein(ET 8;))) = Om since Đ-1 > 0. Plugging in (22a) gives D
y%) Cm. which holds if and only if yo yī since D; for all 1, m..
Left multiplying (Eb (sin E 5%)
sin E 6)))) Dm with (8% o5)) T yields
0 = E 8% E 5%) b (sin(E' 87) - Sin
(23)
Σ b (LE 8 - [E (5)],) (sin([E* 8%1s) sin( 8)11)))
Since by > 0 and sin(-) is strictly increasing in 1-, n, m22), (23) holds if and only if E (5% -))
De. Note that mm(l') 1 [mm())). thus (8; -8, 55)) 1m((1m) Hence, (23) holds if and only
8 = 87. Therefore, for every equilibrium u² lt, there is unique xx = (8", y²) E R² such that f(x", u²) = On.
Let the storage function be S (x, x²) = [ (") (y--vyy yy) --- - b(cos( E 8) cos (ET8
(Ebsin( E 8T)) (Ø 8))). Note that -1 b(cos( E' 8) is strictly convex in H. thus the Bregman distance - b(cos E 6) cos( E (8)) (Ebsin(E) 8*)) 8*)) with equality holds only when ò = 8
The time derivative is
$ (x,xx)) = (y y + (Ebsin EE E 8) Ebsin E 8T))
(y (y- (-) ÿ) d Ebsin( E' 8)) (Ebsin( E 5) -Ebiin (ET 85)) Ty
TT (DD(y* v) - d + u² Eb sin E T 8*))
=0m
@ (y 8¹) D(y (y v¹) (y yy (u u²)
(F¹) (Eb sin E 8) - Eb E 8)*))
- (min Di)lly - y 11 + (y - yy) ) (u - (u=u")
where ! follows from (-D(y' 4) d Eb E 8*)) = O,n by definition of cquilibrium. The relation 2 follows from E "y E E Tyy O. and D, > 0 for all i = 1,... m. Therefore, the dynamics (20) of the power system frequency control satisfies conditions in Assumption 2.
B.2.3 Simulation and Visualization
Simulation Setup We conduct experiments on the IEEE New England 10-machine 39-bus (NE39)
power network with parameters given in [52, 8). We implement control law for power output of generators to realize the track of frequency at 60Hz and reduce the power generation cost. The state ò is initialized as the solution of power flow at the nominal frequency and is initialized O.
21

---

The number of epochs and batch size are 400 and 300, respectively. The step-size in time is set as At 0.01s and the number of time stages in a trajectory for training is K = 400.
Apart from the accumulated frequency deviation, an important metric for the frequency control problem is the maximum frequency deviation (also known the frequency nadir) after distur- bance [8]. Hence, the transient cost is set to be Jy, u) = >i= (maxk=1,....f Iy (kat) - T/ +
0.05 Σ kk=1 (b=tt) ū + 0.005 (u,((t())) The loss function in training is (y. u), such that neural networks are optimized to reduce transient cost. The neural PI controller can be trained by most model-based or model-free algorithms, and we use the model-based framework in [8, 53] by embedding the system dynamic model in the computation graph shown in Figure 4 and training Neural-PI by gradient descent through J [yy u.)
K
Two major goals of this experiment is
1) Verifies the robustness of the controller under parameter changes. Note that the load d is parameter in the dynamics (20). In particular, power system operator emphasizes on the ability of the system withstand big disturbance such step load change. To this end, train and test controllers by randomly picking at most three generators to have a step load change uniformly distributed in uni uniform[- -1, 1] p.u., where Ip.u.=100 MW is the base unit of power for the IEEF-NE39 test system.
2) Verifies the performances under communication constraints. Most systems do not have fully connected teal-time communication capabilities, so the controller needs to respect the communi- cation constraints and we show the flexibility of Neural-PI control under different communication structures.
Controller Performances. We compare the performance of Neural-PI controller where 1) all the nodes can communicate 2) half of the nodes can communicate and 3) none of the nodes can communicate (thus the controller is decentralized), respectively. All neural-PI controllers are pa- rameterized by (7) and (8) where each SCNN has three layers and 20 neurons in each hidden layer. The neural networks are updated using Adam with the learning rate initializes at 0.05 and decays every 50 steps with a base of 0.7. We compare against the following two benchmarks where all the nodes can communicate: 4) DenseNN-Pi-
Dense neural networks (5) with three layers, 20 neurons in each hidden layer, and unconstrained weights. The neural networks are updated using Adam with a learning rate initializes at 0.01 and decays every 50 steps with a base of 0.7. Note that DenseNN needs such small learning rate let the training converge, the reason is that DenseNN may lead to unstable behaviors that we will see later. 5) Linear-Pl-Full: linear PI control where pạÿ - y) := ( - y). r(s) == (s)s) with K P and K2 being the trainable proportional and integral coefficients. The coefficients are updated using Adam with the learning rate initializes at 0.08 and decays every 50 steps with base of 0.7. All of the controllers are trained using 5 random seeds. The training time is shown in Table 2.
Table 2: Training time for power system frequency control
Method
Average Training time (s)
Standard Deviation (s)
Neural-PI-Fuu
4373.52
64.58
Neural-PI-Hal Half
8034.92
115.26
Neural-PI-Dec
23549.34
300.95
DenseNN-PI--H
2193.84
21.22
Linear- PI-Full
981.65
11.19
The average batch loss during epochs of training with 5 seeds is shown in Figure 10(a). All converge. with the Neural-PI-Full achieving the lowest cost. Figure 10(b) shows the average transient cost and steady-state cost with error bar on 100 testing trajectories subject to random step load changes. The steady-state cost is Cly. u) = 0. 0.05||/((5)) Bll + 0.005|/24 (x(55) [2, where we use the variables at the time = 15s since the dynamics approximately enter the steady state after = 15s as we will show later in simulation. Neural-Pl-Fur achieves the lowest transient and steady-state cost. Notably, the steady-state cost significantly decreases with increased communication capability. The reason is that communication serves to better allocated control efforts such that they maintain output
22

---

Neural-Pl-Full Liear-Pl-Full
Neural-Pl-Falf Neural-Pi-Dec
transient Sleady
10.0
PenseNlk-PtFull
Mormaliized Cost
1.5
Loss
1.0
0.5
0.0
100
200
400
Neural-Pt-ull
Neural Pt-Half
Neural- Porral-PP.oo Dec
Linear-Pl-Full
DenseNN--P-Ful
Epochs
(a) Training Loss
(b) Transient and steady cost
Figure 10: (a) Average batch loss during epochs of training with 5 seeds. All converge, with the Neural-PI achieving the lowest cost. (b)The average transient cost and steady-state cost with error bar on 100 testing trajectories subject random step load changes. Neural-PI PI achieves transient cost that is much lower than others. The steady-state cost significantly decreases with increased communication capability. DenseNN without structured design has both high costs in transient and steady-state performances.
Table 3: The average transient cost on 100 testing trajectories starting from randomly generated initial states
Number of training trajectories
Neural-PI
Linear-PI
DenseNN
5
0.1328
0.1915
1.0
10
0.1300
0.1865
0.9833
50
0.1257
0.1838
0.9624
100
0.1234
0.1816
0.9214
300
0.1233
0.1815
0.5347
tracking with smaller control costs. Again, DenseNN without structured design has high costs both in transient and in steady state.
With step load change at 0.5s, Figure 11 shows the dynamics of frequency y and control action u on nodes under the five methods. Again, DenseNN-PI-Ful behavior with large oscillations. As guaranteed by Theorem 2, Neural-PI in Figure 11(a-c) reaches the required frequency ÿ = 60Hz, but the speed convergence is lower for reduced communication capabilities. Hence, the guarantees provided by the structured Neural-PI controllers robust to parameter changes and communication constraints, which have significant practical importance.
Figure (e) exhibits unstable
Performance with different numbers of training trajectories. Table compares the transient cost attained by different controllers trained with different numbers of trajectories. For both Linear-Pi and Neural-PI, training with 5 trajectories for each epoch has already achieved similar cost as training with 300 trajectories. By contrast, unstructured DenseNN requires much larger amount of training data to reduce transient costs on testing trajectories. Therefore, the stabilizing structure significantly reduces the requirement for the number of samples to learn well.
The impact of disturbances and noises. The satisfaction of the Lyapunov condition is robust disturbances in the system parameters and does not need to know how large the disturbances are, as shown in the proof Theorem 2 and Remark 2. Therefore, if there is a sudden change in the load levels, the proposed controller design still stabilizes the system and tracks the required frequency at 60Hz. In Figure 12(a), we demonstrate the system dynamics after two disturbances in load. In Figure 12(b)-(c), we add noises in both data measurement and dynamics with the signal-to-noke -o-sse ratio being 5 dB (much larger than typical measurement noises). The results show that the systems are input-to-state stable, i.e., that bounded noise will lead bounded states. Incorporating noise in rigorous theoretical analysis is an important future direction for us.
23

---

Fequancy
50.98
Action
Accumulsted
-0.5
29.90
99.94
-1.0
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
0.0
2.5
5.0
7.5
timelst
timetsi
timetst
(a) NeuralPI-Ful
60.00
Accumuteted Cost
Frequency
59.98
Action
-0.5
10083
29.96
rod83
59.94
-1.0
0.0
2.5
5.0
7.5
0.0
2.5
5.0
2.5
0.0
2.5
5.0
7.5
timets)
timets
(b) NeuralPI-Han
60.00
0.0
Accumuteted Cost
Frequency
59.98
Action
FC90
-0.5
59.96
-1.0
2.5
5.0
0.0
7.5
(inels)
tmoest
(c) NeuralPI-DDa
0.0
Accumolated Cost
Frequency
60.00
Action a
0.2
59.95
-0.4
wex7 med0
59.90
-0.6
timetsl
tmetsl
ttmots)
(d) Linear-PiFFul Full
TESO
Frequeccy
Actionu
Accumuleted
60.0
-0.2
59.5
-0.4
timels)
tmeiss
tmmtts)
(e) DenseNN-Full
7 nodes with 60Hz and step load change at 0.5s. (a) Neural-PI when nodes can communicate (b) Neural-Pl when half of nodes can communicate, (c) Neural-PI when none nodes can communicate. The control with different communication capability all stabilize the system to the required convergence. (e) MaAiseNN-P--Ful leads large frequency deviations and oscillations.
Figure 11: Dynamics of frequency y. control action and accumulated
60Hz. (d) Linear-Pl-Full is stable but has slower
Frequency
Accumutoadd
*
60.00
59.95
(a) There is a step increase of load on 0.5s and step decrease of load on 3.5s.
0.0
Fequency
59.98
Action a
0.55
Accumubosed
2.
59.96
-1.0
59.94
timetsl
Smetsl
11metsl
(b) Neural-PI with noise data measurement
Aicumuibted Cost
Feequncyy
59.975
00.5
9.
12
59.925
-1.0
2
(imels)
Lirreds)
Gimeet)
Neural-PI with noise system dynamics
Figure 12: Frequency restoration 60Hz aet the disturbances/noises all maintained.


Reviews:
[{"review_id": "JLGygtoX4u", "reviewer": "Reviewer_Cp4m", "summary": "This paper propose a neural-PI controller based on the newly designed strictly convex neural network. Thanks to the EIP condition of the considered systems, the neural-PI controller naturally satisfies the stability and output tracking guarantee with the special structure of the SCNN. The efficacy of the proposed method is demonstrated in several physical models.", "questions": "[1] Since the objective functional incurs the integral of the controlled ODE, can this method scale to the high-dimensional task? \n\n[2] Can this method outperform the existing methods (RL such as PPO or MPC) in terms of training speed or test performance?\n", "limitations": "[1] The current neural method can only handle deterministic systems with EIP condition, which cannot adapt to the real-world scenarios where noise is unavoidable.\n\n[2] The authors did not provide numerical comparison with the existing methods.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1] The proposed neural-PI controller satisfies the stability guarantee by construction of the SCNN instead of adding regularization term to the loss function, which is a hugh improvement to the existing learning controller commuity.\n\n[2] The proposed method does not require the knowledge of the equilibria of the intrisic dynamics but can still provide stability guarantee of the unknown equilibria.\n\n[3] The experiments take communication cost into consideration, which is realistic and instructive.\n\n[4] The presentation of the paper is good, the workflow in the figures are clear and helpful.\n", "weaknesses": "[1] The current framework is restricted in the system with EIP condition. \n\n[2] The authors did not compare the proposed method with the existing methods such as RL, QP and MPC.\n\n[3] The training policy is naïve, the learner should solve the controlled ODE to find the controlled trajectory and then update the parameters.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for thoroughly reading our paper and providing encouraging comments. Below, we provide a point-by-point response.\n\n **Can this method outperform the existing methods (RL such as PPO or MPC)?**\n\n- The Neural-PI controller can be trained by almost all the RL algorithms, including PPO and other variants. **The focus on structure is complementary to training algorithms**: by replacing unstructured neural networks with the proposed Neural-PI structure, the trained controllers will provide provable guarantees of stability and zero steady-state output tracking error. The comparisons then focus on the controller constructions (i.e.,  Neural-PI, DenseNN, and classical linear PI) trained by the same algorithm.\n\n- In addition to the performance guarantees, we show in experiments that **Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN**. This can also help to mitigate the burden of data collection in these RL algorithms.\n\n- In this paper, we consider the systems without real-time communication/computation capabilities. Model predictive control (MPC) typically needs full communication and performs computation in real-time, so it is not directly applicable to systems considered in this paper.\n\n**(W3 and Q1) Questions related to ODE and the training policy**\n\n- We agree that the training method we provided is simple, although it is not very different from what the reviewer suggests here. Essentially, we embed the discretized ODE in a computation graph, which is convenient to train neural networks through backpropagation. We note that other model-based or model-free methods can be used as well.\n\n- We would like to clarify that implementing the control policy in equation (4) does not require solving a controlled ODE. For the integral variable $\\dot{s}=-(y-\\bar{y})$, the computation of $s$ is approximated using the measurement of $s(t) \\approx s(0)+\\sum_{k=0}^{t} -(y(k)-\\bar{y}) \\Delta t$ with $\\Delta t$ be the sampling time interval. This is a common practice and is widely implemented in control. Thus, the Neural-PI takes the measures of $y$ and $s$ as inputs, which is in essence no different from a generic neural network and can scale to high-dimensional tasks. \n\n\n**Adaptation to real-world scenarios where noise is unavoidable.**\n\nWe thank the reviewer for bringing up this important point. Below, we first clarify that the method is robust to disturbances in the system parameters. Then, we show experimental results that the controller also works well for systems with noises both in data measurement and in the system.  \n-  **The satisfaction of the Lyapunov condition is robust to disturbances in the system parameters**, as shown in the proof Theorem 2 and Remark 2. Take the power system experiment as an example, the load $d$ in the system dynamics is a parameter. If there is a sudden change in the load levels, the proposed controller design still stabilizes the system and tracks the required frequency at 60Hz. In Figure 3(a) of the attached PDF in the global response, we demonstrate the system dynamics after two disturbances in load.\n\n- For noises in the data measurements and state transition dynamics, we empirically find that **the systems are input-to-state stable, i.e., that bounded noise will lead to bounded states**.  Incorporating noise in rigorous theoretical analysis is an important future direction for us. We have performed empirical studies with noises in both data measurement and dynamics, as shown in Figure 3(b)-(c) of the attached PDF in the global response.  The signal-to-noise ratio is about 5 dB (much larger than typical measurement noises).  Figure 3 shows that the performance guarantees are robust to noise. \n\nOverall, the EIP property has been found in a large class of physical systems, including the transportation and power systems examples studied in the paper, and many other networked systems in robotics and communication [R1]. The above results show that the design can adapt to real-world scenarios where noise is unavoidable. \n\n[R1] Networks of dissipative systems: compositional certification of stability, performance, and safety. Springer, 2016\n\n**Numerical comparisons with the existing methods.**\n\nA: In supplementary material, we compared the proposed Neural-PI with unstructured neural networks (DenseNN) and classical PI control (Linear-PI). As we mentioned above, the proposed method focuses on the structure, and thus we train the controllers using the same algorithm. To the best of our knowledge, we have not found other existing structured controllers to incorporate in numerical comparisons. \n\nThe results for numerical comparisons are summarized as follows:\n- The proposed Neural-PI control law retains all the stability guarantees of classical linear PI control, but **reduces the average transient cost by approximately 30% compared to Linear-PI**. This is reflected in Figure 7 and Figure 10 of the supplementary material.\n\n- **Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN**.  This is reflected in the following table that compares the transient cost attained by different controllers trained with different numbers of trajectories. For both Linear-PI and Neural-PI, training with 5 trajectories for each epoch has already achieved a similar cost as training with 300 trajectories. By contrast, unstructured DenseNN requires a much larger amount of training data to reduce transient costs on testing trajectories. \n\n   **Table 1: The average transient cost on 100 testing trajectories starting from randomly generated initial states**\n\n   | # of training trajectories  |**Neural-PI** | Linear-PI |DenseNN|\n   |---|---|---|---|\n   | 5 | **0.1328** | 0.1915 |  1.0 |\n   | 10 | **0.1300**| 0.1865 | 0.9833 |\n   | 50 | **0.1257** | 0.1838  | 0.9624 | \n   | 100 | **0.1234**| 0.1816 | 0.9214 | \n   | 300 | **0.1233**| 0.1815 |  0.5347 |\n"}, {"review_id": "GR13XNS0iC", "reviewer": "Reviewer_eDxT", "summary": "The paper \"Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees\" presents a well-motivated and technically sound approach to address the lack of provable guarantees in neural network-based controllers. The combination of structured neural-PI control, equilibrium-independent passivity, and SCNNs with tunable softplus-β activations demonstrates promising results in terms of stability, output tracking, and improved performance.", "questions": "1.\tIn Definition 2, is it critical to demand zero tracking error? That can be augmented easily with a prefilter or an integrator. Even though the controller has an integrator, the system may be of higher type).\n2.\tIn 3.1 it is stated that it is desired to optimize the transient response before the system settles on the tracking point. What are the specifications? Maximum overshot? Settling time? Are there minimum requirements?\n", "limitations": "Not applicable", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clear, well written and accurate in the mathematical control details. The method leverages a specific kind of NN and the approach is clean and seems to hold great potential, at least to an applicable systems class.", "weaknesses": "The proposed approach relies on the assumption of equilibrium-independent passivity, which may limit its applicability to a specific class of physical systems. The paper should explicitly discuss the range of systems to which the approach is applicable and highlight any potential limitations or challenges in extending it to other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for thoroughly reading our paper and providing encouraging\ncomments. Below, we provide a point-by-point response.\n\n\n**The applicability and potential limitations in extending the method to other domains.**\n\n- Theoretically, this method works for systems that satisfy EIP properties, including the transportation and power systems examples studied in the paper, and many other networked systems such as robotics and communication [R1]. For EIP systems, the proposed Neural-PI control law obtains rigorous stability and zero steady-state tracking error guarantees by construction, and results in significantly lower transient cost compared to the linear PI controller and standard NN controller (as shown in the traffic and power system simulations). For extending the proposed method to other domains, potential barriers include the verification of EIP property when a storage function is difficult to be found, and provable guarantees on the robustness to noises. These are all important future directions for us. \n\n  [R1] Murat Arcak, Chris Meissen, and Andrew Packard. Networks of dissipative systems: compositional certification of stability, performance, and safety. Springer, 2016\n\n\n**Q1: In Definition 2, is it critical to demand zero tracking error? That can be augmented easily with a prefilter or an integrator. Even though the controller has an integrator, the system may be of higher type).**\n\n- We agree that zero tracking error can be augmented with a prefilter or an integrator. We will make this more clear in the revised paper.\n\n\n**Q2: What are the specifications to optimize the transient response?**\n\n- The specifications depend on the application case by case, and it generally includes reducing state deviation (e.g., maximum overshot, the accumulated deviation from the tracking point) and control cost. The minimum requirement is that the objective function is differentiable. \n\n- For example, in power system frequency control,  an important metric is the maximum frequency deviation (also known as the frequency nadir) after a disturbance. In addition, we also hope to keep the assumed frequency deviation small with lower control effort.  Then the cost function is the summation of the maximum frequency deviation, the accumulated frequency deviation, and the control effort.\n\n\n"}, {"review_id": "SPJpGndGbV", "reviewer": "Reviewer_aJSd", "summary": "This paper presents an approach to control multi-input, multi-output dynamical systems, introducing a neural network-based PI controller offering provable stability and output tracking guarantees. The neural Proportional-Integral controller is based on strict monotonicity parameterized by strictly convex neural networks (SCNN). The strict monotonicity of the network gives the convergence guarantee by construction.  Testing on traffic and power networks showed improved transient and steady-state performances compared to unstructured networks.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "It is interesting to improve PI control using neural networks. The proposed framework not only keeps the structure (which is well-validated across decades) and robust tracking performance of PI controllers, but also allows optimizing the tracking performance by learning the parameters in g and r functions. The use of monotonic neural networks is also critical. The experiment results demonstrate its advantages over regular fully connected neural networks.", "weaknesses": "- The authors must have known that carefully tuned PI controllers already work very well. Why didn’t they compare in the experiments? It is common that controllers built upon regular fully connected layers will diverge as time progresses, but this is not a problem in simple regular PI controllers. How well the proposed method is, compared to a carefully tuned classical PI controller?\n\n- Even if the proposed framework is better than traditional PI control in the transient phase, how can the authors guarantee its performance in situations outside the training data distribution? In real world applications where data collection can be expensive, if we only have 1000 data points, would the authors choose to train a neural network (this could be dangerous if only have limited data) or tune a classical PI controller using their extensive domain knowledge of the system dynamics (and use the 1000 data points for tuning & experiment).\n\n- In summary, this study is interesting and novel, but its importance / practical contribution to the control field might be minor.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for thoroughly reading our paper and providing encouraging comments. We agree with the reviewer that carefully tuned PI controllers can work well for a large range of applications, and the controller by design has been shown to have good stability properties.  **Classical PI controllers are the impetus for this work, where we want to maintain good theoretical properties and improve over some of the limitations (e.g., slow convergence speed and larger control efforts).** This is the reason why we don’t use a general neural network, but a specific Neural-network structure. \n\nBelow, we first show that the Neural-PI control law retains all the stability guarantees of classical linear PI control, but **achieves much lower transient cost**.  Then, we provide numerical results to demonstrate that **this performance improvement is present even if there is little data to train the controllers**.  \n\n\n**Comparison to Carefully Tuned PI Controller**\n\n- We actually compared the proposed Neural-PI with regular PI controllers (labeled as Linear-PI) in the supplementary material (see Figure 7 and Figure 10). But we should have made it more clear where these results are and leaving them in the appendix may have caused confusion about how well the proposed controller compares with carefully tuned classical linear-PI control law. We will add the comparison with Linear-PI in the revised main paper. \n\n- **We would like to emphasize that the transient cost attained by Neural-PI is significantly lower than Linear-PI.**  In the following, we take the experiments of power systems as an example to show the comparison, where both Neural-PI and Linear-PI are trained with the same algorithm. \n\n   For both the trained Neural-PI and Linear-PI, the table below compares the average and standard deviation of transient cost on 100 testing trajectories starting from randomly generated initial states. Thus, Neural-PI reduces the average transient cost by 32%  compared to Linear-PI.\n\n   **Table 1: The transient cost on 100 testing trajectories starting from randomly generated initial states**\n   | Metric  |**Neural-PI** | Linear-PI |\n   |---|---|---|\n   | Transient cost mean | **0.1233** | 0.1815  | \n   | Transient cost std | **0.0677**| 0.0774 | \n\n\n\n\n**Performance of Neural-PI controller under different numbers of training data**\n\n- The proposed Neural-PI structure provides provable stability and output-tracking guarantees **by construction**. Thus, **such guarantees hold for all cases even if the scenario does not exist in training**. \n- **Compared with Linear-PI, the proposed Neural-PI does not require more data to train well**. This is reflected in the following table that compares the transient cost attained by different controllers trained with different numbers of trajectories (each trajectory evolves 400 time steps). For both Linear-PI and Neural-PI, training with 5 trajectories for each epoch has already achieved a similar cost as training with 300 trajectories. By contrast, unstructured DenseNN requires a much larger amount of training data to reduce transient costs on testing trajectories. Therefore, the stabilizing structure significantly reduces the requirement for the number of samples to learn well.\n\n   **Table 2: The average transient cost on 100 testing trajectories starting from randomly generated initial states**\n\n   | # of training trajectories  |**Neural-PI** | Linear-PI |DenseNN|\n   |---|---|---|---|\n   | 5 | **0.1328** | 0.1915 |  1.0 |\n   | 10 | **0.1300**| 0.1865 | 0.9833 |\n   | 50 | **0.1257** | 0.1838  | 0.9624 | \n   | 100 | **0.1234**| 0.1816 | 0.9214 | \n   | 300 | **0.1233**| 0.1815 |  0.5347 |\n\n- Importantly, tuning a classical PI controller to optimize over a host of objective functions can be challenging even if the exact model and parameters are available. Learning-based methods provide a flexible framework to optimize PI parameters as long as the objective function is differentiable. Considering that Neural-PI trained with 5 trajectories at each epoch already achieves a transient cost much lower than Linear-PI, we envision that Neural-PI is applicable in real-world applications. \n\n\n\n\n **The applicability of Neural-PI compared with Linear-PI**\n\n- The above results show that the Neural-PI control law **retains all the stability guarantees of classical linear PI control, but achieves much lower transient cost**. The control theoretic guarantees also significantly reduce the amount of data required to train well. In other words, the Neural-PI controller cannot do worse than a carefully tuned PI controller. However much data there is to train a Neural-PI controller, the Neural-PI controller archives a much smaller transient cost compared with classical Linear-PI. **If there is domain knowledge to tune a classical PI controller, we can take it as a warm start and train the neural network starting from there**.\n\n- In addition, **the proposed neural network construction makes it tractable to optimize nonlinear monotone functions to improve transient performances**, even when exact parameters for the system are not available. \n\n- Since classical PI control is widely utilized in real-world applications, we expect that the controllers can be transferred to real-world scenarios. For example, our controllers are in the process of being implemented in a real-world test system, the UCSD DER-connect testbed (none of the authors are affiliated with the testbed). \n"}, {"review_id": "BgE79PxKlZ", "reviewer": "Reviewer_F58a", "summary": "This paper proposes a method for optimal control of dynamical systems using a neural network-based controller that guarantees stability and output tracking. The key structure of the proposed neural proportional-integral (PI) controller is based on the strict monotonicity of the proportional and integral terms, which are parameterized as the gradient of a strictly convex neural network. The paper also introduces tunable activations for universal approximation capabilities and communication constraints. Experiments on traffic and power networks show significant improvements in both transient and steady-state performance. The contribution of this paper is mainly in the field of control theory and its application.", "questions": "In real applications, how to judge the passivity of a large-scale nonlinear system is a possible problem to extend the proposed approach.", "limitations": "The paper does not explicitly discuss the scalability of the proposed method to larger, more complex systems. This paper does not provide a comparison with other state-of-the-art control methods, which helps to better understand the advantages and disadvantages of the proposed method.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The strength of this paper is that it presents a method for optimal control using a neural network-based controller with provable stability guarantees. Using a strictly monotonic function parameterized by a strictly convex neural network can improve control performance and output tracking. Experiments on traffic and power networks show significant improvements in both transient and steady-state performance. ", "weaknesses": "The multiple equilibria problem has not been fully addressed in real-world applications. This limitation limits the applicability of the proposed method to systems with general nonlinear systems. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for thoroughly reading our paper and providing encouraging\ncomments. Below, we provide a point-by-point response.\n\n**The multiple equilibria problem**\n\n- The multiple equilibria come from different setpoints of output to track.  For example, a group of vehicles may need to cruise at different speeds. Once a setpoint (e.g., the required cruise speed) is given, the equilibrium of the system becomes unique. Our proposed Neural-PI control has provable guarantees of stability and zero steady-state output tracking error for different setpoints. More specifically, **the performance guarantees in Theorem 2 hold for various $\\bar{y}$, and the satisfaction of the Lyapunov condition does not depend on the specifics of $f(\\cdot)$**. Therefore, our proposed method work for real-world applications with a range of possible equilibria.\n\n\n**How to judge the passivity of a large-scale nonlinear system?**\n\n- We agree that judging the passivity of a large-scale nonlinear system is an important problem. The passivity analysis of large-scale nonlinear systems, especially networked systems, is an active line of research in academia. One benefit of passive analysis is that it provides convenience for a modular analysis of large-scale systems (for details, see  [R1-R3]). **If each of the subsystems is equilibrium-independent passive (EIP),  then the whole system is still EIP if it interconnects the subsystems through parallel, negative feedback, or an undirected graph [R1-R3]**.  This allows the passivity analysis extends to large-scale nonlinear systems [R1], for example, large power networks, traffic networks, internet congestion management, etc. \n\n- More specifically, such modular analysis allows us to divide a large-scale system into smaller subsystems, and then verify the EIP of each subsystem. For a wide range of systems, a storage function has been found to judge EIP [R1]. Even if a storage function has not been found,  it is also possible to learn a storage function parameterized by neural networks for a smaller-scale subsystem, and certify EIP through a satisfiability modulo theories (SMT) solver [R4]. Since there has been lots of success in learning a neural Lyapunov function for small-scale systems(e.g., [R4] and [R5]), we envision that such a modular approach provides the possibility to extend that to larger systems. \n\n  [R1] Arcak et al. Networks of dissipative systems: compositional certification of stability, performance, and safety. Springer, 2016.\n\n  [R2] Hines et al. Equilibrium-independent passivity: new definition and numerical certification. Automatica, 2011.\n\n  [R3] Simpson et al. Equilibrium-independent dissipativity with quadratic supply rates. IEEE Transactions on Automatic Control, 2018.\n\n  [R4] Chang et al. Neural Lyapunov control. Neurips 2019.\n\n  [R5] Zhou et al. Neural Lyapunov control of unknown nonlinear systems with stability guarantees, Neurips 2022. \n\n\n**The scalability of the proposed method to larger, more complex systems**\n\n- As we discussed in the previous response, passivity enables a modular analysis that can scale to large systems.  Namely, we can divide the large-scale systems into smaller subsystems, and then verify EIP of each subsystem. \n\n- Moreover, **Neural-PI control law retains all the stability guarantees of classical linear PI control, but achieves much lower transient cost**. Since classical PI control is widely utilized in real-world applications, we expect that the controllers can be transferred to more complex systems. For example, our controllers are in the process of being implemented in a real-world test system, the UCSD DER-connect testbed (none of the authors are affiliated with the testbed). Rigorous analysis of the scalability of the proposed method to larger, more complex systems is an important future direction for us.\n\n\n**Comparison with other state-of-the-art control methods**\n\n- In supplementary material, we compared the proposed method with unstructured neural networks and classical PI control. Notably, **the focus on structure is complementary to most existing learning algorithms (e.g., model-based and model-free reinforcement learning)**: by replacing unstructured neural networks with the proposed Neural-PI structure, the controllers trained by these algorithms will provide provable guarantees of stability and zero steady-state output tracking error. This is the reason why the comparison focuses on the controller structures (i.e.,  Neural-PI, DenseNN, and classical linear PI) trained by the same algorithm. To the best of our knowledge, we have not found other structured controllers to incorporate in numerical comparisons. \n\n- The key results of comparisons are summarized as follows:\n  - The proposed Neural-PI control law retains all the stability guarantees of classical linear PI control, but **reduces the average transient cost by approximately 30% compared to Linear-PI**. This is reflected in Figure 7 and Figure 10 of the supplementary material.\n  - **Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN**.  This is reflected in the following table that compares the transient cost attained by different controllers trained with different numbers of trajectories. For both Linear-PI and Neural-PI, training with 5 trajectories for each epoch has already achieved a similar cost as training with 300 trajectories. By contrast, unstructured DenseNN requires a much larger amount of training data to reduce transient costs on testing trajectories. \n\n\n   **Table 1: The average transient cost on 100 testing trajectories starting from randomly generated initial states**\n\n   | # of training trajectories  |**Neural-PI** | Linear-PI |DenseNN|\n   |---|---|---|---|\n   | 5 | **0.1328** | 0.1915 |  1.0 |\n   | 10 | **0.1300**| 0.1865 | 0.9833 |\n   | 50 | **0.1257** | 0.1838  | 0.9624 | \n   | 100 | **0.1234**| 0.1816 | 0.9214 | \n   | 300 | **0.1233**| 0.1815 |  0.5347 |\n\n"}, {"review_id": "h7ctncTAvT", "reviewer": "Reviewer_nsH5", "summary": "This paper considers the problem of learning feedback controllers for multi-input, multi-output (MIMO) dynamical systems. At the core of the paper, a PI-like nonlinear control structure is proposed. Specifically, the controller consists of two NN, one which receives the set-point error (denoted as \"P-part\") and another one that receives its integral (\"I-part\") as an input.  For the NNs, a special structure is proposed that ensures strict convexity of the NN. This structure serves as the basis for proving stability and zero steady-state error of the closed loop system employing this controller, irrespective of the parameters used in the NN.  Hence, these guarantees can be given a-priori irrespective of the training.  Training of the NN then aims at optimizing transient behavior. Finally, two numerical examples of MIMO control systems (vehicle platooning, power network) are presented.", "questions": "Q1) Why is it justified to call the structure in (4) a PI-controller, especially, why is it justified to call the first component \"proportional\"? \n\nQ2) What are the 2..3 papers in existing literature that are closest to your approach? And what main limitation of these does you approach address?\n\nQ3) In the experiments, the linear PI performs rather well. Why is the more involved Neural-PI controller even needed there?\n\nQ4) Do you have an example, where the Neural-PI controller clearly outperforms linear approaches?\n\nQ5) Why did you decide to not include the experimental results of the linear PI in the main paper? ", "limitations": "Limitations (such as no consideration of disturbance/noise) are mentioned briefly in the conclusion, but not discussed in detail.  I would have liked to see a discussion on: what happens when there is noise/disturbance in the sensing, the process, and/or the data; do the authors expect that their controllers can be transferred to the real world, or what hurdles do they see?\n\nI don't see any particular potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1) Developing NN control architectures that are guaranteed to satisfy key control properties by design (here: stability and steady-state optimality) is very relevant. The combination of PI-like structure with strictly convex NN seems novel.\n\nS2) Leveraging passivity properties is an interesting idea for analyzing and achieving system-wide properties. This is well-known in the control community, but should also be relvant for the NeurIPS community when considering distributed learning problems.\n\nS3) I like the idea of having a system design that is stable and comes with base performance by design and without communication, and then improving performance further when communication is available.", "weaknesses": "W1)  The introduction makes some imprecise or misleading statements about the state of the art as well as technical aspects.  All statements should be carefully checked and (where needed) be backed up with proper argumentation or references. Examples:\n\n* ll. 33-34: \"Currently, learning-based approaches mostly parameterize nonlinear controllers as neural networks and train them to optimize performance\".  While NN are a popular function approximator in learning-based control, they are actually used in different flavors (model learning, controller parametrization, surrogate model, etc.).  Furthermore, there are many other ML techniques that are also used, such as Gaussian processes, which are particularly popular when it comes to guarantees (examples: A Lederer, J Umlauft, S Hirche, \"Uniform error bounds for Gaussian process regression with application to safe control\"; C Fiedler, CW Scherer, S Trimpe, \"Learning-enhanced robust controller synthesis with rigorous statistical and control-theoretic guarantees\")  Hence, the characterization of the area of learning-based control as described in this above sentence is too narrow in my opinion.\n* ll. 47-49: \"...often overlooking the steady-state behavior (cost at the equilibrium state).\"  The statement is not supported by references.  In fact, I would disagree with it, as there are works that specifically focus on the long term performance of controllers or RL policies (examples: A Doerr et al, \"Optimizing long-term predictions for model-based policy search\"; N Lambert et al, \"Learning accurate long-term dynamics for model-based reinforcement learning\")\n* ll. 49-51: The description of classical P- and I-control is inaccurate.  While it is correct that I-control is often used in practice to make steady-state error zero, the actual behavior depends on the controller *and* the dynamical process.  Depending on the process dynamics (e.g., if it has an integrator itself), also a P-controller may lead to zero steady-state error. Also, the statement that P-control is for the transient, but disregarding steady-state performance is inaccurate.  P-control also affects the steady-state in general.  Again, the general description of how P- and I-control are often used in practice is not wrong, but the authors should more carefully word these sentences to avoid technical inaccuracies.\n* The statement of the main research question that is addressed in this paper is too general to be representative of the paper's contribution: \"Can we learn nonlinear controllers that guarantee transient stability and zero steady-state error for MIMO systems?\"  (ll. 55-56).  First, the authors state themselves in the next sentence that this question can't be answered for general nonlinear system.  Second, there are existing works in learning-based control that focus specifically on MIMO PI-control (see below). IMO, the research question should thus be rephrased to already clarify the scope of this work. \n* ll. 67-68, \"This class of functions is too restrictive to use for controller design\" -- Why?  Appropriate reasoning or reference is missing to support the statement.\n\n\n\nW2) Related work:  There is no explicit discussion of related work as a separate section/subsection, which is not a requirement of course, but could be helpful. Some references are mentioned in the introduction, but IMO the following aspects require further discussion:\n\n* The control problem and the objectives that are stated are super general in control (stability, zero steady-state error, fast transient). Hence, there will be a lot of works in learning-based control addressing similar problems.  While the paper does mention references and related work, I do not find this discussion very structured and have trouble pin-pointing what exactly is the difference to prior art and thus contribution of this paper.  Just saying, for example, that \"guaranteeing stability and output tracking\" (l. 144) is not helpful in this regard as the statement is too general and not specific to the contribution at hand.  It might help to first clearly specify the problem that is addressed, and afterwards discuss the related work specifically to the problem.\n* Given that the consideration of MIMO systems is a main aspect of this work, there should be a review specific to this.  Indeed, there are existing works in learning-based control that focus specifically on MIMO PI-control (e.g., A Doerr et al, \"Model-based policy search for automatic tuning of multivariate PID controllers\", I Carlucho, M De Paula, GG Acosta, \"An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots\", and many other works on \"self-tuning\" for MIMO systems).\n\nOverall, I do not get a clear idea how this work compares to the state of the art. \n\n\n\nW3) Specification of controller structure: The control law u = g(y) is not precisely specified.  Concretely, is g(.) a static function, or does it itself involve states and memory (i.e., is it itself dynamic)?  This is critical, as dynamic controllers (such as combining a state estimator and state-feedback controller) are more powerful than static maps in general. Later on, it seems that static NN are used as components of g(.).  However, the controller also involves an integrator, which would not be captured by a static function g(.) and is actually a dynamic component.  If, on the other hand, dynamic output-feedback controllers are admissible, one could presumably learn much richer control behavior.  IMO, it is ok to restrict the controller structure (from what I understand, to an integrator + static map), but it needs to be clearly and correctly stated.\n\n\n\nW4) I am wondering whether it is really justified to call the proposed structure a proportional + integral (PI) controller. In particular, the term that is called \"proportional\" isn't really proportional to the error for a general nonlinear map.  I can see that what the authors propose is a generalization of well-known PI-control structure.  However, I find it a bit misleading to call this a \"proportional control\".  In fact, p(-y+ybar) is a general nonlinear function of the tracking error.  Hence, it will in general not be proportional to the tracking error.  Please elaborate, why you call the architecture \"PI\", and ideally provide references if this is a common naming.\n\n\n\nW5) I do not find the notion of \"end-to-end\" guarantees conclusive and well justified.  The authors use this notion to denote the fact that they can provide certain guarantees a-priori before training (by combining a certain system class (passive) with the proposed controller architecture).  As far as I know, the term \"end-to-end\" is not commonly used for this (if you have references where it is used in this way, please provide them).  If a controller has theoretical guarantees, these are  most of the time \"by construction\", so I don't understand why this is called \"end-to-end\".\n\n* Minor: In the introduction, it is not yet clear what the authors mean when mentioning \"end-to-end guarantees\" on l. 80. It should be explained when introduced for the first time.\n\n\n\nW6) The presentation of experiments in the main paper is not self-contained and, IMO, insufficient. There needs to be a basic description of the problem in the main paper, which can be understood without consulting the supplementary material.  But this is currently missing.  Because of this, one cannot judge how challenging the control problem actually is, and whether sophisticated nonlinear control is actually needed.  Furthermore, the verification of assumptions is critical because a main claim of the paper are the theoretical guarantees.  This should be in the main paper.  As these are essential aspects of the contribution, the experimental section isn't self contained.  If the authors cannot fit all essential components, then a conference might be the wrong format for this manuscript.\n\n\n\nW7) Experimental results are not fully conclusive; they compare against a standard neural network, but more classical controller structures as baseline are missing in the main paper. Thus, I find the results only partially convincing for showing the benefit of the proposed Neural-PI controller.\n\n* It remains unclear whether the chosen problems require sophisticated nonlinear control such as the proposed Neural-PI controller. First, the problem description does not convey this (see above).  Second, in the supplementary material, the authors report the results also of a standard linear PI, whose parameters have been optimized similarly to the optimization of the NNs.  Apparently, the linear PI achieves almost the same performance as the Neural-PI for vehicle platooning, and very close performance for the power grid.  The results suggests that at least for these systems, there is no significant benefit of the proposed and more involved Neural-PI structure.  Hence, the result are not conclusive regarding the proposed architecture in this regards.\n* Baseline comparisons (i.e., the linear PI from the supplementary, and maybe also a model-based controller such as LQR) should be included in the main paper to judge the performance of the proposed Neural-PI.\n* I find it surprising that the DenseNN cannot stabilize the vehicle platooning system and actually has some significant deviation at the end of the horizon (Fig. 5(b), 15 sec).  Such deviation should also be visible in the objective function, so I would expect that the NN (if trained enough) can accommodate for this. \n* A discussion of assumptions and theoretical guarantees is missing in the main paper. In ll. 177/178, it is claimed that in the experimental section, it will be discussed how EIP (the main assumption) can be verified if the system is unknown.  While I agree that this discussion is very important with regards of the claimed contribution of this paper, I could not find this in Sec. 6.\n* The above points hold for both examples (platooning, power grid).\n\n\n\nMinor: \n\n* l 192: As usually, p is a gain (e.g., a real number), it would be good to emphasize when introducing it that, here, it is meant as a function of the tracking error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed comments and thank the reviewer for the time spent on this paper. \n\n**(W1) We thank the reviewers for identifying imprecise statements. We have updated them and that would be reflected in the revised paper. Below we focus on some clarifications**:\n\n- W1-2: Steady-state behavior is generally not equivalent to long-term dynamics. It is difficult to quantify how long the trajectory is enough to reach a steady state, thus enforcing steady-state output tracking by penalizing a long-term cost is challenging. Therefore we enforce steady-state tracking by construction, instead of relying on training. \n- W1-4: Note that previous works, including those suggested by the reviewer, require the controllers to be linear. Here, we study nonlinear systems with nonlinear controllers. \n- W1-5: We study monotone functions defined as functions satisfying  $(f(x) - f(x'))^T (x - x) >0 $,  a much broader class compared to the ones considered by existing works where the function needs to satisfy $f(x) \\leq f(x')$ if $x \\leq x'$. \n\n**(Q2) The 3 papers closest to our approach and their limitations**\n\n- [R1]   Zhou et al. Neural Lyapunov control of unknown nonlinear systems with stability guarantees, Neurips 2022.\n  - This work learns a Lyapunov function and uses it to certify stability through a satisfiability modulo theories (SMT) solver. **But it is difficult to scale to high-dimensional systems and the learned Lyapunov function only works for a single equilibrium.** \n- [R2] Cui et al.Reinforcement learning for optimal primary frequency control: A Lyapunov approach. IEEE Trans. on Power Systems, 2022.\n  - This work shows that a class of monotone controllers can provide stability guarantees for power system frequency control problems, but it **relies on a tailor-made Lyapunov function and is **limited to SISO systems**. \n- [R3] Bürger et al. Duality and network theory in passivity-based cooperative control. Automatica, 2014.\n  -  This work derives edge feedback control for networked systems with each node being equilibrium-independent passive. **It did not optimize over the monotone functions**. For example, the paper chose the function to be tanh(), but this particular function actually is far from the optimal choice in their application. \n\n**(W2) Related work and our main contributions**\n\n- **Learning-based control with stability guarantees.**  For nonlinear systems, many works add soft penalties on the violation of stability conditions in the cost function, but it is nontrivial to certify stability for \\textit{all} the possible initial states in a compact set.  Learning and certifying a Lyapunov function using SMT solvers [R1] is difficult to scale to high-dimensional systems and the learned Lyapunov function only works for a single equilibrium. The approach in [R4] for control-affine systems requires that all the states are accessible by a central agent and many systems are not control affine.**Our proposed controller guarantees stability for a set of equilibria by construction, and only needs access to the outputs (not the full states).**\n\n  [R4] Lederer et al. Uniform error bounds for gaussian process regression with application to safe control. Neurips, 2019.\n\n\n- **Algorithm to tune nonlinear and MIMO PI Controllers.** Classical PID control structures are widely used, but tuning the PID parameters is tedious. Learning-based methods become popular to tune PID parameters, although still restricted to linear control gains (including the two papers suggested by the reviewer). **Our contribution is the more generalized PI control and the MIMO neural network for parameterization**.\n\n\n**(W3, W5-7) We agree and will clarify these in the revised paper.**\n\n**(W4, Q1)  The notion of proportional + integral (PI) controller.**\n\n- We call it PI, because the name can make it easier for readers to understand the structure of the controller that is a summation of a function of the proportional variable $ -y+\\bar{y}$ and a function of the integral variable $s$. Note that monotone $p(  -y+\\bar{y})$ covers linear functions as a special case. Our terminology is not new, see for example, [R5], where the controller is not linear in the tracking error. \n\n  [R5] Generalized proportional integral control for periodic signals under active disturbance rejection approach, ISA Transactions, 2014.\n\n**(Q3, Q4) Comparison with Neural-PI controller**\n\n- We would like to emphasize that ** the transient cost attained by Neural-PI is significantly lower than Linear-PI.**  As we show in the (G1) of global response, Neural-PI reduces the average transient cost by approximately 30%  compared to Linear-PI (please see Fig 1 in the attached PDF in the global response). \n\n**(W7-3) DenseNN in (Fig. 5(b), 15 sec) has some significant deviation, does the objective function cover this?**\n\n- Fig. 5(b) is one representative unstable case, but it does not indicate that all cases are unstable. The objective function is useful to capture the behavior of the majority of the cases, but it is not sufficient to guarantee the behavior of all possible cases. Learning can only trains the neural network through a finite number of trajectories over a finite time horizon, so it cannot guarantee the stability of trajectories from **all** possible states. \n\n**(Q5) Why did you decide to not include the experimental results of the linear PI in the main paper?**\n\n- We did not include Linear PI in the main paper because the structure of Neural-PI involves the Linear-PI as a special case, so there is no surprise that optimizing Neural-PI would work better than Linear-PI. And we did not add it to the paper because of space limitations. We agree that including Linear-PI in the main paper can demonstrate the improvement of Neural-PI. We will include the comparisons in the revised main paper. \n\n**The consideration of noises/disturbances and real-world applications** \n\n- Please refer to (G2) and (G5) of the global response and the attached pdf.\n"}]

Author rebuttal:
We thank the reviewers for the detailed comments. We notice there are some common concerns and respond to these questions in the general response below.  

We have attached a **one-page PDF containing the figures and updated numerical results**. Because of the space limits, all the numerical results are demonstrated in the case of power system frequency control. Note that we are not allowed to update the main paper and supplementary material in this round, so the changes would be reflected when possible. 

**(G1) The comparison with Linear-PI**

Classical PI controllers are the impetus for this work, where we want to maintain good theoretical properties and improve over some of the limitations (e.g., slow convergence speed and larger control efforts). This is the reason why we don’t use a general neural network, but a specific Neural-network structure. The key comparison results are as follows:

- The structure of Neural-PI involves the classical PI controllers (i.e., Linear-PI) as a special case, and the theoretical results in Theorem 2 show that  **Neural-PI control law retains all the stability guarantees of classical linear PI control**. 

- Experimental results in the supplementary material showed that the **Neural-PI achieves much lower transient cost compared with Linear-PI**. In particular, **the performance improvement is present even if there is little data to train the controllers**.  For Neural-PI, Linear-PI, and DenseNN trained with different amounts of trajectories, the table below compares the average transient cost on testing trajectories starting from randomly generated initial states. For all the cases, Neural-PI reduces the average transient cost by approximately 30% compared to Linear-PI. Moreover, Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN. The comparison with the error bar and the dynamics are visualized in Figure 1 and Figure 2 of the attached PDF, respectively.

   **Table 1: The average transient cost on 100 testing trajectories starting from randomly generated initial states**

   | # of training trajectories  |**Neural-PI** | Linear-PI |DenseNN|
   |---|---|---|---|
   | 5 | **0.1328** | 0.1915 |  1.0 |
   | 10 | **0.1300**| 0.1865 | 0.9833 |
   | 50 | **0.1257** | 0.1838  | 0.9624 | 
   | 100 | **0.1234**| 0.1816 | 0.9214 | 
   | 300 | **0.1233**| 0.1815 |  0.5347 |





**(G2) The impact of noises/disturbances**


- The satisfaction of the Lyapunov condition is **robust to disturbances in the system parameters and does not need to know how large the disturbances** are, as shown in the proof Theorem 2 and Remark 2. Take the power system experiment as an example, the load $d$ in the system dynamics is a parameter. If there is a sudden change in the load levels, the proposed controller design still stabilizes the system and tracks the required frequency at 60Hz. In Figure 3(a) of the attached PDF in the global response, we demonstrate the system dynamics after two disturbances in load.

- For noises in the data measurements and state transition dynamics, we empirically find that **the systems are input-to-state stable, i.e., that bounded noise will lead to bounded states**.  Incorporating noise in rigorous theoretical analysis is an important future direction for us. We have performed empirical studies with noises in both data measurement and dynamics, as shown in Figure 3(b)-(c) of the attached PDF in the global response.  The signal-to-noise ratio is about 5 dB (much larger than typical measurement noises).  Figure 3 shows that the performance guarantees are robust to noise. 




**(G3) Clarification of numerical comparisons**

- Although there is a vast literature on training algorithms (e.g., model-based and model-free reinforcement learning) that either train a neural network or a classical PID controller, the stability guarantees that can be achieved in the structure of neural networks has rarely been explored. Indeed, **The focus on structure is complementary to training algorithms**: by replacing unstructured neural networks with the proposed Neural-PI structure, the trained controllers will provide provable guarantees of stability and zero steady-state output tracking error.  The comparisons then focus on the structure of controllers  (i.e.,  Neural-PI, DenseNN, and classical linear PI) trained by the same algorithm.

**(G4) Summary of contributions**
- Our contribution is the Neural-PI controller that has provable guarantees of stability and zero steady-state output tracking error, and features 1)  the stability guarantee holds for a set of equilibria, 2) multi input multi output (MIMO) monotone neural network architecture and universal approximation theorem, 3) capability to respect the communication constraints. 

**(G5) The applicability to the real world**

- The above results show that the **Neural-PI control law retains all the stability guarantees of classical linear PI control, but achieves much lower transient cost**. The control theoretic guarantees also significantly reduce the amount of data required to train well. Since classical PI control is widely utilized in real-world applications, we expect that the controllers can be transferred to real-world scenarios. For example, our controllers are in the process of being implemented in a real-world test system, the UCSD DER-connect testbed (none of the authors are affiliated with the testbed). 
- Potential barriers include the verification of EIP when a storage function is difficult to be found and provable guarantees on the robustness to noises. These are all important future directions for us. 



Meta-review: The paper proposes a generalization of the classic proportional + integral (PI) control law to be parametrized by neural networks. It is empirically observed that performance is better than the classical PI method on several example systems. The proposed structure also comes with certain correct-by-construction guarantees. Overall, this is an interesting way to combine machine learning and control.
Decision: Accept (poster)