SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenqi Cui $^{1}$  Yan Jiang $^{1}$  Baosen Zhang $^{1}$  Yuanyuan Shi $^{2}$
<sup>1</sup>University of Washington, WA 98195 <sup>2</sup>University of California San Diego, CA 92093
wenqicui@uw.edu jiangyan@uw.edu zhangbao@uw.edu yyshi@eng.ucsd.edu
We study the optimal control of multiple-input and multiple-output dynamical systems via the design of neural network-based controllers with stability and output tracking guarantees. While neural network-based nonlinear controllers have shown superior performance in various applications, their lack of provable guarantees has restricted their adoption in high-stake real-world applications. This paper bridges the gap between neural network-based controllers and the need for stabilization guarantees. Using equilibrium-independent passivity, a property present in a wide range of physical systems, we propose neural Proportional-Integral (PI) controllers that have provable guarantees of stability and zero steady-state output tracking error. The key structure is the strict monotonicity on proportional and integral terms, which is parameterized as gradients of strictly convex neural networks (SCNN). We construct SCNN with tunable softplus- $\beta$  activations, which yields universal approximation capability and is also useful in incorporating communication constraints. In addition, the SCNNs serve as Lyapunov functions, providing end-to-end stability and tracking guarantees. Experiments on traffic and power networks demonstrate that the proposed approach improves both transient and steady-state performances, while unstructured neural networks lead to unstable behaviors.
Learning-based methods have the potential to solve difficult problems in control and have received significant attention from both the machine learning and control communities. A common problem in many applications is to design controllers that—with as little control effort as possible—stabilize a system at a prescribed output. A canonical example is the LQR problem and its variants, which find optimal linear controllers for linear systems [1, 2].
For nonlinear systems, the problem is considerably harder. Neither of the two control goals, system stabilization and output tracking at the steady state, is easy to achieve. Their combination, optimizing controller costs while guaranteeing stability and output tracking at the steady state, is even more challenging. For example, vehicles in a platoon should stay in formation (stability) while cruising at the desired speed (output tracking) [3]. The optimization is then done over the set of all controllers that achieves both of these goals, e.g., reaching a platoon while consuming the least amount of fuel. Another pertinent application is the stability of electric grids with high amounts of renewables. Unlike systems with conventional generators, the power electronic interfaces of the inverters need to be actively controlled such that the grid synchronizes to the same frequency (output tracking), at the minimum operational costs. Currently, learning becomes a popular tool to parameterize nonlinear controllers as neural networks and train them to optimize performance, but provable guarantees on stability and steady-state error for these controllers are nontrivial to obtain [4-7].
Many real-world applications have multiple (possibly a continuous set of) equilibria that may be of interest. For example, a group of vehicles may need to cruise at different speeds. Moreover, the
internal states may not be directly accessible and sometimes there are communication limit. These constraints make it difficult to enforce stability by middle steps including learning a Lyapunov function and learning a model of the system (for details, see related work in Section 1.1). Therefore, we seek to achieve "end-to-end" guarantees: the neural network-based controller should guarantee stability and steady-state error by constructions, for a range of possible tracking points. There are some works that showed a class of monotone controllers can provide stability guarantees for varying equilibria [8-10], but they rely on tailor-made Lyapunov functions that are found in specific applications. Especially, these results are limited to single-input and single-output (SISO) control. In practice, however, lots of complex systems need controllers that are multiple-input and multiple-output (MIMO), sometimes with high input and output dimensions.
Moreover, current learning-based approaches typically focus on optimizing transient performance (the cost and speed at which a system reaches an equilibrium), often overlooking the steady-state behavior (cost at the equilibrium state) [11-13]. In classical control terminology, these controller is proportional. Steady-state requirements are difficult to incorporate since training can only occur over a finite horizon. To enforce output tracking at the steady-state, an integral term is commonly needed to drive the system outputs to the desired value at the steady state [14-17]. This reasoning underlies the widespread use of linear Proportional-Integral (PI) controllers in practical applications [18-20]. However, linear parameterization inherently limits the controllers' degrees of freedom, potentially leading to suboptimal performance. This work addresses the following question: Can we learn nonlinear controllers that guarantee transient stability and zero steady-state error for MIMO systems?
Contributions. Clearly, it is not possible to design a controller for all nonlinear systems and the answer to the question above depends on the class of systems under consideration. This paper focuses on systems that satisfy the equilibrium independent passivity (EIP) [21, 22], which is present in many critical societal-scale systems including transportation [23], power systems [24], and communication network [25]. This abstraction allows us to design generalized controllers without considering the detailed physical system dynamics. We propose a structured Neural-PI controller that has provable guarantees of stability and zero steady-state output tracking error. The key structure we use is strictly monotone functions with vector-valued inputs and outputs. Experiments on traffic and power systems demonstrate that Neural-PI control can reduce the transient cost by at least  $30\%$  compared to the best linear controllers and maintain stability guarantees. Unstructured neural networks, on the other hand, lead to unstable behaviors. We summarize our major contributions as follows.
1) We propose a generalized PI structure for neural network-based controllers in MIMO systems, with proportional and integral terms being strictly monotone functions constructed through the gradient of strictly convex neural networks (SCNN). We construct SCNN with a tunable softplus-  $\beta$  activation function and prove their universal approximation capability in Theorem 1.
2) For a multi-agent system with an underlying communication graph, we show how to restrict the controllers to respect the communication constraints through the composition of SCNNs.
3) Using EIP and the monotone functions structured by SCNNs, we design a generalized Lyapunov function that works for a range of equilibria. This provides end-to-end guarantees on asymptotic stability and zero steady-state output tracking error proved in Theorem 2. The structured neural networks can be trained for transient optimization without jeopardizing the guarantees, establishing a framework for coordinating transient optimization and steady-state requirement.
Learning-based control with stability guarantees. Recently, there has been a large interest in enforcing stability in learning-based controller [26-30]. Many works add soft penalties on the violation of stability conditions in the cost function, but it is nontrivial to certify stability for all the possible initial states in a compact set [30]. Some recent works learn a Lyapunov function and use it to certify stability through a satisfiability modulo theories (SMT) solver [26-29]. But this approach is difficult to scale to high-dimensional systems and the learned Lyapunov function only works for a single equilibrium. For control-affine systems, the work in [31] designs feedback linearizing policy with integral control to guarantee stabilizing to a range of equilibria. But it requires that all the state are accessible and many systems are not control affine. Our proposed controller guarantees stability for a set of equilibria by construction, and only needs access to the outputs (not the full states).
Optimizing long-term behavior. To regulate long-term behavior, existing works train neural networks using a cost function defined over a long time horizon [32, 33]. However, it is difficult
to quantify how long the trajectory is enough to reach the steady state, thus enforcing steady-state tracking performance by adding a long-term cost is challenging. Our proposed controller enforces steady-state tracking by construction, instead of relying on training.
Algorithm to tune MIMO PI controller. Classical Proportional Integral (PI) control structures are widely used in industrial applications, while tuning PI controller parameters in MIMO systems is tedious and relies on heuristic rules [34, 35]. Learning-based methods become popular to tune PI parameters, although still restricted to linear control gains [34, 35]. Our contribution is the more generalized PI control and the MIMO neural network for parameterization.
Monotone neural network. Even though scalar-valued monotone functions have been studied in the learning community [36, 37], their generalization to the vector-valued case has not. Numerous papers studied vector-valued functions that are monotone in every input, that is,  $\pmb{q}(\pmb{x}) \leq \pmb{q}(\pmb{x}')$  if  $\pmb{x} \leq \pmb{x}'$  [38-41]. This is only a small subclass of vector-valued monotone functions that defined as functions satisfying  $(\pmb{q}(\pmb{x}) - \pmb{q}(\pmb{x}'))^\top (\pmb{x} - \pmb{x}') \geq 0$ . In this paper, we construct the general class of monotone functions (often called monotone operators) [42] that satisfy the inner product inequality.
We consider a dynamic system described by:
$$
\dot {\boldsymbol {x}} = \boldsymbol {f} (\boldsymbol {x}, \boldsymbol {u}), \quad \boldsymbol {y} = \boldsymbol {h} (\boldsymbol {x}), \tag {1}
$$
with state  $\boldsymbol{x}(t) \in \mathbb{R}^n$ , control action  $\boldsymbol{u}(t) \in \mathbb{R}^m$ , and output  $\boldsymbol{y}(t) \in \mathbb{R}^v$ . We also sometimes omit the time index  $t$  for simplicity<sup>1</sup>. In many practical applications, not all of the internal states are directly accessible. Hence, we consider control actions that follow output-feedback control laws of the form  $\boldsymbol{u} = \boldsymbol{g}(\boldsymbol{y})$ , which is a static function of the output  $\boldsymbol{y}$  and not the state  $\boldsymbol{x}$ .
A state  $\pmb{x}^*$  such that  $\pmb{f}(\pmb{x}^*,\pmb{u}^*) = \mathbb{0}_n, \pmb{y}^* = \pmb{h}(\pmb{x}^*), \pmb{u}^* = \pmb{g}(\pmb{y}^*)$  is called an equilibrium since the system stops changing at this state. Throughout this paper, the superscript * indicates the equilibrium values of variables.
Definition 1 (Local asymptotic stability [16], Definition 4.1). The system (1) is asymptotically stable around an equilibrium  $\mathbf{x}^*$  if,  $\forall \epsilon > 0$ ,  $\exists \delta > 0$  such that  $\| \mathbf{x}(0) - \mathbf{x}^* \| < \delta$  ensures  $\| \mathbf{x}(t) - \mathbf{x}^* \| < \epsilon$ ,  $\forall t \geq 0$ , and  $\exists \delta' > 0$  such that  $\| \mathbf{x}(0) - \mathbf{x}^* \| < \delta'$  ensures  $\lim_{t \to \infty} \| \mathbf{x}(t) - \mathbf{x}^* \| = 0$ .
One of the main tools to prove asymptotic stability is the Lyapunov's direct method [27, 16]:
Lemma 1 (Lyapunov functions and asymptotic stability [16], Theorem 4.1). Consider the system (1) with an equilibrium  $\pmb{x}^{*}\in \mathcal{X}\subset \mathbb{R}^{n}$ . Suppose there exists a continuously differentiable function  $V:\mathcal{X}\mapsto \mathbb{R}$  that satisfies the following conditions: 1)  $V(\pmb{x}^{*}) = 0,V(\pmb {x}) > 0,\forall \pmb {x}\in \mathcal{X}\backslash \pmb{x}^{*};2)$ $\dot{V} (\pmb {x}) = (\nabla_{\pmb{x}}V(\pmb {x}))^{\top}\dot{\pmb{x}}\leq 0,\forall \pmb {x}\in \mathcal{X}$  with the equality holds when  $\pmb {x} = \pmb{x}^{*}$ . Then the system is asymptotically stable around  $\pmb{x}^{*}$ .
The key challenge to using Lyapunov theory is in constructing such a function and verifying the satisfaction of the conditions in Lemma 1. An important part of this paper is to show how to systematically use neural networks to construct Lyapunov functions for a class of nonlinear systems.
Besides stability, we are often interested in achieving a specific equilibrium such that the output converges to a prescribed setpoint.
Definition 2 (Output tracking to  $\bar{y}$ ). The dynamical system (1) is said to track a setpoint  $\bar{y}$  if  $\lim_{t\to \infty}\pmb {y}(t) = \bar{\pmb{y}}$
For output tracking, the dimension of the input  $\mathbf{u}$  should be at least the dimension of the output  $\mathbf{y}$ , otherwise there is not enough degrees of freedom in the input to track all the outputs. If the dimensions of the input are strictly larger than the output, it is always possible to define "dummy" outputs (e.g., by appending zeros) to match the input dimension. Therefore, it is common to assume the same input and output dimensions [22]. In the remainder of this paper, we make the following assumption.
Assumption 1 (Identical input and output dimension). The input  $\mathbf{u}$  and the output  $\mathbf{y}$  have the same dimension, namely,  $\mathbf{u}(t)$  and  $\mathbf{y}(t)$  are both vectors in  $\mathbb{R}^m$ .
Throughout this paper, vectors are denoted in lower case bold and matrices are denoted in upper case bold, unless otherwise specified. Vectors of all ones and zeros are denoted as  $\mathbb{1}_n$ ,  $\mathbb{0}_n \in \mathbb{R}^n$ , respectively.
Figure 1: (a) The controller aims to improve transient performances after disturbances while guaranteeing stabilization to the desired steady-state output  $\bar{y}$ . (b) Two examples of physical systems with output tracking tasks. (c) We provide end-to-end stability and output tracking guarantees by enforcing stabilizing PI structure in the design of neural networks. The key structure is strictly monotone functions, which are parameterized by the gradient of SCNNs. (d) The transient performance is optimized by training the neural networks.
We will show that strictly monotone functions play an important role in guaranteeing stability and output tracking in a range of systems. Using neural networks for constructing monotone functions from  $\mathbb{R}$  to  $\mathbb{R}$  has been well-studied in [36, 37]. However, since we consider MIMO systems in this paper, we use a generalized notion of monotonicity for vector-valued functions as defined in [42].
Definition 3 (Strictly monotone function [42]). A continuous function  $\pmb{q} : \mathbb{R}^m \mapsto \mathbb{R}^m$  is strictly monotone on  $\mathcal{D} \subset \mathbb{R}^m$  if  $(\pmb{q}(\pmb{\eta}) - \pmb{q}(\pmb{\xi}))^\top (\pmb{\eta} - \pmb{\xi}) \geq 0, \forall \pmb{\eta}, \pmb{\xi} \in \mathcal{D}$ , with the equality holds if and only if  $\pmb{\eta} = \pmb{\xi}$ .
In this paper, we will show a way to construct (strictly) monotone functions using the gradient of (strictly) convex neural networks.
3.1 Transient and steady-state requirements
We aim to optimize the controller in  $\mathbf{u}$  to improve the transient response after disturbances while guaranteeing steady-state performance, as shown in Figure 1(a). By steady-state performance, we mean the system should settle at the desired setpoint  $\bar{\mathbf{y}}$ . By transient performance, we want the system to quickly reaching the steady state with small control efforts.
The inverted pendulum in Figure 1(b) serves as a good motivating example. The input  $u$  is the force on the pendulum. The objectives include 1) the angle  $y$  should reach a setpoint  $\bar{y}$  (e.g.,  $5^{\circ}$ ) and stays there; 2) minimizing the control cost and deviation of  $y$  from  $\bar{y}$  before reaching the steady state. These objectives are common in many real-world applications, such as vehicle platooning (Fig. 1(b)) and power system control in our experiments.
Transient performance optimization. During the transient period, our goal is to quickly stabilize the system to the desired setpoint  $\bar{y}$ , while trading off with using large control efforts in  $\pmb{u}$ . Let  $J(\cdot)$  be the cost function of the system. The transient optimization problem up to time  $T$  is,
$$
\min  _ {\boldsymbol {u}} \sum_ {t = 0} ^ {T} J (\boldsymbol {y} (t) - \bar {\boldsymbol {y}}, \boldsymbol {u} (t)), \tag {2a}
$$
$$
\text {s . t . d y n a m i c s i n} (1): \dot {\boldsymbol {x}} = \boldsymbol {f} (\boldsymbol {x}, \boldsymbol {u}), \quad \boldsymbol {y} = \boldsymbol {h} (\boldsymbol {x}), \tag {2b}
$$
$$
\text {s t a b i l i t y} \quad 1 - 2: \lim  _ {t \rightarrow \infty} \boldsymbol {y} (t) = \bar {\boldsymbol {y}} \tag {2c}
$$
Note that in (2a) we sum up to time  $T$ , which should be long enough to cover the transient period. Since we require tracking in (2c), the exact value of  $T$  is not critical in the optimization problem. In
2Including other forms of differentiable cost functions do not change the analysis.
practice, the system dynamics (1) (and sometimes even the cost function) can be highly nonlinear and nonconvex. Therefore, the current state-of-the-art is to learn  $\mathbf{u}$  by parameterizing it as a neural network and training it by minimizing the cost in (2a) [43, 44]. But the key challenge with applying these neural network-based controllers is guaranteeing stability and output tracking [45, 26, 30]. Even if the learned policy may appear "stable" during training, it is not necessarily stable during testing. This can be observed in both the vehicle and power system experiments in Section 6.
Stability for a range of tracking points. We emphasize that the setpoint  $\bar{y}$  may vary in practice (e.g., the setpoint velocity of vehicles may change), and thus the controller is required to guarantee stability and output tracking to a range of equilibria corresponding to the setpoints. This is difficult to achieve through existing works that enforce stability by learning a Lyapunov function [26, 27, 30], since the learned Lyapunov function is for a single equilibrium (normally setup as  $x^{*} = \mathbb{0}_{n}$ ). These methods also require that all the states  $\pmb{x}$  are observed, which may not be achieved in practice.
Communication requirement. A typical constraint in a large system is limits on communications. For example, vehicles in a group may only be able to measure the output of their neighbors (line-of-sight) or nodes in a power system may only have real-time communication from a subset of other nodes. Therefore, the control action  $\boldsymbol{u}_i$  may be constrained to be a function of a subset of the outputs  $\boldsymbol{y}$ . We show later in Subsection 4.3 about how these communication constraints can be naturally accommodated in our controller design.
3.2 Bridging controller design and stability through passivity analysis
As shown in Figure 1(a), optimizing transient and steady-state performances are two problems in two different time-scales. Therefore, coordinating them is a central challenge.
End-to-end performance guarantees. We propose to overcome this challenge through a structured controller design that provides end-to-end stability and tracking guarantees, as shown in Figure 1(c-d). "End-to-end" means that the guarantees are provided by construction, and do not depend on how the controller is trained. Thus, the neural networks can be trained to optimize the transient performance without jeopardizing the stability and steady-state guarantees. In particular, the construction works for a range of equilibria and can conveniently incorporate communication characteristics.
Instead of working on specific systems individually, we seek to find a family of physical systems that allows us to derive a generalized controller design. It turns out that the notion of equilibrium independent passivity (EIP) [21, 22] provides a concise and useful abstraction of physical systems for stability analysis. This paper thus focuses on systems satisfying the following assumptions:
Assumption 2 (Equilibrium-Independent Passivity [21]). The system (1) is strictly equilibrium-independent passive (EIP), which satisfies: (i) there exists a nonempty set  $\mathcal{U}^* \subset \mathbb{R}^m$  such that for every equilibrium  $\mathbf{u}^* \in \mathcal{U}^*$ , there is a unique  $\mathbf{x}^* \in \mathbb{R}^n$  such that  $\mathbf{f}(\mathbf{x}^*, \mathbf{u}^*) = \mathbb{0}_n$ , and (ii) there exists a positive definite storage function  $S(\mathbf{x}, \mathbf{x}^*)$  and a positive scalar  $\rho$  such that
$$
S \left(\boldsymbol {x} ^ {*}, \boldsymbol {x} ^ {*}\right) = 0 \quad a n d \quad \dot {S} \left(\boldsymbol {x}, \boldsymbol {x} ^ {*}\right) \leq - \rho \| \boldsymbol {y} - \boldsymbol {y} ^ {*} \| ^ {2} + \left(\boldsymbol {y} - \boldsymbol {y} ^ {*}\right) ^ {\top} \left(\boldsymbol {u} - \boldsymbol {u} ^ {*}\right). \tag {3}
$$
The EIP property in Assumption 2 has been found in a large class of physical systems, including transportation [23], power systems [24, 46], robotics [47], communication [25], and others. We will conduct experiments on vehicle platoons and power systems to show how EIP can be validated.
The condition (3) of EIP systems provides us a generalized approach to construct Lyapunov functions without knowing the specifies of the dynamics  $f(\cdot)$ . In turn, we are able to find the right structure for the controllers. In Section 4, we show what these structures are for PI controllers and how to enforce such structures in the design of neural networks. Then Section 5 formally demonstrates the theoretical guarantees and the training procedure for transient optimization.
4 Structured Neural-PI Control
In this section, we construct controllers with a proportional and an integral term, which are both vector-valued strictly monotone functions parameterized by the gradient of strictly convex functions. Then, we present a neural network architecture that is strictly convex by construction and can conveniently incorporate communication constraints.
4.1 Generalized PI control
To realize output tracking, we introduce an integral variable  $\dot{s} = \bar{y} - y$ . Intuitively,  $s$  is the accumulated tracking error and will remain unchanged at the steady-state when  $y = \bar{y}$ . On this basis, we consider a generalized PI controller  $\boldsymbol{u} = \boldsymbol{p}(\bar{y} - \boldsymbol{y}) + \boldsymbol{r}(s)$ . The first component is the proportional term, where  $\boldsymbol{p}(\bar{y} - \boldsymbol{y})$  is a function of the tracking error  $\bar{y} - y$  between the current output  $\boldsymbol{y}$  and desired output  $\bar{y}$ . The second component is the integral term  $\boldsymbol{r}(s)$  as a function of the integral of historical tracking errors. Intuitively, the proportional term drives  $\boldsymbol{y}$  close to  $\bar{y}$ , and the integral term ensures the tracking error equals zero at the steady state.
Generalized PI Controller. Compactly, the control law is
$$
\boldsymbol {u} = \underbrace {\boldsymbol {p} (\bar {\boldsymbol {y}} - \boldsymbol {y})} _ {\text {P r o p o r t i a n c o n t r o l}} + \underbrace {\boldsymbol {r} (\boldsymbol {s})} _ {\text {I n t e g r a l c o n t r o l}} \tag {4a}
$$
$$
\dot {s} = \bar {y} - y. \tag {4b}
$$
Remark 1. The above controller can be envisioned as a nonlinear generalization of widely adopted linear Proportional-Integral (PI) controllers, where  $\pmb{p}(\bar{\pmb{y}} - \pmb{y}) \coloneqq \pmb{K}_P(\bar{\pmb{y}} - \pmb{y})$ ,  $\pmb{r}(s) \coloneqq \pmb{K}_I(s)$  with  $\pmb{K}_P$  and  $\pmb{K}_I$  being the proportional and integral coefficients (scalar for SISO control). Linear PI control laws are almost always used in existing works [14-16], but the performance of linear PI controllers can be poor for large-scale nonlinear systems[6, 48].