SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To achieve provable stability guarantees with output tracking, we need to construct a Lyapunov function that works for the closed-loop system formed by (1) and (4). Therefore, we further design structures in the proportional function  $\pmb{p}(\cdot)$  and the integral function  $r(\cdot)$  that can be utilized in Lyapunov stability analysis. The structures are strictly monotone functions, which generalizes conventional linear functions. In the next subsection, we will show how to parameterize strictly monotone functions. In section 5.1, we will show how the PI controllers structured with monotone functions provide end-to-end stability and output tracking guarantees.
4.2 Strictly monotone function through gradients of strictly convex neural networks
It is not trivial to parameterize strictly monotone functions since this is an infinite-dimensional function space. Scalar-valued monotone functions mapping from  $\mathbb{R}$  to  $\mathbb{R}$  [36, 37] has been proposed, but it is difficult to extend these designs to MIMO systems.
In this paper, we propose a new parameterization of strictly monotone functions by leveraging the fact in Proposition 1 that the gradient of a strictly convex function is strictly monotone.
Proposition 1 (Gradients of strictly convex functions). Let  $g: \mathbb{R}^m \mapsto \mathbb{R}$  be a strictly convex function, then  $(\nabla_{\eta}g(\pmb{\eta}) - \nabla_{\xi}g(\pmb{\xi}))^\top (\pmb{\eta} - \pmb{\xi}) \geq 0 \forall \pmb{\eta}, \pmb{\xi} \in \mathbb{R}^m$ , with equality holds if and only if  $\pmb{\eta} = \pmb{\xi}$ . Namely,  $\nabla g: \mathbb{R}^m \mapsto \mathbb{R}^m$  is a strictly monotone increasing function.
Hence, the strictly monotone property of  $p(\cdot)$  and  $r(\cdot)$  can be guaranteed by design if they are the gradient of a strictly convex function, as shown by Figure 2. The next proposition shows how to construct a strictly convex neural network (SCNN).
Figure 2: Strictly monotone function constructed by strictly convex neural networks (SCNN).
Proposition 2 (Strictly convex neural networks). Consider a function  $g(z;\pmb {\theta}): \mathbb{R}^m\mapsto \mathbb{R}$  parameterized by  $k$ -layer, fully connected neural
network with the input  $\mathbf{z} \in \mathbb{R}^m$ . The output  $\mathbf{o}_l$  of layer  $l = 0, \dots, k - 1$  and the function  $g(\mathbf{z}; \boldsymbol{\theta})$  is represented as
$$
\boldsymbol {o} _ {l + 1} = \sigma_ {l} \left(W _ {l} ^ {(o)} \boldsymbol {o} _ {l} + W _ {l} ^ {(z)} \boldsymbol {z} + b _ {l}\right), \quad g (\boldsymbol {z}; \boldsymbol {\theta}) = o _ {k} \tag {5}
$$
where  $\pmb{o}_0, W_0^{(o)} \equiv 0, \pmb{\theta} = \left\{W_{0:k-1}^{(z)}, W_{1:k-1}^{(o)}, b_{0:k-1}\right\}$  are trainable weights and biases, and  $\sigma_i$  are non-linear activation functions. The function  $g(\pmb{z}; \pmb{\theta})$  is strictly convex in  $\pmb{z}$  provided that all  $W_{1:k-1}^{(o)}$  are positive,  $W_0^{(z)}$  is nonzero, and all functions  $\sigma_l$  are strictly convex and increasing.
Figure 3: Communication embedded controller. Take  $m = 4$  where  $\pmb{z} = (z_{1}, z_{2}, z_{3}, z_{4})$  and  $\mathbf{P}$  network  $\pmb{p} = \nabla_{\pmb{z}} g^{(P)}(\pmb{z}; \pmb{\theta}^{(P)}, \beta^{(P)})$  as an example. (a) Global communication, where  $\mathcal{V} = \{(1, 2, 3, 4)\}$  and  $\pmb{p}$  can be a function of the full  $\pmb{z}$ . (b) Decentralized,  $\mathcal{V} = \{(1), \dots, (4)\}$  and  $p_i$  is a function of  $z_i, \forall i$ . (c) Partial communication,  $\mathcal{V} = \{(1, 2), (2, 3, 4)\}$ , where there exists communication within  $(1, 2)$  and  $(2, 3, 4)$ .
The construction of SCNN follows the general structure of input-convex neural networks [49, 50], where the conditions on activation functions and weights are modified to achieve strictly convexity. The proof follows from the fact that positive sums of strictly convex functions are also strictly convex and that the composition of a strictly convex and convex increasing function is also strictly convex.
The next theorem demonstrates the universal approximation property of SCNN in Proposition 2.
Theorem 1 (Universal approximation). Let  $\mathcal{Z}$  be a compact domain in  $\mathbb{R}^m$  and  $q(z): \mathcal{Z} \mapsto \mathbb{R}$  be a Lipschitz continuous and strictly convex function. For any  $\epsilon > 0$ , there exists a function  $g(z; \theta): \mathcal{Z} \mapsto \mathbb{R}$  constructed by (5) where the activation function is the softplus- $\beta$  function
$$
\sigma_ {l} ^ {\text {S o f t p l u s} \beta} (x) := \frac {1}{\beta} \log \left(1 + e ^ {\beta x}\right), \tag {6}
$$
such that  $|q(z) - g(z;\pmb {\theta})| <   \epsilon$  for all  $\pmb {z}\in \mathcal{Z}$
The proof is given in Appendix A.1. We sketch the proof as follows. We leverage the results in [49, 50] that the structure (5) with ReLU activation is a universal approximation for any convex function. However, ReLU activations are not strictly convex, and thus we design the Softplus-  $\beta$  activation. By showing that the structure (5) with Softplus-  $\beta$  activation can approximate neural networks with the ReLU activations arbitrarily closely when  $\beta$  is sufficiently large, we then prove that the structure (5) with Softplus-  $\beta$  can universally approximate any strictly convex function.
Notably,  $\beta$  in the activation function is a tunable parameter. Hence, we write down the SCNN (5) with activation function being Softplus- $\beta$  function in (6) as  $g(z; \theta, \beta)$ , where  $\beta$  is an extra trainable parameter.
Thus, we parameterize the structured Neural-PI control law  $\pmb{u} = \pmb{p}(\bar{\pmb{y}} -\pmb {y}) + \pmb {r}(\pmb {s})$  in (4) as
$$
\boldsymbol {p} (\bar {\boldsymbol {y}} - \boldsymbol {y}) = \nabla_ {\boldsymbol {z}} g ^ {(P)} (\boldsymbol {z}; \boldsymbol {\theta} ^ {(P)}, \beta^ {(P)}) | _ {\boldsymbol {z} = \bar {\boldsymbol {y}} - \boldsymbol {y}}, \tag {7}
$$
$$
\boldsymbol {r} (\boldsymbol {s}) = \nabla_ {\boldsymbol {z}} g ^ {(I)} (\boldsymbol {z}; \boldsymbol {\theta} ^ {(I)}, \beta^ {(I)}) | _ {\boldsymbol {z} = \boldsymbol {s}}.
$$
This way,  $\pmb{p}(\cdot)$  and  $\pmb{r}(\cdot)$  by construction are strictly monotone. In particular, the convex functions  $g^{(P)}(z; \pmb{\theta}^{(P)}, \beta^{(P)})$ ,  $g^{(I)}(z; \pmb{\theta}^{(I)}, \beta^{(I)})$  play a vital role in constructing a generalized Lyapunov function and showing the satisfaction of the Lyapunov condition in Subsection 5.1. This subsequently provides end-to-end stability guarantees that do not depend on the training algorithm.
4.3 Embedding Communication Constraints
For some large-scale physical systems, not all of the control inputs have access to all of the output measurements at real-time. Therefore, some  $u_{i}$ 's cannot be a function of the full vector  $\mathbf{y}$  and  $s$  because of this lack of global communication, as elaborated in Subsection 3.1.
Constructing the controllers as the gradient of convex functions provides us with a convenient way to embed these communication constraints. Let  $\mathcal{V}$  be the set of indexes with communications. Take  $m = 4$  where  $\pmb {z} = (z_{1},z_{2},z_{3},z_{4})$  and the proportional term  $\pmb {p} = \nabla_{\pmb{z}}g^{(P)}(\pmb {z};\pmb{\theta}^{(P)},\pmb{\beta}^{(P)})$  as an example. Figure 3(a) shows the case with global communication, where  $\mathcal{V} = \{(1,2,3,4)\}$  and  $\pmb{p}$  can be a function of the full  $\pmb{z}$ . Figure 3(b) shows the case without communication where
$\mathcal{V} = \{(1),\dots ,(4)\}$  and  $p_i$  can only be a function of  $z_{i}$  for all  $i$ . Figure 3(c) shows the case  $\mathcal{V} = \{(1,2),(2,3,4)\}$ , where there exists communication within the indexes (1,2) and within (2,3,4). By defining SCNN  $g(\pmb {z}_{\pmb {v}_j};\pmb {\theta}_j^{(P)},\pmb {\beta}_j^{(P)})$  for each group in  $\pmb {v}_j\in \mathcal{V}$ , the gradient  $\nabla_{\pmb{z}}g(\pmb{z}_{\pmb{v}_j};\pmb{\theta}_j^{(P)},\pmb{\beta}_j^{(P)})$  will only be a function of  $\pmb{z}_{\pmb{v}_j}$  and thus satisfying the commutation constraints. Therefore, we construct the functions in (7) as
$$
\nabla_ {\boldsymbol {z}} g ^ {(P)} (\boldsymbol {z}; \boldsymbol {\theta} ^ {(P)}, \boldsymbol {\beta} ^ {(P)}) = \nabla_ {\boldsymbol {z}} \sum_ {\boldsymbol {v} _ {j} \in \mathcal {V}} g ^ {(P)} \left(\boldsymbol {z} _ {\boldsymbol {v} _ {j}}; \boldsymbol {\theta} _ {j} ^ {(P)}, \beta_ {j} ^ {(P)}\right), \tag {8}
$$
where  $\pmb{\theta}^{(P)}\coloneqq \{\pmb{\theta}_1^{(P)},\dots ,\pmb{\theta}_{|\mathcal{V}|}^{(P)}\}$ ,  $\pmb{\beta}^{(P)}\coloneqq \{\beta_1^{(P)},\dots ,\beta_{|\mathcal{V}|}^{(P)}\}$ , and  $\pmb{\theta}_j^{(P)},\pmb{\theta}_j^{(I)}$  are parameters of the SCNNs for group  $\pmb{v}_j$  within the communication network.
The function  $g^{(I)}(\pmb{z}; \pmb{\theta}^{(I)}, \beta^{(I)})$  is also constructed in a similar way as  $g^{(P)}(\pmb{z}; \pmb{\theta}^{(P)}, \beta^{(P)})$  in (8) to incorporate the communication constraints in  $\mathcal{V}$ . Strict convexity still holds since a sum of strictly convex functions is also strictly convex.
5 Training with End-to-End Guarantees
5.1 End-to-end stability and output-tracking guarantees
The convex function  $g^{(I)}(\pmb{s};\pmb{\theta}^{(I)},\pmb{\beta}^{(I)})$  constructed from SCNNs can be utilized to construct a Lyapunov function for proving stability using the Bregman distance defined in the following Lemma [51]:
Lemma 2 (Bregman distance). The Bregman distance associated with the convex function  $g^{(I)}(\pmb{s};\pmb{\theta}^{(I)},\pmb{\beta}^{(I)})$  is defined by
$$
B (\boldsymbol {s}, \boldsymbol {s} ^ {*}; \boldsymbol {\theta} ^ {(I)}, \beta^ {(I)}) = g ^ {(I)} (\boldsymbol {s}; \boldsymbol {\theta} ^ {(I)}, \beta^ {(I)}) - g ^ {(I)} (\boldsymbol {s} ^ {*}; \boldsymbol {\theta} ^ {(I)}, \beta^ {(I)}) - \nabla_ {\boldsymbol {s}} g ^ {(I)} (\boldsymbol {s} ^ {*}; \boldsymbol {\theta} ^ {(I)}, \beta^ {(I)}) ^ {\top} (\boldsymbol {s} - \boldsymbol {s} ^ {*}),
$$
which is positive definite with equality holds if and only if  $s = s^*$ .
The Bregman distance allows us to construct Lyapunov functions without specifying the equilibrium  $s^*$ . Combining the storage function in Assumption 2 and  $B(s, s^*; \theta^{(I)}, \beta^{(I)})$  in Lemma 2, next theorem shows that the Neural-PI controller stabilizes the system to the desired output  $\bar{y}$ .
Theorem 2. Suppose Assumption 2 holds and the input  $\mathbf{u}$  follows the structured PI control law  $\mathbf{u} = \mathbf{p}(\bar{\mathbf{y}} -\mathbf{y}) + \mathbf{r}(\mathbf{s})$  where  $\mathbf{p}(\cdot)$  and  $\mathbf{r}(\cdot)$  are constructed as the gradients of strictly convex neural networks in (7). If the system (1) has a feasible equilibrium, then the system is locally asymptotically stable around it. In particular, the steady-state outputs track the setpoint  $\bar{\mathbf{y}}$ , namely  $\mathbf{y}^{*} = \bar{\mathbf{y}}$ .
Theorem 2 shows that Neural-PI control has provable guarantees on stability and zero steady-state output tracking error from the structures in (7). Detailed proof could be found in Appendix A.2 and we sketch the proof below. At an equilibrium, the right side of (4b) equals to zero gives  $\pmb{y}^{*} = \bar{\pmb{y}}$ . We show that an equilibrium is asymptotically stable by constructing a Lyapunov function  $V(\pmb{x},\pmb{s})|_{\pmb{x}^{*},\pmb{s}^{*}} = S(\pmb{x},\pmb{x}^{*}) + B(\pmb{s},\pmb{s}^{*};\pmb{\theta}^{(I)},\pmb{\beta}^{(I)})$ . Since  $\pmb{r}(\pmb{s}) = \nabla_{\pmb{s}}g^{(I)}(\pmb{s};\pmb{\theta}^{(I)},\pmb{\beta}^{(I)})$  by construction in (7), the time derivative of the Bregman distance term is  $\dot{B} (\pmb {s},\pmb {s}^{*};\pmb{\theta}^{(I)},\pmb{\beta}^{(I)}) = (\pmb {r}(\pmb {s}) - \pmb {r}(\pmb {s}^{*}))^{\top}(-\pmb {y} - \pmb{y}^{*})$ . Combining with the fact  $\dot{S} (\pmb {x},\pmb{x}^{*})\leq -\rho \| \pmb {y} - \pmb{y}^{*}\|^{2} + (\pmb {y} - \pmb{y}^{*})^{\top}(\pmb {u} - \pmb{u}^{*})$  (from the EIP assumption), and  $\pmb {u} = \pmb {p}(\bar{\pmb{y}} -\pmb {y}) + \pmb {r}(\pmb {s})$ , we can conclude that the Lyapunov stability condition holds.
Remark 2. The satisfaction of the Lyapunov condition does not depend on the specifics of  $\pmb{f}(\cdot)$  (as long as it is EIP), making the stability certification robust to parameter changes for systems satisfying Assumption 2. We will demonstrate this in the experiment on power system control.
5.2 Training to improve transient performances
The Neural-PI controller can be trained by most model-based or model-free algorithms, since the stability and output-tracking are guaranteed by construction. Fig 4 visualizes the detailed construction and the computation graph in the dynamical system in (1). The trainable parameters  $\Theta \coloneqq \{\pmb{\theta}^{(P)},\pmb{\beta}^{(P)},\pmb{\theta}^{(I)},\pmb{\beta}^{(I)}\}$  are contained in  $\pmb{p}(\cdot)$  and  $\pmb{r}(\cdot)$  functions, where both are parameterized as the gradients of SCNNs.  $\pmb{u} = \nabla_{\bar{\pmb{y}} -\pmb{y}}g^{(P)}(\bar{\pmb{y}} -\pmb {y};\pmb{\theta}^{(P)},\pmb{\beta}^{(P)}) + \nabla_{\pmb{s}}g^{(I)}(\pmb {s};\pmb{\theta}^{(I)},\pmb{\beta}^{(I)})$  serves as control signal in the system defined in (1) that evolves through time. The loss function is defined as
$$
\operatorname {L o s s} (\boldsymbol {\Theta}) = \sum_ {t = 0} ^ {T} J (\boldsymbol {y} (t) - \bar {\boldsymbol {y}}, \boldsymbol {u} (t)), \tag {9}
$$
where  $J(\cdot)$  is the transient cost function defined in (2a). The parameters  $\Theta$  are optimized via gradient descent to minimize this loss function (9).
Figure 4: The computation graph for training the Neural-PI controllers.
We end the paper with case studies demonstrating the effectiveness of the proposed Neural-PI control in two large-scale systems: vehicle platooning and power system frequency control. Detailed problem formulation, verification of assumptions, simulation setting, and results are provided in Appendix B.1 and B.2 in the supplementary material. All experiments are run with an NVIDIA Tesla T4 GPU with 16GB memory. Code is available at this link.
6.1 Vehicle platooning
Experiment setup. We conduct experiments on the vehicle platooning problem in Figure 1(b) with 20 vehicles  $(m = 20)$ , where  $\pmb{u} \in \mathbb{R}^m$  is the control signal to adjust the velocities of vehicles and the output  $\pmb{y} \in \mathbb{R}^m$  is their actual velocities. The state is  $\pmb{x} = (\zeta, \pmb{y})$ , where  $\zeta \in \mathbb{R}^m$  is the relative position of vehicles. The dynamic model is  $\dot{\zeta} = \Gamma \pmb{y}$ ,  $\dot{\pmb{y}} = \hat{\kappa}(-(\pmb{y} - \pmb{\lambda}^0) + \hat{\rho}(\pmb{u} - \pmb{EDE}^\top \pmb{\zeta}))$ , where  $\hat{\kappa}, \hat{\rho}, \hat{D}, E, \Gamma$  are constant matrices with their physical meaning given in Appendix B.1.1. The vector  $\pmb{\lambda}^0 \in \mathbb{R}^m$  reflects the default velocity of vehicles. In Appendix B.1.2, We verify that this system is EIP (i.e., satisfying Assumption 2) using the storage function  $S(x, x^*) = \frac{1}{2}(y - y^*)^\top \hat{\kappa}^{-1} \hat{\rho}^{-1}(y - y^*) + \frac{1}{2} \zeta^\top EDE^\top \zeta$ . The objective is for the vehicles to reach the same setpoint velocity quickly with acceptable control effort. We train for 400 epochs, where each epoch trains with the loss (9) averaged on 300 trajectories, and each trajectory evolves 6s from random initial velocities.
Controller performance. We compare the performance of 1) Neural-PI: the learned structured Neural-PI controllers parametrized by (7) with three layers and 20 neurons in each hidden layer. 2) DenseNN: Dense neural networks (5) the same as Neural-PI with unconstrained weights. Both of them have no communication constraints. 3) Linear-PI: linear PI control where  $\pmb{p}(\bar{\pmb{y}} - \pmb{y}) \coloneqq \pmb{K}_P(\bar{\pmb{y}} - \pmb{y})$ ,  $\pmb{r}(s) \coloneqq \pmb{K}_I(s)$  with  $\pmb{K}_P$  and  $\pmb{K}_I$  being the trainable proportional and integral coefficients. Figure 5(a) shows the transient and steady-state costs on 100 testing trajectories starting from randomly generated initial states. Neural-PI attains much lower costs even though the weights in DenseNN are not constrained. Compared with Linear-PI, Neural-PI achieves similar steady-state cost (since it retains all the stability guarantees of classical linear PI control), but reduces the transient cost by approximately  $30\%$ . Given  $\bar{y} = 5\mathrm{m / s}$ , Figure 5(b) and 5(c) show the dynamics of selected nodes under DenseNN and Neural-PI, respectively. All the outputs track under Neural-PI but they are unstable under DenseNN (even though the training cost was not prohibitively high). In particular, DenseNN appears to be stable until about 10s, but states blows up quickly after that. Therefore, enforcing stabilizing structures is essential.
6.2 Power systems frequency control
Experiment Setup. The second experiment is the power system frequency control on the IEEE 39-bus New England system [52], where  $\pmb{u} \in \mathbb{R}^m$  is the control signal to adjust the power injection from generators and the output  $\pmb{y} \in \mathbb{R}^m$  is the rotating speed (i.e., frequency) of generators. The objective is to stabilize generators at the required frequency  $\bar{\pmb{y}} = 60\mathrm{Hz}$  at the steady state while minimizing the transient control cost. The state is  $\pmb{x} = (\delta, \pmb{y})$ , where  $\delta \in \mathbb{R}^m$  is the rotating angle of
(a) Transient and steady-state cost
(b) Velocities under DenseNN
Figure 5: (a) The average transient cost and steady-state cost with error bar on 100 testing trajectories starting from randomly generated initial states. (b) The dynamics of DenseNN. (c) The dynamics of Neural-PI.
(c) Velocities under Neural-PI
generators. The dynamics of the system is  $\dot{\delta} = \Gamma y$ ,  $\hat{M}\dot{y} = -\hat{D} (y - \bar{y}) - d + u - E\hat{b}\sin (E^{\top}\delta)$ , where  $\hat{M},\hat{D}\hat{b},E,\Gamma$  are constant matrices with their physical meaning given in Appendix B.2.1. The vector  $d$  is the net load of the system. In Appendix B.2.2, We verify that this system is EIP (i.e., satisfying Assumption 2) using the storage function  $S(x,x^{*}) = \frac{1}{2} (y - y^{*})^{\top}\hat{M} (y - y^{*})-$ $\mathbb{1}_e^\top \hat{b} (\cos (E^\top \delta) - \cos (E^\top \delta^*)) - (E\hat{b}\sin (E^\top \delta^*))^\top (\delta -\delta^*)$ ). Below we show the impact of communication constraints on the performance of Neural-PI control. More simulation results can be found in Appendix B.2.3 to demonstrate 1) Neural-PI control is robust to parameter changes, disturbances and noises. 2) Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN.
Impact of communication constraints. Most systems do not have fully connected real-time communication capabilities, so the controller needs to respect the communication constraints and we show the flexibility of Neural-PI control under different communication structures. We compare the performance of Neural-PI controller where 1) all the nodes can communicate 2) half of the nodes can communicate and 3) none of the nodes can communicate (thus the controller is decentralized), respectively. The transient and steady-state costs are illustrated in Figure 6(a). Neural-PI-Full achieves the lowest transient and steady-state cost. Notably, the steady-state cost significantly decreases with increased communication capability. The reason is that communication serves to better allocated control efforts such that they can maintain output tracking with smaller control costs. The frequency dynamics are shown in Figure 6(b)-(d), where under all settings Neural-PI controllers can stabilize to the setpoint  $(60\mathrm{Hz})$  and it converges the fastest under full communication.
(a) Transient and steady cost
(b) Full Comm
(c) Half Comm
(d) Decentralized
Figure 6: (a) The average transient and steady-state cost with error bar on 100 testing trajectories for Neural-PI controller with different communication constraints. The dynamics of Neural-PI where (b) all nodes can communicate, (c) half of nodes can communicate, (d) none nodes can communicate.
This paper proposes structured Neural-PI controllers for multiple-input multiple-output dynamic systems. We parameterize the P and I terms as the gradients of strictly convex neural networks. For a wide range of dynamic systems, we show that this controller structure provides end-to-end stability and zero output tracking error guarantees. Experiments demonstrate that the Neural-PI control law retains all the stability guarantees of classical linear PI control, but achieves much lower transient cost. Unstructured neural networks, however, lead to unstable behavior and much higher costs. The theoretic guarantees of Neural-PI control also significantly reduce the amount of data required to train well. Since classical PI control is widely utilized in real-world applications, we expect that the controllers can be transferred to real-world scenarios. Potential barriers to the application in real-world scenarios include the verification of EIP when a storage function is difficult to find and provable guarantees on the robustness to noises. These are all important future directions for us.
The authors are partially supported by the NSF grants ECCS-1930605, 1942326, 2200692, and 2153937.