Title: Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees
Venue: NeurIPS 2023
Accepted: yes
Keywords: Control

Stability

Tracking

Passivity

Neural network-based controllers

Power systems
Abstract: We study the optimal control of multiple-input and multiple-output dynamical systems via the design of neural network-based controllers with  stability and output tracking guarantees. While neural network-based nonlinear controllers have shown superior performance in various applications, their lack of provable guarantees has restricted their adoption in high-stake real-world applications. This paper bridges the gap between neural network-based controllers and the need for stabilization guarantees. Using equilibrium-independent passivity, a property present in a wide range of physical systems, we propose neural Proportional-Integral (PI) controllers that have provable guarantees of stability and zero steady-state output tracking error. The key structure is the strict monotonicity on proportional and integral terms, which is parameterized as gradients of strictly convex neural networks (SCNN). We construct SCNN with tunable softplus-$\beta$ activations, which yields universal approximation capability and is also useful in incorporating communication constraints. In addition, the SCNNs serve as Lyapunov functions, giving us end-to-end performance guarantees.  Experiments on traffic and power networks demonstrate that the proposed approach improves both transient and steady-state performances, while unstructured neural networks lead to unstable behaviors.

Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees

Abstract

We study the optimal control of multiple-input and multiple-output dynamical systems via the design of neural network-based controllers with stability and output tracking guarantees. While neural network-based nonlinear controllers have shown superior performance in various applications, their lack of provable guarantees has restricted their adoption in high-stake real-world applications. This paper bridges the gap between neural network-based controllers and the need for stabilization guarantees. Using equilibrium-independent passivity, a property present in a wide range of physical systems, we propose neural Proportional-Integral (PI) controllers that have provable guarantees of stability and zero steady-state output tracking error. The key structure is the strict monotonicity on proportional and integral terms, which is parameterized as gradients of strictly convex neural networks (SCNN). We construct SCNN with tunable softplusβ activations, which yields universal approximation capability and is also useful in incorporating communication constraints. In addition, the SCNNs serve as Lyapunov functions, providing end-to-end stability and tracking guarantees. Experiments on traffic and power networks demonstrate that the proposed approach improves both transient and steady-state performances, while unstructured neural networks lead to unstable behaviors.

1 Introduction

Learning-based methods have the potential to solve difficult problems in control and have received significant attention from both the machine learning and control communities. A common problem in many applications is to design controllers that-with as little control effort as possible-stabilize a system at a prescribed output. A canonical example is the LQR problem and its variants, which finds optimal linear controllers for linear systems [1, 2].

For nonlinear systems, the problem is considerably harder. Neither of the two control goals, system stabilization and output tracking at the steady state, is easy to achieve. Their combination, optimizing controller costs while guaranteeing stability and output tracking at the steady state, is even more challenging. For example, vehicles in a platoon should stay in formation (stability) while cruising at the desired speed (output tracking) [3]. The optimization is then done over the set of all controllers that achieves both of these goals, e.g., reaching a platoon while consuming the least amount of fuel. Another pertinent application is the stability of electric grids with high amounts of renewables. Unlike systems with conventional generators, the power electronic interfaces of the inverters need to be actively controlled such that the grid synchronizes to the same frequency (output tracking), at the minimum operational costs. Currently, learning becomes a popular tool to parameterize nonlinear controllers as neural networks and train them to optimize performance, but provable guarantees on stability and steady-state error for these controllers are nontrivial to obtain [4-7].

Many real-world applications have multiple (possibly a continuous set of) equilibria that may be of interest. For example, a group of vehicles may need to cruise at different speeds. Moreover, the

internal states may not be directly accessible and sometimes there are communication limit. These constraints make it difficult to enforce stability by middle steps including learning a Lyapunov function and learning a model of the system (for details, see related work in Section 1.1). Therefore, we seek to achieve 'end-to-end' guarantees: the neural network-based controller should guarantee stability and steady-state error by constructions , for a range of possible tracking points. There are some works that showed a class of monotone controllers can provide stability guarantees for varying equilibria [8-10], but they rely on tailor-made Lyapunov functions that are found in specific applications. Especially, these results are limited to single-input and single-output (SISO) control. In practice, however, lots of complex systems need controllers that are multiple-input and multiple-output (MIMO), sometimes with high input and output dimensions.

Moreover, current learning-based approaches typically focus on optimizing transient performance (the cost and speed at which a system reaches an equilibrium), often overlooking the steady-state behavior (cost at the equilibrium state) [11-13]. In classical control terminology, these controller is proportional. Steady-state requirements are difficult to incorporate since training can only occur over a finite horizon. To enforce output tracking at the steady-state, an integral term is commonly needed to drive the system outputs to the desired value at the steady state [14-17]. This reasoning underlies the widespread use of linear Proportional-Integral (PI) controllers in practical applications [18-20]. However, linear parameterization inherently limits the controllers' degrees of freedom, potentially leading to suboptimal performance. This work addresses the following question: Can we learn nonlinear controllers that guarantee transient stability and zero steady-state error for MIMO systems?

Contributions. Clearly, it is not possible to design a controller for all nonlinear systems and the answer to the question above depends on the class of systems under consideration. This paper focuses on systems that satisfy the equilibrium independent passivity (EIP) [21, 22], which is present in many critical societal-scale systems including transportation [23], power systems [24], and communication network [25]. This abstraction allows us to design generalized controllers without considering the detailed physical system dynamics. We propose a structured Neural-PI controller that has provable guarantees of stability and zero steady-state output tracking error. The key structure we use is strictly monotone functions with vector-valued inputs and outputs. Experiments on traffic and power systems demonstrate that Neural-PI control can reduce the transient cost by at least 30% compared to the best linear controllers and maintain stability guarantees. Unstructured neural networks, on the other hand, lead to unstable behaviors. We summarize our major contributions as follows.

We propose a generalized PI structure for neural network-based controllers in MIMO systems, with proportional and integral terms being strictly monotone functions constructed through the gradient of strictly convex neural networks (SCNN). We construct SCNN with a tunable softplusβ activation function and prove their universal approximation capability in Theorem 1.

For a multi-agent system with an underlying communication graph, we show how to restrict the controllers to respect the communication constraints through the composition of SCNNs.

Using EIP and the monotone functions structured by SCNNs, we design a generalized Lyapunov function that works for a range of equilibria. This provides end-to-end guarantees on asymptotic stability and zero steady-state output tracking error proved in Theorem 2. The structured neural networks can be trained for transient optimization without jeopardizing the guarantees, establishing a framework for coordinating transient optimization and steady-state requirement.

1.1 Related works

Learning-based control with stability guarantees. Recently, there has been a large interest in enforcing stability in learning-based controller [26-30]. Many works add soft penalties on the violation of stability conditions in the cost function, but it is nontrivial to certify stability for all the possible initial states in a compact set [30]. Some recent works learn a Lyapunov function and use it to certify stability through a satisfiability modulo theories (SMT) solver [26-29]. But this approach is difficult to scale to high-dimensional systems and the learned Lyapunov function only works for a single equilibrium. For control-affine systems, the work in [31] designs feedback linearizing policy with integral control to guarantee stabilizing to a range of equilibria. But it requires that all the state are accessible and many systems are not control affine. Our proposed controller guarantees stability for a set of equilibria by construction , and only needs access to the outputs (not the full states).

Optimizing long-term behavior. To regulate long-term behavior, existing works train neural networks using a cost function defined over a long time horizon [32, 33]. However, it is difficult

to quantify how long the trajectory is enough to reach the steady state, thus enforcing steady-state tracking performance by adding a long-term cost is challenging. Our proposed controller enforces steady-state tracking by construction , instead of relying on training.

Algorithm to tune MIMO PI controller. Classical Proportional Integral (PI) control structures are widely used in industrial applications, while tuning PI controller parameters in MIMO systems is tedious and relies on heuristic rules [34, 35]. Learning-based methods become popular to tune PI parameters, although still restricted to linear control gains [34, 35]. Our contribution is the more generalized PI control and the MIMO neural network for parameterization.

Monotone neural network. Even though scalar-valued monotone functions have been studied in the learning community [36, 37], their generalization to the vector-valued case has not. Numerous papers studied vector-valued functions that are monotone in every input, that is, q ( x ) ≤ q ( x ′ ) if x ≤ x ′ [38-41]. This is only a small subclass of vector-valued monotone functions that defined as functions satisfying ( q ( x ) -q ( x ′ )) ⊤ ( x -x ′ ) ≥ 0 . In this paper, we construct the general class of monotone functions (often called monotone operators) [42] that satisfy the inner product inequality.

2 Background and Preliminaries

We consider a dynamic system described by:

with state x ( t ) ∈ R n , control action u ( t ) ∈ R m , and output y ( t ) ∈ R v . We also sometimes omit the time index t for simplicity 1 . In many practical applications, not all of the internal states are directly accessible. Hence, we consider control actions that follow output-feedback control laws of the form u = g ( y ) , which is a static function of the output y and not the state x .

A state x ∗ such that f ( x ∗ , u ∗ ) = 0 n , y ∗ = h ( x ∗ ) , u ∗ = g ( y ∗ ) is called an equilibrium since the system stops changing at this state. Throughout this paper, the superscript ∗ indicates the equilibrium values of variables.

Definition 1 (Local asymptotic stability [16], Definition 4.1) . The system (1) is asymptotically stable around an equilibrium x ∗ if, ∀ ϵ > 0 , ∃ δ > 0 such that ∥ x (0) -x ∗ ∥ < δ ensures ∥ x ( t ) -x ∗ ∥ < ϵ , ∀ t ≥ 0 , and ∃ δ ′ > 0 such that ∥ x (0) -x ∗ ∥ < δ ′ ensures lim t →∞ ∥ x ( t ) -x ∗ ∥ = 0 .

One of the main tools to prove asymptotic stability is the Lyapunov's direct method [27, 16]:

Lemma 1 (Lyapunov functions and asymptotic stability [16], Theorem 4.1) . Consider the system (1) with an equilibrium x ∗ ∈ X ⊂ R n . Suppose there exists a continuously differentiable function V : X ↦→ R that satisfies the following conditions: 1) V ( x ∗ ) = 0 , V ( x ) > 0 , ∀ x ∈ X\ x ∗ ; 2) ˙ V ( x ) = ( ∇ x V ( x )) ⊤ ˙ x ≤ 0 , ∀ x ∈ X with the equality holds when x = x ∗ . Then the system is asymptotically stable around x ∗ .

The key challenge to using Lyapunov theory is in constructing such a function and verifying the satisfaction of the conditions in Lemma 1. An important part of this paper is to show how to systematically use neural networks to construct Lyapunov functions for a class of nonlinear systems.

Besides stability, we are often interested in achieving a specific equilibrium such that the output converges to a prescribed setpoint.

Definition 2 (Output tracking to ¯ y ) . The dynamical system (1) is said to track a setpoint ¯ y if lim t →∞ y ( t ) = ¯ y .

For output tracking, the dimension of the input u should be at least the dimension of the output y , otherwise there is not enough degrees of freedom in the input to track all the outputs. If the dimensions of the input are strictly larger than the output, it is always possible to define 'dummy' outputs (e.g., by appending zeros) to match the input dimension. Therefore, it is common to assume the same input and output dimensions [22]. In the remainder of this paper, we make the following assumption.

Assumption 1 (Identical input and output dimension) . The input u and the output y have the same dimension, namely, u ( t ) and y ( t ) are both vectors in R m .

(a) Control Goals

(b) Physical Systems

Transient

SteadyState

y

Target

Stable (Def.1)

position

&

y1

y2

y3

Tracking(Def.2)

IDisturbances

Time

Exp1:Invertedpendulum

Exp2:Avehicleplatoon

Transient

Guaranteed

Optimization

by Design

End-to-End Stability Output Tracking

Physics (EIP)

(d) Training

Neural

(c) Structured Neural Pl Controller Construction

Network

StabilizingStructures

Strictly

Parameterize

Strictly

convex

Proportional

Trained

NN

Vzg(z; 0)

functions

monotone

g(z; 0)

(Def. 3)

Integral

We will show that strictly monotone functions play an important role in guaranteeing stability and output tracking in a range of systems. Using neural networks for constructing monotone functions from R to R has been well-studied in [36, 37]. However, since we consider MIMO systems in this paper, we use a generalized notion of monotonicity for vector-valued functions as defined in [42].

Definition 3 (Strictly monotone function [42]) . A continuous function q : R m ↦→ R m is strictly monotone on D ⊂ R m if ( q ( η ) -q ( ξ )) ⊤ ( η -ξ ) ≥ 0 , ∀ η , ξ ∈ D , with the equality holds if and only if η = ξ .

In this paper, we will show a way to construct (strictly) monotone functions using the gradient of (strictly) convex neural networks.

3 Problem Statement

3.1 Transient and steady-state requirements

We aim to optimize the controller in u to improve the transient response after disturbances while guaranteeing steady-state performance, as shown in Figure 1(a). By steady-state performance, we mean the system should settle at the desired setpoint ¯ y . By transient performance, we want the system to quickly reaching the steady state with small control efforts.

The inverted pendulum in Figure 1(b) serves as a good motivating example. The input u is the force on the pendulum. The objectives include 1) the angle y should reach a setpoint ¯ y (e.g., 5 ◦ ) and stays there; 2) minimizing the control cost and deviation of y from ¯ y before reaching the steady state. These objectives are common in many real-world applications, such as vehicle platooning (Fig. 1(b)) and power system control in our experiments.

Transient performance optimization. During the transient period, our goal is to quickly stabilize the system to the desired setpoint ¯ y , while trading off with using large control efforts in u . Let J ( · ) be the cost function of the system. 2 The transient optimization problem up to time T is,

Note that in (2a) we sum up to time T , which should be long enough to cover the transient period. Since we require tracking in (2c), the exact value of T is not critical in the optimization problem. In

practice, the system dynamics (1) (and sometimes even the cost function) can be highly nonlinear and nonconvex. Therefore, the current state-of-the-art is to learn u by parameterizing it as a neural network and training it by minimizing the cost in (2a) [43, 44]. But the key challenge with applying these neural network-based controllers is guaranteeing stability and output tracking [45, 26, 30]. Even if the learned policy may appear 'stable' during training, it is not necessarily stable during testing. This can be observed in both the vehicle and power system experiments in Section 6.

Stability for a range of tracking points. We emphasize that the setpoint ¯ y may vary in practice (e.g., the setpoint velocity of vehicles may change), and thus the controller is required to guarantee stability and output tracking to a range of equilibria corresponding to the setpoints. This is difficult to achieve through existing works that enforce stability by learning a Lyapunov function [26, 27, 30], since the learned Lyapunov function is for a single equilibrium (normally setup as x ∗ = 0 n ). These methods also require that all the states x are observed, which may not be achieved in practice.

Communication requirement. A typical constraint in a large system is limits on communications. For example, vehicles in a group may only be able to measure the output of their neighbors (line-ofsight) or nodes in a power system may only have real-time communication from a subset of other nodes. Therefore, the control action u i may be constrained to be a function of a subset of the outputs y . We show later in Subsection 4.3 about how these communication constraints can be naturally accommodated in our controller design.

3.2 Bridging controller design and stability through passivity analysis

As shown in Figure 1(a), optimizing transient and steady-state performances are two problems in two different time-scales. Therefore, coordinating them is a central challenge.

End-to-end performance guarantees. We propose to overcome this challenge through a structured controller design that provides end-to-end stability and tracking guarantees, as shown in Figure 1(c-d). "End-to-end" means that the guarantees are provided by construction, and do not depend on how the controller is trained. Thus, the neural networks can be trained to optimize the transient performance without jeopardizing the stability and steady-state guarantees. In particular, the construction works for a range of equilibria and can conveniently incorporate communication characteristics.

Instead of working on specific systems individually, we seek to find a family of physical systems that allows us to derive a generalized controller design. It turns out that the notion of equilibrium independent passivity (EIP) [21, 22] provides a concise and useful abstraction of physical systems for stability analysis. This paper thus focuses on systems satisfying the following assumptions:

Assumption 2 (Equilibrium-Independent Passivity [21] ) . The system (1) is strictly equilibriumindependent passive (EIP), which satisfies: (i) there exists a nonempty set U ∗ ⊂ R m such that for every equilibrium u ∗ ∈ U ∗ , there is a unique x ∗ ∈ R n such that f ( x ∗ , u ∗ ) = 0 n , and (ii) there exists a positive definite storage function S ( x , x ∗ ) and a positive scalar ρ such that

The EIP property in Assumption 2 has been found in a large class of physical systems, including transportation [23], power systems [24, 46], robotics [47], communication [25], and others. We will conduct experiments on vehicle platoons and power systems to show how EIP can be validated.

The condition (3) of EIP systems provides us a generalized approach to construct Lyapunov functions without knowing the specifies of the dynamics f ( · ) . In turn, we are able to find the right structure for the controllers. In Section 4, we show what these structures are for PI controllers and how to enforce such structures in the design of neural networks. Then Section 5 formally demonstrates the theoretical guarantees and the training procedure for transient optimization.

4 Structured Neural-PI Control

In this section, we construct controllers with a proportional and an integral term, which are both vector-valued strictly monotone functions parameterized by the gradient of strictly convex functions. Then, we present a neural network architecture that is strictly convex by construction and can conveniently incorporate communication constraints.

4.1 Generalized PI control

To realize output tracking, we introduce an integral variable ˙ s = ¯ y -y . Intuitively, s is the accumulated tracking error and will remain unchanged at the steady-state when y = ¯ y . On this basis, we consider a generalized PI controller u = p (¯ y -y )+ r ( s ) . The first component is the proportional term, where p (¯ y -y ) is a function of the tracking error ¯ y -y between the current output y and desired output ¯ y . The second component is the integral term r ( s ) as a function of the integral of historical tracking errors. Intuitively, the proportional term drives y close to ¯ y , and the integral term ensures the tracking error equals zero at the steady state.

Generalized PI Controller. Compactly, the control law is

Remark 1. The above controller can be envisioned as a nonlinear generalization of widely adopted linear Proportional-Integral (PI) controllers, where p (¯ y -y ) := K P ( ¯ y -y ) , r ( s ) := K I ( s ) with K P and K I being the proportional and integral coefficients (scalar for SISO control). Linear PI control laws are almost always used in existing works [14-16], but the performance of linear PI controllers can be poor for large-scale nonlinear systems[6, 48].

To achieve provable stability guarantees with output tracking, we need to construct a Lyapunov function that works for the closed-loop system formed by (1) and (4). Therefore, we further design structures in the proportional function p ( · ) and the integral function r ( · ) that can be utilized in Lyapunov stability analysis. The structures are strictly monotone functions, which generalizes conventional linear functions. In the next subsection, we will show how to parameterize strictly monotone functions. In section 5.1, we will show how the PI controllers structured with monotone functions provide end-to-end stability and output tracking guarantees.

4.2 Strictly monotone function through gradients of strictly convex neural networks

It is not trivial to parameterize strictly monotone functions since this is an infinite-dimensional function space. Scalar-valued monotone functions mapping from R to R [36, 37] has been proposed, but it is difficult to extend these designs to MIMO systems.

In this paper, we propose a new parameterization of strictly monotone functions by leveraging the fact in Proposition 1 that the gradient of a strictly convex function is strictly monotone.

Proposition 1 (Gradients of strictly convex functions) . Let g : R m ↦→ R be a strictly convex function, then ( ∇ η g ( η ) - ∇ ξ g ( ξ )) ⊤ ( η -ξ ) ≥ 0 ∀ η , ξ ∈ R m , with equality holds if and only if η = ξ . Namely, ∇ g : R m ↦→ R m is a strictly monotone increasing function.

Hence, the strictly monotone property of p ( · ) and r ( · ) can be guaranteed by design if they are the gradient of a strictly convex function, as shown by Figure 2. The next proposition shows how to construct a strictly convex neural network (SCNN).

Proposition 2

(Strictly convex neural networks)

Consider a function

.

(

z

;

θ

)

:

R

m

rameterized by

k

g

R

pa-

-layer, fully connected neural network with the input z ∈ R m . The output o l of layer l = 0 , . . . , k -1 and the function g ( z ; θ ) is represented as

where o 0 , W ( o ) 0 ≡ 0 , θ = { W ( z ) 0: k -1 , W ( o ) 1: k -1 , b 0: k -1 } are trainable weights and biases, and σ i are non-linear activation functions. The function g ( z ; θ ) is strictly convex in z provided that all W ( o ) 1: k -1 are positive, W ( z ) 0 is nonzero, and all functions σ l are strictly convex and increasing.

(a) Full

(b) Decentralized

(c)Partial

Communication

Communication

Z

2

v;EV

Z

g(P) (ZD; 0j,β)

Zv1

Z

SCNN

4

Z

asla

Z

g(P)(2;0,β;)

Z

Z

g(P) (Z; 0j,β)

Zvi

SCNN

6

Zv2

Zv3

SCNN

+

g

ZV1

SCNN

+

g

Z

SCNN

Zv2

SCNN

SCNN

The construction of SCNN follows the general structure of input-convex neural networks [49, 50], where the conditions on activation functions and weights are modified to achieve strictly convexity. The proof follows from the fact that positive sums of strictly convex functions are also strictly convex and that the composition of a strictly convex and convex increasing function is also strictly convex.

The next theorem demonstrates the universal approximation property of SCNN in Proposition 2.

Theorem 1 (Universal approximation ) . Let Z be a compact domain in R m and q ( z ) : Z ↦→ R be a Lipschitz continuous and strictly convex function. For any ϵ > 0 , there exists a function g ( z ; θ ) : Z ↦→ R constructed by (5) where the activation function is the softplusβ function

such that | q ( z ) -g ( z ; θ ) | < ϵ for all z ∈ Z .

The proof is given in Appendix A.1. We sketch the proof as follows. We leverage the results in [49, 50] that the structure (5) with ReLU activation is a universal approximation for any convex function. However, ReLU activations are not strictly convex, and thus we design the Softplusβ activation. By showing that the structure (5) with Softplusβ activation can approximate neural networks with the ReLU activations arbitrarily closely when β is sufficiently large, we then prove that the structure (5) with Softplusβ can universally approximate any strictly convex function.

Notably, β in the activation function is a tunable parameter. Hence, we write down the SCNN (5) with activation function being Softplusβ function in (6) as g ( z ; θ , β ) , where β is an extra trainable parameter.

Thus, we parameterize the structured Neural-PI control law u = p (¯ y -y ) + r ( s ) in (4) as

This way, p ( · ) and r ( · ) by construction are strictly monotone. In particular, the convex functions g ( P ) ( z ; θ ( P ) , β ( P ) ) , g ( I ) ( z ; θ ( I ) , β ( I ) ) play a vital role in constructing a generalized Lyapunov function and showing the satisfaction of the Lyapunov condition in Subsection 5.1. This subsequently provides end-to-end stability guarantees that do not dependent on the training algorithm.

4.3 Embedding Communication Constraints

For some large-scale physical systems, not all of the control inputs have access to all of the output measurements at real-time. Therefore, some u i 's cannot be a function of the full vector y and s because of this lack of global communication, as elaborated in Subsection 3.1.

Constructing the controllers as the gradient of convex functions provides us with a convenient way to embed these communication constraints. Let V be the set of indexes with communications. Take m = 4 where z = ( z 1 , z 2 , z 3 , z 4 ) and the proportional term p = ∇ z g ( P ) ( z ; θ ( P ) , β ( P ) ) as an example. Figure 3(a) shows the case with global communication, where V = { (1 , 2 , 3 , 4) } and p can be a function of the full z . Figure 3(b) shows the case without communication where

V = { (1) , · · · , (4) } and p i can only be a function of z i for all i . Figure 3(c) shows the case V = { (1 , 2) , (2 , 3 , 4) } , where there exists communication within the indexes (1 , 2) and within (2 , 3 , 4) . By defining SCNN g ( z v j ; θ ( P ) j , β ( P ) j ) for each group in v j ∈ V , the gradient ∇ z g ( z v j ; θ ( P ) j , β ( P ) j ) will only be a function of z v j and thus satisfying the commutation constraints. Therefore, we construct the functions in (7) as

where θ ( P ) := { θ ( P ) 1 , · · · , θ ( P ) |V| } , β ( P ) := { β ( P ) 1 , · · · , β ( P ) |V| } , and θ ( P ) j , θ ( I ) j are parameters of the SCNNs for group v j within the communication network.

The function g ( I ) ( z ; θ ( I ) , β ( I ) ) is also constructed in a similar way as g ( P ) ( z ; θ ( P ) , β ( P ) ) in (8) to incorporate the communication constraints in V . Strict convexity still holds since a sum of strictly convex functions is also strictly convex.

5 Training with End-to-End Guarantees

5.1 End-to-end stability and output-tracking guarantees

The convex function g ( I ) ( s ; θ ( I ) , β ( I ) ) constructed from SCNNs can be utilized to construct a Lyapunov function for proving stability using the Bregman distance defined in the following Lemma [51]: Lemma 2 (Bregman distance) . The Bregman distance associated with the convex function g ( I ) ( s ; θ ( I ) , β ( I ) ) is defined by

The Bregman distance allows us to construct Lyapunov functions without specifying the equilibrium s ∗ . Combining the storage function in Assumption 2 and B ( s , s ∗ ; θ ( I ) , β ( I ) ) in Lemma 2, next theorem shows that the Neural-PI controller stabilizes the system to the desired output ¯ y .

Theorem 2. Suppose Assumption 2 holds and the input u follows the structured PI control law u = p (¯ y -y ) + r ( s ) where p ( · ) and r ( · ) are constructed as the gradients of strictly convex neural networks in (7) . If the system (1) has a feasible equilibrium, then the system is locally asymptotically stable around it. In particular, the steady-state outputs track the setpoint ¯ y , namely y ∗ = ¯ y .

Theorem 2 shows that Neural-PI control has provable guarantees on stability and zero steady-state output tracking error from the structures in (7). Detailed proof could be found in Appendix A.2 and we sketch the proof below. At an equilibrium, the right side of (4b) equals to zero gives y ∗ = ¯ y . We show that an equilibrium is asymptotically stable by constructing a Lyapunov function V ( x , s ) | x ∗ , s ∗ = S ( x , x ∗ ) + B ( s , s ∗ ; θ ( I ) , β ( I ) ) . Since r ( s ) = ∇ s g ( I ) ( s ; θ ( I ) , β ( I ) ) by construction in (7), the time derivative of the Bregman distance term is ˙ B ( s , s ∗ ; θ ( I ) , β ( I ) ) = ( r ( s ) -r ( s ∗ )) ⊤ ( -( y -y ∗ )) . Combining with the fact ˙ S ( x , x ∗ ) ≤ -ρ ∥ y -y ∗ ∥ 2 +( y -y ∗ ) ⊤ ( u -u ∗ ) (from the EIP assumption), and u = p (¯ y -y ) + r ( s ) , we can conclude that the Lyapunov stability condition holds.

Remark 2. The satisfaction of the Lyapunov condition does not depend on the specifics of f ( · ) (as long as it is EIP), making the stability certification robust to parameter changes for systems satisfying Assumption 2. We will demonstrate this in the experiment on power system control.

5.2 Training to improve transient performances

The Neural-PI controller can be trained by most model-based or model-free algorithms, since the stability and output-tracking are guaranteed by construction. Fig 4 visualizes the detailed construction and the computation graph in the dynamical system in (1). The trainable parameters Θ := { θ ( P ) , β ( P ) , θ ( I ) , β ( I ) } are contained in p ( · ) and r ( · ) functions, where both are parameterized as the gradients of SCNNs. u = ∇ ¯ y -y g ( P ) (¯ y -y ; θ ( P ) , β ( P ) ) + ∇ s g ( I ) ( s ; θ ( I ) , β ( I ) ) serves as control signal in the system defined in (1) that evolves through time. The loss function is defined as

where J ( · ) is the transient cost function defined in (2a). The parameters Θ are optimized via gradient descent to minimize this loss function (9).

-y+y

p()network

p(-y+y)

s(0)defined

s(1)defined

s(T)defined

via(4b)

via(4b)

via(4b)

SCNN

6

e

+

z

SCNN

y(0)

y(1)

y(T)

p()

r()

p()

r()

P()

r()

network

network

network

network

network

network

S

r() r

network

r(s)

+

+

SCNN

6

a

u(0)

u(1)

u(T)

+

dz

SCNN

System(1)attime stept=0

System(1)attime stept=1

System(1)attimestept=T

6 Experiments

We end the paper with case studies demonstrating the effectiveness of the proposed Neural-PI control in two large-scale systems: vehicle platooning and power system frequency control. Detailed problem formulation, verification of assumptions, simulation setting, and results are provided in Appendix B.1 and B.2 in the supplementary material. All experiments are run with an NVIDIA Tesla T4 GPU with 16GB memory. Code is available at this link.

6.1 Vehicle platooning

Experiment setup. We conduct experiments on the vehicle platooning problem in Figure 1(b) with 20 vehicles ( m = 20 ), where u ∈ R m is the control signal to adjust the velocities of vehicles and the output y ∈ R m is their actual velocities. The state is x = ( ζ , y ) , where ζ ∈ R m is the relative position of vehicles. The dynamic model is ˙ ζ = Γ y , ˙ y = ˆ κ ( -( y -λ 0 ) + ˆ ρ ( u -E ˆ DE ⊤ ζ )) , where ˆ κ , ˆ ρ , ˆ D , E , Γ are constant matrices with their physical meaning given in Appendix B.1.1. The vector λ 0 ∈ R m reflects the default velocity of vehicles. In Appendix B.1.2, We verify that this system is EIP (i.e., satisfying Assumption 2) using the storage function S ( x , x ∗ ) = 1 2 ( y -y ∗ ) ⊤ ˆ κ -1 ˆ ρ -1 ( y -y ∗ ) + 1 2 ζ ⊤ E ˆ DE ⊤ ζ . The objective is for the vehicles to reach the same setpoint velocity quickly with acceptable control effort. We train for 400 epochs, where each epoch trains with the loss (9) averaged on 300 trajectories, and each trajectory evolves 6s from random initial velocities.

Controller performance. We compare the performance of 1) Neural-PI: the learned structured Neural-PI controllers parametrized by (7) with three layers and 20 neurons in each hidden layer. 2) DenseNN: Dense neural networks (5) the same as Neural-PI with unconstrained weights. Both of them have no communication constriants. 3) Linear-PI: linear PI control where p (¯ y -y ) := K P (¯ y -y ) , r ( s ) := K I ( s ) with K P and K I being the trainable proportional and integral coefficients. Figure 5(a) shows the transient and steady-state costs on 100 testing trajectories starting from randomly generated initial states. Neural-PI attains much lower costs even though the weights in DenseNN are not constrained. Compared with Linear-PI, Neural-PI achieves similar steady-state cost (since it retains all the stability guarantees of classical linear PI control), but reduces the transient cost by approximately 30%. Given ¯ y = 5 m/s, Figure 5(b) and 5(c) show the dynamics of selected nodes under DenseNN and Neural-PI, respectively. All the outputs track under Neural-PI but they are unstable under DenseNN (even though the training cost was not prohibitively high). In particular, DenseNN appears to be stable until about 10s, but states blows up quickly after that. Therefore, enforcing stabilizing structures is essential.

6.2 Power systems frequency control

Experiment Setup. The second experiment is the power system frequency control on the IEEE 39 -bus New England system [52], where u ∈ R m is the control signal to adjust the power injection from generators and the output y ∈ R m is the rotating speed (i.e., frequency) of generators. The objective is to stabilize generators at the required frequency ¯ y = 60 Hz at the steady state while minimizing the transient control cost. The state is x = ( δ , y ) , where δ ∈ R m is the rotating angle of

(a) Transient and steady-state cost

(b) Velocities under DenseNN

(c) Velocities under Neural-PI

Cost

transient

1.5

steady

6

6

Normallized

>

>

1.0

speed

5

speed

5

4

node2

node9

node2

node9

0.5

S

node3

node10

S

4

node3

node10

3

node4

node15

node4

node15

node5

node18

3

node5

node18

0.0

0

5

10

15

0

5

10

15

Neural-PI

Linear-PI

DenseNN

time(s)

time(s)

generators. The dynamics of the system is ˙ δ = Γ y , ˆ M ˙ y = -ˆ D ( y -¯ y ) -d + u -E ˆ b sin( E ⊤ δ ) , where ˆ M , ˆ D ˆ b , E , Γ are constant matrices with their physical meaning given in Appendix B.2.1. The vector d is the net load of the system. In Appendix B.2.2, We verify that this system is EIP (i.e., satisfying Assumption 2) using the storage function S ( x , x ∗ ) = 1 2 ( y -y ∗ ) ⊤ ˆ M ( y -y ∗ ) -1 ⊤ e ˆ b (cos( E ⊤ δ ) -cos( E ⊤ δ ∗ )) -( E ˆ b sin( E ⊤ δ ∗ )) ⊤ ( δ -δ ∗ )) . Below we show the impact of communication constraints on the performance of Neural-PI control. More simulation results can be found in Appendix B.2.3 to demonstrate 1) Neural-PI control is robust to parameter changes, disturbances and noises. 2) Neural-PI significantly reduces the number of sampled trajectories to train well compared with DenseNN.

Impact of communication constraints. Most systems do not have fully connected real-time communication capabilities, so the controller needs to respect the communication constraints and we show the flexibility of Neural-PI control under different communication structures. We compare the performance of Neural-PI controller where 1) all the nodes can communicate 2) half of the nodes can communicate and 3) none of the nodes can communicate (thus the controller is decentralized), respectively. The transient and steady-state costs are illustrated in Figure 6(a). Neural-PI-Full achieves the lowest transient and steady-state cost. Notably, the steady-state cost significantly decreases with increased communication capability. The reason is that communication serves to better allocated control efforts such that they can maintain output tracking with smaller control costs. The frequency dynamics are shown in Figure 6(b)-(d), where under all settings Neural-PI controllers can stabilize to the setpoint (60Hz) and it converges the fastest under full communication.

(a) Transient and steady cost

(b) Full Comm

(c) Half Comm

(d) Decentralized

Cos

0.3

transient

60.00

60.00

60.00

Normallized

0.2

steady

Frequency

Frequency

Frequency

node3

59.98

59.98

59.98

node4

node5

0.1

node6

59.96

59.96

59.96

node7

node8

0.0

nodeg

59.94

59.94

59.94

Neural-Pl-Full

Neural-PI-Half

0

2

0

2

0

2

Neural-Pl-Decenralized

time(s)

time(s)

time(s)

7 Conclusions

This paper proposes structured Neural-PI controllers for multiple-input multiple-output dynamic systems. We parameterize the P and I terms as the gradients of strictly convex neural networks. For a wide range of dynamic systems, we show that this controller structure provides end-to-end stability and zero output tracking error guarantees. Experiments demonstrate that the Neural-PI control law retains all the stability guarantees of classical linear PI control, but achieves much lower transient cost. Unstructured neural networks, however, lead to unstable behavior and much higher costs. The theoretic guarantees of Neural-PI control also significantly reduce the amount of data required to train well. Since classical PI control is widely utilized in real-world applications, we expect that the controllers can be transferred to real-world scenarios. Potential barriers to the application in real-world scenarios include the verification of EIP when a storage function is difficult to find and provable guarantees on the robustness to noises. These are all important future directions for us.

Acknowledgments and Disclosure of Funding

The authors are partially supported by the NSF grants ECCS-1930605, 1942326, 2200692, and 2153937.

References

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467-1476. PMLR, 2018.

Yingying Li, Tianpeng Zhang, Subhro Das, Jeff Shamma, and Na Li. Safe adaptive learning for linear quadratic regulators with constraints. Technical report, Technical report, 2023.

Samuel Coogan and Murat Arcak. A dissipativity approach to safety verification for interconnected systems. IEEE Transactions on Automatic Control , 60(6):1722-1727, 2014.

Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control , pages 321-384, 2021.

Guannan Qu, Yiheng Lin, Adam Wierman, and Na Li. Scalable multi-agent reinforcement learning for networked systems with average reward. Advances in Neural Information Processing Systems , 33:2074-2086, 2020.

Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. Advances in Neural Information Processing Systems , 34:3271-3284, 2021.

Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. In International Conference on Learning Representations , 2019.

Wenqi Cui, Yan Jiang, and Baosen Zhang. Reinforcement learning for optimal primary frequency control: A Lyapunov approach. IEEE Transactions on Power Systems , 2022.

Yuanyuan Shi, Guannan Qu, Steven Low, Anima Anandkumar, and Adam Wierman. Stability constrained reinforcement learning for real-time voltage control. American Control Conference (ACC) , 2022.

Jie Feng, Yuanyuan Shi, Guannan Qu, Steven H Low, Anima Anandkumar, and Adam Wierman. Stability constrained reinforcement learning for real-time voltage control in distribution systems. IEEE Transactions on Control of Network Systems , 2023.

Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics , 2023.

Guannan Qu, Yuanyuan Shi, Sahin Lale, Anima Anandkumar, and Adam Wierman. Stable online control of linear time-varying systems. In Learning for Dynamics and Control , pages 742-753. PMLR, 2021.

Christian Fiedler, Carsten W Scherer, and Sebastian Trimpe. Learning-enhanced robust controller synthesis with rigorous statistical and control-theoretic guarantees. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 5122-5129. IEEE, 2021.

Changhong Zhao, Enrique Mallada, and Florian Dörfler. Distributed frequency control for stability and economic dispatch in power networks. In Proc. of American Control Conference , pages 2359-2364, July 2015. doi: 10.1109/ACC.2015.7171085.

Martin Andreasson, Dimos V Dimarogonas, Henrik Sandberg, and Karl Henrik Johansson. Distributed control of networked dynamical systems: Static feedback, integral action and consensus. IEEE Transactions on Automatic Control , 59(7):1750-1764, 2014.

Hassan K Khalil. Nonlinear control , volume 406. Pearson New York, 2015.

Yan Jiang, Wenqi Cui, Baosen Zhang, and Jorge Cortés. Stable reinforcement learning for optimal frequency control: A distributed averaging-based integral approach. IEEE Open Journal of Control Systems , 1:194-209, 2022.

Sönke Thomsen, Nils Hoffmann, and Friedrich Wilhelm Fuchs. Pi control, pi-based state space control, and model-based predictive control for drive systems with elastically coupled loads-a comparative study. IEEE transactions on industrial electronics , 58(8):3647-3657, 2010.

Jian Wang, Xin Xu, Daxue Liu, Zhenping Sun, and Qingyang Chen. Self-learning cruise control using kernel-based least squares policy iteration. IEEE Transactions on Control Systems Technology , 22(3):1078-1087, 2013.

Deany Putri Aulia, Alfiyah Shaldzabila Yustin, Amien Marzuq Hilman, Aulia Rahma Annisa, and Eng Wahyu Kunto Wibowo. Fuzzy gain scheduling for cascaded pi-control for dc motor. In 2021 IEEE Conference on Energy Conversion (CENCON) , pages 158-162. IEEE, 2021.

Murat Arcak, Chris Meissen, and Andrew Packard. Networks of dissipative systems: compositional certification of stability, performance, and safety . Springer, 2016.

George H Hines, Murat Arcak, and Andrew K Packard. Equilibrium-independent passivity: A new definition and numerical certification. Automatica , 47(9):1949-1956, 2011.

Mathias Bürger, Daniel Zelazo, and Frank Allgöwer. Duality and network theory in passivitybased cooperative control. Automatica , 50(8):2051-2061, 2014.

Wenqi Cui and Baosen Zhang. Equilibrium-independent stability analysis for distribution systems with lossy transmission lines. IEEE Control Systems Letters , 6:3349-3354, 2022.

John W Simpson-Porco. Equilibrium-independent dissipativity with quadratic supply rates. IEEE Transactions on Automatic Control , 64(4):1440-1455, 2018.

Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural lyapunov control. Advances in neural information processing systems , 32, 2019.

Ruikun Zhou, Thanin Quartz, Hans De Sterck, and Jun Liu. Neural lyapunov control of unknown nonlinear systems with stability guarantees. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=QvlcRh8hd8X .

Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barrier functions. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pages 1724-1735. PMLR, 08-11 Nov 2022. URL https://proceedings.mlr.press/v164/dawson22a.html .

Qiuhua Huang, Renke Huang, Weituo Hao, Jie Tan, Rui Fan, and Zhenyu Huang. Adaptive power system emergency control using deep reinforcement learning. IEEE Transactions on Smart Grid , 2019.

Wanxin Jin, Zhaoran Wang, Zhuoran Yang, and Shaoshuai Mou. Neural certificates for safe control policies. arXiv preprint arXiv:2006.08465 , 2020.

Armin Lederer, Jonas Umlauft, and Sandra Hirche. Uniform error bounds for gaussian process regression with application to safe control. Advances in Neural Information Processing Systems , 32, 2019.

Andreas Doerr, Christian Daniel, Duy Nguyen-Tuong, Alonso Marco, Stefan Schaal, Toussaint Marc, and Sebastian Trimpe. Optimizing long-term predictions for model-based policy search. In Conference on Robot Learning , pages 227-238. PMLR, 2017.

Nathan Lambert, Albert Wilcox, Howard Zhang, Kristofer SJ Pister, and Roberto Calandra. Learning accurate long-term dynamics for model-based reinforcement learning. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 2880-2887. IEEE, 2021.

Andreas Doerr, Duy Nguyen-Tuong, Alonso Marco, Stefan Schaal, and Sebastian Trimpe. Model-based policy search for automatic tuning of multivariate pid controllers. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 5295-5301. IEEE, 2017.

Ignacio Carlucho, Mariano De Paula, and Gerardo G Acosta. An adaptive deep reinforcement learning approach for mimo pid control of mobile robots. ISA transactions , 102:280-294, 2020.

Joseph Sill. Monotonic networks. In Proceedings of the 10th International Conference on Neural Information Processing Systems , pages 661-667, 1997.

Antoine Wehenkel and Gilles Louppe. Unconstrained monotonic neural networks. Advances in neural information processing systems , 32, 2019.

Xingchao Liu, Xing Han, Na Zhang, and Qiang Liu. Certified monotonic neural networks. Advances in Neural Information Processing Systems , 33:15427-15438, 2020.

Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions on Neural Networks , 21(6):906-917, 2010.

Aishwarya Sivaraman, Golnoosh Farnadi, Todd Millstein, and Guy Van den Broeck. Counterexample-guided learning of monotonic neural networks. Advances in Neural Information Processing Systems , 33:11936-11948, 2020.

Hong Zhang and Zhen Zhang. Feedforward networks with monotone constraints. In IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No. 99CH36339) , volume 3, pages 1820-1823. IEEE, 1999.

Ernest K Ryu and Stephen Boyd. Primer on monotone operator methods. Appl. comput. math , 15(1):3-43, 2016.

Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. In 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages 7559-7566. IEEE, 2018.

Lucian Bu¸ soniu, Tim de Bruin, Domagoj Toli´ c, Jens Kober, and Ivana Palunko. Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control , 46:8-28, 2018.

Priya L Donti, Melrose Roderick, Mahyar Fazlyab, and J Zico Kolter. Enforcing robust control guarantees within neural network policies. In International Conference on Learning Representations , 2020.

Pulkit Nahata, Raffaele Soloperto, Michele Tucci, Andrea Martinelli, and Giancarlo FerrariTrecate. A passivity-based approach to voltage stabilization in dc microgrids with zip loads. Automatica , 113:108770, 2020.

Chris Meissen, Kristian Klausen, Murat Arcak, Thor I Fossen, and Andrew Packard. Passivitybased formation control for uavs with a suspended load. IFAC-PapersOnLine , 50(1):1315013155, 2017.

Wei He, Hejia Gao, Chen Zhou, Chenguang Yang, and Zhijun Li. Reinforcement learning control of a flexible two-link manipulator: an experimental investigation. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 51(12):7326-7336, 2020.

Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning , pages 146-155. PMLR, 2017.

Yize Chen, Yuanyuan Shi, and Baosen Zhang. Optimal control via neural networks: A convex approach. In International Conference on Learning Representations , 2019.

Stephen Boyd and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

T Athay, Robin Podmore, and Sudhir Virmani. A practical method for the direct analysis of transient stability. IEEE Transactions on Power Apparatus and Systems , PAS-98(2):573-584, 1979.

Ján Drgona, Aaron Tuor, and Draguna Vrabie. Learning constrained adaptive differentiable predictive control policies with guarantees. arXiv preprint arXiv:2004.11184 , 2020.

Erik Weitenberg, Claudio De Persis, and Nima Monshizadeh. Exponential convergence under distributed averaging integral frequency control. Automatica , 98:103-113, Dec. 2018.

Erik Weitenberg, Yan Jiang, Changhong Zhao, Enrique Mallada, Claudio De Persis, and Florian Dörfler. Robust decentralized secondary frequency control in power systems: Merits and tradeoffs. IEEE Transactions on Automatic Control , 64(10):3967-3982, Oct. 2019.

Florian Dörfler and Sergio Grammatico. Gather-and-broadcast frequency control in power systems. Automatica , 79:296-305, May 2017. ISSN 0005-1098. doi: https://doi.org/10.1016/j. automatica.2017.02.003.

Peter W Sauer, Mangalore A Pai, and Joe H Chow. Power system dynamics and stability: with synchrophasor measurement and power system toolbox . John Wiley & Sons, 2017.

A Proof

A.1 Proof of Theorem 1

We leverage the results in [49, 50] that the structure (5) with ReLU activation is a universal approximation for any convex function. However, ReLU activations are not strictly convex, and thus we design the Softplusβ activation. By showing that the structure (5) with Softplusβ activation can approximate neural networks with the ReLU activations arbitrarily closely when β is sufficiently large, we then prove that the structure (5) with Softplusβ can universally approximate any strictly convex function.

To prepare for the proof of Theorem 1, we first derive the following Lemma about the difference between ReLU activations and Softplusβ activation.

Lemma 3. Consider the ReLU activation σ ReLU l ( x ) := max( x, 0) . For all x ∈ R and ∆ > 0 , we have 0 < ( σ Softplus β l ( x +∆) -σ ReLU l ( x ) ) < ∆+ 1 β log(2) .

Proof. Note that

The proof of Theorem 1 is given below.

Proof. Previous works [49, 50] have shown that the structure (5) with ReLU activation is a universal approximation for any convex function. Hence, for any q ( z ) : Z ↦→ R , there exists g ( z ; θ ) ReLU constructed by (5) where the activation function is ReLU and satisfying | g ( z ; θ ) ReLU -q ( z ) | < 1 2 ϵ . Note that

then it suffices to prove | g ( z ; θ ) Softplus β -q ( z ) | < ϵ by showing the existence of g ( z ; θ ) Softplus β such that | g ( z ; θ ) Softplus β -g ( z ; θ ) ReLU | < 1 2 ϵ for all z ∈ Z .

Next, we compute the difference of the structure (5) with softplusβ and ReLU activations by tracing through the first layer to the last layer, under the same weights θ = { W ( z ) 0: k -1 , W ( o ) 1: k -1 , b 0: k -1 } .

The difference between the output of the first layer in g ( z ; θ ) Softplus β and g ( z ; θ ) ReLU is

which yields 0 < o Softplus β 1 -o ReLU 1 ≤ 1 β log(2) 1 by Lemma 3.

The difference of the second layer is

Since all the element of W ( o ) 1 is positive, we have 0 < W ( o ) 1 ( o Softplus β 1 -o ReLU 1 ) ≤ 1 β log(2) W ( o ) 1 1 . Applying Lemma 3 element-wise yields 0 ≤ o Softplus β 2 -o ReLU 2 ≤ 1 β log(2) W ( o ) 1 1 + 1 β log(2) 1 .

Similarily 0 ≤ o Softplus β 3 -o ReLU 3 ≤ 1 β log(2) W ( o ) 2 W ( o ) 1 1 + 1 β log(2) W ( o ) 2 1 + 1 β log(2) 1 . By induction

Hence,

where ∏ k -1 j = i W ( o ) j := W ( o ) k -1 W ( o ) k -2 · · · W ( o ) i .

Let β > 2 ϵ log(2) ( 1 + ( ∑ k -1 i =1 ∏ k -1 j = i W ( o ) j ) 1 ) , then 0 ≤ g ( z ; θ ) Softplus β -g ( z ; θ ) ReLU ≤ 1 2 ϵ . We complete the proof using

A.2 Proof of Theorem 2

Proof. At an equilibrium, the right side of (4b) equals to zero gives y ∗ = ¯ y and the corresponding set of equlibrium E = { x ∗ , s ∗ | f ( x ∗ , r ( s ∗ )) = 0 , y ∗ = ¯ y , h ( x ∗ ) = y ∗ } . We construct a Lyapunov function to prove that if there is a feasible equilibrium in E , then the system is locally asymptotically stable around it.

We construct a Lyapunov function using the storage function in Assumption 2 and the Bregman distance in Lemma 2 as

where the functions by construction satisfy S ( x , x ∗ ) ≥ 0 , B ( s , s ∗ ; θ ( I ) , β ( I ) ) ≥ 0 with equality holds only when x = x ∗ and s = s ∗ , respectively. Hence, V ( x , s ) | x ∗ , s ∗ is a well-defined function that is positive definite and equals to zero at the equilibrium.

To prepare for the calculation of the time derivative of the Lyapunov function, we start by calculating the time derivative of the function B ( s , s ∗ ; θ ( I ) , β ( I ) ) :

where 1 follows from ∇ s g ( I ) ( s ; θ ( I ) , β ( I ) ) = r ( s ) and ˙ s = ( -( y -¯ y )) = ( -( y -y ∗ )) . The time derivative of the Lyapunov function is

where 1 follows from the strict EIP property and equations derived in (15). The equality 2 is derived by plugging in the controller design in (4a) where u = p ( -y + ¯ y ) + r ( s ) and u ∗ = p ( -y ∗ + ¯ y ) + r ( s ∗ ) . The inequality 3 uses strictly monotone property of p ( · ) .

Therefore, ˙ V ( x , s ) | x ∗ , s ∗ ≤ 0 with equality only holds at the equilibrium. By Lyapunov stability theory in Proposition 1, the system is locally asymptotically stable around the equilibrium.

B Experiments

We demonstrate the effectiveness of the proposed neural-PI control in two dynamical systems: vehicle platooning and power system frequency control. All experiments are run with an NVIDIA Tesla T4 GPU with 16GB memory. For completeness, the figures highlighted in Section 6 are also shown below with more thorough discussions. Code is available at this link.

B.1 Vehicle platooning

B.1.1 Problem statement

The first experiment is the vehicle platoon control in [3, 23] with m vehicles, where u ∈ R m is the control signal to adjust the velocities of vehicles and the output y ∈ R m is their actual velocities. The state is x = ( ζ , y ) , where ζ ∈ R m is the relative position of vehicles with ζ (0) ⊥ Im ( 1 m ) (namely, the vehicles are not in the same position at the time step 0). The dynamic model is

where ˆ κ = diag ( κ 1 , · · · , κ m ) , ˆ ρ = diag ( ρ 1 , · · · , ρ m ) ∈ R m × m are constant diagonal matrices and κ i > 0 , ρ i > 0 for all i = 1 , · · · , m . The vector λ 0 = ( λ 0 1 , · · · , λ 0 m ) ∈ R m reflects the default velocity of vehicles. The matrix E ∈ R m × e is the incidence matrix that indicates the neighbouring relations for e pairs of neighbouring vehicles and satisfy ker ( E ⊤ ) = Im ( 1 m ) . The matrix Γ := I m -1 m 1 m 1 ⊤ m extracts the relative velocities of vehicles by Γ y . The diagonal matrix ˆ D ∈ R m × m represents the sensitivity to the relative distance of neighbouring vehicles.

B.1.2 Verification of Assumption 2

We start by verifying the uniqueness of x ∗ for any u ∗ ∈ R m . At the equilibrium, the right side of (17) equals zero gives

̸

For a given u ∗ ∈ R m , suppose there exists x ∗ a = ( ζ ∗ a , y ∗ a ) and x ∗ b = ( ζ ∗ b , y ∗ b ) , x ∗ a = x ∗ b such that (18) holds. Plugging in (18) gives

Note that Γ E = E . Left multiplying (19a) with ( E ˆ DE ⊤ ( ζ ∗ a -ζ ∗ b )) ⊤ Γ yields ( E ˆ DE ⊤ ( ζ ∗ a -ζ ∗ b )) ⊤ ˆ ρE ˆ DE ⊤ ( ζ ∗ a -ζ ∗ b ) = 0 , which holds if and only if E ˆ DE ⊤ ( ζ ∗ a -ζ ∗ b ) = 0 m since ˆ ρ ≻ 0 .

Hence, ( y ∗ a -y ∗ b ) = -ˆ ρE ˆ DE ⊤ ( ζ ∗ a -ζ ∗ b ) = 0 m and therefore y ∗ a = y ∗ b . Note that ker ( E ˆ DE ⊤ ) = Im ( 1 m ) and Im (Γ) ⊥ Im ( 1 m ) , thus ( ζ ∗ a -ζ ∗ b ) ⊥ Im ( 1 m ) . Hence, E ˆ DE ⊤ ( ζ ∗ a -ζ ∗ b ) = 0 m if and only if ζ ∗ a = ζ ∗ b . Theorefore, for every equilibrium u ∗ ∈ R m , there is a unique x ∗ = ( ζ ∗ , y ∗ ) ∈ R n such that f ( x ∗ , u ∗ ) = 0 n .

Let the storage function be S ( x , x ∗ ) = 1 2 ( y -y ∗ ) ⊤ ˆ κ -1 ˆ ρ -1 ( y -y ∗ ) + 1 2 ζ ⊤ E ˆ DE ⊤ ζ . Then

where 1 follows from ( -( y ∗ -λ 0 ) + ˆ ρ ( u ∗ -E ˆ DE ⊤ ζ ∗ )) = 0 m and E ⊤ y ∗ = E ⊤ Γ y ∗ = 0 e by definition of equilibrium. The inequality 2 follows from ˆ ρ > 0 . Therefore, the dynamics in (17) satisfy conditions in Assumption 2.

B.1.3 Simulation and Visualization

Simulation and training setup We adopt the model setup and parameters in [3, 23]. The number of vehicles is m = 20 . The sensitivity parameter is κ i = 1 for all vehicles. The parameters λ 0 i and ρ i are randomly generated by λ 0 i ∼ uniform [5 , 6] and ρ i ∼ uniform [1 , 2] , respectively. We train for 400 epochs, where each epoch trains with 300 trajectories with initial velocities y i (0) ∼ uniform [5 , 6] and initial integral variable s i (0) = 0 . The stepsize in time is set as ∆ t = 0 . 02 s and for K = 300 steps in a trajectory (Namely, each trajectory evolves 6s).

We implement control law in u to realize a specific output agreement at ¯ y and reduce the transient cost. The transient cost is set to be J ( y , u ) = ∑ K k =1 || y ( k ∆ t ) -¯ y || 1 + ˆ c || u ( k ∆ t ) || 2 2 , where ˆ c = diag ( c 1 , · · · , c m ) with c i ∼ uniform [0 . 025 , 0 . 075] . The loss function in training is set to be the same as J ( y , u ) , such that neural networks are optimized to reduce transient cost through training. The neural PI controller can be trained by most model-based or model-free algorithms, and we use the model-based framework in [8, 53] by embedding the system dynamic model in the computation graph shown in Figure 4 and training Neural-PI by gradient descent through J ( y , u ) .

Controller performances. We compare the performance of 1) Neural-PI: the learned structured Neural-PI controllers parametrized by (7) with three layers and 20 neurons in each hidden layer. The neural networks are updated using Adam with learning rate initializes at 0.05 and decays every 50 steps with a base of 0.7. 2) DenseNN-PI: The proportional and integral terms are parameterized by dense neural networks (5) with three layers, 20 neurons in each hidden layer, and unconstrained weights. The neural networks are updated using Adam with learning rate initializes at 0.035 and decays every 50 steps with a base of 0.7. 3) Linear-PI: linear PI control where p (¯ y -y ) := K P (¯ y -y ) , r ( s ) := K I ( s ) with K P and K I being the trainable proportional and integral coefficients. The coefficients are updated using Adam with learning rate initializes at 0.03 and decays every 50 steps with a base of 0.7. All of them have no communication constraints. All of the controllers are trained using 5 random seeds. The training time is shown in Table 1.

The average batch loss during epochs of training with 5 seeds is shown in Figure 7(a). All of the three methods converge, with the Neural-PI achieves the lowest cost. Figure 7(b) shows the transient

DenseNN

(a) Training Loss

(b) Average transient and steady cost

Neural-PI

100

Linear-PI

Cost

transient

DenseNN-PI

1.5

steady

Loss

Normallized

1.0

10

0.5

0.0

0

100

200

300

400

Epochs

Neural-PI

Linear-PI

and steady-state cost on 100 testing trajectories starting from randomly generated initial states. The steady-state cost is C ( y , u ) = || y (15) -¯ y || 1 + ˆ c || u (15) || 2 2 , where we use the variables at the time t = 15 s since the dynamics approximately enter the steady state after t = 15 s as we will show later in simulation. Neural-PI and Linear-PI have the lowest steady-state cost, and the output reaches ¯ y as guaranteed by Theorem 2. Neural-PI also achieves a transient cost that is much lower than others. By contrast, DenseNN-PI without structured design has both high costs in transient and steady-state performances.

(a) Neural-PI

(b) Linear-PI

(c) DenseNN-PI

6

5

speed

>

5

Acction

node2

node9

0

4

node3

node10

node4

node15

3

node5

node18

0

5

10

15

0

5

10

15

time(s)

time(s)

15

6

speed

Acction

10

node2

node9

node3

node10

5

4

node4

node15

node5

node18

0

0

5

10

15

0

5

10

15

time(s)

time(s)

6

>

Acction

0

speed

node2

node9

4

node3

node10

-5

node4

node15

-10

node5

node18

0

5

10

15

0

5

10

15

time(s)

time(s)

Given ¯ y = 5 m/s, Figure 8 shows the dynamics of velocity y and control action u on 8 nodes under the three methods. As guaranteed by Theorem 2, Neural-PI in Figure 8(a) and Linear-PI in Figure 8(b) reaches the required speed ¯ y = 5 m/s. However, Linear-PI has slower convergence and much larger control efforts compared with Neural-PI. Even though DenseNN-PI achieves finite loss both in training and testing, Figure 8(c) actually exhibits unstable behaviors. In particular, DenseNN-PI appears to be stable until about 10s, but states blows up quickly after that. Therefore, enforcing stabilizing structures is essential.

B.2 Power systems frequency control

B.2.1 Problem statement

The second experiment is the power system frequency control on the IEEE 39 -bus New England system [52] shown in Figure 9, where u ∈ R m is the control signal to adjust the power injection from generators and the output y ∈ R m is the rotating speed (i.e., frequency) of generators. The objective is to stabilize generators at the required frequency ¯ y = 60 Hz at the steady state while minimizing the transient control cost. The state is x = ( δ , y ) , where δ ∈ R m is the rotating angle of generators in the center-of-inertia coordinates with δ (0) ⊥ Im ( 1 m ) [54]. The model of power systems reflects the transmission of electricity from generators to loads through power transmission lines and is represented as follows:

where ˆ M = diag ( M 1 , · · · , M m ) , ˆ D = diag ( D 1 , · · · , D m ) with M j > 0 and D j > 0 being the inertia and damping constant of generator j , respectively. The vector d is the net load of the system. The matrix E ∈ R m × e is the incidence matrix corresponding to the topology of the power network with e transmission lines and satisfying ker ( E ⊤ ) = Im ( 1 m ) . The matrix Γ := I m -1 m 1 m 1 ⊤ m extracts the relative rotating speed of generators by Γ y . The diagonal matrix ˆ b = diag ( b 1 , · · · , b e ) ∈ R e × e with b j > 0 being the susceptance of the j -th transmission line.

Bus 30

Bus 37

WW

Bus 25

Bus 26

Bus 28

Bus 29

Bus 2

Bus 27

Bus 38

Bus 1

Bus 18

Bus 17

Bus 16

Bus 39

Bus 3

Bus15

Bus 21

Bus 4

Bus 14

Bus 12

Bus 9

Bus 22

Bus 5

Bus 13

BUS24

Bus 8

Bus 35

Bus 19

Bus

WW

Bus 7

Bus 10

Bus 20

Bus 23

Bus 6

Bus31

Bus 32

Bus

34

Bus 33

Bus 36

We adopt a common assumption in literature that the power system operates with δ satisfying H = { δ | [ E ⊤ δ ] j ∈ ( -π/ 2 , π/ 2) ∀ j = 1 , · · · , e } , where [ E ⊤ δ ] j is the angle difference between the generators in head and tail of the j -th transmission line [55-57]. This range is sufficiently large to include almost all practical scenarios [55-57].

B.2.2 Verification of Assumption 2

At the equilibrium, the right side of (17) equals zero gives

̸

We start by verifying the uniqueness of x ∗ for any u ∗ ∈ U where (21) has a feasible solution such that δ ∈ H . For a given u ∗ ∈ U , suppose there exists x ∗ a = ( δ ∗ a , y ∗ a ) and x ∗ b = ( δ ∗ b , y ∗ b ) , x ∗ a = x ∗ b such that (21) holds. Plugging in (21) gives

Note that Γ E = E . Left multiplying (22a) with ( E ˆ b ( sin( E ⊤ δ ∗ a ) -sin( E ⊤ δ ∗ b ) ) ) ⊤ Γ ˆ D -1 yields ( E ˆ b ( sin( E ⊤ δ ∗ a ) -sin( E ⊤ δ ∗ b ) ) ) ⊤ ˆ D -1 ( E ˆ b ( sin( E ⊤ δ ∗ a ) -sin( E ⊤ δ ∗ b ) ) ) = 0 , which holds if and only if ( E ˆ b ( sin( E ⊤ δ ∗ a ) -sin( E ⊤ δ ∗ b ) ) ) = 0 m since ˆ D -1 ≻ 0 . Plugging in (22a) gives ˆ D ( y ∗ a -y ∗ b ) = 0 m , which holds if and only if y ∗ a = y ∗ b since D i > 0 for all i = 1 , · · · , m .

Left multiplying ( E ˆ b ( sin( E ⊤ δ ∗ a ) -sin( E ⊤ δ ∗ b ) ) ) = 0 m with ( δ ∗ a -δ ∗ b )) ⊤ yields

Since b j > 0 and sin( · ) is strictly increasing in ( -π/ 2 , π/ 2) , (23) holds if and only if E ⊤ ( δ ∗ a -δ ∗ a ) = 0 e . Note that Im (Γ) ⊥ Im ( 1 m ) , thus ( δ ∗ a -δ ∗ b ) ⊥ Im ( 1 m ) . Hence, (23) holds if and only if δ ∗ a = δ ∗ b . Therefore, for every equilibrium u ∗ ∈ U , there is a unique x ∗ = ( δ ∗ , y ∗ ) ∈ R n such that f ( x ∗ , u ∗ ) = 0 n .

Let the storage function be S ( x , x ∗ ) = 1 2 ( y -y ∗ ) ⊤ ˆ M ( y -y ∗ ) -1 ⊤ e ˆ b (cos( E ⊤ δ ) -cos( E ⊤ δ ∗ )) -( E ˆ b sin( E ⊤ δ ∗ )) ⊤ ( δ -δ ∗ )) . Note that -1 ⊤ e ˆ b (cos( E ⊤ δ ) is strictly convex in H , thus the Bregman distance -1 ⊤ e ˆ b (cos( E ⊤ δ ) -cos( E ⊤ δ ∗ )) -( E ˆ b sin( E ⊤ δ ∗ )) ⊤ ( δ -δ ∗ )) ≥ 0 with equality holds only when δ = δ ∗ .

The time derivative is

where 1 follows from ( -ˆ D ( y ∗ -¯ y ) -d + u ∗ -E ˆ b sin( E ⊤ δ ∗ )) = 0 m by definition of equilibrium. The relation 2 follows from E ⊤ y ∗ = E ⊤ Γ y ∗ = 0 e and D i > 0 for all i = 1 , · · · , m . Therefore, the dynamics (20) of the power system frequency control satisfies conditions in Assumption 2.

B.2.3 Simulation and Visualization

Simulation Setup We conduct experiments on the IEEE New England 10-machine 39-bus (NE39) power network with parameters given in [52, 8]. We implement control law for power output u of generators to realize the track of frequency at 60Hz and reduce the power generation cost. The state δ is initialized as the solution of power flow at the nominal frequency and s is initialized as 0.

The number of epochs and batch size are 400 and 300, respectively. The step-size in time is set as ∆ t = 0 . 01 s and the number of time stages in a trajectory for training is K = 400 .

Apart from the accumulated frequency deviation, an important metric for the frequency control problem is the maximum frequency deviation (also known as the frequency nadir) after a disturbance [8]. Hence, the transient cost is set to be J ( y , u ) = ∑ n i =1 ( max k =1 , ··· ,K | y i ( k ∆ t ) -¯ y | + 0 . 05 ∑ K k =1 | y i ( k ∆ t ) -¯ y | +0 . 005 ∑ K k =1 ( u i ( k ∆ t )) 2 ) . The loss function in training is J ( y , u ) , such that neural networks are optimized to reduce transient cost. The neural PI controller can be trained by most model-based or model-free algorithms, and we use the model-based framework in [8, 53] by embedding the system dynamic model in the computation graph shown in Figure 4 and training Neural-PI by gradient descent through J ( y , u ) .

Two major goals of this experiment is

Verifies the robustness of the controller under parameter changes . Note that the load d is a parameter in the dynamics (20). In particular, power system operator emphasizes on the ability of the system to withstand a big disturbance such as a step load change. To this end, we train and test controllers by randomly picking at most three generators to have a step load change uniformly distributed in uniform [ -1 , 1] p.u., where 1p.u.=100 MW is the base unit of power for the IEEE-NE39 test system.

Verifies the performances under communication constraints . Most systems do not have fully connected real-time communication capabilities, so the controller needs to respect the communication constraints and we show the flexibility of Neural-PI control under different communication structures.

Controller Performances. We compare the performance of Neural-PI controller where 1) all the nodes can communicate 2) half of the nodes can communicate and 3) none of the nodes can communicate (thus the controller is decentralized), respectively. All neural-PI controllers are parameterized by (7) and (8) where each SCNN has three layers and 20 neurons in each hidden layer. The neural networks are updated using Adam with the learning rate initializes at 0.05 and decays every 50 steps with a base of 0.7. We compare against the following two benchmarks where all the nodes can communicate: 4) DenseNN-PI-Full: Dense neural networks (5) with three layers, 20 neurons in each hidden layer, and unconstrained weights. The neural networks are updated using Adam with a learning rate initializes at 0.01 and decays every 50 steps with a base of 0.7. Note that DenseNN needs such a small learning rate to let the training converge, the reason is that DenseNN may lead to unstable behaviors that we will see later. 5) Linear-PI-Full: linear PI control where p (¯ y -y ) := K P (¯ y -y ) , r ( s ) := K I ( s ) with K P and K I being the trainable proportional and integral coefficients. The coefficients are updated using Adam with the learning rate initializes at 0.08 and decays every 50 steps with a base of 0.7. All of the controllers are trained using 5 random seeds. The training time is shown in Table 2.

The average batch loss during epochs of training with 5 seeds is shown in Figure 10(a). All converge, with the Neural-PI-Full achieving the lowest cost. Figure 10(b) shows the average transient cost and steady-state cost with error bar on 100 testing trajectories subject to random step load changes. The steady-state cost is C ( y , u ) = 0 . 05 || y (15) -¯ y || 1 +0 . 005 || u (15) || 2 2 , where we use the variables at the time t = 15 s since the dynamics approximately enter the steady state after t = 15 s as we will show later in simulation. Neural-PI-Full achieves the lowest transient and steady-state cost. Notably, the steady-state cost significantly decreases with increased communication capability. The reason is that communication serves to better allocated control efforts such that they can maintain output

(a) Training Loss

(b) Transient and steady cost

Neural-Pl-Full

Neural-PI-Falf

Linear-Pl-Full

Neural-PI-DeC

transient

10.0

DenseNN-PI-Full

Cost

1.5

steady

Normallized

1.0

1.0

0.5

0.0

0

100

200

300

400

Epochs

Neural-Pl-Full

Neural-PI-Half

Neural-PI-Dec

Linear-PI-Full

DenseNN-PI-Full

tracking with smaller control costs. Again, DenseNN without structured design has high costs both in transient and in steady state.

With a step load change at 0.5s, Figure 11 shows the dynamics of frequency y and control action u on 7 nodes under the five methods. Again, DenseNN-PI-Full in Figure 11(e) exhibits unstable behavior with large oscillations. As guaranteed by Theorem 2, Neural-PI in Figure 11(a-c) reaches the required frequency ¯ y = 60 Hz, but the speed of convergence is lower for reduced communication capabilities. Hence, the guarantees provided by the structured Neural-PI controllers are robust to parameter changes and communication constraints, which have significant practical importance.

Performance with different numbers of training trajectories. Table 3 compares the transient cost attained by different controllers trained with different numbers of trajectories. For both Linear-PI and Neural-PI, training with 5 trajectories for each epoch has already achieved a similar cost as training with 300 trajectories. By contrast, unstructured DenseNN requires a much larger amount of training data to reduce transient costs on testing trajectories. Therefore, the stabilizing structure significantly reduces the requirement for the number of samples to learn well.

The impact of disturbances and noises. The satisfaction of the Lyapunov condition is robust to disturbances in the system parameters and does not need to know how large the disturbances are, as shown in the proof Theorem 2 and Remark 2. Therefore, if there is a sudden change in the load levels, the proposed controller design still stabilizes the system and tracks the required frequency at 60Hz. In Figure 12(a), we demonstrate the system dynamics after two disturbances in load. In Figure 12(b)-(c), we add noises in both data measurement and dynamics with the signal-to-noise ratio being 5 dB (much larger than typical measurement noises). The results show that the systems are input-to-state stable, i.e., that bounded noise will lead to bounded states. Incorporating noise in rigorous theoretical analysis is an important future direction for us.

(a) NeuralPI-Full

(b) NeuralPI-Half

(c) NeuralPI-Dec

(d) Linear-PI-Full

0.0

Accumulated Cost

60.00

Frequency

Actionu

4

59.98

FS'0-

node3

node7

59.96

node4

node8

node5

node9

59.94

1.0

node6

0

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

time(s)

time(s)

time(s)

60.00

0.0

Accumulated Cost

Frequency

Actionu

4

59.98

-0.5

node3

node7

2

59.96

node4

node8

node5

6apou

node6

59.94

-1.0

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

time(s)

time(s)

time(s)

60.00

F0:0

Accumulated Cost

6

Frequency y

Actionu

59.98

0.5

node3

node7

59.96

node4

node5

node8

node9

node6

-1.0 -

0

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

time(s)

time(s)

time(s)

0.0

Accumulated Cost

10

》60.00

Frequency

-0.2

Action

59.95

0.4

eapou

node7

—node8

59.90

-0.6

node5

node6

node9

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

time(s)

time(s)

time(s)

Frequencyy

60.5

0.0

node3

node4

node7

node8

Accumulated Cost

150

Actionu

node5

node9

60.0

-0.2

node6

100

50

59.5

-0.4

10

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

time(s)

time(s)

time(s)

(e) DenseNN-Full

(a) There is a step increase of load on 0.5s and a step decrease of load on 3.5s.

(b) Neural-PI with noise on data measurement

AccumulatedCost

6

>60.05

Frequency

Actionu

4

60.00

0

node3

node7

node4

node8

59.95

-1

node5

node6

node9

0

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

time(s)

time(s)

time(s)

0.0

AccumulatedCost

60.00

Frequency

Action u

4

59.98

-0.5

node3

node7

59.96

node4

node8

-1.0

node5

node9

59.94

0

2

4

6

8

0

2

node6

4

6

8

0

0

2

4

6

8

time(s)

time(s)

time(s)

60.000

0.0

AccumulatedCost

Frequency

59.975

Action u

4

-0.5

59.950

node3

node4

node7

node8

2

59.925

-1.0

node5

node6

node9

0

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

time(s)

time(s)

time(s)

(c) Neural-PI with noise on system dynamics

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of learning feedback controllers for multi-input, multi-output (MIMO) dynamical systems. At the core of the paper, a PI-like nonlinear control structure is proposed. Specifically, the controller consists of two NN, one which receives the set-point error (denoted as \"P-part\") and another one that receives its integral (\"I-part\") as an input.  For the NNs, a special structure is proposed that ensures strict convexity of the NN. This structure serves as the basis for proving stability and zero steady-state error of the closed loop system employing this controller, irrespective of the parameters used in the NN.  Hence, these guarantees can be given a-priori irrespective of the training.  Training of the NN then aims at optimizing transient behavior. Finally, two numerical examples of MIMO control systems (vehicle platooning, power network) are presented.\n\nSTRENGTHS:\nS1) Developing NN control architectures that are guaranteed to satisfy key control properties by design (here: stability and steady-state optimality) is very relevant. The combination of PI-like structure with strictly convex NN seems novel.\n\nS2) Leveraging passivity properties is an interesting idea for analyzing and achieving system-wide properties. This is well-known in the control community, but should also be relvant for the NeurIPS community when considering distributed learning problems.\n\nS3) I like the idea of having a system design that is stable and comes with base performance by design and without communication, and then improving performance further when communication is available.\n\nWEAKNESSES:\nW1)  The introduction makes some imprecise or misleading statements about the state of the art as well as technical aspects.  All statements should be carefully checked and (where needed) be backed up with proper argumentation or references. Examples:\n\n* ll. 33-34: \"Currently, learning-based approaches mostly parameterize nonlinear controllers as neural networks and train them to optimize performance\".  While NN are a popular function approximator in learning-based control, they are actually used in different flavors (model learning, controller parametrization, surrogate model, etc.).  Furthermore, there are many other ML techniques that are also used, such as Gaussian processes, which are particularly popular when it comes to guarantees (examples: A Lederer, J Umlauft, S Hirche, \"Uniform error bounds for Gaussian process regression with application to safe control\"; C Fiedler, CW Scherer, S Trimpe, \"Learning-enhanced robust controller synthesis with rigorous statistical and control-theoretic guarantees\")  Hence, the characterization of the area of learning-based control as described in this above sentence is too narrow in my opinion.\n* ll. 47-49: \"...often overlooking the steady-state behavior (cost at the equilibrium state).\"  The statement is not supported by references.  In fact, I would disagree with it, as there are works that specifically focus on the long term performance of controllers or RL policies (examples: A Doerr et al, \"Optimizing long-term predictions for model-based policy search\"; N Lambert et al, \"Learning accurate long-term dynamics for model-based reinforcement learning\")\n* ll. 49-51: The description of classical P- and I-control is inaccurate.  While it is correct that I-control is often used in practice to make steady-state error zero, the actual behavior depends on the controller *and* the dynamical process.  Depending on the process dynamics (e.g., if it has an integrator itself), also a P-controller may lead to zero steady-state error. Also, the statement that P-control is for the transient, but disregarding steady-state performance is inaccurate.  P-control also affects the steady-state in general.  Again, the general description of how P- and I-control are often used in practice is not wrong, but the authors should more carefully word these sentences to avoid technical inaccuracies.\n* The statement of the main research question that is addressed in this paper is too general to be representative of the paper's contribution: \"Can we learn nonlinear controllers that guarantee transient stability and zero steady-state error for MIMO systems?\"  (ll. 55-56).  First, the authors state themselves in the next sentence that this question can't be answered for general nonlinear system.  Second, there are existing works in learning-based control that focus specifically on MIMO PI-control (see below). IMO, the research question should thus be rephrased to already clarify the scope of this work. \n* ll. 67-68, \"This class of functions is too restrictive to use for controller design\" -- Why?  Appropriate reasoning or reference is missing to support the statement.\n\n\n\nW2) Related work:  There is no explicit discussion of related work as a separate section/subsection, which is not a requirement of course, but could be helpful. Some references are mentioned in the introduction, but IMO the following aspects require further discussion:\n\n* The control problem and the objectives that are stated are super general in control (stability, zero steady-state error, fast transient). Hence, there will be a lot of works in learning-based control addressing similar problems.  While the paper does mention references and related work, I do not find this discussion very structured and have trouble pin-pointing what exactly is the difference to prior art and thus contribution of this paper.  Just saying, for example, that \"guaranteeing stability and output tracking\" (l. 144) is not helpful in this regard as the statement is too general and not specific to the contribution at hand.  It might help to first clearly specify the problem that is addressed, and afterwards discuss the related work specifically to the problem.\n* Given that the consideration of MIMO systems is a main aspect of this work, there should be a review specific to this.  Indeed, there are existing works in learning-based control that focus specifically on MIMO PI-control (e.g., A Doerr et al, \"Model-based policy search for automatic tuning of multivariate PID controllers\", I Carlucho, M De Paula, GG Acosta, \"An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots\", and many other works on \"self-tuning\" for MIMO systems).\n\nOverall, I do not get a clear idea how this work compares to the state of the art. \n\n\n\nW3) Specification of controller structure: The control law u = g(y) is not precisely specified.  Concretely, is g(.) a static function, or does it itself involve states and memory (i.e., is it itself dynamic)?  This is critical, as dynamic controllers (such as combining a state estimator and state-feedback controller) are more powerful than static maps in general. Later on, it seems that static NN are used as components of g(.).  However, the controller also involves an integrator, which would not be captured by a static function g(.) and is actually a dynamic component.  If, on the other hand, dynamic output-feedback controllers are admissible, one could presumably learn much richer control behavior.  IMO, it is ok to restrict the controller structure (from what I understand, to an integrator + static map), but it needs to be clearly and correctly stated.\n\n\n\nW4) I am wondering whether it is really justified to call the proposed structure a proportional + integral (PI) controller. In particular, the term that is called \"proportional\" isn't really proportional to the error for a general nonlinear map.  I can see that what the authors propose is a generalization of well-known PI-control structure.  However, I find it a bit misleading to call this a \"proportional control\".  In fact, p(-y+ybar) is a general nonlinear function of the tracking error.  Hence, it will in general not be proportional to the tracking error.  Please elaborate, why you call the architecture \"PI\", and ideally provide references if this is a common naming.\n\n\n\nW5) I do not find the notion of \"end-to-end\" guarantees conclusive and well justified.  The authors use this notion to denote the fact that they can provide certain guarantees a-priori before training (by combining a certain system class (passive) with the proposed controller architecture).  As far as I know, the term \"end-to-end\" is not commonly used for this (if you have references where it is used in this way, please provide them).  If a controller has theoretical guarantees, these are  most of the time \"by construction\", so I don't understand why this is called \"end-to-end\".\n\n* Minor: In the introduction, it is not yet clear what the authors mean when mentioning \"end-to-end guarantees\" on l. 80. It should be explained when introduced for the first time.\n\n\n\nW6) The presentation of experiments in the main paper is not self-contained and, IMO, insufficient. There needs to be a basic description of the problem in the main paper, which can be understood without consulting the supplementary material.  But this is currently missing.  Because of this, one cannot judge how challenging the control problem actually is, and whether sophisticated nonlinear control is actually needed.  Furthermore, the verification of assumptions is critical because a main claim of the paper are the theoretical guarantees.  This should be in the main paper.  As these are essential aspects of the contribution, the experimental section isn't self contained.  If the authors cannot fit all essential components, then a conference might be the wrong format for this manuscript.\n\n\n\nW7) Experimental results are not fully conclusive; they compare against a standard neural network, but more classical controller structures as baseline are missing in the main paper. Thus, I find the results only partially convincing for showing the benefit of the proposed Neural-PI controller.\n\n* It remains unclear whether the chosen problems require sophisticated nonlinear control such as the proposed Neural-PI controller. First, the problem description does not convey this (see above).  Second, in the supplementary material, the authors report the results also of a standard linear PI, whose parameters have been optimized similarly to the optimization of the NNs.  Apparently, the linear PI achieves almost the same performance as the Neural-PI for vehicle platooning, and very close performance for the power grid.  The results suggests that at least for these systems, there is no significant benefit of the proposed and more involved Neural-PI structure.  Hence, the result are not conclusive regarding the proposed architecture in this regards.\n* Baseline comparisons (i.e., the linear PI from the supplementary, and maybe also a model-based controller such as LQR) should be included in the main paper to judge the performance of the proposed Neural-PI.\n* I find it surprising that the DenseNN cannot stabilize the vehicle platooning system and actually has some significant deviation at the end of the horizon (Fig. 5(b), 15 sec).  Such deviation should also be visible in the objective function, so I would expect that the NN (if trained enough) can accommodate for this. \n* A discussion of assumptions and theoretical guarantees is missing in the main paper. In ll. 177/178, it is claimed that in the experimental section, it will be discussed how EIP (the main assumption) can be verified if the system is unknown.  While I agree that this discussion is very important with regards of the claimed contribution of this paper, I could not find this in Sec. 6.\n* The above points hold for both examples (platooning, power grid).\n\n\n\nMinor: \n\n* l 192: As usually, p is a gain (e.g., a real number), it would be good to emphasize when introducing it that, here, it is meant as a function of the tracking error.\n\nQUESTIONS:\nQ1) Why is it justified to call the structure in (4) a PI-controller, especially, why is it justified to call the first component \"proportional\"? \n\nQ2) What are the 2..3 papers in existing literature that are closest to your approach? And what main limitation of these does you approach address?\n\nQ3) In the experiments, the linear PI performs rather well. Why is the more involved Neural-PI controller even needed there?\n\nQ4) Do you have an example, where the Neural-PI controller clearly outperforms linear approaches?\n\nQ5) Why did you decide to not include the experimental results of the linear PI in the main paper?\n\nLIMITATIONS:\nLimitations (such as no consideration of disturbance/noise) are mentioned briefly in the conclusion, but not discussed in detail.  I would have liked to see a discussion on: what happens when there is noise/disturbance in the sensing, the process, and/or the data; do the authors expect that their controllers can be transferred to the real world, or what hurdles do they see?\n\nI don't see any particular potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper proposes a method for optimal control of dynamical systems using a neural network-based controller that guarantees stability and output tracking. The key structure of the proposed neural proportional-integral (PI) controller is based on the strict monotonicity of the proportional and integral terms, which are parameterized as the gradient of a strictly convex neural network. The paper also introduces tunable activations for universal approximation capabilities and communication constraints. Experiments on traffic and power networks show significant improvements in both transient and steady-state performance. The contribution of this paper is mainly in the field of control theory and its application.

STRENGTHS:
The strength of this paper is that it presents a method for optimal control using a neural network-based controller with provable stability guarantees. Using a strictly monotonic function parameterized by a strictly convex neural network can improve control performance and output tracking. Experiments on traffic and power networks show significant improvements in both transient and steady-state performance.

WEAKNESSES:
The multiple equilibria problem has not been fully addressed in real-world applications. This limitation limits the applicability of the proposed method to systems with general nonlinear systems.

QUESTIONS:
In real applications, how to judge the passivity of a large-scale nonlinear system is a possible problem to extend the proposed approach.

LIMITATIONS:
The paper does not explicitly discuss the scalability of the proposed method to larger, more complex systems. This paper does not provide a comparison with other state-of-the-art control methods, which helps to better understand the advantages and disadvantages of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to control multi-input, multi-output dynamical systems, introducing a neural network-based PI controller offering provable stability and output tracking guarantees. The neural Proportional-Integral controller is based on strict monotonicity parameterized by strictly convex neural networks (SCNN). The strict monotonicity of the network gives the convergence guarantee by construction.  Testing on traffic and power networks showed improved transient and steady-state performances compared to unstructured networks.\n\nSTRENGTHS:\nIt is interesting to improve PI control using neural networks. The proposed framework not only keeps the structure (which is well-validated across decades) and robust tracking performance of PI controllers, but also allows optimizing the tracking performance by learning the parameters in g and r functions. The use of monotonic neural networks is also critical. The experiment results demonstrate its advantages over regular fully connected neural networks.\n\nWEAKNESSES:\n- The authors must have known that carefully tuned PI controllers already work very well. Why didn’t they compare in the experiments? It is common that controllers built upon regular fully connected layers will diverge as time progresses, but this is not a problem in simple regular PI controllers. How well the proposed method is, compared to a carefully tuned classical PI controller?\n\n- Even if the proposed framework is better than traditional PI control in the transient phase, how can the authors guarantee its performance in situations outside the training data distribution? In real world applications where data collection can be expensive, if we only have 1000 data points, would the authors choose to train a neural network (this could be dangerous if only have limited data) or tune a classical PI controller using their extensive domain knowledge of the system dynamics (and use the 1000 data points for tuning & experiment).\n\n- In summary, this study is interesting and novel, but its importance / practical contribution to the control field might be minor.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper "Structured Neural-PI Control with End-to-End Stability and Output Tracking Guarantees" presents a well-motivated and technically sound approach to address the lack of provable guarantees in neural network-based controllers. The combination of structured neural-PI control, equilibrium-independent passivity, and SCNNs with tunable softplus-β activations demonstrates promising results in terms of stability, output tracking, and improved performance.

STRENGTHS:
The paper is clear, well written and accurate in the mathematical control details. The method leverages a specific kind of NN and the approach is clean and seems to hold great potential, at least to an applicable systems class.

WEAKNESSES:
The proposed approach relies on the assumption of equilibrium-independent passivity, which may limit its applicability to a specific class of physical systems. The paper should explicitly discuss the range of systems to which the approach is applicable and highlight any potential limitations or challenges in extending it to other domains.

QUESTIONS:
1.	In Definition 2, is it critical to demand zero tracking error? That can be augmented easily with a prefilter or an integrator. Even though the controller has an integrator, the system may be of higher type).
2.	In 3.1 it is stated that it is desired to optimize the transient response before the system settles on the tracking point. What are the specifications? Maximum overshot? Settling time? Are there minimum requirements?

LIMITATIONS:
Not applicable
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a neural-PI controller based on the newly designed strictly convex neural network. Thanks to the EIP condition of the considered systems, the neural-PI controller naturally satisfies the stability and output tracking guarantee with the special structure of the SCNN. The efficacy of the proposed method is demonstrated in several physical models.\n\nSTRENGTHS:\n[1] The proposed neural-PI controller satisfies the stability guarantee by construction of the SCNN instead of adding regularization term to the loss function, which is a hugh improvement to the existing learning controller commuity.\n\n[2] The proposed method does not require the knowledge of the equilibria of the intrisic dynamics but can still provide stability guarantee of the unknown equilibria.\n\n[3] The experiments take communication cost into consideration, which is realistic and instructive.\n\n[4] The presentation of the paper is good, the workflow in the figures are clear and helpful.\n\nWEAKNESSES:\n[1] The current framework is restricted in the system with EIP condition. \n\n[2] The authors did not compare the proposed method with the existing methods such as RL, QP and MPC.\n\n[3] The training policy is naïve, the learner should solve the controlled ODE to find the controlled trajectory and then update the parameters.\n\nQUESTIONS:\n[1] Since the objective functional incurs the integral of the controlled ODE, can this method scale to the high-dimensional task? \n\n[2] Can this method outperform the existing methods (RL such as PPO or MPC) in terms of training speed or test performance?\n\nLIMITATIONS:\n[1] The current neural method can only handle deterministic systems with EIP condition, which cannot adapt to the real-world scenarios where noise is unavoidable.\n\n[2] The authors did not provide numerical comparison with the existing methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}