Title: Complexity of Derivative-Free Policy Optimization for Structured $\mathcal{H}_\infty$ Control
Venue: NeurIPS 2023
Accepted: yes
Keywords: Structured $\mathcal{H}_\infty$ Control

Nonsmooth Optimization

Complexity Analysis
Abstract: The applications of direct policy search in reinforcement learning and continuous control have received increasing attention.
In this work, we present novel theoretical results on the complexity of derivative-free policy optimization on an important class of robust control tasks, namely the structured $H_\infty$ synthesis with static output feedback. 
Optimal $H_\infty$ synthesis under structural constraints leads to a constrained nonconvex nonsmooth problem and is typically
addressed using subgradient-based policy search techniques that are built upon the concept of Goldstein subdifferential or other notions of enlarged subdifferential.  In this paper, we study the complexity of finding $(\delta,\epsilon)$-stationary points for such nonsmooth robust control design tasks using policy optimization methods which can only access the zeroth-order oracle (i.e. the $H_\infty$ norm of the closed-loop system). First, we study the exact oracle setting and identify the coerciveness of the cost function to prove high-probability feasibility/complexity bounds for derivative-free policy optimization on this problem. Next, we derive a sample complexity result for the multi-input multi-output (MIMO)  $H_\infty$-norm estimation. We combine this with our analysis to obtain the first sample complexity of model-free, trajectory-based, zeroth-order policy optimization on finding $(\delta,\epsilon)$-stationary points for structured $H_\infty$ control. 
Numerical results are also provided to demonstrate our theory.

Complexity of Derivative-Free Policy Optimization for Structured H∞ Control

Abstract

The applications of direct policy search in reinforcement learning and continuous control have received increasing attention. In this work, we present novel theoretical results on the complexity of derivative-free policy optimization on an important class of robust control tasks, namely the structured H ∞ synthesis with static output feedback. Optimal H ∞ synthesis under structural constraints leads to a constrained nonconvex nonsmooth problem and is typically addressed using subgradient-based policy search techniques that are built upon the concept of Goldstein subdifferential or other notions of enlarged subdifferential. In this paper, we study the complexity of finding ( δ, ϵ ) -stationary points for such nonsmooth robust control design tasks using policy optimization methods which can only access the zeroth-order oracle (i.e. the H ∞ norm of the closed-loop system). First, we study the exact oracle setting and identify the coerciveness of the cost function to prove high-probability feasibility/complexity bounds for derivative-free policy optimization on this problem. Next, we derive a sample complexity result for the multi-input multi-output (MIMO) H ∞ -norm estimation. We combine this with our analysis to obtain the first sample complexity of model-free, trajectory-based, zeroth-order policy optimization on finding ( δ, ϵ ) -stationary points for structured H ∞ control. Numerical results are also provided to demonstrate our theory.

1 Introduction

Policy optimization techniques have received increasing attention due to their impressive performance in reinforcement Learning (RL) and continuous control tasks [65, 48, 64, 45]. Despite the empirical successes, the theoretical properties of policy-based RL methods have not been fully understood, even on relatively simple linear control benchmarks. This has motivated a line of recent work developing sample complexity theory of model-free policy optimization on benchmark linear control problems such as linear quadratic regulator (LQR) [22, 47, 72, 50, 24, 33], stabilization [57, 56], linear robust/risk-sensitive control [30, 76, 77], Markov jump linear quadratic control [59], and distributed LQR [44]. These existing sample complexity results heavily rely on the fact that the cost functions in these benchmark problems are differentiable over the entire feasible set, and hence cannot be applied to cover the H ∞ robust control setting where the objective function is nonsmooth in the first place [2, 3, 18]. In this paper, we make a meaningful initial step to bridge this gap by

developing a novel complexity theory for an important class of H ∞ robust control problems, namely the structured H ∞ synthesis with static output feedback.

H ∞ synthesis is arguably the most fundamental paradigm for robust control [80, 20]. Therefore, it is important to understand the complexity of policy optimization on such problems. In this work, we will develop complexity theory for derivative-free policy optimization on the structured static output-feedback H ∞ synthesis problem [3]. Notice that this is an important class of H ∞ control problems with practical importance [2, 3, 4, 31, 12, 18] due to the following reasons.

Structured control refers to fixing the structure of the controller/policy without using the order information of the plant to be controlled. In practice, the state dimension of the true system is typically unknown (e.g. the system is assumed to be rigid body dynamics, but there will always be flexible modes with unknown state order for the true plant due to elasticity or other unmodeled dynamics). It can be very challenging to learn the exact order of the plant [55, 61]. Therefore, it is preferred to fix the structure of the controller/policy beforehand.

Static output feedback is one of the most important forms of structured controller used in control, and it has a long history dating back to [42, 67, 49, 60]. Static output feedback has been extensively studied and applied in various areas, including aerospace [23], robotics [25], and chemical engineering [1]. In addition, the static output-feedback setting covers several important problems as special cases such as distributed controller design [14, 34] and proportional-integral-derivative (PID) control [73].

Studying the complexity of policy optimization methods on structured H ∞ synthesis is of great importance for several reasons. First, this problem cannot be solved using convexification techniques [62, 7, 26, 63] and hence it is natural to pose this problem as policy optimization [2, 3]. This is significantly different from the full-order H ∞ control design which yields convex reparameterizations [80, 20]. Specifically, if the order (i.e. state dimension) of the plant is exactly known, one can use a dynamical controller whose order is the same as the order of the plant, and the resultant full-order H ∞ synthesis can be lifted as a convex optimization problem. However, if the plant order is unknown and the controller structure is fixed beforehand, the resultant structured H ∞ problem is non-convex by nature [2, 3]. Second, the objective function in structured H ∞ synthesis can be non-differentiable over important feasible points (e.g. stationary points), and hence the structured H ∞ synthesis is a nonconvex nonsmooth optimization problem by nature. In the model-based setting, various advanced nonconvex nonsmooth optimization methods have been developed to solve this problem [2, 3, 4, 31, 12, 18, 52, 13, 40], leading to practical toolboxes such as HIFOO [4, 31]. Some convergence analysis has also been recently developed in [32]. However, in the model-free setting, the sample complexity of this problem remains largely unknown. The nonsmoothness of the cost function raises new challenges in developing the complexity analysis. There is a gap between the existing sample complexity theory and the model-free policy optimization on nonsmooth H ∞ synthesis. Our work will bridge this gap via offering the first sample complexity result of model-free policy optimization on nonsmooth H ∞ robust control. Specifically, we study the complexity of derivative-free policy optimization for structured H ∞ static output feedback problem when we can only access the zeroth-order oracle. We consider both exact and inexact zeroth-order oracle settings. Our contributions are two-fold:

Exact zeroth-order oracle: We establish the coerciveness of the cost function and then provide high-probability feasibility and complexity bounds by leveraging the relationship between Goldstein's subdifferential and uniform smoothing. Specifically, we show that to find a ( δ, ϵ ) -stationary point, the required number of iterations is on the order of Θ( 1 δϵ 4 ) .

Inexact zeroth-order oracle: We first derive sample complexity bounds for the MIMO H ∞ estimation, which are then combined to establish the high-probability sample complexity bounds on the model-free, trajectory-based zeroth-order policy optimization methods for structured H ∞ problem. In particular, we show that the sample complexity bounds for finding a ( δ, ϵ ) -stationary point under inexact oracle setting are on the order of Ω ( 1 δ 3 ϵ 8 ) . To the best of our knowledge, this is the first sample complexity result for model-free policy optimization on nonsmooth H ∞ synthesis.

More discussions on related work. Our paper focuses on the sample complexity of model-free policy optimization. Other than sample complexity, many other results on optimization landscape and

algorithm convergence of policy optimization for control have also been recently developed in the literature [9, 21, 79, 78, 75, 74, 16, 58, 39, 37, 38, 69, 66, 36]. See [35] for a comprehensive survey.

2 Preliminaries and Problem Formulation

2.1 Notation

Let R d denote the d -dimensional real vectors, R m × n denote the set of real matrices with dimension m × n . For a matrix A , the notations A T , σ max ( A )( ∥ A ∥ ) , ∥ A ∥ F , ρ ( A ) denote its transpose, largest singular value (spectral norm), Frobenius norm, and spectral radius, respectively. The symbol B r ( x ) denotes a closed Euclidean ball of radius r around a point x . Consider a sequence w := { w 0 , w 1 , · · · } with w t ∈ R n w for all t . This sequence belongs to ℓ n w 2 if ∑ ∞ t =0 ∥ w t ∥ 2 < ∞ where ∥ w t ∥ is the standard 2-norm of a vector. In addition, the 2-norm for w ∈ ℓ n w 2 is defined as ∥ w ∥ 2 := ∑ ∞ t =0 ∥ w t ∥ 2 . We say a function f ( x ) = O ( g ( x )) if lim sup x →∞ f ( x ) g ( x ) < ∞ , f ( x ) = Ω( g ( x )) if lim inf x →∞ f ( x ) g ( x ) > 0 , and f ( x ) = Θ( g ( x )) if f ( x ) = O ( g ( x )) and f ( x ) = Ω( g ( x )) . Given a matrix sequence { P k ∈ C n × n } k ∈ Z + , let T ( P ) denote the infinite Toeplitz matrix T ( P ) = ( P i -j ) ∞ ,i i,j =0 and T N ( P ) denote the N × N block Toeplitz matrix T N ( P ) = ( P i -j ) N -1 ,i i,j =0 .

2.2 Static output feedback H ∞ control

Consider the following linear time-invariant (LTI) system

where x t ∈ R n x is the system state, u t ∈ R n u is the control action, w t ∈ R n w is the disturbance, and y t ∈ R n y is the output measurement. We have A ∈ R n x × n x , B ∈ R n x × n u , C ∈ R n y × n x , and n w = n x . The initial condition is fixed as x 0 = 0 . We denote w := { w 0 , w 1 , · · · } 1 .

In this work, we consider the static output feedback defined as u t = -Ky t = -KCx t , where K ∈ R n u × n y is a constant matrix. Then the closed loop system is given by

The structured H ∞ synthesis with static output feedback is defined as the following minimax problem

where K is the set of all linear static output feedback stabilizing policies, i.e. K = { K ∈ R n u × n y : ρ ( A -BKC ) < 1 } and we consider the standard quadratic cost function with worst case disturbance w that satisfies the ℓ 2 norm bound ∥ w ∥ ≤ 1 . This is different than the conventional Linear Quadratic Regulator (LQR) problem where only the stochastic disturbance is considered. Throughout this paper, we also adopt the following standard assumption [21].

Assumption 2.1. There exists a K 0 ∈ K . The matrices Q and R are positive definite. The matrix C is full row rank.

In general, the problem of finding a stablizing static output feedback policy is known to be NPhard [5]. For the developments of policy optimization theory, it is standard to assume that such an initial stabilizing policy is available.

In the frequency domain, we can show that the above cost function (3) is equivalent to the H ∞ norm of the associated closed-loop transfer function [3]. Specifically, the structured H ∞ static output-feedback control problem can be formulated as

where J ( K ) is the H ∞ norm of the associated closed-loop transfer function that can be calculated as

More details on the derivation of (5) can be found in the supplementary material. We emphasize that the policy search problem (4) is a nonconvex nonsmooth optimization problem. There are two sources of nonsmoothness for the cost (5). Specifically, computing the largest singular value and taking supremum over the frequency ω ∈ [0 , 2 π ] can both be nonsmooth. Consequently, the objective function (5) can be non-differentiable over some important feasible points [2, 3, 4, 31, 12, 18].

2.3 Subgradient methods in the model-based setting

Akey concept in the nonconvex nonsmooth optimization theory is the so-called Clarke subdifferential [15, 43]. A function J : K → R is said to be locally Lipschitz if for any bounded set S ⊂ K , there exists a constant L such that | J ( K ) -J ( K ′ ) | ≤ L ∥ K -K ′ ∥ F for all K,K ′ ∈ S . It is well known that the closed-loop H ∞ cost (5) is locally Lipschitz over the set of stabilizing controllers [3]. For a locally Lipschitz function, the Clarke subdifferential exists and is defined as ∂J ( K ) := conv { lim l →∞ ∇ J ( K l ) : K l → K,K l ∈ dom( ∇ J ) ⊂ K} , where conv stands for the convex hull [15]. In the model-based setting, structural H ∞ synthesis is typically solved using advanced nonsmooth optimization algorithms that generate good descent directions via enlarging the Clarke subdifferential. Two main types of algorithms are briefly reviewed as follows.

Goldstein's subgradient method and gradient sampling. The main workhorse for the HIFOO toolbox is the gradient sampling method [11], which is developed based on the concept of Goldstein subdifferential [28]. Specifically, the Goldstein δ -subdifferential for a point K ∈ K is defined as

which implicitly requires B δ ( K ) ⊆ K . It is well known that the minimum norm element of the Goldstein subdifferential generates a good descent direction, i.e. J ( K -δH/ ∥ H ∥ F ) ≤ J ( K ) -δ ∥ H ∥ F for H being the minimum norm element of ∂ δ J ( K ) [28]. Although calculating the exact minimum norm element from the Goldstein subdifferential can be difficult, one can still estimate a good descent direction from approximating ∂ δ J ( K ) as the convex hull of randomly sampled gradients over B δ ( K ) . This leads to the gradient sampling method.

Frequency-domain methods. Another popular technique is generateing the descent directions via enlarging the Clarke subdifferential in the frequency domain [17, 54]. Such an enlargement method relies on standard chain rules to exploit the fact that the closed-loop H ∞ cost can be rewritten as a composition of a smooth mapping and a convex norm. The Matlab function Hinfstruct from the robust control package [3] is developed based on such a frequency-domain technique.

2.4 Randomized smoothing

Our analysis relies on the concept of randomized smoothing techniques, which have been widely used in convex/nonconvex optimization problems [19, 27]. The smoothed version of J ( K ) via uniform randomized smoothing is defined as below.

Definition 2.2. Consider an L -Lipschitz function J (possibly nonconvex and nonsmooth) and a uniform distribution P on { U ∈ R n u × n y : ∥ U ∥ F = 1 } . Then the smoothed version of J is defined as J δ is defined as

The above definition requires both K and K + δU to belong to the feasible set K for all U ∼ P . Very recently, [46] established the relationship between Goldstein subdifferential and uniform smoothing. We briefly restate such connections below.

Lemma 2.3. Suppose J ( K ) is L -Lipschitz over some bounded set S ⊂ K , and for any K ∈ S , we have K + δU ∈ K for all U with ∥ U ∥ F = 1 such that J δ ( K ) is well defined (see Definition 2.2). Let ∂ δ J ( K ) be the Goldstein δ -subdifferential at K , then for any K ∈ S , we have: (i) | J ( K ) -J δ ( K ) | ≤ δL , √

(ii) J δ is differentiable and L -Lipschitz with the cL d δ -Lipschtiz gradient, where d = n y n u is the problem dimension and c > 0 is a constant, (iii) ∇ J δ ( K ) ∈ ∂ δ J ( K ) .

This lemma is essentially [46, Proposition 2.3, Theorem 3.1]. Based on the definition of Goldstein δ -subdifferential (6), we say a point K is a ( δ, ϵ ) -stationary point if dist(0 , ∂ δ J ( K )) ≤ ϵ . Then Lemma 2.3 (iii) implies that if K is a ϵ -stationary point of J δ ( K ) (i.e., ∥∇ J δ ( K ) ∥ F ≤ ϵ ), then K is also a ( δ, ϵ ) -stationary point of the original function J ( K ) .

Suppose that we can compute ∇ J δ ( K ) , then we can simply apply gradient descent based on the gradient of J δ ( K )

to find a ϵ -stationary point of J δ ( K ) . However, how to choose the stepsize η and smooth radius δ to guarantee the feasibility and convergence to an ( δ, ϵ ) -stationary points of the original cost function (5) is unknown. We will discuss this setup more in Section 3.2.

2.5 Problem formulation: two zeroth-order oracle assumptions

In this paper, we aim to minimize the cost function J ( K ) in the policy space directly via derivativefree methods. Specifically, we consider two different zeroth-order oracle settings:

Exact zeroth-order oracle: This oracle assumption is standard for zeroth-order optimization literature and natural for the model-based control setting. In particular, we assume that we can exactly calculate J ( K ) (which is the closed-loop H ∞ norm) for every stabilizing K . When the system dynamics are known, such an oracle is available since the closedloop H ∞ norm can be efficiently calculated using existing robust control packages in MATLAB (currently, the state-of-the-art techniques for model-based H ∞ norm calculations rely on using the relation between the singular values of the transfer function matrix and the eigenvalues of a related Hamiltonian matrix [6, 8]). Under the same oracle setting, the non-derivative sampling method has been successfully applied to state-feedback H ∞ control problem in [32] without complexity guarantee. In this work, we close this gap by showing that, for constrained policy optimization problem (4), our algorithm returns a ( δ, ϵ ) -stationary point with high-probability feasibility/complexity bounds.

Inexact zeroth-order oracle: This oracle assumption is relevant for the model-free learningbased control setting, where the system dynamics are unknown, and J ( K ) (the closed-loop H ∞ norm) can only be estimated from the input/output data of a black-box simulator of the underlying system. In particular, we use the model-free time-reversal power-iteration-based H ∞ estimation from [70] to serve as the inexact oracle for J ( K ) . Despite the existence of such algorithms, the prior literature lacks sample complexity bounds for general MIMO systems. Therefore, we presented the first sample complexity result for H ∞ norm estimation for general MIMO systems. Building upon this, we obtain the first sample complexity results for model-free policy optimization of H ∞ control with noisy function values.

3 Main Results

In this section, we present our main result for the exact zeroth-order oracle case. We will start with the optimization landscape of the policy optimization problem (4).

3.1 Optimization landscape

Proposition 3.1. The set K = { K : ρ ( A -BKC ) < 1 } is open and nonconvex. It can be disconnected. In general, it can be either bounded or unbounded. The cost function (5) is locally Lipschitz over K and hence is continuous in K .

The proof of the above proposition can be found in [10, 22, 3]. Next, we identify the coerciveness of the cost function J ( K ) .

Lemma 3.2. The H ∞ objective function J ( K ) defined by (5) is coercive over the set K in the sense that for any sequence { K l } ∞ l =1 ⊂ K we have

if either ∥ K l ∥ F → + ∞ , or K l converges to an element in the boundary ∂ K .

It is worth mentioning that the above lemma relies on the assumption that both Q and R are positive definite. The formal proof of Lemma 3.2 is deferred to the supplementary material. The continuity and coerciveness of J ( K ) directly lead to the following results.

Lemma 3.3. Consider the H ∞ output feedback policy search problem (4) with the objective function J ( K ) defined in (5) . Under Assumption 2.1, for any γ > J ∗ , the sublevel set defined as S γ := { K ∈ K : J ( K ) ≤ γ } is compact.

Remark 3.4 . Based on Lemma 3.3 and the fact that the set K is open, we can show that there is a strict separation between the sublevel set S γ and ∂ K . It is obvious that ∂ K is closed. Since S γ is compact and S γ ∩ ∂ K = ∅ , we have dist( S γ , ∂ K ) > 0 .

3.2 Warm-up: convergence of smoothed function

In this section, we assume that the exact gradient oracle ∇ J δ ( K ) is available for each K ∈ K such that J δ ( K ) is well defined. Then we analyze the finite-time convergence and feasibility of the vanilla gradient descent method (8). To this end, let K 0 ∈ K be an arbitrary feasible initial controller. Without loss of generality, define two feasible sets:

Denote ∆ 0 = dist( S 0 , ∂ K ) and ∆ 1 = dist( S 1 , ∂ K ) . Let L 0 and L 1 be the Lipschitz constants of J ( K ) associated with the sublevel set S 0 and S 1 , respectively. It is obvious that ∆ 0 ≥ ∆ 1 and L 0 ≤ L 1 . In addition, we have ∆ = dist( S 0 , ∂ S 1 ) > 0 by Remark 3.4.

Remark 3.5 . Since J ( K ) is a continuous function of K by Proposition 3.1 and the sublevel sets S 0 and S 1 are compact by Lemma 3.3, this implies that there exists a constant ξ > 0 such that for any K ∈ S 0 , and K ′ with ∥ K -K ′ ∥ F ≤ ξ , we have K ′ ∈ S 1 .

Now we are ready to state the following result.

Theorem 3.6. Let K 0 ∈ K be an arbitrary feasible initial controller and suppose that J δ ( K ) defined as in (7) is L 1 -Lipschitz with the cL 1 √ d δ -Lipschtiz gradient on the sublevel set S 1 . Choose δ = min { ∆ 1 , 49 J ( K 0 ) 2 L 1 } and η = min { ξ L 1 , δ cL 1 √ d } . Then the iterative method (8) stays in K and we have:

In other words, we have min 0 ≤ t ≤ T -1 ∥∇ J δ ( K t ) ∥ F ≤ ϵ after T = O ( 1 ϵ 2 ) .

The proof of Theorem 3.6 can be found in the supplementary material. Since we have a constrained optimization problem, it is crucial to guarantee the feasibility of the gradient descent method (8). The careful choice of smooth radius δ and step size η ensures that K t stays inside the feasible set.

3.3 Analysis for the exact oracle case

In this section, we consider the exact zeroth-order oracle setup, where we can obtain the exact H ∞ norm of the closed-loop system for a given K ∈ K . One straightforward extension of (8) is to compute an unbiased estimation of the gradient ∇ J δ ( K t ) via the zeroth-order oracle. Then we can perform the one-step gradient descent. In particular, we analyze the feasibility and convergence of the Algorithm 1 in the following result.

0

Theorem 3.7. Let K ∈ K be an arbitrary feasible initial controller and suppose J ( K ) is L 1 -Lipschtiz with the cL 1 √ d δ -Lipschtiz gradient on the sublevel set S 1 . Let 1 ≤ υ ≤ 80 and suppose we choose

Then the following two statements hold:

(1) the controllers { K t } T -1 t =0 generated by Algorithm 1 are all stabilizing with probability at least 0 . 95 -0 . 01 υ .

(2) the output of Algorithm 1 K R satisfies

with probability at least 0 . 87 -0 . 17 υ -1 2 -0 . 01 υ .

The proof of Theorem 3.7 can be found in the supplementary material. Now we provide some discussions regarding Theorem 3.7 in order:

Probability Bounds: Statement 1 suggests that as T increases, the probability that all the generated controllers are stabilizing will decrease. This is because our algorithm uses a zeroth-order oracle to build an estimator of the smoothed function gradient. As T increases, the biases and variance of the gradient estimation accumulate, resulting in a larger failure probability. In addition, Statement 2 suggests that as T increases, the probability of finding a ( δ, ϵ ) -stationary point will first increase and then decrease. Indeed, when T is too small, more iterations will improve the performance of the generated controllers, while for large T , the probability of generating unstable controllers becomes dominant. In addition, the constant factors in the probability bounds are not restrictive, they can be further improved by e.g., increasing the level of S 0 , using smaller step size η or using smaller smooth radius δ in the analysis.

Feasibility: Statement 1 states that all the output controllers { K t } T -1 t =0 are feasible with high probability under the choice of algorithm parameters. It is worth mentioning that, to ensure the feasibility of the iterates { K t } T -1 t =0 , we need to show that { J δ ( K t ) } T -1 t =0 are well defined so that we can apply the smoothness property of J δ ( K ) in our theoretical analysis. Furthermore, this also guarantees that { J ( K t ± δW t ) } T -1 t =0 used in Algorithm 1 are well defined. This turns out to be a nontrivial task and one needs to design algorithm parameters carefully to resolve the feasibility issue. More discussions will be provided in the end of this section.

Output Controller: Since our cost function J ( K ) is nonconvex, we consider an output controller K R uniformly sampled from { K t } T -1 t =0 . Such random output technique has been used in smooth/nonsmooth nonconvex optimization problems for theoretical analysis [27, 46, 44]. In particular, Statement 2 implies that the output controller K R is a ( δ, ϵ ) -stationary point with high probability. In practical implementation, one can just select the iterate that gives the lowest cost J ( K t ) for t = 1 , · · · , T . Our numerical experiments suggest that selecting K T often yields satisfactory performance (see Section 5).

Sample Complexity: For sufficiently small ϵ , the number of iterations to guarantee (11) is given by:

where d ≥ 1 is the problem dimension, L 1 is the Lipschtiz parameter. We also ignore the numerical constants since they are conservative and not restrictive.

Finally, we close this section by highlighting the main technical contributions of Theorem 3.7 and 3.6. In our control setup, unlike the unconstrained optimization problems [46], we need to ensure that the iterate K t and the perturbed iterate K t ± δW t stay within a non-convex feasible set K . Previous work on policy optimization theory of H ∞ control addresses this feasibility issue via using the coerciveness of J ( K ) and mainly relies on the fact that J ( K ) is a barrier function on the non-convex set of stabilizing policies [32]. In particular, such previous results rely on model-based algorithms (such as Goldstein's subgradient method) which can decrease the value of J ( K ) directly. However, the zeroth-order policy search can only decrease the value of the smoothed function J δ ( K ) , which is not coercive over the non-convex feasible set and hence cannot be used as a barrier function. Importantly, the descent of J δ ( K ) does not imply the descent of the original function value and hence cannot ensure feasibility by itself. In Theorem 3.7 and 3.6, by carefully choosing the smooth radius

δ and step size η , we manage to show that the iterate K t and the perturbed iterate K t ± δW t stay within a non-convex feasible set with high probability.

4 Sample Complexity for the Model-Free Case

In this section, we assume that the system dynamics in (1) are unknown and we can only use samples to obtain an estimation of the cost function ˆ J ( K ) with some error ˆ J ( K ) = J ( K ) + ζ ( K ) . To obtain an end-to-end sample complexity result of Algorithm 1, we first derive the sample complexity of H ∞ norm estimation for the general MIMO systems such that ζ ( K ) ≤ κ for all K ∈ K .

4.1 Complexity of H ∞ norm estimation for MIMO systems

The H ∞ norm estimation method via input/output data can be roughly categorized into two major approaches: (a) try to find the worst case ℓ 2 -norm signal using power-iteration algorithm [71, 53]. (b) discretizing the interval [0 , 2 π ] and search for the maximizing frequency using multi-armed bandit [51]. However, there are no sample complexity results of the aforementioned methods for the general MIMO system. In this section, we analyze the H ∞ norm estimation method proposed in [53] and establish its sample complexity bounds for the general MIMO systems.

To this end, let P ( z ) = ( Q + C T K T RKC ) 1 / 2 ( zI -A + BKC ) -1 be the transfer function associated with the system (1) to the cost function (3), let T ( P ) denote the corresponding Toeplitz (convolution) operator and T N ( P ) be the N × N upper-left submatrix of T ( P ) . Then it is known that J ( K ) = ∥ T ( P ) ∥ and ∥ T N ( P ) ∥ → ∥ T ( P ) ∥ as N →∞ . With large enough N , the largest singular value of the T N ( P ) can be used as a reasonable estimation of J ( K ) , which can be approximated by running the power iteration algorithm n steps 2 . Therefore, there are two error terms based on this approach. The first term ζ 1 ( K ) comes from the difference between ∥ T N ( P ) ∥ and ∥ T ( P ) ∥ , and the second term ζ 2 ( K ) is induced by computing the largest singular value of T N ( P ) via the power iteration algorithm:

where ˆ J ( K ) is the approximated largest singular value of T N ( P ) and hence an estimation of J ( K ) . Therefore, if we can choose N and n large enough such that ζ 1 ( K ) ≤ κ/ 2 and ζ 2 ( K ) ≤ κ/ 2 , we will have ζ ( K ) ≤ κ holds by (13). The finite-time condition on N has been established in [68] for single input single output (SISO) systems only. In the following result, we extend the result in [68] to the general MIMO system. Furthermore, we provide the finite-time condition on the number of power iteration iterations n .

Theorem 4.1. Let P ( z ) = ∑ ∞ k =0 P k z -k be the corresponding transfer function of system (1) to the cost function (3) with stability radius ι ∈ (0 , 1) . And choose γ ∈ ( ι, 1) and D ≥ σ max ( P 0 ) . Let ∥ P γ ∥ ∞ denotes the H ∞ norm of the system P γ ( z ) := γP ( γz ) . Then for N ≥ 3 , we have

are errors due to approximating T by T N and using power iteration to estimate ∥ T N ∥ , respectively. Here C 1 = 3 √ 2 π (2 + 3 π 4 ) , C 2 = 9 √ 2 π 2 are universal constants and C 3 depends on an angle between power iteration initialization and the eigenvector corresponding to the biggest eigenvalue of T ∗ N ( P ) T N ( P ) .

The proof of Theorem 4.1 can be found in the supplementary material. Theorem 4.1 implies that the approximation error | ζ ( K ) | ≤ κ is guaranteed by choosing

where

4.2 Sample complexity of zeroth-order optimization

In this section, we consider that Algorithm 1 can only access the inexact zeroth-order oracle of the cost function J ( K ) . In this case, we do not have an unbiased estimation of ∇ J δ ( K t ) . Nevertheless, we can make the estimation as small as possible and assume that the estimation error ζ ( K ) is uniformly upper bounded for all K ∈ S 1 :

Clearly, the smaller κ is, the larger power iteration number n and approximation horizon N are needed. We will specify the choice of κ in the next result, where we obtain the feasibility/sample complexity bounds of Algorithm 1 via inexact zeroth-order oracle on finding a ( δ, ϵ ) -stationary point.

Theorem 4.2. Let K 0 ∈ K be an arbitrary feasible initial controller and suppose J is L 1 -Lipschtiz on the sublevel set S 1 . Consider the Algorithm 1 with inexact zeroth-order oracle and suppose that the estimation error κ ≤ δϵ 2 100 dL 1 . Denote Γ := cL 1 √ d (16 √ 2 πdL 2 1 +( dκ δ ) 2 ) . Let 1 ≤ υ ≤ 25 and suppose we choose

Then the following two statements hold:

(1) the controllers { K t } T -1 t =0 generated by Algorithm 1 with inexact zeroth-order oracle are all stabilizing with probability at least 0 . 95 -0 . 03 υ . R

(2) the output of Algorithm 1 K satisfies

with probability at least 0 . 8 -0 . 15 υ -1 2 -0 . 03 υ .

The proof of Theorem 4.2 can be found in the supplementary material. For sufficiently small ϵ , we can obtain the following lower bound on the number of iterations to guarantee (19):

Since we require κ ≤ δϵ 2 100 dL 1 , we have the following lower bounds on the power iteration number n and approximation horizon N by (17):

Finally, by combining the above sample complexity bounds for MIMO H ∞ estimation with (20), the number of samples to guarantee (19) with high probability is given by:

In this section, we consider the sample complexity of Algorithm 1 with the inexact oracle case. This is particularly relevant for the model-free control setting. Specifically, we are using imperfect estimates of J ( K ) that are calculated using the model-free MIMO power iteration method. Therefore, an extra statistical error term appears in the iterations of zeroth-order policy optimization and requires special treatment. Such an extra term has not been considered in the literature on zeroth-order optimization for nonconvex nonsmooth problems. To address this extra technical difficulty, we first establish sample complexity bounds for H ∞ norm estimation of the general MIMO system (Theorem 4.1). Then we carefully propagate such sample complexity bounds to obtain an error bound for ∇ J δ ( K ) in terms of ϵ and δ . Built upon this, we demonstrate that Algorithm 1 remains effective even with an inexact oracle, ensuring the feasibility of the iterates while achieving finite-time sample complexity with high probability (Theorem 4.2).

5 Numerical Experiments

In this section, we present the numerical study to show the effectiveness of Algorithm 1. The left plot of Figure 1 displays the relative error trajectories of Algorithm 1 with an exact oracle for system dimensions n x = { 10 , 50 , 100 } . The system is of form (1) with parameters ( A,B,C ) , where the entries of ( A,B,C ) are sampled from a standard normal distribution N (0 , 1) . The results show that Algorithm 1 performs well even for larger-scale systems. In the middle plot, we compare the trajectories of Algorithm 1 with both exact and inexact zeroth-order oracle. It can be observed that the inexact oracle case closely tracks the performance of the exact oracle case. The right plot illustrates the zeroth-order complexity of the exact oracle case with varying ϵ . As shown, the number of oracle calls increases as ϵ decreases. It is important to note that the complexity bounds derived in (12) are not tight. We conjecture that by leveraging other advanced properties of the cost function (5), further improvements in complexity can be achieved.

10 -2

10 -1

10 0

10 1

10 1

10 2

10 3

100

10

Exact Oracle

(Y)

8

Relative Error

10-1

6

0

50

100

150

200

10-2

10

InexactOralce

n=50

n=10

8

n=100

6

10-3

0

2000

4000

Iterations

6000

8000

10000

0

50

100

150

200

Iterations

In addition, we conducted a comparison of our derivative-free method with the model-based methods HIFOO and Hinfstruct, using several benchmark examples from COMPl e ib [41]. Table 1 shows the corresponding optimal closed-loop H ∞ norm. It can be seen that our derivative-free method yields comparable results with the model-based packages even without the knowledge of system models. More details about the numerical experiments can be found in the supplementary material.

6 Conclusion and Future Work

This paper presents the feasibility and complexity bounds for the derivative-free policy optimization method on the structured H ∞ synthesis, considering both exact and inexact zeroth-order oracles. Despite the fact that this structured H ∞ synthesis is a constrained nonconvex nonsmooth optimization problem, we leverage the intriguing connections between randomized smoothing and ( δ, ϵ ) -stationarity, enabling the first sample analysis for model-free, trajectory-based, zeroth-order policy optimization in structured H ∞ synthesis. One limitation of this work is the uncertain tightness of the sample complexity bounds obtained.

For future studies, it is important to tighten the sample complexity results by exploring more properties of H ∞ control problems. In addition, it is interesting to explore the behavior of the structured H ∞ problem with a dynamic output feedback controller.

References

C. Antoniades and P. Christofides. Integrating nonlinear output feedback control and optimal actuator/sensor placement for transport-reaction processes. Chemical Engineering Science , 56(15):4517-4535, 2001.

P. Apkarian and D. Noll. Controller design via nonsmooth multidirectional search. SIAM Journal on Control and Optimization , 44(6):1923-1949, 2006.

P. Apkarian and D. Noll. Nonsmooth H ∞ synthesis. IEEE Transactions on Automatic Control , 51(1):71-86, 2006.

D. Arzelier, G. Deaconu, S. Gumussoy, and D. Henrion. H 2 for HIFOO. In International Conference on Control and Optimization With Industrial Applications , 2011.

V. Blondel and J. Tsitsiklis. Np-hardness of some linear control design problems. SIAM journal on control and optimization , 35(6):2118-2127, 1997.

S. Boyd, V. Balakrishnan, and P. Kabamba. A bisection method for computing the h ∞ norm of a transfer matrix and related problems. Mathematics of Control, Signals and Systems , 2(3):207-219, 1989.

S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan. Linear Matrix Inequalities in System and Control Theory , volume 15. SIAM, 1994.

N. Bruinsma and M. Steinbuch. A fast algorithm to compute the h ∞ -norm of a transfer function matrix. Systems & Control Letters , 14(4):287-293, 1990.

J. Bu, A. Mesbahi, M. Fazel, and M. Mesbahi. LQR through the lens of first order methods: Discrete-time case. arXiv preprint arXiv:1907.08921 , 2019.

J. Bu, A. Mesbahi, and M. Mesbahi. On topological and metrical properties of stabilizing feedback gains: the MIMO case. arXiv preprint arXiv:1904.02737 , 2019.

J. Burke, D. Henrion, A. Lewis, and M. Overton. HIFOO - A MATLAB package for fixed-order controller design and H ∞ optimization. IFAC Proceedings Volumes , 39(9):339-344, 2006.

J. V. Burke, F. E. Curtis, A. S. Lewis, M. L. Overton, and L. E. Simões. Gradient sampling methods for nonsmooth optimization. Numerical Nonsmooth Optimization , pages 201-225, 2020.

J. V. Burke, A. S. Lewis, and M. L. Overton. A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM Journal on Optimization , 15(3):751-779, 2005.

P. Christofides and A. Armaou. Global stabilization of the kuramoto-sivashinsky equation via distributed output feedback control. Systems & Control Letters , 39(4):283-294, 2000.

F. H. Clarke. Optimization and nonsmooth analysis . SIAM, 1990.

L. Cui and L. Molu. Mixed H 2 / H ∞ LQ games for robust policy optimization under unknown dynamics. arXiv preprint arXiv:2209.04477 , 2022.

J. Cullum, W. Donath, and P. Wolfe. The minimization of certain nondifferentiable sums of eigenvalues of symmetric matrices. Nondifferentiable optimization , pages 35-55, 1975.

F. E. Curtis, T. Mitchell, and M. L. Overton. A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles. Optimization Methods and Software , 32(1):148-181, 2017.

J. Duchi, P. Bartlett, and M. Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization , 22(2):674-701, 2012.

G. Dullerud and F. Paganini. A Course in Robust Control Theory: A Convex Approach . Springer, 1999.

I. Fatkhullin and B. Polyak. Optimizing static linear feedback: Gradient method. SIAM Journal on Control and Optimization , 59(5):3887-3911, 2021.

M. Fazel, R. Ge, S. Kakade, and M. Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 1467-1476, 2018.

F. Fonte, S. Ricci, and P. Mantegazza. Gust load alleviation for a regional aircraft through a static output feedback. Journal of Aircraft , 52(5):1559-1574, 2015.

L. Furieri, Y. Zheng, and M. Kamgarpour. Learning the globally optimal distributed LQ regulator. In Learning for Dynamics and Control , pages 287-297, 2020.

J. Gadewadikar, F. Lewis, K. Subbarao, K. Peng, and B. Chen. H-infinity static output-feedback control for rotorcraft. Journal of Intelligent and Robotic Systems , 54:629-646, 2009.

P. Gahinet and P. Apkarian. A linear matrix inequality approach to H ∞ control. International Journal of Robust and Nonlinear Control , 4:421-448, 1994.

S. Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

A. Goldstein. Optimization of lipschitz continuous functions. Mathematical Programming , 13(1):14-22, 1977.

G. Golub and C. Van Loan. Matrix computations . JHU press, 2013.

B. Gravell, P. M. Esfahani, and T. Summers. Learning optimal controllers for linear systems with multiplicative noise via policy gradient. IEEE Transactions on Automatic Control , 66(11):5283-5298, 2020.

S. Gumussoy, D. Henrion, M. Millstone, and M. L. Overton. Multiobjective robust control with HIFOO 2.0. IFAC Proceedings Volumes , 42(6):144-149, 2009.

X. Guo and B. Hu. Global convergence of direct policy search for state-feedback H ∞ robust control: A revisit of nonsmooth synthesis with goldstein subdifferential. In Conference on Neural Information Processing Systems , 2022.

B. Hambly, R. Xu, and H. Yang. Policy gradient methods for the noisy linear quadratic regulator over a finite horizon. SIAM Journal on Control and Optimization , 59(5):3359-3391, 2021.

K. Hengster-Movric, F. Lewis, and M. Sebek. Distributed static output-feedback control for state synchronization in networks of identical lti systems. Automatica , 53:282-290, 2015.

B. Hu, K. Zhang, N. Li, M. Mesbahi, M. Fazel, and T. Ba¸ sar. Towards a theoretical foundation of policy optimization for learning control policies. arXiv preprint arXiv:2210.04810 , 2022.

B. Hu and Y. Zheng. Connectivity of the feasible and sublevel sets of dynamic output feedback control with robustness constraints. IEEE Control Systems Letters , 7:442-447, 2022.

J. P. Jansch-Porto, B. Hu, and G. Dullerud. Policy learning of MDPs with mixed continuous/discrete variables: A case study on model-free control of Markovian jump systems. In Learning for Dynamics and Control , pages 947-957, 2020.

J. P. Jansch-Porto, B. Hu, and G. Dullerud. Policy optimization for Markovian jump linear quadratic control: Gradient-based methods and global convergence. IEEE Transactions on Automatic Control , 2020.

J. P. Jansch-Porto, B. Hu, and G. E. Dullerud. Convergence guarantees of policy optimization methods for Markovian jump linear systems. In American Control Conference , pages 2882-2887, 2020.

K. C. Kiwiel. Convergence of the gradient sampling algorithm for nonsmooth nonconvex optimization. SIAM Journal on Optimization , 18(2):379-388, 2007.

F. Leibfritz and W. Lipinski. COMPleib 1.0-user manual and quick reference, 2004.

W. Levine and M. Athans. On the determination of the optimal constant output feedback gains for linear multivariable systems. IEEE Transactions on Automatic control , 15(1):44-48, 1970.

J. Li, A. So, and W. Ma. Understanding notions of stationarity in nonsmooth optimization: A guided tour of various constructions of subdifferential for nonsmooth functions. IEEE Signal Processing Magazine , 37(5):18-31, 2020.

Y. Li, Y. Tang, R. Zhang, and N. Li. Distributed reinforcement learning for decentralized linear quadratic control: A derivative-free policy optimization approach. IEEE Transactions on Automatic Control , 2021.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

T. Lin, Z. Zheng, and M. Jordan. Gradient-free methods for deterministic and stochastic nonsmooth nonconvex optimization. In Conference on Neural Information Processing Systems , 2022.

D. Malik, A. Pananjady, K. Bhatia, K. Khamaru, P. Bartlett, and M. Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In International Conference on Artificial Intelligence and Statistics , pages 2916-2925, 2019.

V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, M. Bellemare, A. Graves, M. Riedmiller, A. Fidjeland, G. Ostrovski, and S. Petersen. Human-level control through deep reinforcement learning. Nature , 518:529-533, 2015.

D. Moerder and A. Calise. Convergence of a numerical algorithm for calculating optimal output feedback gains. IEEE Transactions on Automatic Control , 30(9):900-903, 1985.

H. Mohammadi, A. Zare, M. Soltanolkotabi, and M. R. Jovanovic. Convergence and sample complexity of gradient methods for the model-free linear quadratic regulator problem. IEEE Transactions on Automatic Control , 2021.

M. I. Müller, P. E. Valenzuela, A. Proutiere, and C. R. Rojas. A stochastic multi-armed bandit approach to nonparametric H ∞ -norm estimation. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC) , pages 4632-4637. IEEE, 2017.

D. Noll and P. Apkarian. Spectral bundle methods for non-convex maximum eigenvalue functions: second-order methods. Mathematical Programming , 104(2):729-747, 2005.

T. Oomen, R. van der Maas, C. R. Rojas, and H. Hjalmarsson. Iteratively learning the H ∞ -norm of multivariable systems applied to model-error-modeling of a vibration isolation system. In 2013 American Control Conference , pages 6703-6708. IEEE, 2013.

F. Oustry. A second-order bundle method to minimize the maximum eigenvalue function. Mathematical Programming , 89:1-33, 2000.

S. Oymak and N. Ozay. Non-asymptotic identification of lti systems from a single trajectory. In 2019 American control conference (ACC) , pages 5655-5661. IEEE, 2019.

I. K. Ozaslan, H. Mohammadi, and M. R. Jovanovi´ c. Computing stabilizing feedback gains via a model-free policy gradient method. IEEE Control Systems Letters , 7:407-412, 2022.

J. Perdomo, J. Umenberger, and M. Simchowitz. Stabilizing dynamical systems via policy gradient methods. Advances in Neural Information Processing Systems , 34, 2021.

G. Qu, C. Yu, S. Low, and A. Wierman. Exploiting linear models for model-free nonlinear control: A provably convergent policy gradient approach. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 6539-6546, 2021.

S. Rathod, M. Bhadu, and A. De. Global convergence using policy gradient methods for model-free Markovian jump linear quadratic control. arXiv preprint arXiv:2111.15228 , 2021.

T. Rautert and E. W. Sachs. Computational design of optimal output feedback controllers. SIAM Journal on Optimization , 7(3):837-852, 1997.

T. Sarkar and A. Rakhlin. Near optimal finite time identification of arbitrary linear dynamical systems. In International Conference on Machine Learning , pages 5610-5618, 2019.

C. Scherer, P. Gahinet, and M. Chilali. Multiobjective output-feedback control via lmi optimization. IEEE Transactions on automatic control , 42(7):896-911, 1997.

C. Scherer and S. Wieland. Linear matrix inequalities in control. Lecture notes for a course of the dutch institute of systems and control, Delft University of Technology, 2004.

J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of Go without human knowledge. Nature , 550(7676):354-359, 2017.

Y. Tang and Y. Zheng. On the global optimality of direct policy search for nonsmooth h ∞ output-feedback control. arXiv preprint arXiv:2304.00753 , 2023.

H. Toivonen. A globally convergent algorithm for the optimal constant output feedback problem. International Journal of Control , 41(6):1589-1599, 1985.

S. Tu, R. Boczar, and B. Recht. On the approximation of Toeplitz operators for nonparametric H ∞ -norm estimation. In 2018 Annual American Control Conference (ACC) , pages 1867-1872. IEEE, 2018.

J. Umenberger, M. Simchowitz, J. C. Perdomo, K. Zhang, and R. Tedrake. Globally convergent policy search over dynamic filters for output estimation. arXiv preprint arXiv:2202.11659 , 2022.

B. Wahlberg, M. Syberg, and H. Hjalmarsson. Non-parametric methods for l 2 -gain estimation using iterative experiments. Automatica , 46(8):1376-1381, 2010.

B. Wahlberg, M. B. Syberg, and H. Hjalmarsson. Non-parametric methods for l 2 -gain estimation using iterative experiments. Automatica , 46(8):1376-1381, 2010.

Z. Yang, Y. Chen, M. Hong, and Z. Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems , 32, 2019.

H. Zhang, Y. Shi, and A. Mehr. Robust static output feedback control and remote pid design for networked motor systems. IEEE Transactions on Industrial Electronics , 58(12):5396-5405, 2011.

K. Zhang, B. Hu, and T. Ba¸ sar. On the stability and convergence of robust adversarial reinforcement learning: A case study on linear quadratic systems. Advances in Neural Information Processing Systems , 33, 2020.

K. Zhang, B. Hu, and T. Basar. Policy optimization for H 2 linear control with H ∞ robustness guarantee: Implicit regularization and global convergence. SIAM Journal on Control and Optimization , 59(6):40814109, 2021.

K. Zhang, X. Zhang, B. Hu, and T. Ba¸ sar. Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

F. Zhao and K. You. Primal-dual learning for the model-free risk-constrained linear quadratic regulator. In Learning for Dynamics and Control , pages 702-714, 2021.

Y. Zheng, Y. Sun, M. Fazel, and N. Li. Escaping high-order saddles in policy optimization for linear quadratic gaussian (lqg) control. In IEEE 61st Conference on Decision and Control (CDC) , pages 5329-5334, 2022.

Y. Zheng, Y. Tang, and N. Li. Analysis of the optimization landscape of linear quadratic gaussian (LQG) control. arXiv preprint arXiv:2102.04393 , 2021.

K. Zhou, J. Doyle, and K. Glover. Robust and Optimal Control . Prentice-Hall, 1996.

Supplementary Material

A More on Problem Formulation

A.1 Derivation of the cost function in frequency domain

The goal of the structured H ∞ synthesis with static output feedback is to find the optimal controller gain K ∗ to solve the following minimax problem:

Now if we define z t = ( Q + C T K T RKC ) 1 2 x t , we have ∥ z t ∥ 2 = x T t ( Q + C T K T RKC ) x t = x T t Qx t + u T t Ru t . Then the closed-loop LTI system (2) can be viewed as a linear operator G K mapping any disturbance sequence { w t } to another sequence { z t } . If K is stabilizing, i.e. ρ ( A -BKC ) < 1 , then G K is bounded in the sense that it maps any ℓ 2 sequence w to another sequence z in ℓ n x 2 . For any stabilizing K , the ℓ 2 → ℓ 2 induced norm of G K can be defined as:

̸

Since G K is a linear operator, it is straightforward to show

Therefore, the minimax optimization problem (A.1) can be rewritten as the policy optimization problem: min K ∈K ∥ G K ∥ 2 2 → 2 . We can just drop the square in the cost function and reformulate (A.1) as min K ∈K ∥ G K ∥ 2 → 2 . This is exactly the policy optimization formulation for structured H ∞ with static output-feedback control. In the frequency domain, G K can be viewed as a transfer function that maps signal w t to z t . Applying the frequency-domain formula for the H ∞ norm, we can calculate ∥ G K ∥ 2 → 2 as

which is the same as we defined in (5).

A.2 LTI system with measurement noise

In this section, we extend the LTI system (1) considered in the main paper to the one with measurement noise as below:

with n -1 = 0 . We denote n := { n -1 , n 0 , · · · } . We will show that the problem with measurement noise falls within our framework via a proper state augmentation trick. In particular, it is obvious that (A.4) can be rewritten as:

In this setup, the control inputs become

and the closed-loop system can be represented as

Then, the min-max cost function can be written as

If we define z t = [ Q + C T K T RKC C T K T RK K T RKC K T RK ] 1 2 [ x t n t -1 ] , in frequency domain, we can rewrite the above cost function as

We can slightly modify our proof of Lemma 3.2 to show that (A.5) remains coercive. Therefore, all the results presented in the paper follow.

B Detailed Proofs

B.1 Proof of Lemma 3.2

Suppose we have a sequence { K l } satisfying ∥ K l ∥ F → + ∞ . Let w l = { w l 0 , 0 , 0 , · · · } with ∥ w l 0 ∥ = 1 such that σ max ( K l C ) = ∥ K l Cw l 0 ∥ . Then we have:

where inequality ( i ) holds since we plugged into a specific w l over the max operation and the matrix Q + ( K l C ) T RK l C is positive definite. Inequality ( ii ) uses the fact that R ≥ λ min ( R ) I , where λ min ( R ) is the minimum eigenvalue of R . Since ∥ K l ∥ F → + ∞ by equivalence of norms σ max ( K l ) → + ∞ , and knowing C is a full row rank matrix guarantees that λ min ( CC T ) > 0 , therefore J ( K l ) → + ∞ as ∥ K l ∥ F → + ∞ .

Next, we assume K l → K where K is on the boundary ∂ K . Clearly we have ρ ( A -BKC ) = 1 . We will use a frequency-domain argument to prove J ( K l ) → + ∞ . Since ρ ( A -BKC ) = 1 , there exists some ω 0 such that the matrix ( e jω 0 I -A + BKC ) becomes singular. Obviously, for the same ω 0 , the matrix ( e jω 0 I -A + BKC ) is also singular. Therefore, we have:

Clearly, the above argument relies on the fact that Q is positive definite and Q ≥ λ min ( Q ) I . Notice that we have ρ ( A -BK l C ) < 1 for each l , and hence we have σ min ( ( e jω 0 I -A + BK l C ) ) > 0 ,

i.e. the smallest singular values of ( e jω 0 I -A + BK l C ) are strictly positive for all l . By the continuity of the σ min ( · ) , we must have σ min ( ( e jω 0 I -A + BK l C ) ) → 0 as lim l →∞ K l ∈ ∂ K . Hence we have σ max ( ( e jω 0 I -A + BK l C ) -1 ) → + ∞ as l → ∞ . Therefore, we have J ( K l ) → + ∞ as K l → K ∈ ∂ K . This completes the proof.

B.2 Proof of Lemma 3.3

We first prove Lemma 3.3. Since J is continuous, for any γ > J ∗ , we know S γ = { K ∈ K : J ( K ) ≤ γ } is a closed set. It remains to show S γ is bounded. Suppose that S γ is unbounded. Then there exists a sequence { K l } ∞ l =1 ⊂ S such that ∥ K l ∥ 2 → + ∞ as l →∞ . But by coerciveness of J ( K ) , we must have J ( K l ) → + ∞ as well, which contradicts that J ( K l ) ≤ γ for all l . Hence S γ is bounded. Therefore, S γ is compact.

B.3 Proof of Theorem 3.6

We first use induction to show that the iterative method (8) stays in K for all t . Since K 0 ∈ S 0 and δ ≤ ∆ 1 , we know that J δ ( K 0 ) is well defined. In addition, we have:

where we use the fact that J δ is L 1 -Lipschitz on the sublevel set S 1 and η ≤ ξ L 1 . Therefore we have K 1 ∈ S 1 by Remark 3.5 and J δ ( K 1 ) is well defined. Since J δ is L 1 -Lipschitz with the cL 1 √ d δ -Lipschtiz gradient, we have:

where the last inequality holds since we have 0 < η = min { ξ L 1 , δ cL 1 √ d } . Therefore, we have J δ ( K 1 ) ≤ J δ ( K 0 ) . This implies:

Hence we have K 1 ∈ S 0 . Repeating this argument, leads to the fact that K t ∈ S 0 for all t . Thus:

Rearranging terms of the above inequality yields:

Summing up the above inequality over t = 0 , 1 , · · · , T -1 gives:

where we use the inequality J δ ( K T ) ≥ J ∗ by definition of J δ ( K ) . Therefore, we have:

Therefore, we have:

and the sample complexity result follows. This completes the proof.

B.4 Proof of Theorem 3.7

We first provide several technical lemmas which will be useful for proving Theorem 3.7. To this end, let F t denote the filtration σ ( K s , s ≤ t ) for each t = 1 , 2 , · · · , T .

Lemma B.1. Suppose J δ ( K ) ≤ 49 J ( K 0 ) and δ ≤ J ( K 0 ) L 1 . Then we have K ∈ S 0 .

Proof. Since J δ ( K ) is well defined, we must have K ∈ K by definition of J δ . Then Proposition 2.3 (i) implies that J ( K ) ≤ J δ ( K ) + δL 1 ≤ 49 J ( K 0 ) + J ( K 0 ) L 1 L 1 = 50 J ( K 0 ) . Hence we have K ∈ S 0 .

Lemma B.2. Suppose that J ( K ) is L 1 -Lipschitz on the sublevel set S 1 and let { g t } T -1 t =0 and { K t } T -1 t =0 be generated by Algorithm 1 such that { K t } T -1 t =0 are feasible and { J δ ( K t ) } T -1 t =0 are well defined. Then, we have

The proof can be found in [46, Lemma D.1]. We omit it here.

Proof.

Lemma B.3. J ( K ) is L 0 -Lipschitz on the sublevel set S 0 , let η ≤ 2 δξ d (100 J ( K 0 ) -J ∗ ) and δ ≤ min { ∆ 1 , ∆ } , then as long as K t ∈ S 0 , we will have K t +1 ∈ S 1 and

where Z = η (8 √ 2 π ) cd 3 / 2 L 3 1 δ -1 .

Proof. Since ∥ W t ∥ F = 1 , we have:

where the second inequality holds since we have K t ± δW t ∈ S 1 when δ ≤ ∆ . This implies that K t +1 ∈ S 1 by Remark 3.5. In addition, K t +1 ∈ S 1 and δ ≤ ∆ 1 ensures that J δ ( K t +1 ) is well defined. By Proposition 2.3 (ii), we know that J δ ( K ) is differentiable and L 1 -Lipschitz with cL 1 √ d δ -Lipschitz gradient on the sublevel set S 1 . Then we have:

Taking the expectation on both sides conditioned on F t and using Lemma B.2 yields

with Z = η (8 √ 2 π ) cd 3 / 2 L 3 1 δ -1 . This completes the proof.

Now we are ready to prove the Theorem 3.7. We will first prove Statement 1 in Theorem 3.7: all the generated controllers are within the feasible set with high probability. Then we will show Statement 2: the Algorithm 1 returns a ( δ, ϵ ) -stationary point with high probability.

Proof of Statement 1 We first define a stopping time τ as below:

Based on Lemma B.1, it can be seen that as long as τ ≥ T -1 , the iterates { K t } T -1 t =0 generated by Algorithm 1 are feasible. Therefore our goal becomes bounding the probability Pr( τ ≤ T -1) . To this end, we can define a nonnegative supermartingale Y ( t ) as below

To show it is a supermartingale, noticing that we have

where in equality (i), J δ ( K t +1 ) is well defined by the definition of the τ ; in particular, we have J δ ( K t ) ≤ 49 J ( K 0 ) , hence K t ∈ S 0 by Lemma B.1 and we can apply Lemma B.3 to obtain inequality (ii). Then Doob's maximal inequality for super-martingales gives

where inequality (i) uses Lemma 2.3 and (ii) is true since δ ≤ J ( K 0 ) L 1 . For sufficiently small ϵ , we have η = δϵ 2 3500 cd 3 / 2 L 3 1 , then one can verify that TηZ = 4 √ 2 πυ 25 J ( K 0 ) . Therefore, we have

This implies that all the iterates K t are stabilizing with probability at least 1 -(0 . 05 + 0 . 01 υ ) = 0 . 95 -0 . 01 υ . This completes the proof for Statement 1.

Proof of Statement 2 We first show that we can extend (B.5) as

E [ J δ ( K t +1 ) ✶ τ>t +1 | F t ] ≤ J δ ( K t ) ✶ τ>t -η ∥∇ J δ ( K t ) ∥ 2 F ✶ τ>t + ηZ, (B.9) If τ > t , then we know J δ ( K t ) ≤ 49 J ( K 0 ) , hence J ( K t ) ≤ J δ ( K t ) + δL 1 ≤ 50 J ( K 0 ) , we have K t ∈ S 0 and K t +1 ∈ S 1 . Therefore, we can apply Lemma B.3 to show that:

On the other hand, if τ ≤ t , we have

E [ J δ ( K t +1 ) ✶ τ>t +1 | F t ] = 0 ≤ J δ ( K t ) ✶ τ>t -η ∥∇ J δ ( K t ) ∥ 2 F ✶ τ>t + ηZ (B.10) since Z ≥ 0 . Therefore, (B.9) holds. Taking the expectation and rearranging the terms of (B.9) gives:

Summing up the above inequality over t = 0 , 1 , · · · , T -1 yields

Since the cost function J ( K ) is nonnegative, we have E [ J δ ( K T ) τ>T ] ≥ 0 and:

Hence we conclude that

By the choice of η and T , the above inequality becomes:

which implies

Therefore, we have

where the second inequality uses the Markov's inequality. By Proposition 2.3 (iii), we have ∇ J δ ( K R ) ∈ ∂ δ J ( K R ) . This implies that

Therefore our algorithm can find a ( δ, ϵ ) -stationary point with probability at least 1 -(0 . 17 υ -1 2 + 0 . 13 + 0 . 01 υ ) = 0 . 87 -0 . 17 υ -1 2 -0 . 01 υ . This completes the proof for Statement 2.

Remark B.4 . It is worth mentioning that the choice of sublevels in sets in (9) is WLOG. Tuning these two numbers entails trade-offs. On the one hand, a larger sublevel in set S 1 yields a higher probability as observed from the proofs of Theorem 3.7 where we are bounding the term Pr( τ ≤ T -1) as an example. On the other hand, a larger sublevel set also results in a larger Lipschitz constant L 1 and a smaller smooth radius δ , thereby worsening the complexity bounds.

B.5 Proof of Theorem 4.1

We first state several technical lemmas which will be useful in proving of Theorem 4.1.

Lemma B.5. Let P ( z ) = ∑ ∞ k =0 P k z -k be a stable, discrete-time MIMO LTI system, where P = { P k ∈ C n x × n x } k ∈ Z + is the impulse response of P ( z ) and T ( P ) = ( P i -j ) ∞ ,i i,j =0 denote the Toeplitz operator associated with system P ( z ) . Suppose for any given Euclidean unit vectors y ∈ C n x and x ∈ C n x , we define a scalar sequence g = { g k = y ∗ P k x } k ∈ Z + . Let G ( z ) = ∑ ∞ k =0 g k z -k denote the z -transform of the impulse response g and T ( g ) = ( g i -j ) ∞ ,i i,j =0 be the corresponding Toeplitz matrix. For any given Euclidean unit vectors x and y , we can show:

where T N ( P ) and T N ( g ) are Nn x × Nn x and N × N upper-left submatrix of T ( P ) and T ( g ) , respectively, ∥ · ∥ is the spectral norm of a given matrix.

Proof. Using the definition of g k = y ∗ P k x , we can write T N ( g ) as:

∗

which completes the proof.

Corollary B.6. Let P ( z ) and G ( z ) be defined as lemma B.5, fix a γ ∈ ( ι, 1) and define the systems P γ ( z ) := γP ( γz ) and G γ ( z ) := γG ( γz ) . We can show that:

Proof. This result can be shown by applying lemma B.5 to G γ ( z ) and P γ ( z ) and considering this fact that ∥ T N ( g γ ) ∥ and ∥ T N ( P γ ) ∥ converge to ∥ G γ ∥ ∞ and ∥ P γ ∥ ∞ as N →∞ , respectively.

Now we are ready to extend [68] results of H ∞ -norm approximation of SISO system using the corresponding truncated Toeplitz matrix to more general MIMO systems.

Lemma B.7. Let P ( z ) = ∑ ∞ k =0 P k z -k be a stable, discrete-time MIMO LTI system with stability radius ι ∈ (0 , 1) . Fix a γ ∈ ( ι, 1) and suppose that σ max ( P 0 ) ≤ D . For all N ≥ 3 , we have that

where ∥ P γ ( z ) ∥ ∞ denotes the H ∞ -norm of the system P γ ( z ) := γP ( γz ) , C 1 = 3 √ 2 π (2 + 3 π 4 ) , C 2 = 9 √ 2 π 2 are universal constants.

Proof. We can write ∥ P ∥ ∞ as follows

This means that there exists Euclidean unit vectors x and y such that ∥ P ∥ ∞ = y ∗ P ( e iω opt ) x . Now define a SISO LTI system G ( z ) = y ∗ P ( z ) x , which has the impulse response g = { g k = y ∗ P k x } k ∈ Z + . Since P ( z ) is a stable function, so is G ( z ) and we have ∥ P ∥ ∞ = ∥ G ∥ ∞ . Additionally, regarding the stability radius of G , it can be derived that, ι g ≤ ι and γ ∈ ( ι g , 1) . Consequently:

Now we invoke Theorem 4.1 from [68]; for a stable, SISO LTI system G ( z ) = ∑ ∞ k =0 g k z -k , with a fix γ ∈ ( ι g , 1) and ∥ g 0 ∥ ≤ D , we have:

From lemma B.5 and corollary B.6, we know that ∥ T N ( g ) ∥ ≤ ∥ T N ( P ) ∥ and ∥ G γ ∥ ∞ ≤ ∥ P γ ∥ ∞ . Also using this fact that ∥ G ∥ ∞ = ∥ P ∥ ∞ , we have

Algorithm 2: Power iteration method to find the maximum eigenvalue of a matrix

Require: Given a positive semi-definite matrix M ∈ R Nn x × Nn x and an initial vector v (0) ∈ R Nn x

Lemma B.8. Given a positive semi-definite matrix M ∈ R Nn x × Nn x with spectral factorization

Where V = [ v 1 | ... | v Nn x ] is orthogonal, λ 1 ≥ λ 2 ≥ · · · ≥ λ Nnx ≥ 0 and λ 1 > 0 . Suppose we choose a ψ such that λ 1 , · · · , λ j ∈ [ λ 1 -ψ, λ 1 ] . Let the vectors v ( i ) be specified by Algorithm 2 and define θ 0 ∈ [0 , π/ 2) by cos( θ 0 ) = | v T 1 v (0) | . For i = 0 , 1 , ... we have

Proof. The idea of our proof is based on the results in section 8.2.1 of [29]. Suppose v (0) has the eigenvector expansion v (0) = a 1 v 1 + a 2 v 2 + ... + a Nn x v Nn x , then

̸

and and so

where the inequality ( i ) follows from the fact that tan( θ 0 ) 2 = 1 -a 2 1 a 2 1 and this completes the proof.

Corollary B.9. Let matrix M be given as lemma B.8, we can show that for each iteration i , ψ ( i ) min = λ 1 ( 1 -1 2 i -1 √ 2 i ) minimize the upper bound in power iteration algorithm (B.20) , and (B.20) can be written as:

Proof. Since the right hand side of inequality (B.20) is a convex function of ψ , we can minimize it by setting its derivative with respect to ψ equal to zero. Define f ( ψ ) = tan( θ 0 ) 2 ( ψ + λ 1 ( λ 1 -ψ λ 1 ) 2 i ) , we have:

It is straightforward to show that ψ ( i ) min = λ 1 ( 1 -1 2 i -1 √ 2 i ) makes f ′ ( ψ ) = 0 and substitute ψ ( i ) min in f ( ψ ) , gives f ( ψ ( i ) min ) = tan( θ 0 ) 2 λ 1 ( 1 -1 2 i -1 √ 2 i ( 1 -1 2 i ) ) , which completes the proof.

Now we are ready to prove Theorem 4.1. As mentioned before, cost function J ( K ) is equal to H ∞ -norm of P ( z ) , ∥ P ∥ ∞ . From lemma B.7, We can find the bound ζ 1 ( K ) for | J ( K ) -∥ T N ( P ) ∥| ≤ ζ 1 ( K ) . Also, using Algorithm 2 we can obtain the model-free cost function J ( K ) as ˆ J ( K ) := √ λ ( n ) . It is also well-known that √ λ 1 ≤ ∥ P ∥ ∞ and √ λ 1 →∥ P ∥ ∞ as N →∞ . Now invoke lemma B.8, corollary B.9 and use M := T N ( P ) ∗ T N ( P ) , we have

It is easy to show that

In addition, for n ≥ 1 , one can verify that

and hence | ˆ J ( K ) -∥ T N ( P ) ∥| ≤ C 3 ∥ P ∥ ∞ 1 √ n = ζ 2 ( K ) with C 3 = tan( θ 0 ) 2 . This gives us sample complexity (16). Finally, applying triangle inequality, we have

ˆ ˆ which completes the proof.

B.6 Proof of Theorem 4.2

We consider that we have an inaccurate function value estimation ˆ J ( K ) = J ( K ) + ζ ( K ) , here we assume that the estimation error ζ ( K ) is uniformly upper bounded for all K ∈ S 1 as stated in (18). Then we have:

where we denote ζ t 1 = ζ ( K t + δW t ) and ζ t 2 = ζ ( K t -δW t ) for simplicity. To prove Theorem 4.2, we first present some technical lemmas.

Lemma B.10. Suppose that J ( K ) is L 1 -Lipschitz on the sublevel set S 1 and let { g t } T -1 t =0 and { K t } T -1 t =0 be generated by Algorithm 1 such that { K t } T -1 t =0 are feasible and { J δ ( K t ) } T -1 t =0 are well defined. Then, we have

Proof. By definition of ˆ g t , we have:

This completes the proof.

Lemma B.11. J ( K ) is L 1 -Lipschitz on the sublevel set S 1 , let η ≤ δξ dκ (100 J ( K 0 ) -J ∗ ) and δ ≤ min { ∆ 1 , ∆ } , then as long as K t ∈ S 0 , we will have K t +1 ∈ S 1 and

Proof. Since ∥ W t ∥ F = 1 , we have:

where the second inequality holds since we have K t ± δW t ∈ S 1 when δ ≤ ∆ and | ζ t 1 -ζ t 2 | ≤ 2 κ . This implies that K t +1 ∈ S 1 by Remark 3.5. In addition, K t +1 ∈ S 1 and δ ≤ ∆ 1 ensures that J δ ( K t +1 ) is well defined. By Proposition 2.3 (ii), we know that J δ ( K ) is differentiable and L 1 -Lipschitz with cL 1 √ d δ -Lipschitz gradient on the sublevel set S 1 . Then we have:

Taking the expectation of both sides conditioned on K t , we have

where inequality (i) and (iii) use the bounds in Lemma B.10, and (ii) holds by Cauchy-Schwarz inequality. This completes the proof.

Now we are ready to prove the Theorem 4.2. We will first prove Statement 1 in Theorem 4.2: all the generated controllers are within the feasible set with high probability. Then we will show Statement 2: the Algorithm 1 returns a ( δ, ϵ ) -stationary point with high probability.

Proof of Statement 1 We first define a stopping time τ as below:

Based on Lemma 3.2, it can be seen that as long as τ ≥ T -1 , the iterates { K t } T -1 t =0 generated by Algorithm 1 are feasible. Therefore our goal becomes bounding the probability Pr( τ ≤ T -1) . To this end, we can define a nonnegative supermartingale Y ( t ) as below

To show it is a supermartingale, noticing that we have

where the inequality ( i ) uses (B.25). This is valid since τ > t implies J δ ( K t ) ≤ 49 J ( K 0 ) by the definition of τ . Then we have J ( K t ) ≤ J δ ( K t ) + δL 1 ≤ 50 J ( K 0 ) . Hence we have K t ∈ S 0 . The choice of η guarantees that K t +1 ∈ S 1 . Hence K t +1 is feasible and J δ ( K t +1 ) is well defined when δ ≤ ∆ 1 . Then Doob's maximal inequality for super-martingales gives

For sufficiently small ϵ , we have η = δϵ 2 100Γ , then one can verify that Tη ( Z 1 + Z 2 ) ≤ υJ ( K 0 ) by the choice of T , η , and κ . Therefore, we have

This implies that all the iterates K t are stabilizing with probability at least 1 -(0 . 05 + 0 . 03 υ ) = 0 . 95 -0 . 03 υ . This completes the proof for Statement 1.

Proof of Statement 2 We first show that we can extend (B.25) as

E [ J δ ( K t +1 ) ✶ τ>t +1 | F t ] ≤ J δ ( K t ) ✶ τ>t -η ∥∇ J δ ( K t ) ∥ 2 F ✶ τ>t + ηZ 1 + ηZ 2 , (B.28) If τ > t , then we know J δ ( K t ) ≤ 49 J ( K 0 ) , hence J ( K t ) ≤ J δ ( K t ) + δL 1 ≤ 50 J ( K 0 ) , we have K t ∈ S 0 and K t +1 ∈ S 1 . Therefore, we can apply Lemma B.3 to show that:

On the other hand, if τ ≤ t , we have

E [ J δ ( K t +1 ) ✶ τ>t +1 | F t ] = 0 ≤ J δ ( K t ) ✶ τ>t -η ∥∇ J δ ( K t ) ∥ 2 F ✶ τ>t + ηZ 1 + ηZ 2 (B.29) since Z 1 ≥ 0 and Z 2 ≥ 0 . Therefore, (B.28) holds. Taking the expectation and rearranging the terms of (B.28) gives:

Since the cost function J ( K ) is nonnegative, we have E [ J δ ( K T ) τ>T ] ≥ 0 and:

By the choice of η and T , the above inequality becomes:

which implies

where the second inequality uses the Markov's inequality. By Proposition 2.3 (iii), we have ∇ J δ ( K R ) ∈ ∂ δ J ( K R ) . This implies that

Therefore our algorithm can find a ( δ, ϵ ) -stationary point with probability at least 1 -(0 . 15 υ -1 2 + 0 . 2 + 0 . 03 υ ) = 0 . 8 -0 . 15 υ -1 2 -0 . 03 υ . This completes the proof for Statement 2.

C Numerical Experiments

In this section, we provide more details on the numerical experiments. All the experiments are performed on a desktop computer with a 3.7 GHz Intel i5-9600K processor.

C.1 Two-dimension example

In this subsection, we first present an example where the iterates of Algorthm 1 can be visualized in controller space directly. In particular, we consider the following system

The weight matrices Q and R are set as identity matrices with proper dimensions. In this case, we have K = [ k 1 k 2 ] ∈ R 1 × 2 . Since A is a stable matrix, we initialize K 0 = [0 0] . For Algorthm 1, we set η = 1 × 10 -3 , δ = 1 × 10 -4 , and ϵ = 1 × 10 -3 . The iterates and contour lines of the H ∞ norm are drawn in Figure 2. It can be seen that Algorithm 1 yields to a local minimum and after 5000 iterations, it converges to a controller K ∗ = [ -0 . 1429 -0 . 6425] .

6

6

6

6

6.5

6.5

6.5

7

7

7.5

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

-0.2

-0.1

0

0.1

0.2

C.2 Experiments for Figure 1

In this subsection, we discuss more details about the three different experiments displayed in Figure 1 to show the effectiveness of Algorithm 1 with exact and inexact oracle.

Left plot of Figure 1 For the left plot of Figure 1, we show the relative error of the Algorithm 1 with exact oracle for different system dimensions n x = { 10 , 50 , 100 } . The system is of form (1) with parameters ( A,B,C ) . The entries of ( A,B,C ) are sampled from a standard normal distribution N (0 , 1) . The system matrix A is scaled when necessary to make ρ ( A ) < 1 such that we can set K 0 as zero matrices. The cost matrices Q and R are set to be identities with appropriate dimensions. Table 2 shows the detailed algorithm parameters.

Middle plot of Figure 1 For the second experiment, we perform the Algorithm 1 with both exact and inexact zeroth-order oracles. In particular, we consider the following specific MIMO system:

The weight matrices Q and R are set as:

Since the system matrix A is Schur stable, we set the initial point K 0 as a zero matrix. We also set the smooth radius δ = 0 . 001 and the stepsize η = 0 . 0001 . It can be seen that the inexact oracle case works well and tracks the exact oracle scenario.

Right plot of Figure 1 Finally, we examine how the sample complexity varies with different ϵ for the exact oracle setting with system matrices (C.2). It is worth mentioning that in Algorithm 1, we cannot really determine whether an iterate K t is a ( δ, ϵ ) -stationary point or not since we cannot calculate the exact ∇ J δ ( K t ) . In practice, we use the magnitude of g t as a stationarity criterion. In particular, we terminate the algorithm when the norm of g t is less than ϵ .

40

30

20

10

0

5000

4500

4000

0

AC15

HF2D11

9.5 ×104

Algorithm 1

9

HIFOO

一Hinfstruct

8.5

8

7.5

7

0

1000

2000

3000

4000

5000

120

Iterations

HE4

100

80

60

40

20

0

Algorithm 1

HIFOO

-—Hinfstruct

2000

3000

4000

5000

Iterations

Algorithm 1

HIFOO

-Hinfstruct

2000

3000

4000

5000

Iterations

DLR2

Algorithm 1

HIFOO

一Hinfstruct

1000 2000 3000 4000 5000

Iterations

C.3 Comparison with model-based methods

In this subsection, we compare our derivative-free methods to model-based packages HIFOO and Hinfstruct, which are available in MATLAB. In particular, we select several benchmark models from COMPl e ib including aircraft model AC15 , 2D heat flow model HF2D11 , second-order model DLR2 , and helicopter models HE4 [41]. Table 3 summarized more details on the selected examples.

It is worth mentioning that the models in COMPl e ib are all continuous time models. In addition, the implementations of HIFOO and Hinfstruct are also based on continuous-time models. Therefore, we

1000

1000

implemented the continuous-time version of Algorithm 1. In other words, when we compute the H ∞ norm of the closed-loop system, we are computing the H ∞ norm of a continuous-time system instead of a discrete-time system. Figure 3 shows the trajectories of the four selected examples when we run Algorithm 1 for 5000 iterations. It can be seen that our algorithm converges to the ones computed by the model-based methods HIFOO and Hinfstruct with proper initialization. In practice, one can use multiple initial points and run Algorithm 1 multiple times then just report the best case with the lowest J ( K ) . Such a strategy is used in HIFOO and Hinfstruct. It can be seen that our derivative-free method can achieve comparable results even without the knowledge of the system dynamics.

Peer reviews:
comments: SUMMARY:
This paper considers solving the $H_\infty$ control problem using zero-th-order policy optimization. The main results are sample complexity bounds for both the exact Oracle setting and the model-free setting. Numerical simulations are conducted to demonstrate the effectiveness of the algorithm.

STRENGTHS:
This paper is well-written. The $H_\infty$ control problem is known to be challenging and the policy optimization results are likely to be of great interest to the learning-in-control community. The sample complexity result, while not necessarily optimal, is the first non-asymptotic result in the literature.

WEAKNESSES:
(1) Related Work: The related work section could be more structured. I went through [32] to complete the review of this work. It seems that the main technical tools are already developed in [32]. However, [32] does not provide any sample complexity result. What are the major technical challenges (and the ideas used to overcome them) in going beyond the global convergence in [32] to the sample complexity results in this work?

(2) Theorem 3.7 and Theorem 4.2: Usually for high probability bounds, as the tolerance level $v$ decreases, more iterations are needed. However, for Theorem 3.7 and Theorem 4.2, as $v$ decreases, $T$ also decreases, which seems counter-intuitive. Moreover, the bound has a polynomial tail rather than an exponential tail. Is this an artifact of the proof or are exponential tail bounds not achievable?

QUESTIONS:
From the numerical simulations, is it possible to tell what the convergence rate of the algorithm is?

LIMITATIONS:
The sample complexity bounds are not necessarily optimal, which was pointed out by the authors.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the structured $H_\\infty$ control problem. They provide sample complexity bounds for policy optimization in $H_\\infty$ control problem. \nThe results are provided for two separate scenarios namely: \n- Exact Oracle Setting  (exact $J(K)$ for any $K$ is available, for the given closed loop system)\n- Inexact Oracle Setting (the system matrices are not known)\n\nThe theoritical results provide the sample complexity of $H_\\infty$ norm estimation.\nFinally, the paper provides a few numerical experiments supporting their theoritical results along with comparison to some model-based approaches.\n\nSTRENGTHS:\n- $H_\\infty$ control problem is one of the important setting in linear systems , which is well studied in adaptive control literature but has received less attention in recent learning theory literature unlike the standard LQR setting. The paper highlights the various challenges involved in the analysis of $H_\\infty$ control due to non-convexity and non-smoothness.\n\n- The Sample complexity of the $H_\\infty$ norm estimation are provided by exploiting the randomized smoothing techniques.\n\nWEAKNESSES:\n- The algorithm relies on access to an oracle. More discussion on the oracle is warranted. \n- The simulation section in the main body (as well as the appendix) of the paper is meager. The presentation quality of plots included can be improved. \n- Authors mention in the appendix that (line 806) that when necessary, one can reinitiate to avoid bad local minima, but how would one know that they are at a 'bad' local minima.\n\nQUESTIONS:\nFollowing are my concerns/ questions regarding the paper.\n- Can authors explain why they call this approach as \"derivative free\" approach since calculating $g_t$  is required in the algorithm?\n- I would like to know the motivation behind considering the two separate type of oracle scenarios. \n- Authors mention that their probability bounds can be sharpened (line 248), I request authors to present the best possible results in the main body of the paper.\n\nI also request authors to address the comments in the weakness section if possible.\n\nLIMITATIONS:\nThe results rely on access to an oracle.\n\nSocial Impact: NA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies the static output feedback $\mathcal{H}_{\infty}$ control problem. It proposes a derivative-free policy optimization algorithm via randomized smoothing and further provides sample complexity analysis for the cases with exact and inexact zeroth-order oracles. To validate the performance of the new algorithm, the authors also conduct some numerical experiments and compete against the model-based methods in the literature.

STRENGTHS:
1. As the authors claim, the proposed algorithm is the first derivative-free policy optimization algorithm for constrained structured $\mathcal{H}_{\infty}$ control problem (there have been some works on the unconstrained setting).
2. The authors also consider the inexact oracle setting.
3. The paper provides both theoretical analysis and numerical experiments.

WEAKNESSES:
I am not very familiar with the problem studied by this paper and it looks fine to me. However, I feel like the paper is mainly a combination of existing techniques (such as randomized smoothing and gradient sampling with zeroth order feedback). Would the authors highlight any novel techniques they apply in the analysis or explain what makes the problem different from other nonsmooth nonconvex problems such that this paper is not simply A+B?

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27