SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 3.5. Since  $J(K)$  is a continuous function of  $K$  by Proposition 3.1 and the sublevel sets  $S^0$  and  $S^1$  are compact by Lemma 3.3, this implies that there exists a constant  $\xi > 0$  such that for any  $K \in S^0$ , and  $K'$  with  $\| K - K' \|_F \leq \xi$ , we have  $K' \in S^1$ .
Now we are ready to state the following result.
Theorem 3.6. Let  $K^0 \in \mathcal{K}$  be an arbitrary feasible initial controller and suppose that  $J_{\delta}(K)$  defined as in (7) is  $L_1$ -Lipschitz with the  $\frac{cL_1\sqrt{d}}{\delta}$ -Lipschitz gradient on the sublevel set  $S^1$ . Choose  $\delta = \min \{\Delta_1, \frac{49J(K^0)}{2L_1}\}$  and  $\eta = \min \{\frac{\xi}{L_1}, \frac{\delta}{cL_1\sqrt{d}}\}$ . Then the iterative method (8) stays in  $\mathcal{K}$  and we have:
$$
\min  _ {t = 0, 1, \dots , T - 1} \| \nabla J _ {\delta} \left(K ^ {t}\right) \| _ {F} ^ {2} \leq \frac {2 \left(J _ {\delta} \left(K ^ {0}\right) - J ^ {*}\right)}{\eta T}. \tag {10}
$$
In other words, we have  $\min_{0\leq t\leq T - 1}\| \nabla J_{\delta}(K^t)\| _F\leq \epsilon$  after  $T = O(\frac{1}{\epsilon^2})$
The proof of Theorem 3.6 can be found in the supplementary material. Since we have a constrained optimization problem, it is crucial to guarantee the feasibility of the gradient descent method (8). The careful choice of smooth radius  $\delta$  and step size  $\eta$  ensures that  $K^t$  stays inside the feasible set.
3.3 Analysis for the exact oracle case
In this section, we consider the exact zeroth-order oracle setup, where we can obtain the exact  $\mathcal{H}_{\infty}$  norm of the closed-loop system for a given  $K \in \mathcal{K}$ . One straightforward extension of (8) is to compute an unbiased estimation of the gradient  $\nabla J_{\delta}(K^t)$  via the zeroth-order oracle. Then we can perform the one-step gradient descent. In particular, we analyze the feasibility and convergence of the Algorithm 1 in the following result.
Theorem 3.7. Let  $K^0 \in \mathcal{K}$  be an arbitrary feasible initial controller and suppose  $J(K)$  is  $L_1$ -Lipschitz with the  $\frac{cL_1\sqrt{d}}{\delta}$ -Lipschitz gradient on the sublevel set  $S^1$ . Let  $1 \leq v \leq 80$  and suppose we choose
$$
\delta = \min \{\Delta_ {1}, \Delta , \frac {J (K ^ {0})}{L _ {1}} \}, T = \frac {\upsilon \delta J (K ^ {0})}{5 0 \eta^ {2} c d ^ {3 / 2} L _ {1} ^ {3}}, \eta = \min \{\frac {2 \delta \xi}{d (1 0 0 J (K ^ {0}) - J ^ {*})}, \frac {\delta \epsilon^ {2}}{3 5 0 0 c d ^ {3 / 2} L _ {1} ^ {3}} \}.
$$
Then the following two statements hold:
(1) the controllers  $\{K^t\}_{t=0}^{T-1}$  generated by Algorithm 1 are all stabilizing with probability at least  $0.95 - 0.01v$ .
(2) the output of Algorithm 1  $K^R$  satisfies
$$
\min  \left\{\| H \| _ {F}: H \in \partial_ {\delta} J \left(K ^ {R}\right) \right\} \leq \epsilon \tag {11}
$$
with probability at least  $0.87 - 0.17v^{-\frac{1}{2}} - 0.01v$
Algorithm 1: Derivative-Free Methods for Policy Optimization Problem (4)
Require: feasible initial point  $K^0\in \mathbb{R}^{n_u\times n_y}$  , stepsize  $\eta >0$  , problem dimension  $d\coloneqq n_u\times n_y\geq 1$  , smoothing parameter  $\delta$  and iteration number  $T\geq 1$    
for  $t = 0,1,\dots ,T - 1$  do Sample  $W^{t}\in \mathbb{R}^{n_{u}\times n_{y}}$  uniformly at random over matrices such that  $\| W\| _F = 1$  Compute  $g^{t} = \frac{d}{2\delta} (J(K^{t} + \delta W^{t}) - J(K^{t} - \delta W^{t}))W^{t}$  Update  $K^{t + 1} = K^t -\eta g^t$    
end for   
Output:  $K^R$  where  $R\in \{0,1,2,\dots ,T - 1\}$  is uniformly sampled.
The proof of Theorem 3.7 can be found in the supplementary material. Now we provide some discussions regarding Theorem 3.7 in order:
Probability Bounds: Statement 1 suggests that as  $T$  increases, the probability that all the generated controllers are stabilizing will decrease. This is because our algorithm uses a zeroth-order oracle to build an estimator of the smoothed function gradient. As  $T$  increases, the biases and variance of the gradient estimation accumulate, resulting in a larger failure probability. In addition, Statement 2 suggests that as  $T$  increases, the probability of finding a  $(\delta, \epsilon)$ -stationary point will first increase and then decrease. Indeed, when  $T$  is too small, more iterations will improve the performance of the generated controllers, while for large  $T$ , the probability of generating unstable controllers becomes dominant. In addition, the constant factors in the probability bounds are not restrictive, they can be further improved by e.g., increasing the level of  $S^0$ , using smaller step size  $\eta$  or using smaller smooth radius  $\delta$  in the analysis.
Feasibility: Statement 1 states that all the output controllers  $\{K^t\}_{t=0}^{T-1}$  are feasible with high probability under the choice of algorithm parameters. It is worth mentioning that, to ensure the feasibility of the iterates  $\{K^t\}_{t=0}^{T-1}$ , we need to show that  $\{J_{\delta}(K^t)\}_{t=0}^{T-1}$  are well defined so that we can apply the smoothness property of  $J_{\delta}(K)$  in our theoretical analysis. Furthermore, this also guarantees that  $\{J(K^t \pm \delta W^t)\}_{t=0}^{T-1}$  used in Algorithm 1 are well defined. This turns out to be a nontrivial task and one needs to design algorithm parameters carefully to resolve the feasibility issue. More discussions will be provided in the end of this section.
Output Controller: Since our cost function  $J(K)$  is nonconvex, we consider an output controller  $K^R$  uniformly sampled from  $\{K^t\}_{t=0}^{T-1}$ . Such random output technique has been used in smooth/nonsmooth nonconvex optimization problems for theoretical analysis [27, 46, 44]. In particular, Statement 2 implies that the output controller  $K^R$  is a  $(\delta, \epsilon)$ -stationary point with high probability. In practical implementation, one can just select the iterate that gives the lowest cost  $J(K^t)$  for  $t = 1, \dots, T$ . Our numerical experiments suggest that selecting  $K^T$  often yields satisfactory performance (see Section 5).
Sample Complexity: For sufficiently small  $\epsilon$ , the number of iterations to guarantee (11) is given by:
$$
T = \Theta \left(\frac {d ^ {\frac {3}{2}} L _ {1} ^ {3}}{\delta \epsilon^ {4}}\right), \tag {12}
$$
where  $d \geq 1$  is the problem dimension,  $L_{1}$  is the Lipschitz parameter. We also ignore the numerical constants since they are conservative and not restrictive.
Finally, we close this section by highlighting the main technical contributions of Theorem 3.7 and 3.6. In our control setup, unlike the unconstrained optimization problems [46], we need to ensure that the iterate  $K^t$  and the perturbed iterate  $K^t \pm \delta W^t$  stay within a non-convex feasible set  $\mathcal{K}$ . Previous work on policy optimization theory of  $\mathcal{H}_{\infty}$  control addresses this feasibility issue via using the coerciveness of  $J(K)$  and mainly relies on the fact that  $J(K)$  is a barrier function on the non-convex set of stabilizing policies [32]. In particular, such previous results rely on model-based algorithms (such as Goldstein's subgradient method) which can decrease the value of  $J(K)$  directly. However, the zeroth-order policy search can only decrease the value of the smoothed function  $J_{\delta}(K)$ , which is not coercive over the non-convex feasible set and hence cannot be used as a barrier function. Importantly, the descent of  $J_{\delta}(K)$  does not imply the descent of the original function value and hence cannot ensure feasibility by itself. In Theorem 3.7 and 3.6, by carefully choosing the smooth radius
$\delta$  and step size  $\eta$ , we manage to show that the iterate  $K^t$  and the perturbed iterate  $K^t \pm \delta W^t$  stay within a non-convex feasible set with high probability.
4 Sample Complexity for the Model-Free Case
In this section, we assume that the system dynamics in (1) are unknown and we can only use samples to obtain an estimation of the cost function  $\hat{J} (K)$  with some error  $\hat{J} (K) = J(K) + \zeta (K)$ . To obtain an end-to-end sample complexity result of Algorithm 1, we first derive the sample complexity of  $\mathcal{H}_{\infty}$  norm estimation for the general MIMO systems such that  $\zeta (K)\leq \kappa$  for all  $K\in \mathcal{K}$ .
4.1 Complexity of  $\mathcal{H}_{\infty}$  norm estimation for MIMO systems
The  $\mathcal{H}_{\infty}$  norm estimation method via input/output data can be roughly categorized into two major approaches: (a) try to find the worst case  $\ell_2$ -norm signal using power-iteration algorithm [71, 53]. (b) discretizing the interval  $[0, 2\pi]$  and search for the maximizing frequency using multi-armed bandit [51]. However, there are no sample complexity results of the aforementioned methods for the general MIMO system. In this section, we analyze the  $\mathcal{H}_{\infty}$  norm estimation method proposed in [53] and establish its sample complexity bounds for the general MIMO systems.
To this end, let  $P(z) = (Q + C^T K^T RKC)^{1/2}(zI - A + BKC)^{-1}$  be the transfer function associated with the system (1) to the cost function (3), let  $T(P)$  denote the corresponding Toeplitz (convolution) operator and  $T_N(P)$  be the  $N \times N$  upper-left submatrix of  $T(P)$ . Then it is known that  $J(K) = \|T(P)\|$  and  $\|T_N(P)\| \to \|T(P)\|$  as  $N \to \infty$ . With large enough  $N$ , the largest singular value of the  $T_N(P)$  can be used as a reasonable estimation of  $J(K)$ , which can be approximated by running the power iteration algorithm  $n$  steps. Therefore, there are two error terms based on this approach. The first term  $\zeta_1(K)$  comes from the difference between  $\|T_N(P)\|$  and  $\|T(P)\|$ , and the second term  $\zeta_2(K)$  is induced by computing the largest singular value of  $T_N(P)$  via the power iteration algorithm:
$$
| \zeta (K) | = | J (K) - \hat {J} (K) | \leq | J (K) - \| T _ {N} (P) \| | + \| T _ {N} (P) \| - \hat {J} (K) | = \zeta_ {1} (K) + \zeta_ {2} (K), \tag {13}
$$
where  $\hat{J}(K)$  is the approximated largest singular value of  $T_{N}(P)$  and hence an estimation of  $J(K)$ . Therefore, if we can choose  $N$  and  $n$  large enough such that  $\zeta_1(K) \leq \kappa/2$  and  $\zeta_2(K) \leq \kappa/2$ , we will have  $\zeta(K) \leq \kappa$  holds by (13). The finite-time condition on  $N$  has been established in [68] for single input single output (SISO) systems only. In the following result, we extend the result in [68] to the general MIMO system. Furthermore, we provide the finite-time condition on the number of power iteration iterations  $n$ .
Theorem 4.1. Let  $P(z) = \sum_{k=0}^{\infty} P_k z^{-k}$  be the corresponding transfer function of system (1) to the cost function (3) with stability radius  $\iota \in (0,1)$ . And choose  $\gamma \in (\iota,1)$  and  $D \geq \sigma_{\max}(P_0)$ . Let  $\| P^\gamma \|_\infty$  denotes the  $\mathcal{H}_\infty$  norm of the system  $P^\gamma(z) \coloneqq \gamma P(\gamma z)$ . Then for  $N \geq 3$ , we have
$$
| \hat {J} (K) - J (K) | \leq \zeta_ {1} (K) + \zeta_ {2} (K) \tag {14}
$$
$$
\zeta_ {1} (K) = C _ {1} \frac {D \| P ^ {\gamma} \| _ {\infty} (1 - \gamma^ {2}) + \| P ^ {\gamma} \| _ {\infty} ^ {2} \gamma}{\| P \| _ {\infty} (1 - \gamma) ^ {4}} \frac {1}{N ^ {2}} + C _ {2} \frac {\| P ^ {\gamma} \| _ {\infty} ^ {2}}{\| P \| _ {\infty} (1 + \gamma) (1 - \gamma) ^ {5}} \frac {1}{N ^ {3}}, \tag {15}
$$
$$
\zeta_ {2} (K) = C _ {3} \| P \| _ {\infty} n ^ {- \frac {2}{3}} \tag {16}
$$
are errors due to approximating  $T$  by  $T_{N}$  and using power iteration to estimate  $\| T_N\|$ , respectively. Here  $C_1 = 3\sqrt{2}\pi (2 + 3\pi^4)$ ,  $C_2 = 9\sqrt{2}\pi^2$  are universal constants and  $C_3$  depends on an angle between power iteration initialization and the eigenvector corresponding to the biggest eigenvalue of  $T_N^* (P)T_N(P)$ .
The proof of Theorem 4.1 can be found in the supplementary material. Theorem 4.1 implies that the approximation error  $|\zeta(K)| \leq \kappa$  is guaranteed by choosing
$$
N \geq \inf  _ {\iota <   \gamma <   1} \Omega \left(\frac {1}{(1 - \gamma) ^ {2}} \sqrt {\frac {\| P ^ {\gamma} \| _ {\infty} ^ {2}}{\| P \| _ {\infty}} \frac {2}{\kappa}}\right), \quad n \geq \Omega \left(\frac {(2 C _ {3}) ^ {\frac {3}{2}} \| P \| _ {\infty} ^ {\frac {3}{2}}}{\kappa^ {\frac {3}{2}}}\right). \tag {17}
$$
For completeness, the pseudo-code of the power iteration algorithm is provided in the supplementary material.
4.2 Sample complexity of zeroth-order optimization
In this section, we consider that Algorithm 1 can only access the inexact zeroth-order oracle of the cost function  $J(K)$ . In this case, we do not have an unbiased estimation of  $\nabla J_{\delta}(K^t)$ . Nevertheless, we can make the estimation as small as possible and assume that the estimation error  $\zeta (K)$  is uniformly upper bounded for all  $K\in S^{1}$ :
$$
| \zeta (K) | \leq \kappa , \forall K \in \mathcal {S} ^ {1}. \tag {18}
$$
Clearly, the smaller  $\kappa$  is, the larger power iteration number  $n$  and approximation horizon  $N$  are needed. We will specify the choice of  $\kappa$  in the next result, where we obtain the feasibility/sample complexity bounds of Algorithm 1 via inexact zeroth-order oracle on finding a  $(\delta ,\epsilon)$  -stationary point.
Theorem 4.2. Let  $K^0 \in \mathcal{K}$  be an arbitrary feasible initial controller and suppose  $J$  is  $L_{1}$ -Lipschitz on the sublevel set  $S^{1}$ . Consider the Algorithm 1 with inexact zeroth-order oracle and suppose that the estimation error  $\kappa \leq \frac{\delta\epsilon^2}{100dL_1}$ . Denote  $\Gamma \coloneqq cL_1\sqrt{d}(16\sqrt{2\pi} dL_1^2 + (\frac{d\kappa}{\delta})^2)$ . Let  $1 \leq v \leq 25$  and suppose we choose
$$
\delta = \min \{\Delta_ {1}, \Delta , \frac {J (K ^ {0})}{L _ {1}} \}, T = \frac {\upsilon \delta J (K ^ {0})}{2 \eta^ {2} \Gamma}, \eta = \min \{\frac {\delta \xi}{d \kappa (1 0 0 J (K ^ {0}) - J ^ {*})}, \frac {\delta \epsilon^ {2}}{1 0 0 \Gamma} \}.
$$
(1) the controllers  $\{K^t\}_{t=0}^{T-1}$  generated by Algorithm 1 with inexact zeroth-order oracle are all stabilizing with probability at least  $0.95 - 0.03v$ .
(2) the output of Algorithm 1  $K^R$  satisfies
$$
\min  \left\{\| H \| _ {F}: H \in \partial_ {\delta} J \left(K ^ {R}\right) \right\} \leq \epsilon \tag {19}
$$
with probability at least  $0.8 - 0.15v^{-\frac{1}{2}} - 0.03v$
The proof of Theorem 4.2 can be found in the supplementary material. For sufficiently small  $\epsilon$ , we can obtain the following lower bound on the number of iterations to guarantee (19):
$$
T = \Omega \left(\frac {d ^ {\frac {3}{2}} L _ {1} ^ {3}}{\delta \epsilon^ {4}}\right), \tag {20}
$$
Since we require  $\kappa \leq \frac{\delta\epsilon^2}{100dL_1}$ , we have the following lower bounds on the power iteration number  $n$  and approximation horizon  $N$  by (17):
$$
N = \Omega \left(\frac {d ^ {1 / 2} L _ {1} ^ {1 / 2}}{\delta^ {1 / 2} \epsilon}\right), n = \Omega \left(\frac {(d L _ {1}) ^ {\frac {3}{2}}}{\delta^ {\frac {3}{2}} \epsilon^ {3}}\right).
$$
Finally, by combining the above sample complexity bounds for MIMO  $\mathcal{H}_{\infty}$  estimation with (20), the number of samples to guarantee (19) with high probability is given by:
$$
n N T = \Omega \left(\frac {d ^ {7 / 2} L _ {1} ^ {5}}{\delta^ {3} \epsilon^ {8}}\right). \tag {21}
$$
In this section, we consider the sample complexity of Algorithm 1 with the inexact oracle case. This is particularly relevant for the model-free control setting. Specifically, we are using imperfect estimates of  $J(K)$  that are calculated using the model-free MIMO power iteration method. Therefore, an extra statistical error term appears in the iterations of zeroth-order policy optimization and requires special treatment. Such an extra term has not been considered in the literature on zeroth-order optimization for nonconvex nonsmooth problems. To address this extra technical difficulty, we first establish sample complexity bounds for  $\mathcal{H}_{\infty}$  norm estimation of the general MIMO system (Theorem 4.1). Then we carefully propagate such sample complexity bounds to obtain an error bound for  $\nabla J_{\delta}(K)$  in terms of  $\epsilon$  and  $\delta$ . Built upon this, we demonstrate that Algorithm 1 remains effective even with an inexact oracle, ensuring the feasibility of the iterates while achieving finite-time sample complexity with high probability (Theorem 4.2).
In this section, we present the numerical study to show the effectiveness of Algorithm 1. The left plot of Figure 1 displays the relative error trajectories of Algorithm 1 with an exact oracle for system dimensions  $n_x = \{10,50,100\}$ . The system is of form (1) with parameters  $(A,B,C)$ , where the entries of  $(A,B,C)$  are sampled from a standard normal distribution  $\mathcal{N}(0,1)$ . The results show that Algorithm 1 performs well even for larger-scale systems. In the middle plot, we compare the trajectories of Algorithm 1 with both exact and inexact zeroth-order oracle. It can be observed that the inexact oracle case closely tracks the performance of the exact oracle case. The right plot illustrates the zeroth-order complexity of the exact oracle case with varying  $\epsilon$ . As shown, the number of oracle calls increases as  $\epsilon$  decreases. It is important to note that the complexity bounds derived in (12) are not tight. We conjecture that by leveraging other advanced properties of the cost function (5), further improvements in complexity can be achieved.
Figure 1: Left: The relative error trajectories of Algorithm 1 with exact zeroth-order oracle and  $n_x = \{10, 50, 100\}$ , the solid lines represent the mean values and the shade represents  $98\%$  confidence intervals; Middle: The trajectory of Algorithm 1 with exact and inexact zeroth-order oracle; Right: Number of zeroth-order oracle calls required to find a  $(\delta, \epsilon)$ -stationary point with varying  $\epsilon$ .
Table 1: Comparison of Algorithm 1 with model-based methods
Example	(nx, nu, ny)	HIFOO	Hinfstruct	Algorithm 1 (Ours)
AC15	(4,2,3)	15.2919	15.2	15.4141
HF2D11	(5,2,3)	7.7237 × 104	7.72 × 104	7.7223 × 104
DLR2	(40,2,2)	4.0066 × 103	4.01 × 103	4.0094 × 103
HE4	(8,4,6)	22.8382	22.8	22.8538
In addition, we conducted a comparison of our derivative-free method with the model-based methods HIFOO and Hinfstruct, using several benchmark examples from COMPlib [41]. Table 1 shows the corresponding optimal closed-loop  $\mathcal{H}_{\infty}$  norm. It can be seen that our derivative-free method yields comparable results with the model-based packages even without the knowledge of system models. More details about the numerical experiments can be found in the supplementary material.
This paper presents the feasibility and complexity bounds for the derivative-free policy optimization method on the structured  $\mathcal{H}_{\infty}$  synthesis, considering both exact and inexact zeroth-order oracles. Despite the fact that this structured  $\mathcal{H}_{\infty}$  synthesis is a constrained nonconvex nonsmooth optimization problem, we leverage the intriguing connections between randomized smoothing and  $(\delta, \epsilon)$ -stationarity, enabling the first sample analysis for model-free, trajectory-based, zeroth-order policy optimization in structured  $\mathcal{H}_{\infty}$  synthesis. One limitation of this work is the uncertain tightness of the sample complexity bounds obtained.
For future studies, it is important to tighten the sample complexity results by exploring more properties of  $\mathcal{H}_{\infty}$  control problems. In addition, it is interesting to explore the behavior of the structured  $\mathcal{H}_{\infty}$  problem with a dynamic output feedback controller.
The work of Xingang Guo and Bin Hu is generously supported by the NSF award CAREER-2048168 and the IBM/IIDAI award 110662-01. The work of Darioush Kevian and Geir Dullerud was supported by NSF under Grant ECCS 1932735. The work of Peter Seiler was supported by the U.S. Office of Naval Research (ONR) under Grant N00014-18-1-2209.
A More on Problem Formulation
A.1 Derivation of the cost function in frequency domain
The goal of the structured  $\mathcal{H}_{\infty}$  synthesis with static output feedback is to find the optimal controller gain  $K^{*}$  to solve the following minimax problem:
$$
\min  _ {K \in \mathcal {K}} \max  _ {\mathbf {w}: \| \mathbf {w} \| \leq 1} \sum_ {t = 0} ^ {\infty} x _ {t} ^ {\top} (Q + C ^ {\top} K ^ {\top} R K C) x _ {t}, \tag {A.1}
$$
Now if we define  $z_{t} = (Q + C^{\top}K^{\top}RKC)^{\frac{1}{2}}x_{t}$ , we have  $\| z_t\| ^2 = x_t^\top (Q + C^\top K^\top RKC)x_t = x_t^\top Qx_t + u_t^\top Ru_t$ . Then the closed-loop LTI system (2) can be viewed as a linear operator  $G_{K}$  mapping any disturbance sequence  $\{w_{t}\}$  to another sequence  $\{z_{t}\}$ . If  $K$  is stabilizing, i.e.  $\rho (A - BKC) < 1$ , then  $G_{K}$  is bounded in the sense that it maps any  $\ell_2$  sequence  $\mathbf{w}$  to another sequence  $\mathbf{z}$  in  $\ell_2^{n_x}$ . For any stabilizing  $K$ , the  $\ell_2\rightarrow \ell_2$  induced norm of  $G_{K}$  can be defined as:
$$
\left\| G _ {K} \right\| _ {2 \rightarrow 2} := \sup  _ {0 \neq \| \mathbf {w} \| \leq 1} \frac {\left\| \mathbf {z} \right\|}{\left\| \mathbf {w} \right\|} \tag {A.2}
$$
Since  $G_{K}$  is a linear operator, it is straightforward to show
$$
\| G _ {K} \| _ {2 \rightarrow 2} ^ {2} := \max  _ {\mathbf {w}: \| \mathbf {w} \| \leq 1} \sum_ {t = 0} ^ {\infty} x _ {t} ^ {\intercal} (Q + C ^ {\intercal} K ^ {\intercal} R K C) x _ {t} = \max  _ {\mathbf {w}: \| \mathbf {w} \| \leq 1} \sum_ {t = 0} ^ {\infty} (x _ {t} ^ {\intercal} Q x _ {t} + u _ {t} ^ {\intercal} R u _ {t}).
$$
Therefore, the minimax optimization problem (A.1) can be rewritten as the policy optimization problem:  $\min_{K\in \mathcal{K}}\| G_K\|_{2\to 2}^2$ . We can just drop the square in the cost function and reformulate (A.1) as  $\min_{K\in \mathcal{K}}\| G_K\|_{2\to 2}$ . This is exactly the policy optimization formulation for structured  $\mathcal{H}_{\infty}$  with static output-feedback control. In the frequency domain,  $G_{K}$  can be viewed as a transfer function that maps signal  $w_{t}$  to  $z_{t}$ . Applying the frequency-domain formula for the  $\mathcal{H}_{\infty}$  norm, we can calculate  $\| G_K\|_{2\to 2}$  as
$$
\left\| G _ {K} \right\| _ {2 \rightarrow 2} = \sup  _ {\omega \in [ 0, 2 \pi ]} \sigma_ {\max } \left(\left(Q + C ^ {\top} K ^ {\top} R K C\right) ^ {\frac {1}{2}} \left(e ^ {j \omega} I - A + B K\right) ^ {- 1}\right), \tag {A.3}
$$
which is the same as we defined in (5).
A.2 LTI system with measurement noise
In this section, we extend the LTI system (1) considered in the main paper to the one with measurement noise as below:
$$
x _ {t + 1} = A x _ {t} + B u _ {t} + w _ {t} \quad x _ {0} = 0, \tag {A.4}
$$
$$
y _ {t} = C x _ {t} + n _ {t - 1},
$$
with  $n_{-1} = 0$ . We denote  $\mathbf{n} \coloneqq \{n_{-1}, n_0, \dots \}$ . We will show that the problem with measurement noise falls within our framework via a proper state augmentation trick. In particular, it is obvious that (A.4) can be rewritten as:
$$
\left[ \begin{array}{c} x _ {t + 1} \\ n _ {t} \end{array} \right] = \left[ \begin{array}{c c} A & 0 \\ 0 & 0 \end{array} \right] \left[ \begin{array}{c} x _ {t} \\ n _ {t - 1} \end{array} \right] + \left[ \begin{array}{c} B \\ 0 \end{array} \right] u _ {t} + \left[ \begin{array}{c} w _ {t} \\ n _ {t} \end{array} \right]
$$
$$
y _ {t} = \left[ \begin{array}{c c} C & I \end{array} \right] \left[ \begin{array}{c} x _ {t} \\ n _ {t - 1} \end{array} \right].
$$
In this setup, the control inputs become
$$
u _ {t} = - K y _ {t} = - K C x _ {t} - K n _ {t - 1}
$$
and the closed-loop system can be represented as
$$
\left[ \begin{array}{c} x _ {t + 1} \\ n _ {t} \end{array} \right] = \left[ \begin{array}{c c} A - B K C & - B K \\ 0 & 0 \end{array} \right] \left[ \begin{array}{c} x _ {t} \\ n _ {t - 1} \end{array} \right] + \left[ \begin{array}{c} w _ {t} \\ n _ {t} \end{array} \right].
$$
Then, the min-max cost function can be written as