Title: Distributionally Robust Linear Quadratic Control
Venue: NeurIPS 2023
Accepted: yes
Keywords: linear quadratic control

distributionally robust optimization

optimal transport

Wasserstein distance
Abstract: Linear-Quadratic-Gaussian (LQG) control is a fundamental control paradigm that is studied in various fields such as engineering, computer science, economics, and neuroscience. It involves controlling a system with linear dynamics and imperfect observations, subject to additive noise, with the goal of minimizing a quadratic cost function for the state and control variables. In this work, we consider a generalization of the discrete-time, finite-horizon LQG problem, where the noise distributions are unknown and belong to Wasserstein ambiguity sets centered at nominal (Gaussian) distributions. The objective is to minimize a worst-case cost across all distributions in the ambiguity set, including non-Gaussian distributions. Despite the added complexity, we prove that a control policy that is linear in the observations is optimal for this problem, as in the classic LQG problem. We propose a numerical solution method that efficiently characterizes this optimal control policy. Our method uses the Frank-Wolfe algorithm to identify the least-favorable distributions within the Wasserstein ambiguity sets and computes the controller's optimal policy using Kalman filter estimation under these distributions.

Distributionally Robust Linear Quadratic Control

Abstract

Linear-Quadratic-Gaussian (LQG) control is a fundamental control paradigm that has been studied and applied in various fields such as engineering, computer science, economics, and neuroscience. It involves controlling a system with linear dynamics and imperfect observations, subject to additive noise, with the goal of minimizing a quadratic cost function depending on the state and control variables. In this work, we consider a generalization of the discrete-time, finite-horizon LQG problem, where the noise distributions are unknown and belong to Wasserstein ambiguity sets centered at nominal (Gaussian) distributions. The objective is to minimize a worst-case cost across all distributions in the ambiguity set, including non-Gaussian distributions. Despite the added complexity, we prove that a control policy that is linear in the observations is optimal, as in the classic LQG problem. We propose a numerical solution method that efficiently characterizes this optimal control policy. Our method uses the Frank-Wolfe algorithm to identify the leastfavorable distributions within the Wasserstein ambiguity sets and computes the controller's optimal policy using Kalman filter estimation under these distributions.

1. Introduction

The Linear Quadratic Regulator (LQR) is a classic control problem that has served as a building block for numerous applications in engineering and computer science [3, 12], economics [29], or neuroscience [47]. It involves controlling a system with linear dynamics and imperfect observations affected by additive noise, with the goal of minimizing a quadratic state and control cost. Under the assumption that noise terms are independent and normally distributed (a case referred to as Linear-Quadratic-Gaussian, or LQG), it is well known that the optimal control policy depends linearly on the observations and can be obtained efficiently by using the Kalman filtering procedure and dynamic programming [8].

Motivated by practical settings where noise distributions may not be readily available or may not be Gaussian, this paper considers a discrete-time, finite-horizon generalization of the LQG setting where noise distributions are unknown and are chosen adversarially from ambiguity sets characterized by a Wasserstein distance and centered around nominal (Gaussian) distributions.

We show that, even under distributional ambiguity, the optimal control policy remains linear in the system's observations. Our proof is novel and does not rely on traditional recursive dynamic programming arguments. Instead, we re-parametrize the control policy in terms of the purified state observations and we derive an upper bound for the resulting minimax formulation by relaxing the ambiguity set (from a Wasserstein ball into a Gelbrich ball) while simultaneously restricting the controller to linear dependencies. We then use convex duality to prove that this upper bound matches

Ça˘ gıl Koçyi˘ git

a lower bound obtained by restricting the ambiguity set in the dual of the minimax formulation. This implies the optimality of linear output feedback controllers, thus generalizing the classic results to a distributionally robust setting.

We also find that the worst-case distribution is actually Gaussian, which leads to a very efficient algorithm for finding optimal controllers. Specifically, we propose an algorithm based on the FrankWolfe first-order method that at every step solves sub-problems corresponding to classic LQG control problems, using Kalman filtering and dynamic programming. We show that this algorithm enjoys a sublinear convergence rate and is susceptible to parallelization. Lastly, we implement the algorithm leveraging PyTorch's automatic differentiation module and we find that it yields uniformly lower runtimes than a direct method (based on solving semidefinite programs) across all problem horizons.

1.1. Literature Review

This paper is related to the ample literature in control theory and engineering aimed at designing controllers that are robust to noise. The classic LQR/LQG theory, developed in the 1960s, examined linear dynamical systems in either time or frequency domain, seeking to minimize a combination of quadratic state and control costs (in time-domain) or the H 2 norm of the system's transfer function (in frequency domain). Motivated by findings that LQG controllers do not provide the guaranteed robust stability properties of LQR controllers [15], much effort has been devoted subsequently to designing controllers that are robust to worst-case perturbations, typically evaluated in terms of the H ∞ norm of the system's transfer function (see, e.g., [16, 53] for a comprehensive review of H ∞ and H 2 controllers). Because H ∞ controllers tend to be overly conservative [32], various approaches have been proposed to balance the performance of nominal and robust controllers, e.g., by combining H 2 and H ∞ approaches [7, 17]. A parallel stream of literature has considered risk-sensitive control [51], which minimizes an entropic risk measure instead of the expected quadratic cost. Although risksensitive control has a distributionally robust flavor (as the entropic risk measure is equivalent to a distributionally robust quadratic objective penalized via Kullback-Leibler divergence), risk-sensitive control models do not admit a distributionally robust formulation because the entropic risk measure is convex, but not coherent [22]. In contrast, our distributionally robust model provides a direct interpretation of the exact set of noise distributions against which the controller provides safeguards, and leads to a computationally tractable framework for finding the optimal controller.

In this sense, our work is more directly related to the literature on distributionally robust control, which seeks controllers that minimize expected costs under worst-case noise distributions [11, 33, 34, 41, 50, 52]. Closest to our work are [28, 33]. [33] proves the optimality of linear state-feedback control policies for a related minimax LQR model with a Wasserstein distance but with perfect state observations. With perfect observations, the optimal policies in the classic LQR formulation are independent of the noise distribution and are thus inherently already robust, so considering imperfect observations is what makes the problem significantly more challenging in our case. [28] studies a minimax formulation based on the Wasserstein distance with both state and observation noise but without any control policy, and focuses solely on the problem of estimating the states. Several papers have considered robust formulations with imperfect observations but for constrained systems [5, 6, 34], which are more challenging; the common approach is to restrict attention to linear feedback policies for computational tractability, and without proving their optimality.

Also related is the recent literature stream on distributionally robust optimization using the Wasserstein distance [36]. Within this stream, the closest work is [38, 44], which consider the problem of minimax mean-squared-error estimation when ambiguity is modeled with a Wasserstein distance from a nominal Gaussian distribution. Our proof builds on some ideas from these papers (e.g., relying on the Gelbrich distance in the construction of the upper bound), which it combines with ideas from control theory on purified output-feedback to obtain the overall construction. Also related is [2], which studies multistage distributionally robust problems with ambiguity sets given by a nested Wasserstein distance for stochastic processes and identifies computationally tractable cases. For a broader overview of developments related to optimal transport and Wasserstein distance with an emphasis on computational tractability and applications in machine learning, we refer to [42].

Finally, our paper is also related to literature that documents the optimality of linear/affine policies in (distributionally) robust dynamic optimization models. [10, 30] prove optimality for one-dimensional linear systems affected by additive noise and with perfect state observations, but with general (convex) state and/or control costs, [27, 49] provide computationally tractable approaches to quantifying the

suboptimality of affine controllers in finite or infinite-horizon settings, and [9, 21, 25] characterize the performance of affine policies in two-stage (distributionally) robust dynamic models.

Notation. All random objects are defined on a probability space (Ω , F , P ) . Thus, the distribution of any random vector ξ : Ω → R d is given by the pushforward distribution P ξ = P ◦ ξ -1 of P with respect to ξ . The expectation under P is denoted by E P [ · ] . For any t ∈ Z + , we set [ t ] = { 0 , . . . , t } .

2. Problem Definition

We consider a discrete-time linear dynamical system

with states x t ∈ R n , control inputs u t ∈ R m , process noise w t ∈ R n and system matrices A t ∈ R n × n and B t ∈ R n × m . The controller only has access to imperfect state measurements

corrupted by observation noise v t ∈ R p , where C t ∈ R p × n and usually p ≤ n (so that observing y t would not allow reconstructing x t even if there were no observation noise). The control inputs u t are causal, i.e., depend on the past observations y 0 , . . . , y t but not on the future observations y t +1 , . . . , y T -1 . More precisely, the set of feasible control inputs U y is the set of random vectors ( u 0 , u 1 , . . . , u T -1 ) where for every t there exists a measurable control policy φ t : R p ( t +1) → R m such that u t = φ t ( y 0 , . . . , y t ) . Controlling the system generates costs that depend quadratically on the states and the controls:

where Q t ∈ S n + and R t ∈ S m ++ represent the state and input cost matrices, respectively. The exogenous random vectors x 0 , { w t } T -1 t =0 and { v t } T -1 t =0 are mutually independent and follow probability distributions given by P x 0 , { P w t } T -1 t =0 , and { P v t } T -1 t =0 , respectively. As the control inputs are causal, the system equations (2) imply that x t , u t and y t can be expressed as measurable functions of the exogenous uncertainties x 0 as well as w s and v s , s ∈ [ t ] , for every t . From now on we may thus assume without loss of generality that Ω = R n × R n × T × R p × T is the space of realizations of the exogenous uncertainties, F is the Borel σ -algebra on Ω and P = P x 0 ⊗ ( ⊗ T -1 t =0 P w t ) ⊗ ( ⊗ T t =0 P v t ) , where P 1 ⊗ P 2 denotes the independent coupling of the distributions P 1 and P 2 .

In this context, the classic LQG model assumes that P is known and Gaussian, and seeks u ∈ U y that minimizes E P [ J ] . Appendix §A reviews the standard approach for computing optimal control inputs by estimating states through Kalman filtering techniques and using dynamic programming.

In contrast, we assume that P is only known to belong to an ambiguity set W , and we formulate a distributionally robust LQG problem that seeks u ∈ U y to minimize the worst-case expected cost:

We construct the ambiguity set W as a ball based on the Wasserstein distance. Specifically, we assume that a nominal Gaussian distribution ˆ P = ˆ P x 0 ⊗ ( ⊗ T -1 t =0 ˆ P w t ) ⊗ ( ⊗ T t =0 ˆ P v t ) is available so that ˆ P x 0 = N (0 , ˆ X 0 ) , ˆ P w t = N (0 , ˆ W t ) , and ˆ P v t = N (0 , ˆ V t ) for all t ∈ [ T -1] , and W is given by:

where

and W is the 2-Wasserstein distance. Thus, by construction, all exogenous random variables x 0 , w 0 , . . . , w T -1 , v 0 , . . . , v T -1 are independent under every distribution in W .

Definition 1 (2-Wasserstein distance) . The 2-Wasserstein distance between two distributions P 1 and P 2 on R d with finite second moments is given by

where Π( P 1 , P 2 ) denotes the set of all couplings, that is, all joint distributions of the random variables ξ 1 and ξ 2 with marginal distributions P 1 and P 2 , respectively.

Our model strictly generalizes the classic LQG setting, 1 which can be recovered by choosing ρ x 0 = ρ w t = ρ v t = 0 . The parameters ρ thus allow quantifying the uncertainty about the nominal model and building robustness to mis-specification. We emphasize that the Wasserstein ambiguity set W contains many non-Gaussian distributions and it is not readily obvious that the worst-case distribution in (4) is in fact Gaussian. However, the set W is also non-convex, as it contains only distributions under which the exogenous uncertainties are independent, which makes the distributionally robust LQG problem potentially difficult to solve.

3. Nash Equilibrium and Optimality of Linear Output Feedback Controllers

We henceforth view the distributionally robust LQG problem as a zero-sum game between the controller, who chooses causal control inputs, and nature, who chooses a distribution P ∈ W . In this section we show that this game admits a Nash equilibrium, where nature's Nash strategy is a Gaussian distribution P ⋆ ∈ W and the controller's Nash strategy is a linear output feedback policy based on the Kalman filter evaluated under P ⋆ .

Purified Observations. Before outlining our proof strategy, we first simplify the problem formulation by re-parametrizing the control inputs in a more convenient form (following [5, 6, 27]). Note that the control inputs in the LQG formulation are subject to cyclic dependencies, as u t depends on y t , while y t depends on x t through (2), and x t depends again on u t through (1), etc. Because these dependencies make the problem hard to analyze, it is preferable to instead consider the controls as functions of a new set of so-called purified observations instead of the actual observations y t .

Specifically, we first introduce a fictitious noise-free system

with states ˆ x t ∈ R n and outputs ˆ y t ∈ R p , which is initialized by ˆ x 0 = 0 and controlled by the same inputs u t as the original system (2). We then define the purified observation at time t as η t = y t -ˆ y t and we use η = ( η 0 , . . . , η T -1 ) to denote the trajectory of all purified observations.

As the inputs u t are causal, the controller can compute the fictitious state ˆ x t and output ˆ y t from the observations y 0 , . . . , y t . Thus, η t is representable as a function of y 0 , . . . , y t . Conversely, one can show by induction that y t can also be represented as a function of η 0 , . . . , η t . Moreover, any measurable function of y 0 , . . . , y t can be expressed as a measurable function of η 0 , . . . , η t and viceversa [27, Proposition II.1]. So if we define U η as the set of all control inputs ( u 0 , u 1 , . . . , u T -1 ) so that u t = ψ t ( η 0 , . . . , η t ) for some measurable function ψ t : R p ( t +1) → R m for every t ∈ [ T -1] , the above reasoning implies that U η = U y .

In view of this, we can rewrite the distributionally robust LQG problem equivalently as

where x = ( x 0 , . . . , x T ) , u = ( u 0 , . . . , u T -1 ) , y = ( y 0 , . . . , y T -1 ) , w = ( x 0 , w 0 , . . . , w T -1 ) , v = ( v 0 , . . . , v T -1 ) , η = ( η 0 , . . . , η T -1 ) , and R , Q , H , G and C are suitable block matrices

(see Appendix §B for their precise definitions). The latter reformulation involving the purified observations η is useful because these are independent of the inputs. Indeed, by recursively combining the equations of the original and the noise-free systems, one can show that η = Dw + v for some block triangular matrix D (see Appendix §B for its construction). This shows that the purified observations depend (linearly) on the exogenous uncertainties but not on the control inputs. Hence, the cyclic dependencies complicating the original system are eliminated in (5).

Subsequently, we also study the dual of (5), defined as

The classic minimax inequality implies that p ⋆ ≥ d ⋆ . If we can prove that p ⋆ = d ⋆ , that (5) has a solution u ⋆ and that (6) has a solution P ⋆ , then ( u ⋆ , P ⋆ ) must be a Nash equilibrium of the zero-sum game at hand [43, Theorem 2]. However, because U η is an infinite-dimensional function space and W is an infinite-dimensional, non-convex set of non-parametric distributions, the existence of a Nash equilibrium (in pure strategies) is not at all evident. Instead, our proof strategy will rely on constructing an upper bound for p ⋆ and a lower bound for d ⋆ , and showing that these match.

Upper Bound for p ⋆ . We obtain an upper bound for p ⋆ by suitably enlarging the ambiguity set W and restricting the controllers u t to linear dependencies. We enlarge W by ignoring all information about the distributions in W except for their covariance matrices, and by replacing the Wasserstein distance with the Gelbrich distance. To that end, we first define the Gelbrich distance on the space of covariance matrices.

Definition 2 (Gelbrich distance) . The Gelbrich distance between the two covariance matrices Σ 1 , Σ 2 ∈ S d + is given by

We are interested in the Gelbrich distance because of its close connection to the 2-Wasserstein distance. Indeed, it is known that the 2-Wasserstein distance between two distributions with zero means is bounded below by the Gelbrich distance between the respective covariance matrices.

Proposition 3.1 (Gelbrich bound [24, Theorem 2.1]) . For any two distributions P 1 and P 2 on R d with zero means and covariance matrices Σ 1 , Σ 2 ∈ S d + , respectively, we have W ( P 1 , P 2 ) ≥ G (Σ 1 , Σ 2 ) .

Recalling that ˆ X 0 , ˆ W t and ˆ V t respectively denote the covariance matrices for x 0 , w t and v t under the nominal distribution ˆ P , we can then define the following Gelbrich ambiguity set for the exogenous uncertainties:

where

By construction, the random vectors x 0 , { w t } T -1 t =0 and { v t } T -1 t =0 are thus mutually independent under any P ∈ G . In addition and as a direct consequence of Proposition 3.1, G constitutes an outer approximation for the Wasserstein ambiguity set W , as summarized in the next result.

Corollary 1 (Gelbrich hull) . We have W ⊆ G .

Because G covers W , we henceforth refer to it as the Gelbrich hull of the Wasserstein ambiguity set W . To finalize our construction of the upper bound on p ⋆ , we focus on linear policies 2 of the form

u = q + Uη = q + U ( Dw + v ) , where q = ( q 0 , . . . , q T -1 ) , and U is a block lower triangular matrix

The block lower triangularity of U ensures that the corresponding controller is causal, which in turn ensures that u ∈ U η . In the following, we denote by U the set of all block lower triangular matrices of the form (7). An upper bound on problem (5) can now be obtained by restricting the controller's feasible set to causal controllers that are linear in the purified observations η and by relaxing nature's feasible set to the Gelbrich hull G of W . The resulting bounding problem is given by

As we obtained (8) by restricting the feasible set of the outer minimization problem and relaxing the feasible set of the inner maximization problem in (5), it is clear that p ⋆ ≥ p ⋆ . Recall also that problem (5) constitutes an infinite-dimensional zero-sum game, where the agents optimize over measurable policies and non-parametric distributions, respectively. In contrast, the next proposition shows that problem (8) is equivalent to a finite-dimensional zero-sum game.

Proposition 3.2. Problem (8) is equivalent to the optimization problem

where

We emphasize that Proposition 3.2 remains valid even if the nominal distribution ˆ P fails to be normal. Note also that, while nature's feasible set in (8) is non-convex due to the independence conditions, the sets G W and G V are convex and even semidefinite representable thanks to the properties of the squared Gelbrich distance. 3 By dualizing the inner maximization problem, one can therefore reformulate the minimax problem (9) as a convex semidefinite program (SDP). Even though this SDP is computationally tractable in theory, it involves O ( T ( mp + n 2 + p 2 )) decision variables. For practically interesting problem dimensions, it thus quickly exceeds the capabilities of existing solvers.

Lower Bound for d ⋆ . To derive a tractable lower bound on d ⋆ , we restrict nature's feasible set to the family W N of all normal distributions in the Wasserstein ambiguity set W . The resulting bounding problem is thus given by

As we obtained (10) by restricting the feasible set of the outer maximization problem in (6), it is clear that d ⋆ ≤ d ⋆ . Next, we show that (10) can be recast as a finite-dimensional zero-sum game. This result critically relies on the following known fact regarding the 2-Wasserstein distance between two normal distributions, which coincides with the Gelbrich distance between their covariance matrices.

Proposition 3.3 (Tightness for normal distributions [26, Proposition 7]) . For any two normal distributions P 1 = N (0 , Σ 1 ) and P 2 = N (0 , Σ 2 ) with zero means we have W ( P 1 , P 2 ) = G (Σ 1 , Σ 2 ) .

With this, we can provide a finite-dimensional reformulation, as summarized in the next result.

Proposition 3.4. Problem (10) is equivalent to the optimization problem

where G W and G V are defined exactly as in Proposition 3.2.

Proposition 3.4 relies on Proposition 3.3 and thus fails to hold unless ˆ P is normal. Also, one can again reformulate (11) as a tractable SDP by dualizing the inner minimization problem.

Conclusions. Propositions 3.2 and 3.4 reveal that problems (9) and (11) are dual to each other, that is, they can be transformed into one another by interchanging minimization and maximization. The following main theorem shows that strong duality holds irrespective of the problem data.

Theorem 3.5 (Strong duality of (9) and (11)) . We have p ⋆ = d ⋆ .

Theorem 3.5 follows immediately from Sion's classic minimax theorem [45], which applies because G W and G V are convex as well as compact thanks to [38, Lemma A.6].

By weak duality and the construction of the bounding problems (9) and (11), we trivially have d ⋆ ≤ d ⋆ ≤ p ⋆ ≤ p ⋆ . Theorem 3.5 reveals that all of these inequalities are in fact equalities, each of which gives rise to a non-trivial insight. The first key insight is that (5) and (6) are strong duals.

Corollary 2 (Strong duality of (5) and (6)) . We have p ⋆ = d ⋆ .

We stress that, unlike Theorem 3.5, Corollary 2 establishes strong duality between two infinitedimensional zero-sum games. The second key implication of Theorem 3.5 is that the distributionally robust LQG problem (5) is solved by a linear output-feedback controller.

Corollary 3 (The controller's Nash strategy is linear in the observations) . There exist U ⋆ ∈ U and q ⋆ ∈ R m such that the distributionally robust LQG problem (5) is solved by u ⋆ = q ⋆ + U ⋆ y .

The identity p ⋆ = p ⋆ readily implies that (5) is solved by a causal controller that is linear in the purified observations. However, any causal controller that is linear in the purified observations η can be reformulated exactly as a causal controller that is linear in the original observations y and vice-versa [6, Proposition 3]. Thus, Corollary 3 follows. The third key implication of Theorem 3.5 is that the dual distributionally robust LQG problem is solved by a normal distribution.

Corollary 4 (Nature's Nash strategy is a normal distribution) . The dual distributionally robust LQG problem (6) is solved by a distribution P ⋆ ∈ W N .

Corollary 4 is a direct consequence of the identity d ⋆ = d ⋆ . Note that the optimal normal distribution P ⋆ is uniquely determined by the covariance matrices W ⋆ and V ⋆ of the exogenous uncertain parameters, which can be computed by solving problem (11). That the worst-case distribution is actually Gaussian is not a-priori expected and is surprising given that the Wasserstein ball contains many non-Gaussian distributions.

4. Efficient Numerical Solution of Distributionally Robust LQG Problems

Having proven these structural results, we next turn attention to the problem of finding the optimal strategies. Our next result shows that, under a mild regularity condition, the optimal controller u ⋆ of the distributionally robust LQG problem (5) can be computed efficiently from P ⋆ .

Proposition 4.1 (Optimality of Kalman filter-based feedback controllers) . If ˆ V t ≻ 0 for all t ∈ [ T -1] , then problem (6) is solved by a Gaussian distribution P ⋆ under which v t has a covariance matrix V ⋆ t ≻ 0 for every t ∈ [ T -1] , and (5) is solved by the optimal LQG controller corresponding to P ⋆ . Additionally, the optimal value of problem (9) and its strong dual (11) does not change if we restrict G W and G V to G + W and G + V , respectively, where

This implies that the optimal controller can be computed by solving a classic LQG problem corresponding to nature's optimal strategy P ⋆ , which can be done very efficiently through Kalman filtering and dynamic programming (see Appendix §A for details). It thus suffices to design an efficient algorithm for computing P ⋆ , which is uniquely determined by the covariance matrices ( W ⋆ , V ⋆ ) that solve problem (11). To this end, we first reformulate (11) as

where we restrict G W and G V to G + W and G + V , respectively, due to Proposition 4.1, and where f ( W,V ) denotes the optimal value function of the inner minimization problem in (11). As (11) is a reformulation of (10) and as the family of all causal purified output-feedback controllers matches the family of causal output-feedback controllers, f ( W,V ) can also be viewed as the optimal value of the classic LQG problem corresponding to the normal distribution P determined by the covariance matrices W and V . These insights lead to the following structural result.

Proposition 4.2. f ( W,V ) is concave and β -smooth in ( W,V ) ∈ G + W ×G + V for some β > 0 .

By Proposition 4.2, it is possible to address problem (12) with a Frank-Wolfe algorithm [13, 18, 19, 20, 23, 35]. Each iteration of this algorithm solves a direction-finding subproblem, that is, a variant of problem (12) that maximizes the first-order Taylor expansion of f ( W,V ) around the current iterates.

The next iterates are then obtained by moving towards a maximizer ( L ⋆ W , L ⋆ V ) of (13), i.e., we update

where α is an appropriate step size. The proposed Frank-Wolfe algorithm enjoys a very low periteration complexity because problem (13) is separable. To see this, we reformulate (13) as

Hence, (13) decomposes into 2 T +1 separate subproblems that can be solved in parallel. That is, for any matrix Z ∈ { X 0 , W 0 , . . . , W T -1 , V 0 , . . . , V T -1 } we solve a separate subproblem of the form

These subproblems can be reformulated as tractable SDPs and are thus amenable to efficient offthe-shelf solvers. By [38, Theorem 6.2], however, one can exploit the structure of the Gelbrich distance in order to reduce (14) to a univariate algebraic equation that can be solved to any desired accuracy δ > 0 by a highly efficient bisection algorithm. We say that L δ Z is a δ -approximate solution of problem (14) for some δ ∈ (0 , 1) if L δ Z is feasible in (14) and if

where L ⋆ Z is an exact maximizer of (14). Note that, by the concavity of f ( W,V ) , the inner product on the right-hand side is nonnegative and vanishes if and only if Z maximizes f ( W,V ) over the feasible set of (14). For further details we refer to Appendix §E in the supplementary material.

Remark 1 (Automatic differentiation) . Recall that f ( W,V ) coincides with the optimal value of the LQG problem corresponding to the normal distribution P determined by the covariance matrices W and V . By using the underlying dynamic programming equations, f ( W,V ) can thus be expressed in closed form as a serial composition of O ( T ) rational functions (see Appendix §A for details). Hence, ∇ Z f ( W,V ) can be calculated symbolically for any Z ∈ { X 0 , W 0 , . . . , W T -1 , V 0 , . . . , V T -1 } by repeatedly applying the chain and product rules. However, the resulting formulas are lengthy and cumbersome. We thus compute the gradients numerically using backpropagation. The cost of evaluating ∇ Z f ( W,V ) is then of the same order of magnitude as the cost of evaluating f ( W,V ) .

A detailed description of the proposed Frank-Wolfe method is given in Algorithm 1 below.

By [31, Theorem 1 and Lemma 7], which applies thanks to the structural properties of f ( W,V ) established in Proposition 4.2, Algorithm 1 attains a suboptimality gap of ϵ within O (1 /ϵ ) iterations.

Algorithm 1 Frank-Wolfe algorithm for solving (12)

10 0

10 1

10 2

Time horizon

10

2

10

1

10 0

10 1

10 2

Execution time (s)

MOSEK

(a)

10 0

10 1

10 2

# of iterations

10

5

10

4

10

3

10

2

10

1

10 0

10 1

10 2

Optimality gap

(b)

5. Numerical Experiments

All experiments are run on an Intel i7-8700 CPU (3.2 GHz) machine with 16GB RAM. All linear SDP problems are modeled in Python 3.8.6 using CVXPY [1, 14] and solved with MOSEK [37]. The gradients of f ( W,V ) are computed via Pymanopt [48] with PyTorch's automated differentiation module [39, 40].

Consider a class of distributionally robust LQG problems with n = m = p = 10 . We set A t = 0 . 1 × A to have ones on the main diagonal and the superdiagonal and zeroes everywhere else ( A i,j = 1 if i = j or i = j -1 and A i,j = 0 otherwise), and the other matrices to B t = C t = Q t = R t = I d . The Wasserstein radii are set to ρ x 0 = ρ w t = ρ v t = 10 -1 . The nominal covariance matrices of the exogenous uncertainties are constructed randomly and with eigenvalues in the interval [1 , 2] (so as to ensure they are positive definite). The code is publicly available in the Github repository https: //github.com/RAO-EPFL/DR-Control .

The optimal value of the distributionally robust LQG problem (5) can be computed by directly solving the SDP reformulation of (11) with MOSEK or by solving the nonlinear SDP (12) with our FrankWolfe method detailed in Algorithm 1. We next compare these two approaches in 10 independent simulation runs, where we set a stopping criterion corresponding to an optimality gap below 10 -3 and we run the Frank-Wolfe method with δ = 0 . 95 . Figure 1a illustrates the execution time for both approaches as a function of the planning horizon T ; runs where MOSEK exceeds 100 s are not reported. Figure 1b visualizes the empirical convergence behavior of the Frank-Wolfe algorithm. The results highlight that the Frank-Wolfe algorithm achieves running times that are uniformly lower than MOSEK across all problem horizons and is able to find highly accurate solutions already after a small number of iterations (50 iterations for problem instances of time horizon T = 10 ).

6. Concluding Remarks and Limitations

In view of the popularity of LQG models, the results in this work carry important theoretical and practical implications. Despite considering a generalization of the classic LQG setting where the noise affecting the system dynamics and the observations follows unknown (and potentially nonGaussian) distributions, our findings suggest that certain classic structural results continue to hold and that highly efficient methods can be adapted to tackle this more realistic (and more challenging) problem. Specifically, that control policies depending linearly on observations continue to be optimal and that the worst-case distribution turns out to be Gaussian is surprising from a theoretical angle and also has direct practical implications, because it allows leveraging the highly efficient Kalman filter in conjunction with dynamic programming and a Frank-Wolfe method to design an efficient computational procedure for solving the problem.

The results also raise several important questions that warrant future exploration. First, it would be highly relevant to consider extensions where the system matrices are also affected by uncertainty, as this captures many applications of practical interest in, e.g., reinforcement learning or revenue management. Second, it would be worth exploring an infinite horizon setting or relaxing the assumption that the nominal distribution is Gaussian, as both assumptions may be limiting the practical appeal of the framework. Third, one could also attempt to prove structural optimality results or design novel algorithms for generating high-quality suboptimal solutions for the more general setting involving constraints on states and/or control inputs. Lastly, one could improve the present algorithmic proposal by exploiting topological properties of the objective so as to guarantee linear convergence rates in the Frank-Wolfe procedure.

Acknowledgements. This research was supported by the Swiss National Science Foundation under the NCCR Automation, grant agreement 51NF40_180545. Dan A. Iancu would like to acknowledge INSEAD for financial support during the duration of the project.

References

Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision , 5(1):42-60, 2018.

Rohit Arora and Rui Gao. Data-driven multistage distributionally robust optimization with nested distance: Time consistency and tractable dynamic reformulations. Available at Optimization Online , 2022.

François Auger, Mickael Hilairet, Josep M. Guerrero, Eric Monmasson, Teresa OrlowskaKowalska, and Seiichiro Katsura. Industrial applications of the Kalman filter: A review. IEEE Transactions on Industrial Electronics , 60(12):5458-5471, 2013.

Richard Bellman. Some inequalities for the square root of a positive definite matrix. Linear Algebra and its Applications , 1(3):321-324, 1968.

Aharon Ben-Tal, Stephen Boyd, and Arkadi Nemirovski. Control of uncertainty-affected discrete time linear systems via convex programming. Available at Optimization Online , 2005.

Aharon Ben-Tal, Stephen Boyd, and Arkadi Nemirovski. Extending scope of robust optimization: Comprehensive robust counterparts of uncertain problems. Mathematical Programming , 107(1):63-89, 2006.

Denis S. Bernstein and Wassim M. Haddad. LQG control with an H ∞ performance bound: A Riccati equation approach. In American Control Conference , pages 796-802, 1988.

Dimitri Bertsekas. Dynamic Programming and Optimal Control , volume I. Athena Scientific, 2017.

Dimitris Bertsimas and Vineet Goyal. On the power and limitations of affine policies in two-stage adaptive optimization. Mathematical Programming , 134(2):491-531, 2012.

Dimitris Bertsimas, Dan A. Iancu, and Pablo A. Parrilo. Optimality of affine policies in multistage robust optimization. Mathematics of Operations Research , 35(2):363-394, 2010.

Dimitris Bertsimas, Dan A. Iancu, and Pablo A. Parrilo. A hierarchy of near-optimal policies for multistage adaptive optimization. IEEE Transactions on Automatic Control , 56(12):2809-2824, 2011.

Shen-Yong Chen. Kalman filter for robot vision: A survey. IEEE Transactions on Industrial Electronics , 59(11):4409-4420, 2012.

Vladimir F. Demyanov and Aleksandr M. Rubinov. Approximate Methods in Optimization Problems . Elsevier, 1970.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

John Doyle. Guaranteed margins for LQG regulators. IEEE Transactions on Automatic Control , 23(4):756-757, 1978.

John Doyle, Keith Glover, Pramod Khargonekar, and Bruce Francis. State-space solutions to standard H 2 and H ∞ control problems. In American Control Conference , pages 1691-1696, 1988.

John Doyle, Kemin Zhou, and Bobby Bodenheimer. Optimal control with mixed H 2 and H ∞ performance objectives. In American Control Conference , pages 2065-2070, 1989.

Joseph C. Dunn. Rates of convergence for conditional gradient algorithms near singular and nonsingular extremals. SIAM Journal on Control and Optimization , 17(2):187-211, 1979.

Joseph C. Dunn. Convergence rates for conditional gradient sequences generated by implicit step length rules. SIAM Journal on Control and Optimization , 18(5):473-487, 1980.

Joseph C. Dunn and S. Harshbarger. Conditional gradient algorithms with open loop step size rules. Journal of Mathematical Analysis and Applications , 62(2):432-444, 1978.

Omar El Housni and Vineet Goyal. On the optimality of affine policies for budgeted uncertainty sets. Mathematics of Operations Research , 46(2):674-711, 2021.

Hans Föllmer and Alexander Schied. Stochastic Finance: An Introduction in Discrete Time . de Gruyter, 2011.

Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logistics , 3(1-2):95-110, 1956.

Matthias Gelbrich. On a formula for the L 2 Wasserstein metric between measures on Euclidean and Hilbert spaces. Mathematische Nachrichten , 147(1):185-203, 1990.

Angelos Georghiou, Angelos Tsoukalas, and Wolfram Wiesemann. On the optimality of affine decision rules in robust and distributionally robust optimization. Available at Optimization Online , 2021.

Clark R. Givens and Rae M. Shortt. A class of Wasserstein metrics for probability distributions. Michigan Mathematical Journal , 31(2):231-240, 1984.

Michael J. Hadjiyiannis, Paul J. Goulart, and Daniel Kuhn. An efficient method to estimate the suboptimality of affine controllers. IEEE Transactions on Automatic Control , 56(12):2841-2853, 2011.

Bingyan Han. Distributionally robust Kalman filtering with volatility uncertainty. arXiv preprint arXiv:2302.05993 , 2023.

Lars Peter Hansen and Thomas J. Sargent. Robust estimation and control under commitment. Journal of Economic Theory , 124(2):258-301, 2005.

Dan A. Iancu, Mayank Sharma, and Maxim Sviridenko. Supermodularity and affine policies in dynamic robust optimization. Operations Research , 61(4):941-956, 2013.

Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In International Conference on Machine Learning , pages 427-435, 2013.

Aren Karapetyan, Andrea Iannelli, and John Lygeros. On the regret of H ∞ control. In IEEE Conference on Decision and Control , pages 6181-6186, 2022.

Kihyun Kim and Insoon Yang. Distributional robustness in minimax linear quadratic control with Wasserstein distance. SIAM Journal on Control and Optimization , 61(2):458-483, 2023.

Georgios Kotsalis, Guanghui Lan, and Arkadi S. Nemirovski. Convex optimization for finitehorizon robust covariance control of linear stochastic systems. SIAM Journal on Control and Optimization , 59(1):296-319, 2021.

Evgenii S. Levitin and Boris T. Polyak. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics , 6(5):1-50, 1966.

Peyman Mohajerin Esfahani and Daniel Kuhn. Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. Mathematical Programming , 171(1-2):115-166, 2018.

MOSEK ApS. The MOSEK Optimization Toolbox. Version 9.2. , 2019.

Viet Anh Nguyen, Soroosh Shafieezadeh-Abadeh, Daniel Kuhn, and Peyman Mohajerin Esfahani. Bridging Bayesian and minimax mean square error estimation via Wasserstein distributionally robust optimization. Mathematics of Operations Research , 48(1):1-37, 2023.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS 2017 Autodiff Workshop , 2017.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pages 8026-8037, 2019.

Ian R. Petersen, Matthiew R. James, and Paul Dupuis. Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Transactions on Automatic Control , 45(3):398-412, 2000.

Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning , 11(5-6):355-607, 2019.

R. Tyrrell Rockafellar. Conjugate Duality and Optimization . SIAM, 1974.

Soroosh Shafieezadeh-Abadeh, Viet Anh Nguyen, Daniel Kuhn, and Peyman Mohajerin Esfahani. Wasserstein distributionally robust Kalman filtering. In Advances in Neural Information Processing Systems , pages 8483-8492, 2018.

Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics , 8(1):171-176, 1958.

Joelle Skaf and Stephen P. Boyd. Design of affine controllers via convex optimization. IEEE Transactions on Automatic Control , 55(11):2476-2487, 2010.

Emanuel Todorov and Michael I. Jordan. Optimal feedback control as a theory of motor coordination. Nature Neuroscience , 5(11):1226-1235, 2002.

James Townsend, Niklas Koep, and Sebastian Weichwald. Pymanopt: A Python toolbox for optimization on manifolds using automatic differentiation. Journal of Machine Learning Research , 17(137):1-5, 2016.

Bart P. G. Van Parys, Paul J. Goulart, and Manfred Morari. Infinite horizon performance bounds for uncertain constrained systems. IEEE Transactions on Automatic Control , 58(11):2803-2817, 2013.

Bart P. G. Van Parys, Daniel Kuhn, Paul J. Goulart, and Manfred Morari. Distributionally robust control of constrained stochastic systems. IEEE Transactions on Automatic Control , 61(2):430-442, 2016.

Peter Whittle. Risk-sensitive linear/quadratic/Gaussian control. Advances in Applied Probability , 13(4):764-777, 1981.

Insoon Yang. Wasserstein distributionally robust stochastic control: A data-driven approach. IEEE Transactions on Automatic Control , 66(8):3863-3870, 2021.

Kemin Zhou and John C. Doyle. Essentials of Robust Control . Prentice Hall, 1998.

Appendix

The supplementary material is structured as follows. Appendix §A presents the well-known solution to the classic LQG problem using dynamic programming and Kalman Filter estimation. Appendix §B provides the definitions of the stacked system matrices utilized in the compact formulation (5) of the distributionally robust LQG problem. Appendix §C contains the proofs of the formal statements in the main text and provides additional technical results. Appendix §D derives the SDP reformulation of the dual problem (11). Appendix §E, finally, elaborates on the bisection algorithm used for solving the linearization oracle of the Frank-Wolfe algorithm.

A. Solution of the LQG Problem

The classic LQG problem can be solved efficiently via dynamic programming; see, e.g., [8]. That is, the unique optimal control inputs satisfy u ⋆ t = K t ˆ x t for every t ∈ [ T -1] , where K t ∈ R n × n is the optimal feedback gain matrix, and ˆ x t = E P [ x t | y 0 , . . . , y t ] is the minimum mean-squared-error estimator of x t given the observation history up to time t . Thanks to the celebrated separation principle, K t can be computed by pretending that the system is deterministic and allows for perfect state observations, and ˆ x t can be computed while ignoring the control problem.

To compute K t , one first solves the deterministic LQR problem corresponding to the LQG problem at hand. Its value function x ⊤ t P t x t at time t is quadratic in x t , and P t obeys the backward recursion

initialized by P T = Q T . The optimal feedback gain matrix K t can then be computed from P t +1 as

Since x t and ( y 0 , . . . , y t ) are jointly normally distributed, the minimum mean-squared-error estimator ˆ x t can be calculated directly using the formula for the mean of a conditional normal distribution. Alternatively, however, one can use the Kalman filter to compute ˆ x t recursively, which is significantly more insightful and efficient. The Kalman filter also recursively computes the covariance matrix Σ t of x t conditional on y 0 , . . . , y t and the covariance matrix Σ t +1 | t of x t +1 conditional on y 0 , . . . , y t evaluated under P . Specifically, these covariance matrices obey the forward recursion

initialized by Σ 0 |-1 = X 0 . Using Σ t | t -1 , we then define the Kalman filter gain as

which allows us to compute the minimum mean-squared-error estimator via the forward recursion

initialized by ˆ x 0 = L 0 y 0 . One can also show that the optimal value of the LQG problem amounts to

B. Definitions of Stacked System Matrices

The stacked system matrices appearing in the distributionally robust LQG problem (5) are defined as follows. First, the stacked state and input cost matrices Q ∈ S n ( T +1) and R ∈ S mT are set to

respectively. Similarly, the stacked matrices appearing in the linear dynamics and the measurement equations C ∈ R pT × n ( T +1) , G ∈ R n ( T +1) × n ( T +1) and H ∈ R n ( T +1) × mT are defined as

and

respectively, where A t s = ∏ t -1 k = s A k for every s < t and A t s = I n for s = t .

Using the stacked system matrices, we can now express the purified observation process η as a linear function of the exogenous uncertainties w and v that is not impacted by u ; see also [5, 46]

Lemma B.1. We have η = Dw + v , where D = CG .

Proof of Lemma B.1. The purified observation process is defined as η = y -ˆ y . Recall now that the observations of the original system satisfy y = Cx + v . Similarly, one readily verifies that the observations of the fictitious noise-free system satisfy ˆ y = C ˆ x . Thus, we have η = C ( x -ˆ x ) + v . Next, recall that the state of the original system satisfies x = Hu + Gw , and note that the state of the fictitious noise-free system satisfies ˆ x = Hu . Combining all of these linear equations finally shows that u cancels out and that η = CGw + v = Dw + v .

C. Proofs

C.1. Additional Technical Results

It is well known that every causal controller that is linear in the original observations y can be reformulated as a causal controller that is linear in the purified observations η and vice versa [5, 46]. Perhaps surprisingly, however, the one-to-one transformation between the respective coefficients of y and η is not linear. To keep this paper self-contained, we review these insights in the next lemma.

Lemma C.1. If u = Uη + q for some U ∈ U and q ∈ R pT , then u = U ′ y + q ′ for U ′ = ( I + UCH ) -1 U and q ′ = ( I + UCH ) -1 q . Conversely, if u = U ′ y + q ′ for some U ′ ∈ U and q ′ ∈ R pT , then u = Uη + q for U = ( I -U ′ CH ) -1 U ′ and q = ( I -U ′ CH ) -1 q ′ .

Proof of Lemma C.1. If u = Uη + q for some U ∈ U and q ∈ R pT , then we have

where the second equality follows from the definition of η , the third equality holds because y = Cx + v , and the last equality exploits our earlier insight that ˆ y = C ˆ x . The last expression depends only on y and u . Solving for u yields u = U ′ y + q ′ , where U ′ = ( I + UCH ) -1 U and q ′ = ( I + UCH ) -1 q . Note that ( I + UCH ) is indeed invertible because I + UCH is a lower triangular matrix with all diagonal entries equal to one, ensuring a determinant of one.

Similarly, if u = U ′ y + q ′ for some U ′ ∈ U and q ′ ∈ R pT , then we have

Solving for u yields u = Uη + q , where U = ( I -U ′ CH ) -1 U ′ and q = ( I -U ′ CH ) -1 q ′ . Note again that ( I -U ′ CH ) is indeed invertible because ( I -U ′ CH ) is a lower triangular matrix with all diagonal entries equal to one.

C.2. Proofs of Section 3

Proof of Proposition 3.2. In problem (8), both u and x are linear in w and v , i.e., u = q + UDw + Uv and x = Hu + Gw = Hq + HUDw + HUv + Gw . By substituting the linear representations of u and x into the objective function of problem (8), we obtain the following equivalent reformulation.

For any fixed P ∈ G , we can express the expectation in the objective function of the above problem in terms of the covariance matrices W = E P [ ww ⊤ ] and V = E P [ vv ⊤ ] . Thus, the problem becomes

Recall now the definition of G , and note that the requirements G ( X 0 , ˆ X 0 ) ≤ ρ x 0 , G ( W t , ˆ W t ) ≤ ρ w t and G ( V t , ˆ V t ) ≤ ρ v t are equivalent to the convex constraints G ( X 0 , ˆ X 0 ) 2 ≤ ρ 2 x 0 , G ( W t , ˆ W t ) 2 ≤ ρ 2 w t and G ( V t , ˆ V t ) 2 ≤ ρ 2 v t , respectively, for all t ∈ [ T -1] . The definition of G also implies that

Problem (A.18) thus constitutes a relaxation of problem (9). Indeed, the feasible set of the inner maximization problem in (A.18) is a subset of the feasible set of the inner maximization problem in (9). Moreover, for any W and V feasible in the inner maximization problem in (9), the distribution P = P x 0 ⊗ ( ⊗ T -1 t =0 P w t ) ⊗ ( ⊗ T t =0 P v t ) defined through P x 0 = N (0 , X 0 ) , P w t = N (0 , W t ) and P v t = N (0 , V t ) , t ∈ [ T -1] , is feasible in the inner maximization problem in (A.18) with the same objective value. The relaxation is thus exact, and the optimal values of (8), (9) and (A.18) coincide.

Proof of Proposition 3.4. Recall that the space U y of all causal output-feedback controllers coincides with the space U η of all causal purified output-feedback controllers. We can thus replace the feasible set U η of the inner minimization problem in (10) with U y . Hence, for any fixed P ∈ W N , the inner minimization problem in (10) constitutes a classic LQG problem. By standard LQG theory [8], it is solved by a linear output-feedback controller of the form u = U ′ y + q ′ for some U ′ ∈ U and q ′ ∈ R pT ; see also Appendix §A. Lemma C.1 shows, however, that any linear output-feedback controller can be equivalently expressed as a linear purified -output feedback controller of the form u = Uη + q for some U ∈ U and q ∈ R pT . In summary, the above reasoning shows that the feasible set of the inner minimization problem in (10) can be reduced to the family of all linear purified-output feedback controllers without sacrificing optimality. Thus, problem (10) is equivalent to

Using a similar reasoning as in the proof of Proposition 3.2, we can now substitute the linear representations of u and x into the objective function and reformulate the above problem as

As W N contains only normal distributions, Proposition 3.3 implies that W ( P x 0 , ˆ P x 0 ) = G ( X 0 , ˆ X 0 ) , W ( P w t , ˆ P w t ) = G ( W t , ˆ W t ) and W ( P v t , ˆ P v t ) = G ( V t , ˆ V t ) for all t ∈ [ T -1] . We may thus replace the requirement W ( P x 0 , ˆ P x 0 ) ≤ ρ x 0 in the definition of W N by G ( X 0 , ˆ X 0 ) ≤ ρ x 0 , which is equivalent to the convex constraint G ( X 0 , ˆ X 0 ) 2 ≤ ρ 2 x 0 . The conditions on the marginal distributions of w t and v t , t ∈ [ T -1] , admit similar reformulations. The definition of W N also implies that

Thus, the feasible set of the outer maximization problem in (11) constitutes a relaxation of that in (10). One readily verifies that the relaxation is exact by using similar arguments as in the proof of Proposition 3.2. Thus, the claim follows.

Proof of Theorem 3.5. By Proposition 3.2, ¯ p ⋆ coincides with the minimum of (9). Similarly, by Proposition 3.4 d ⋆ coincides with the maximum of (11). Note that problems (9) and (11) only differ by the order of minimization and maximization. Note also that U is convex and closed, G W and G V are convex and compact by virtue of [38, Lemma A.6], and the (identical) trace terms in (9) and (11) are bilinear in ( W,V ) and ( U, q ) . The claim thus follows from Sion's minimax theorem [45].

C.3. Proofs of Section 4

Note that Proposition 4.1 is consistent with Corollary 3 because the optimal LQG controller corresponding to P ⋆ is linear in the past observations.

Proof of Proposition 4.1. By [38, Lemma A.3], the inner problem in (9) admits a maximizer ( W ⋆ , V ⋆ ) with X ⋆ 0 ⪰ λ min ( ˆ X 0 ) as well as W ⋆ t ⪰ λ min ( ˆ W t ) and V ⋆ t ⪰ λ min ( ˆ V t ) for all t ∈ [ T -1] . Thus, the optimal value of problem (9) and its strong dual (11) does not change if we restrict G W and G V to G + W and G + V , respectively. We may thus conclude that problem (11) has a maximizer ( W ⋆ , V ⋆ ) with V ⋆ t ⪰ λ min ( ˆ V t ) ≻ 0 for all t ∈ [ T -1] . This in turn implies that problem (6) is solved by a normal distribution P ⋆ under which the covariance matrix of the observation noise v t satisfies V ⋆ t ≻ 0 for every t ∈ [ T -1] . As (5) and (6) are strong duals, the optimal solution u ⋆ of problem (5) forms a Nash equilibrium with P ⋆ , i.e., u ⋆ is a best response to P ⋆ and thus solves the classic LQG problem corresponding to P ⋆ . As R t ≻ 0 for every t ∈ [ T -1] , this best response u ⋆ is unique, and as V ⋆ T ≻ 0 for every t ∈ [ T -1] , u ⋆ is in fact the Kalman filter-based optimal outputfeedback strategy corresponding to P ⋆ (which can be obtained using the techniques highlighted in Appendix §A).

Before proving Proposition 4.2, recall that f ( W,V ) is called β -smooth for some β > 0 if

where ∥ · ∥ F denotes the Frobenius norm.

Proof of Proposition 4.2. The function f ( W,V ) is concave because the objective function of the inner minimization problem in (11) is linear (and hence concave) in W and V and because concavity is preserved under minimization. To prove that f ( W,V ) is β -smooth, we first recall from Proposition 3.3 that it coincides with the optimal value of the inner minimization problem in (10). As U η = U y , f ( W,V ) can thus be viewed as the optimal value of the classic LQG problem corresponding to the normal distribution P determined by the covariance matrices W and V . Hence, f ( W,V ) coincides with (A.17), where Σ t , for t ∈ [ T -1] , is a function of ( W,V ) defined recursively through the Kalman filter equations (A.16). Note that all inverse matrices in (A.16) are well-defined because any V ∈ G + V is strictly positive definite. Therefore, Σ t constitutes a proper rational function (that is, a ratio of two polyonmials with the polynomial in the denominator being strictly positive) for every t ∈ [ T -1] . Thus, f ( W,V ) is infinitely often continuously differentiable on a neighborhood of G + W ×G + V .

As f ( W,V ) is concave and (at least) twice continuously differentiable, it is β -smooth on G + W ×G + V if and only if the largest eigenvalue of the Hessian matrix of -f ( W,V ) is bounded above by β throughout G + W × G + V . Also, the largest eigenvalue of the positive semidefinite Hessian matrix ∇ 2 ( -f ( W,V )) coincides with the spectral norm of ∇ 2 f ( W,V ) . We may thus set

where ∥ · ∥ 2 denotes the spectral norm. The supremum in the above maximization problem is finite and attained thanks to Weierstrass' theorem, which applies because f ( W,V ) is twice continuously differentiable and the spectral norm is continuous, while the sets G + W and G + V are compact by virtue of [38, Lemma A.6]. This observation completes the proof.

D. SDP Reformulation of the Lower Problem (11)

Instead of solving the dual problem (11) with the customized Frank-Wolfe algorithm of Section 4, it can be reformulated as an SDP amenable to off-the-shelf solvers. This reformulation is obtained by dualizing the inner minimization problem and by exploiting the following preliminary lemma.

Lemma D.1. For any ˆ Z ∈ S d and ρ z ≥ 0 , the set G Z = { Z ∈ S d : G ( Z, ˆ Z ) ≤ ρ z }

Proof of Lemma D.1. By Definition 2, we have

Next, introduce an auxiliary variable E z ∈ S d + subject to the matrix inequality E 2 z ⪯ ( ˆ Z 1 2 Z ˆ Z 1 2 ) . By [4, Theorem 1], this inequality can be recast as E z ⪯ ( ˆ Z 1 2 Z ˆ Z 1 2 ) 1 2 . Hence, we can reformulate the nonlinear matrix inequality in the above representation of G Z as Tr( Z + ˆ Z -2 E z ) ≤ ρ 2 z . A standard Schur complement argument reveals that the inequality E 2 z ⪯ ( ˆ Z 1 2 Z ˆ Z 1 2 ) is also equivalent to

The claim then follows by combining all of these insights.

We are now ready to derive the desired SDP reformulation of problem (11).

Proposition D.2. If ˆ V ≻ 0 , then problem (11) is equivalent to the SDP

Here, M denotes the set of all strictly upper block triangular matrices of the form

where M t,s ∈ R m × p for every t, s ∈ Z with 1 ≤ t < s ≤ T .

Proof of Proposition D.2. The proof relies on dualizing the inner minimization problem in (11). Note that strong duality holds because the primal problem is trivially feasible and involves only equality constraints, which implies that any feasible point is in fact a Slater point. In the following we use M ∈ M to denote the Lagrange multiplier of the constraint U ∈ U , which requires all blocks of

the matrix U above the main diagonal to vanish. The Lagrangian function of the inner minimization problem in (11) can therefore be represented as

Recall now that R ≻ 0 and Q ⪰ 0 , and thus R + H ⊤ QH ≻ 0 . Consequently, L is minimized by q ⋆ = 0 for any fixed U and M . In addition, the partial gradient of L with respect U is given by

Recall also that V ∈ G + V is strictly positive, which implies that DWD ⊤ + V ≻ 0 is invertible. As we already know that R + H ⊤ QH ≻ 0 is invertible, as well, L is minimized by

for any fixed M . Substituting both q ⋆ and U ⋆ into L yields the dual objective function

The dual of the inner minimization problem in (11) is thus given by max M ∈M g ( M ) . To linearize the dual objective function, we next introduce an auxiliary variable F ∈ S mT + subject to the matrix inequality F ⪰ ( H ⊤ QGWD ⊤ + M/ 2)( DWD ⊤ + V ) -1 ( H ⊤ QGWD ⊤ + M/ 2) ⊤ . By using a standard Schur complement reformulation, we can then rewrite the dual problem as

Next, by replacing the inner problem in (11) with its strong dual (A.21), we can reformulate (11) as

By Proposition 4.1, the inclusion of the constraints X 0 ⪰ λ min ( ˆ X 0 ) I , W t ⪰ λ min ( ˆ W t ) I and V t ⪰ λ min ( ˆ V t ) I for all t ∈ [ T -1] has no effect on the solution to problem (A.22). In addition, by Lemma D.1, each (non-linear) Gelbrich constraint in (A.22) can be reformulated as an equivalent (linear) SDP constraint. Thus, problem (A.22) reduces to (A.20), and the claim follows.

E. Bisection Algorithm for the Linearization Oracle

We now show that the direction-finding subproblem (14) can be solved efficiently via bisection. To this end, we first establish that (14) can be reduced to the solution of a univariate algebraic equation.

̸

Proposition E.1 ([38, Proposition A.4 (iii)]) . If ˆ Z ∈ S d ++ , Γ Z ∈ S d + , Γ Z = 0 and ρ z ∈ R ++ , then

is uniquely solved by L ⋆ = ( γ ⋆ ) 2 ( γ ⋆ I -Γ Z ) -1 ˆ Z ( γ ⋆ I -Γ Z ) -1 , where γ ⋆ is the unique solution of

in the interval ( λ max (Γ Z ) , ∞ ) .

In practice, we need to solve the algebraic equation (A.24) numerically. The numerical error in approximating γ ⋆ should be contained to ensure that L ⋆ approximates the exact maximizer of problem (A.23). The next proposition shows that, for any tolerance δ ∈ (0 , 1) , a δ -approximate solution of (A.23) can be computed with an efficient bisection algorithm.

̸

Proposition E.2 ([38, Theorem 6.4]) . For any fixed ρ z ∈ R ++ , ˆ Z ∈ S d ++ and Γ Z ∈ S d + , Γ Z = 0 , define G + Z = { Z ∈ S d + : G ( Z, ˆ Z ) ≤ ρ z , Z ⪰ λ min ( ˆ Z ) } as the feasible set of problem (A.23) , and let Z ∈ G + Z be any reference covariance matrix. Additionally, let δ ∈ (0 , 1) be the desired oracle precision, and define φ ( γ ) = γ ( ρ 2 + ⟨ γ ( γI -Γ Z ) -1 -I, ˆ Z ⟩ ) -⟨ Z, Γ Z ⟩ for any γ > λ max (Γ Z ) . Then, Algorithm A.2 returns in finite time a matrix L δ Z ∈ S d + with the following properties. (i) Feasibility: L δ Z ∈ G + Z (ii) δ -Suboptimality: ⟨ L δ Z -Z, Γ Z ⟩ ≥ δ max L ∈G + Z ⟨ Γ Z , L -Z ⟩ .

Algorithm A.2 Bisection algorithm to compute L δ

̸

In summary, for any Z ∈ { X 0 , W 0 , . . . , W T -1 , V 0 , . . . , V T -1 } , Algorithm A.2 computes a δ -approximate solutions to the direction-finding subproblem (14) with Γ Z = ∇ Z f ( W,V ) .

F. Additional Information on Experiments

Generation of Nominal Covariance Matrices. The nominal covariance matrices of the exogenous uncertainties are constructed randomly using the following procedure. For each exogenous uncertainty z ∈ { x 0 , w 0 , . . . , w T -1 , v 0 , . . . , v T -1 } , we denote the dimension of z by d and sample a matrix M Z ∈ R d × d from the uniform distribution on the hypercube [0 , 1] d × d . Next, we define Ξ Z ∈ R d × d as the orthogonal matrix whose columns represent the orthonormal eigenvectors of the symmetric matrix M Z + M ⊤ Z . Finally, we set ˆ Z = Ξ Z Λ Z Ξ ⊤ Z , where Λ Z is a diagonal matrix whose main diagonal is sampled uniformly from the interval [1 , 2] d . The rationale for adopting this cumbersome procedure is to ensure that the covariance matrix ˆ Z is positive definite.

Optimality Gap. The optimality gap of the Frank-Wolfe algorithm visualized in Figure 1b is calculated as the sum of the surrogate optimality gaps ⟨ L δ Z -Z, ∇ Z f ( W,V ) ⟩ across all Z ∈ { X 0 , W 0 . . . , W T -1 , V 0 , . . . , V T -1 } . For more information on the surrogate optimality gaps see [31].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper **Distributionally Robust Linear Quadratic Control** considers the case of finite-time linear quadratic optimal control with process and measurement noises and known time-varying dynamics. The novelty lies in the fact that the distributions of the initial state and noises are unknown but lie in an ambiguity set defined as a 2-Wasserstein ball centered in a known Gaussian distribution and with known radius. The objective is to minimize the expected LQR cost with the distributions chosen adversarially in that ball. The authors call this adversarial problem \"distributionally robust Linear Quadratic Gaussian\". It strict generalizes LQG control, which assumes these distributions are known and Gaussian. \nThe main contribution is twofold. The first part is theoretical, as the authors prove the existence of a Gaussian adversarial distribution, that is, the distributionally robust LQG reduces to classic LQG with unknown Gaussian distributions. A consequence is that the optimal controller is a linear state feedback, like in the classic LQG case. This leads to the second contribution: an algorithm to efficiently estimate the adversarial Gaussian distribution from data. The authors claim that, then, the LQG problem with estimated noise distributions can be solved efficiently by using classical methods.\nThe authors illustrate convergence of their estimation algorithm on a simulated example.\n\nSTRENGTHS:\nThe paper is interesting and the clear exposition makes the reasonings easy to follow. The appendix is well-managed, and I appreciate Appendix A on the solution of classical LQG with a Kalman filter.\nI like the idea of allowing for a whole family of noise distributions rather than assuming a fixed one. The main theoretical result helps mitigate the restrictiveness of the commonly-accepted \"white noise assumption\"; indeed, it shows that allowing for a larger family of noises does not provide any benefits (for distributions close enough to Gaussians). This new problem formulation thus seems relevant. The subsequent algorithm to solve the distributionally robust LQG problem follows naturally and further justifies the interest of the theoretical result. Overall, the reasoning exposed is sound and well-motivated.\nFinally, I appreciate that the authors went the extra mile in Section IV by augmenting the theoretical result with a data-efficient algorithm.\n\nWEAKNESSES:\nThe three main weaknesses of the paper are, in my opinion, 1. the lack of thoroughness of the simulation study; 2. the lack of discussion of the effect of hyperparameters; and 3. some arguments are unclear and should be explicited (although I do not question their conclusions). I detail these three points. These points are not critical for acceptance, but I believe the paper would be improved by addressing them.\n\nW1) The simulation study shows convergence of the algorithm on a randomized use case. This is a good sanity check, but I would also appreciate a simulation of the case when the noise distribution is not normal but still within the Wasserstein ball. The theoretical result ensures that Nature's adversarial strategy _is_ normal, but an illustration of the case when Nature is sub-adversarial would be welcome. In particular, is the cost of the learned policy reduced compared to when the noise is normal?\n\nW2) The role of the choice of ball radii is not discussed. While there is obviously no notion of \"optimal\" radius, since it simply corresponds to a degree of robustness, I would like to see the evolution in performance with increasing radii. Intuition tells that performance should decrease as robustness increases, but a confirmation in simulation would be welcome.\n\nW3) Some arguments are unclear to me, although their conclusions seem to be valid. In particular:\n   1. The authors make multiple claims on convexity of sets of probability measures. Examples are lines 112, 151, 189. I understand that these claims are made by considering Borel measures as a subset of the vector space of signed Borel measures with standard addition and scalar multiplication. I believe this superset should be mentioned explicitly at least ones, as it is rather unusual and notions of convexity of a metric space exist (and differ).\n      This is also relevant on line 151, where $\\mathcal{W}$ is claimed to be infinite-dimensional despite not being a vector space.\n   2. With this understanding of convexity, why is the set $\\mathcal{W}$ non convex, as claimed e.g. on lines 112, 151 and 189? As far as I understand, each set $\\mathcal{W}_{z}$ is convex, with $z\\in\\{x_0, w_t, v_t\\}$. Then, $\\mathcal{W}$ should be convex as the cartesian product of these sets. The only way I understand this non-convexity is if $\\mathcal{W}$ is not _equal_ to the cartesian product, but only isometric to this cartesian product by the mapping that computes the marginal distributions. I believe this should be mentioned explicitly, at least in a footnote, as this questions distracted me from more central claims of the paper for a while.\n   3. I am unsure about the claim on line 131 that the controller can compute the fictious states $\\hat x_0,\\dots,\\hat x_t$ from the real observations $y_0, \\dots, y_t$ without knowing the initial state. Indeed, this would imply in particular that the controller can reconstruct the initial state. Since the time origin is arbitrary, any state could be reconstructed. This claim appear unnecessary for the rest of the argument and should be either removed or clarified in my opinion.\n   4. I find the formulation of the first sentence of Proposition 4.1 extremely confusing. In particular, what comes after \"then\" in the first sentence reads as a logical consequence of what precedes whereas it is actually a definition of the symbols $\\mathbb{P}^\\star$ and $V_t^\\star$. I recommend reformulating.\n   5. In Proposition 4.2, I recommend avoiding using the term \"smooth\". While it has a precise meaning in a specific branch of mathematics, it is often only used informally in control. I would prefer the more standard \"infinitely differentiable with $\\beta$-Lipschitz gradient\".\n\nQUESTIONS:\nMy questions are detailed in the above paragraph on weaknesses.  I would appreciate if the authors could respond to these.\n\nLIMITATIONS:\nThe authors have adequately addressed limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a robust method for controlling LQ systems, where process and observation noise distribution laws are unknown, but noise samples are assumed to be independent, zero mean and have their distributions lying close to a a nominal Gaussian distribution in Wasserstein-2 space.\n\nA pair of relaxed settings is used to prove a strong duality between the minimax\\maximin problems, showing that the worst-case distribution is Gaussian. A numerical method is proposed for computing this distribution for long horizons.\n\nSTRENGTHS:\nThe paper itself is well-written and gives a thorough overview of the problem.\n\nThe fact that the solution for the minimax problem is given by a Gaussian distribution \\ linear controller, even if not surprising considering the nature of the quadratic cost and $W_2$/Gelbrich distance, is still not clear from the outset and is important enough from a theoretical point of view. Its proof seems correct as well.\n\nWEAKNESSES:\nMy main concern here, is that the theoretical contribution is limited to one main result, which is confined to a Gaussian-centered ambiguity set, i.e. the real distribution is assumed to be close to a Gaussian one. This calls for an additional study of other nominal distributions (as the Gelbrich distance equals to $W_2$ for other families of distributions [18], and discussion may be confined to linear filters) or mixtures, or at least for a detailed discussion about the practical implications of this choice (beyond that of the 'concluding remarks' in Sec.6) i.e. about whether this hypothesis may (or might not) be practical in real problems.\n\nHowever, even if limited, the contribution is still novel and important enough to warrant acceptance.\n\nQUESTIONS:\nSome minor comments \\ typos:\n\nIt is somewhat misleading to call $\\mathcal{W}$ a 'Wasserstein ball', e.g. l.33, while it is not even convex, but I guess that's forgivable since this set is defined clearly.\n\nl. 93 dimension should be $p \\times T$. \n\nl.104 it would probably be more didactic to introduce the Wasserstein distance before using it to define  $\\mathcal{W}$.\n\nLIMITATIONS:\nThe authors addressed the limitations, but a more detailed discussion about the assumption that the nominal distribution is Gaussian is required.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a distributionally robust version of the output feedback linear quadratic control problem. The goal is to control a system with known partially observed linear dynamics in the face of stochastic disturbances. The stochastic disturbances (both measurement and state disturbances) are drawn from an unknown distribution. This distribution is assumed to belong to some ambiguity set centered at a known gaussian distribution. The disturbances drawn from any distribution in the ambiguity set are assumed to be mean zero, and independent across time, with measurement disturbances independent from the process disturbances. Furthermore, the marginal distribution for the measurement or process disturbance at each time is assumed to be close to the corresponding nominal marginal distribution (as measured by the 2-Wasserstein distance).\n\nUnder this setting, the paper finds the optimal output feedback controller for the worst case distribution of disturbances in the ambiguity set. It is found that the worst case distribution of disturbances belonging to the ambiguity set described above is Gaussian. Therefore, the corresponding optimal controller is a Linear-Quadratic Gaussian controller designed for this worst case distribution. \n\nIt is then shown that the worst case distribution may be determined in a computationally efficient manner. In particular, it is demonstrated that the Frank-Wolfe algorithm converges to the optimal covariance parameters for the worst case distribution. Furthermore, it is shown that each step of the Frank-Wolfe algorithm may be decomposed into simple, easily parallelizable components.\n\nSTRENGTHS:\nThe formulation of the distributionally robust output feedback LQ control problem is novel. It is related to previously published results on distributionally robust state feedback LQ control. As acknowledged by the authors, the output feedback setting brings an extra technical challenge due to the dependence of the optimal state estimator upon the disturbance distribution. \n\nThe problem and corresponding solution are clearly presented, and easy to follow.  The authors highlight the rather surprising result that the worst case disturbance distribution from the prescribed ambiguity set is Gaussian. \n\nGiven the appropriate context, the results here could be a meaningful step in a unification of worst case and stochastic control.\n\nWEAKNESSES:\nThe contextualization of the setting studied relative to prior work could be improved. In particular, there are a large class of control synthesis approaches for settings where the noise distributions are either not available or are not Gaussian. In particular, consider H-infinity approaches [Zhou et. al, Robust and Optimal Control, 1996], mixed H-2/H-infinity approaches [(Doyle et. al Optimal Control with Mixed H-2/H-infinity Performance Objectives, 1989) (Bernstein and Haddad,  LQG control with an H-infinity performance bound, 1988)], adversarially robust control [Lee et. al, Performance-Robustness Tradeoffs in Adversarially Robust Control and Estimation, 2023], and nonstochastic control [Hazan and Singh, Introduction to Online Nonstochastic Control, 2023]. It would be useful to discuss several of these results and contrast the setting with the distributionally robust setting. \n\nA thorough discussion about the choice for the ambiguity set and what distributions it can model would be beneficial. In particular, considering mean zero disturbances which are independent across time is restrictive. It fails to model e.g. colored noise. \n\nDespite the fact that it was possible to propose a relatively efficient method to optimize for the worst-case covariance of the noise distribution, the required computation time appears to scale quite poorly with the problem horizon, T. Only short horizons, up to T=20, were considered in the simulations, however many control problems have much longer horizons. This appears to limit the practical applicability.\n\nQUESTIONS:\nIs there any concrete theoretical connection between distributionally robust linear quadratic control and the other methods for incorporating robustness mentioned above (e.g. mixed H-2/H-inf)? \n\nAre there any practical examples where the proposed method substantially outperforms conventional approaches for incorporating robustness to unknown disturbance distributions? Such an example would make the setting more compelling for practical use. \n\nFrom the experiments, is it possible to detect any clear trends regarding the worst case covariance relative to the nominal covariance of the ambiguity set? E.g. Do we see or expect to see that the worst-case covariances are larger than the central covariances in Loewner order? Combined with the theoretical results, such an observation might justify crude approximate approaches for distributionally robust control design in practice. For example, one could take their estimate for the covariance for the central distribution in the ambiguity set, and simply scale it up by some constant. The resulting covariances could then be used to design a LQG controller.\n\nLIMITATIONS:\nThere is no negative societal impact that the authors must address. Limitations to the method that would be worth addressing are mentioned in the weaknesses section. To reiterate, it would be helpful to address:\n-\tWhat the ambiguity set can model, and what it cannot\n-\tIn which settings the proposed method would outperform conventional methods from robust control theory for handling unknown disturbance distributions\n-\tAcknowledging the computational burden of the approach relative to e.g. LQG with a known distribution", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
- The paper considers a standard LQ setup with uncertainty in the distributions of system noise, observation noise and initial state.
- Described as Zero-Sum Game between the controller and nature, they show the optimal decisions for both players. Specifically they show that the worst-case distribution is Gaussian and the optimal control law is linear.
- They provide an numerically efficient approach to solve the distributionally robust LQ control problem based on Frank-Wolfe algorithm.
-  Simulation experiments are provided to show the computational efficiency of the proposed algorithm compared to MOSEK.

STRENGTHS:
The results provided in the manuscript make multiple important contribution.

- The optimal control law still remains linear and the worst case distribution is still Gaussian.
- The proof technique is novel which relies on the "purified states" instead of usual dynamic programming approaches, it is interesting to see how this approach can be used in other LQG problems.
- Simulation results show the computational efficiency of the Frank-Wolfe algorithm over MOSEK.

WEAKNESSES:
- There are no obvious major weakness in the manuscript. Some potential minor weakness are currently mentioned in the form of questions in the comment below.

QUESTIONS:
- In general the adaptive linear quadratic control results are provided under assumption of sub-gaussian system noises (Assumption A1, http://proceedings.mlr.press/v19/abbasi-yadkori11a/abbasi-yadkori11a.pdf). Is it possible to extend these results to Sub-Gaussian nominal distribution?

LIMITATIONS:
Authors provide some limitations in form of possible extensions and future work in the last section.

Social Impact: NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08