Title: Online Control in Population Dynamics
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online Control

Theory

Population Dynamics

Linear Dynamical Systems
Abstract: The study of population dynamics originated with early sociological works but has since extended into many fields, including biology, epidemiology, evolutionary game theory, and economics. Most studies on population dynamics focus on the problem of prediction rather than control. Existing mathematical models for population control are often restricted to specific, noise-free dynamics, while real-world population changes can be complex and adversarial. 

To address this gap, we propose a new framework based on the paradigm of online control. We first characterize a set of linear dynamical systems that can naturally model evolving populations. We then give an efficient gradient-based controller for these systems, with near-optimal regret bounds with respect to a broad class of linear policies. Our empirical evaluations demonstrate the effectiveness of the proposed algorithm for population control even in non-linear models such as SIR and replicator dynamics.

Online Control in Population Dynamics

Abstract

The study of population dynamics originated with early sociological works but has since extended into many fields, including biology, epidemiology, evolutionary game theory, and economics. Most studies on population dynamics focus on the problem of prediction rather than control . Existing mathematical models for population control are often restricted to specific, noise-free dynamics, while real-world population changes can be complex and adversarial.

To address this gap, we propose a new framework based on the paradigm of online control. We first characterize a set of linear dynamical systems that can naturally model evolving populations. We then give an efficient gradient-based controller for these systems, with near-optimal regret bounds with respect to a broad class of linear policies. Our empirical evaluations demonstrate the effectiveness of the proposed algorithm for population control even in non-linear models such as SIR and replicator dynamics.

1 Introduction

Dynamical systems involving populations are ubiquitous in describing processes that arise in natural environments. As one example, the SIR model [26] is a fundamental concept in epidemiology, used to describe the spread of infectious diseases within a population. It divides the population into three groups - Susceptible (S), Infected (I) and Removed (R). A susceptible individual has not contracted the disease but has the chance to be infected if interacting with an infected individual. A removed individual either has recovered from the disease and gained immunity or is deceased. The population evolves over time according to three ordinary differential equations:

for constants β, θ ¡ 0 representing the infection and recovery rate, respectively. Numerous extensions of this basic model have been proposed to better capture how epidemics evolve and spread [8].

Beyond epidemiology, population dynamics naturally arise in many other fields, notably evolutionary game theory [23], biology [14, 7], and the analysis of genetic algorithms [39, 25]. For any dynamical system, it is natural to ask how it might best be controlled . For instance, controlling the spread of an infectious disease while minimizing externalities is a problem of significant societal importance.

In many natural models, these dynamics tend to be nonlinear, so the control problem is often computationally intractable. Existing algorithms are designed on a case-by-case basis by positing a specific system of differential equations, and then numerically or analytically solving for the optimal controller. Unfortunately, this approach is not robust to adversarial shocks to the system and cannot adapt to time-varying cost functions.

A new approach for population control. In this paper we propose a generic and robust methodology for population control, drawing on the framework and tools from online non-stochastic control theory to obtain a computationally efficient gradient-based method of control. In online non-stochastic control, at every time t GLYPH<16> 1 , . . . , T , the learner is faced with a state x t and must choose a control u t . The learner then incurs cost according to some time-varying cost function c t p x t , u t q evaluated at the current state/control pair, and the state evolves as:

where f describes the (known) discrete-time dynamics, x t GLYPH<0> 1 is the next state, and w t is an adversarially-chosen perturbation. A policy is a mapping from states to controls. The goal of the learner is to minimize regret with respect to some rich policy class Π , formally defined by

where p x π t , u π t q is the state/control pair at time t had policy π been carried out since time 1 .

As with prior work in online control [1], our method is theoretically grounded by regret guarantees for a broad class of Linear Dynamical Systems (LDSs). The key algorithmic and technical challenge we overcome is that prior methods only give regret bounds against comparator policies that strongly stabilize the LDS (Definition 4). Such policies force the magnitude of the state to decrease exponentially fast in the absence of noise. Unfortunately, for applications to population dynamics, even the assumption that such policies exist - let alone perform well - is fundamentally unreasonable, since it essentially implies that the population can be made to exponentially shrink.

A priori, one might hope to generically overcome this issue, by broadening the comparator class to all policies that marginally stabilize the LDS (informally, these are policies under which the magnitude of the state does not blow up). But we show that, in general, it is impossible to achieve sub-linear regret against that class - a result that may be of independent interest in online control: 6

Theorem 1 (Informal statement of Theorem 25) . There is a distribution D over LDSs with state space and control space given by R , such that any online control algorithm on a system L GLYPH<18> D incurs expected regret Ω p T q against the class of time-invariant linear policies that marginally stabilize L .

For general LDSs, it's not obvious if there is a natural 'intermediate' comparator class that does not require strong stabilizability and does enable control with low regret. However, systems that model populations possess rich additional structure, since they can be interpreted as controlled Markov chains. 7 In this paper, leveraging that structure, we design an algorithm GPC -Simplex for online control that applies to LDSs constrained to the simplex (Definition 3), and achieves strong regret bounds against a natural comparator class of policies with bounded mixing time (Definition 6).

1.1 Our Results

Throughout this work, we model a population as a distribution over d different categories, evolving over T discrete timesteps. For simplicity, we assume that u t is a d -dimensional real vector.

Theoretical guarantees for online population control. We introduce the simplex LDS model (Definition 3), which is a modification of the standard LDS model (Definition 9) that ensures the states p x t q t always represent valid distributions, i.e. never leave the simplex ∆ d . Informally, given state x t P ∆ d and control u t P R d ¥ 0 with } u t } 1 / 1 , the next state is

where A,B are known stochastic matrices, γ t P r 0 , 1 s is the observed perturbation strength , 8 and w t P ∆ d is an unknown perturbation. The perturbation w t can be interpreted as representing an

adversary that can add individuals from a population with distribution w t to the population under study. Intuitively, u t represents a distribution over d possible interventions as well as a 'null intervention'.

For any simplex LDS L and mixing time parameter τ ¡ 0 , we define a class K glyph[triangle] τ p L q (Definition 6), which roughly consists of the linear time-invariant policies under which the state of the system would mix to stationarity in time τ , in the absence of noise. Our main theoretical contribution is an algorithm GPC -Simplex that achieves low regret against this policy class:

Theorem 2 (Informal version of Theorem 7) . Let L be a simplex LDS on ∆ d , and let τ ¡ 0 . For any adversarially-chosen perturbations p w t q t , perturbation strengths p γ t q t , and convex and Lipschitz cost functions p c t q t , the algorithm GPC -Simplex performs T steps of online control on L with regret ˜ O p τ 7 { 2 ? dT q against K glyph[triangle] τ p L q .

Finally, analogously Theorem 1, we show that the mixing time assumption cannot be removed: it is impossible to achieve sub-linear regret (for online control of a simplex LDS) against the class of all linear time-invariant policies (Theorem 8).

Experimental evaluations. To illustrate the practicality of our results, we apply (a generalization of) GPC -Simplex to controlled versions of (a) the SIR model for disease transmission (Section 4), and (b) the replicator dynamics from evolutionary game theory (Appendix H). In the former, closed-form optimal controllers are known in the absence of perturbations [27]. We find that GPC -Simplex learns characteristics of the optimal control (e.g. the 'turning point' phase transition where interventions stop once herd immunity is reached). Moreover, our algorithm is robust even in the presence of adversarial perturbations, where previous theoretical results no longer apply. In the latter, we demonstrate that even when the control affects the population only indirectly, through the replicator dynamics payoff matrix , GPC -Simplex can learn to control the population effectively, and is more robust to noisy cost functions than a one-step best response controller.

1.2 Related work

Online non-stochastic control. In recent years, the machine learning community has witnessed an increasing interest in non-stochastic control problems (e.g. [1, 38, 19]). Unlike the classical setting of stochastic control, in non-stochastic control the dynamics are subject to time-varying, adversarially chosen perturbations and cost functions. See [21] for a survey of prior results. Most relevant to our work is the Gradient Perturbation Controller ( GPC ) for controlling general LDSs [1]. All existing controllers only provide provable regret guarantees against policies that strongly stabilize the system.

Population growth models. There is extensive research on modeling the evolution of populations in sociology, biology and economics. Besides the pioneering work of [33], notable models include the SIR model from epidemiology [26], the Lotka-Volterra model for predator-prey dynamics [32, 40] and the replicator dynamics from evolutionary game theory [23]. Recent years have seen intensive study of controlled versions of the SIR model - see e.g. empirical work [10], vaccination control models [13], and many others [15, 12, 30, 17]. Most relevant to our work is the quarantine control model , where the control reduces the effective transmission rate. Some works consider optimal control in the noiseless setting [27, 5]; follow-up work [34] considers a budget constraint on the control. None of these prior works can handle the general case of adversarial noise and cost functions.

2 Definitions and setup

Notation. Denote S d : GLYPH<16> ! M P r 0 , 1 s d GLYPH<2> d : GLYPH<176> d i GLYPH<16> 1 M i,j GLYPH<16> 1 @ j P r d s ) as the set of d GLYPH<2> d columnstochastic matrices. For a ¡ 0 , define S d a : GLYPH<16> t a GLYPH<4> M : M P S d u and S d / a : GLYPH<16> GLYPH<148> 0 / a 1 / a S d a 1 . Let ∆ d denote the simplex in R d . Similarly, we define ∆ d α : GLYPH<16> α GLYPH<4> ∆ d and ∆ d / α : GLYPH<16> GLYPH<148> 0 / α 1 / α ∆ d α 1 . Given a square matrix M P R d GLYPH<2> d , let M GLYPH<4> ,j denote the j th column of M . We consider the following matrix norms: } M } denotes the spectral norm of M , } M } 2 2 , 1 : GLYPH<16> GLYPH<176> d j GLYPH<16> 1 } M GLYPH<4> ,j } 2 1 is the sum of the squares of the glyph[lscript] 1 norms of the columns of M , and } M } 1 GLYPH<209> 1 : GLYPH<16> sup x P R d : } x } 1 GLYPH<16> 1 } Mx } 1 .

2.1 Dynamical systems

The standard model in online control is the linear dynamical system (LDS) . We define a simplex LDS to be an LDS where the state of the system always lies in the simplex. This requires enforcing certain constraints on the transition matrices, the control, and the noise:

Definition 3 (Simplex LDS) . Let d P N . A simplex LDS on ∆ d is a tuple

where A,B P S d are the transition matrices ; I GLYPH<132> ∆ d / 1 is the valid control set ; x 1 P ∆ d is the initial state ; γ t P r 0 , 1 s , w t P ∆ d are the noise strength and noise value at time t ; and c t : ∆ d GLYPH<2> I GLYPH<209> R is the cost function at time t . These parameters define a dynamical system where the state at time t GLYPH<16> 1 is x 1 . For each t ¥ 1 , given state x t and control u t P I at time t , the state at time t GLYPH<0> 1 is

and the cost incurred at time t is c t p x t , u t q .

Note that since the set of possible controls I is contained in ∆ d / 1 , the states p x t q t are guaranteed to remain within the simplex for all t . In this paper, we will assume that I GLYPH<16> GLYPH<148> α Pr α,α s ∆ d α , for some parameters α, α P r 0 , 1 s , which represent lower and upper bounds on the strength of the control.

Online non-stochastic control. Let L GLYPH<16> p A,B,x 1 , p γ t q t P N , p w t q t P N , p c t q t P N , I q be a simplex LDS and let T P N GLYPH<0> . We assume that the transition matrices A,B are known to the controller at the beginning of time, but the perturbations p w t q T t GLYPH<16> 1 are unknown. At each step 1 / t / T , the controller observes x t and γ t , plays a control u t P I , and then observes the cost function c t and incurs cost c t p x t , u t q . The system then evolves according to Eq. (4). Note that our assumption that the controller observes γ t contrasts with some of the existing work on nonstochastic control [1, 21], in which no information about the adversarial disturbances is known. In Appendix B, we justify the learner's ability to observe γ t by observing that in many situations, the learner observes the counts of individuals in a populations (in addition to their proportions , represented by the state x t ), and that this additional information allows computation of γ t .

The goal of the controller is to minimize regret with respect to some class K GLYPH<16> K p L q of comparator policies. Formally, for any fixed dynamical system and any time-invariant and Markovian policy K : ∆ d GLYPH<209> I , let p x t p K qq t and p u t p K qq t denote the counterfactual sequences of states and controls that would have been obtained by following policy K . Then the regret of the controller on observed sequences p x t q t and p u t q t with respect to K is

The following assumption on the cost functions of L is standard in online control [1, 3, 38, 37]:

Assumption 1. The cost functions c t : ∆ d GLYPH<2> I GLYPH<209> R are convex and L -Lipschitz, in the following sense: for all x, x 1 P ∆ d and u, u 1 P I , we have | c t p x, u qGLYPH<1> c t p x 1 , u 1 q| / L GLYPH<4>p} x GLYPH<1> x 1 } 1 GLYPH<0>} u GLYPH<1> u 1 } 1 q .

2.2 Comparator class and spectral conditions

In prior works on non-stochastic control for linear dynamical systems [1], the comparator class K GLYPH<16> K κ,ρ p L q is defined to be the set of linear, time-invariant policies x GLYPH<222>GLYPH<209> Kx where K P R d GLYPH<2> d p κ, ρ q -strongly stabilizes L :

Definition 4. A matrix M P R d GLYPH<2> d is p κ, ρ q -strongly stable if there is a matrix H P R d GLYPH<2> d so that } H GLYPH<1> 1 MH } / 1 GLYPH<1> ρ and } M } , } H } , } H GLYPH<1> 1 } / κ . A matrix K P R d GLYPH<2> d is said to p κ, ρ q -strongly stabilize an LDS with transition matrices A,B P R d GLYPH<2> d if A GLYPH<0> BK is p κ, ρ q -strongly stable.

The regret bounds against K κ,ρ p L q scale with ρ GLYPH<1> 1 , and so are vacuous for ρ GLYPH<16> 0 [1]. Unfortunately, in the simplex LDS setting, no policies satisfy the analogous notion of strong stability (see discussion in Section 3) unless ρ GLYPH<16> 0 . Intuitively, the reason is that a p κ, ρ q -strongly stable policy with ρ ¡ 0 makes the state converge to 0 in the absence of noise.

What is a richer but still-tractable comparator class for a simplex LDS? We propose the class of linear, time-invariant policies under which (a) the state of the LDS mixes , when viewed as a distribution, and (b) the level of control } u t } 1 is independent of the state x t . Formally, we make the following definitions:

Definition 5. Given t P N and a matrix X P S d with unique stationary distribution π P ∆ d , we define D X p t q : GLYPH<16> sup p P ∆ d } X t p GLYPH<1> π } 1 and ¯ D X p t q : GLYPH<16> sup p,q P ∆ d } X t GLYPH<4> p p GLYPH<1> q q} 1 . Moreover we define t mix p X,ε q : GLYPH<16> min t P N t t : D X p t q / ε u for each glyph[epsilon1] ¡ 0 , and we write t mix p X q : GLYPH<16> t mix p X, 1 { 4 q . 9

Definition 6 (Mixing a simplex LDS) . Let L be a simplex LDS with transition matrices A,B P S d and control set I GLYPH<16> GLYPH<148> α Pr α,α s ∆ d α . A matrix K P S d r α,α s is said to τ -mix L if t mix p A K q / τ, where

We define the comparator class K glyph[triangle] τ GLYPH<16> K glyph[triangle] τ p L q as the set of linear, time-invariant policies x GLYPH<222>GLYPH<209> Kx where K P S d r α,α s τ -mixes L .

Notice that for any K P S d r α,α s , the linear policy u t : GLYPH<16> Kx t always plays controls in the control set I , and the dynamics Eq. (4) under this policy can be written as x t GLYPH<0> 1 GLYPH<16> p 1 GLYPH<1> γ t q GLYPH<4> A K x t GLYPH<0> γ t GLYPH<4> w t .

Notice that by considering the comparator class K glyph[triangle] τ , we require the control norm to be independent of the state. This assumption is needed for technical reasons: without it, since Ax is multiplied by 1 GLYPH<1> } u } 1 in the transition dynamics (see Equation (4)), even a 'linear' policy u : GLYPH<16> Kx does not induce a linear transition. Hence, it would no longer be clear how one might define mixing time of a linear policy. It is a very interesting question whether there is a more natural (yet still tractable) definition of a simplex LDS that avoids this issue.

3 Online Algorithm and Theoretical Guarantee

In this section, we describe our main upper bound and accompanying algorithm for the setting of online control in a simplex LDS L . As discussed above, we assume that the set of valid controls is given by I GLYPH<16> GLYPH<148> α Pr α,α s ∆ d α , for some constants 0 / α / α / 1 , representing lower and upper bounds on the strength of the control. 10

For convenience, we write α t : GLYPH<16> } u t } 1 and u 1 t GLYPH<16> u t { α t P ∆ d (if α t GLYPH<16> 0 , we set u 1 t : GLYPH<16> 0 ). The dynamical system Eq. (4) can then be expressed as follows:

We aim to obtain a regret guarantee as in Eq. (3) with respect to some rich class of comparator policies K glyph[triangle] . As is typical in existing work on linear nonstochastic control, we take K glyph[triangle] to be a class of time-invariant linear policies, i.e. policies that choose control u t : GLYPH<16> Kx t at time t for some matrix K P R d GLYPH<2> d . In the standard setting of nonstochastic control, it is typically further assumed that all policies in the comparator class strongly stabilize the LDS (Definition 4). 11 The naive generalization of such a requirement in our setting would be that A K is strongly stable; however, this is impossible, since no stochastic matrix can be strongly stable. Instead, we aim to compete against the class K glyph[triangle] GLYPH<16> K glyph[triangle] τ p L q of time-invariant linear policies that (a) have fixed level of control in r α, α s , and (b) τ -mix L (Definition 6). We view the second condition as a natural distributional analogue of strong stabilizability; the first condition is needed for τ -mixing to even be well-defined.

Algorithm description. Our main algorithm, GPC -Simplex (Algorithm 1), is a modification of the GPC algorithm [1, 21]. As a refresher, GPC chooses the controls u t by learning a disturbance-action policy : a policy u t : GLYPH<16> ¯ Kx t GLYPH<0> GLYPH<176> H i GLYPH<16> 1 M r i s w t GLYPH<1> i , where ¯ K is a known, fixed matrix that strongly stabilizes the LDS; w t GLYPH<1> 1 , . . . , w t GLYPH<1> H are the recent noise terms; and M r 1 s , . . . , M r H s are learnable, matrix-valued parameters which we abbreviate as M r 1: H s . The key advantage of this parametrization

of policies (as opposed to a simpler parametrization such as u t GLYPH<16> Kx t for a parameter K ) is that the entire trajectory is linear in the parameters, and not a high-degree polynomial. Thus, optimizing the cost of a trajectory over the class of disturbance-action policies is a convex problem in M r 1: H s .

But why is the class of disturbance-action policies expressive enough to compete against the comparator class? This is where GPC crucially uses strong stabilizability. Notice that in the absence of noise, every disturbance-action policy is identical to the fixed policy u t : GLYPH<16> ¯ Kx t . This is fine when ¯ K and the comparator class are strongly stabilizing, since in the absence of noise, all strongly stabilizing policies rapidly force the state to 0 , and thus incur very similar costs in the long run. But in the simplex LDS setting, strong stabilizability is impossible. While all policies in K glyph[triangle] mix the LDS, they may mix to different states, which may incur different costs. There is no reason to expect that an arbitrary ¯ K P K glyph[triangle] , chosen before observing the cost functions, will have low regret against all policies in K glyph[triangle] .

We fix this issue by enriching the class of disturbance-action policies with an additional parameter p P ∆ d which, roughly speaking, represents the desired stationary distribution to which x t would converge, in the absence of noise, as t GLYPH<209> 8 . It is unreasonable to expect prior knowledge of the optimal choice of p , which depends on the not-yet-observed cost functions. Thus, GPC -Simplex instead learns p together with M r 1: H s . We retain the property that the requisite online learning problem is convex in the parameters, and therefore can be efficiently solved via an online convex optimization algorithm (as discussed in Appendix C.1, we use lazy mirror descent, LazyMD ). One advantage of GPC -Simplex over GPC is that the former requires no knowledge of the fixed 'reference' policy ¯ K (which, in the context of GPC , had to be strongly stabilizing). While such ¯ K is needed in the context of GPC to bound a certain approximation error involving the cost functions, in the context of GPC -Simplex this approximation error may be bounded by some simple casework involving properties of stochastic matrices (see Appendix C.3).

Formally, for parameters a 0 P r α, α s and H P N , GPC -Simplex considers a class of policies parametrized by the set X d,H,a 0 ,α : GLYPH<16> GLYPH<148> a Pr a 0 ,α s ∆ d a GLYPH<2> p S d a q H . We abbreviate elements p p, p M r 1 s , . . . , M r H s qq P X d,H,a 0 ,α by p p, M r 1: H s q . The high level idea of GPC -Simplex , like that of GPC , is to perform online convex optimization on the domain X d,H,a 0 ,α (Line 10). At each time t , the current iterate p p t , M r 1: H s t q , which defines a policy π p t ,M r 1: H s t , is used to choose the control u t . The optimization subroutine then receives a new loss function glyph[lscript] t : X d,H,a 0 ,α GLYPH<209> R based on the newly observed cost function c t . As with GPC , showing that this algorithm works requires showing that the policy class is sufficiently expressive. Unlike for GPC , our comparator policies are not strongly stabilizing, so new ideas are required for the proof.

We next formally define the policy π p,M r 1: H s associated with parameters p p, M r 1: H s q , and the loss function glyph[lscript] t used to update the optimization algorithm at time t .

Parametrization of policies. First, for t P r T s and i P N GLYPH<0> , we define the weights

We write w 0 : GLYPH<16> x 1 , γ 0 GLYPH<16> 1 , and w t GLYPH<16> 0 for t GLYPH<160> 0 as a matter of convention. 12 λ t,i can be interpreted as the 'influence of perturbation w t GLYPH<1> i on the state x t ', and ¯ λ t,i can be interpreted as the 'influence of perturbations prior to time step t GLYPH<1> i on the state x t '. An element p p, M r 1: H s q P X d,H,a 0 ,α induces a policy 13 at time t , denoted π p,M r 1: H s t , via the following variant of the disturbance-action control [1]:

In Line 6 of GPC -Simplex , the control u t is chosen to be π p t ,M r 1: H s t t p w t GLYPH<1> 1: t GLYPH<1> H q , which belongs to ∆ d } p t } 1 (using GLYPH<176> H i GLYPH<16> 0 λ t,i GLYPH<16> 1 ) and hence to the constraint set I (since } p t } 1 P r a 0 , α s GLYPH<128> r α, α s ).

r

1:

H

s

Algorithm 1 GPC -Simplex : GPC for Simplex LDS

Require: Linear system A,B , mixing time τ ¡ 0 for comparator class, horizon parameter H P N , set of valid controls I GLYPH<16> GLYPH<148> α Pr α,α s α GLYPH<4> ∆ d , total number of time steps T .

1: Write τ A : GLYPH<16> t mix p A q , and define a 0 : GLYPH<16> max t α, min t α, ✶ t τ A ¡ 4 τ u{p 96 τ quu . 2: Initialize an instance LazyMD of mirror descent (Algorithm 2) for the domain X d,H,a 0 ,α with the regularizer R d,H (defined in Appendix C.1) and step size η GLYPH<16> c a dH ln p d q{p Lτ 2 log 2 p T q ? T q , for a sufficiently small constant c .

3: Initialize p p 1 , M r 1: H s 1 q arg min p p,M r 1: H s qP X d,H,a 0 ,α R d,H p p, M r 1: H s q .

4: Observe initial state x 1 P ∆ d .

5: for 1 / t / T do

7: Receive cost c t p x t , u t q .

6: Choose control u t : GLYPH<16> λ t, 0 GLYPH<4> p t GLYPH<0> GLYPH<176> H i GLYPH<16> 1 M r i s t GLYPH<4> λ t,i GLYPH<4> w t GLYPH<1> i .

8: Observe x t GLYPH<0> 1 , γ t and compute w t GLYPH<16> γ GLYPH<1> 1 t p x t GLYPH<0> 1 GLYPH<1> p 1 GLYPH<1> γ t qrp 1 GLYPH<1> } u t } 1 q Ax t GLYPH<0> Bu t sq . (If γ t GLYPH<16> 0 , then set w t GLYPH<16> 0 .)

9: Define loss function glyph[lscript] t p p, M r 1: H s q : GLYPH<16> c t p x t p p, M r 1: H s q , u t p p, M r 1: H s qq .

10: Update p p t GLYPH<0> 1 , M r 1: H s t GLYPH<0> 1 q LazyMD t p glyph[lscript] t ; p p t , M r 1: H s t qq .

Loss functions. For p p, M r 1: H s q P X d,H,a 0 ,α , we let x t p p, M r 1: H s q and u t p p, M r 1: H s q denote the state and control at step t obtained by following the policy π p,M r 1: H s s at all time steps s prior to t (see Eqs. (20) and (21) in the appendix for precise definitions). We then define glyph[lscript] t p p, M r 1: H s q to be the evaluation of the adversary's cost function c t on the state-action pair p x t p p, M r 1: H s q , u t p p, M r 1: H s qq (Line 9).

Main guarantee and proof overview. Theorem 7 gives our regret upper bound for GPC -Simplex :

Theorem 7. Let d, T P N and τ ¡ 0 . Let L GLYPH<16> p A,B, I , x 1 , p γ t q t P N , p w t q t P N , p c t q t P N q be a simplex LDS with cost functions p c t q t satisfying Assumption 1 for some L ¡ 0 . Set H : GLYPH<16> τ r log p 2 LT 3 q s . Then the iterates p x t , u t q T t GLYPH<16> 1 of GPC -Simplex (Algorithm 1) with input p A,B,τ,H, I , T q satisfy:

where ˜ O pGLYPH<4>q hides only universal constants and poly-logarithmic dependence in T . Moreover, the time complexity of GPC -Simplex is poly p d, T q .

While for simplicity we have stated our results for obliviously chosen p γ t q t , p w t q t , p c t q t , since GPC -Simplex is deterministic the result also holds when these parameters are chosen adaptively by an adversary. See Appendix C for the formal proof of Theorem 7.

Lower bound. We also show that the mixing assumption on the comparator class K glyph[triangle] τ p L q (Definition 6) cannot be removed. In particular, without that assumption, if the valid control set I is restricted to controls u t of norm at mostly roughly O p 1 { T q , then linear regret is unavoidable. 14

Theorem 8 (Informal statement of Theorem 30) . Let β ¡ 0 be a sufficiently large constant. For any T P N , there is a distribution D over simplex LDSs with state space ∆ 2 and control space GLYPH<148> α Pr 0 ,β { T s ∆ 2 α , such that any online control algorithm on a system L GLYPH<18> D incurs expected regret Ω p T q against the class of all time-invariant linear policies x GLYPH<222>GLYPH<209> Kx where K P GLYPH<148> α Pr 0 ,β { T s S d α .

4 Experimental Evaluation

The previous sections focused on linear systems, but in fact GPC -Simplex can be easily modified to control non-linear systems, for similar reasons as in prior work [2]. It suffices for the dynamics to

have the form

for known f , observed γ t , and unknown w t . See Appendix E for discussion of the needed modifications and other implementation details. Relevant code is open-sourced in [16].

As a case study, in this section we apply GPC -Simplex (Algorithm 1) to a disease transmission model - specifically, a controlled generalization of the SIR model introduced earlier. In Appendix H we apply GPC -Simplex to a controlled version of the replicator dynamics from evolutionary game theory.

A controlled disease transmission model. The Susceptible-Infectious-Recovered (SIR) model is a basic model for the spread of an epidemic [26]. The SIR model has been extensively studied since last century [36, 41, 4, 24, 6] and attracted renewed interest during the COVID-19 pandemic [10, 29, 9]. As discussed previously, this model posits that a population consists of susceptible ( S ), infected ( I ), and recovered ( R ) individuals. When a susceptible individual comes into contact with an infected individual, the susceptible individual becomes infected at some 'transmission rate' β . Infected patients become uninfected and gain immunity at some 'recovery rate' θ . We consider a natural generalization of the standard dynamics Eq. (1) where recovered individuals may also lose immunity at a rate of ξ . Formally, in the absence of control, the population evolves over time according to the following system of differential equations:

Typically, β ¡ θ ¡ ξ . We normalize the total population to be 1 , and thus x GLYPH<16> r S, I, R s P ∆ 3 . Next, we introduce a variable called the preventative control u t P ∆ 2 , which has the effect of decreasing the transmission rate β , and adversarial perturbations w t , which allow for model misspecification. Incorporating these changes to the forward discretization of Eq. (10) gives the following dynamics:

The control u t P ∆ 2 represents a distribution over transmission prevention protocols: u t GLYPH<16> r 1 , 0 s represents full-scale prevention, whereas u t GLYPH<16> r 0 , 1 s represents that no prevention measure is imposed. Concretely, the effective transmission rate under control u t is β GLYPH<4> u t p 2 q .

Parameters and cost function. To model a highly infectious pandemic, we consider Eq. (11) with parameters β GLYPH<16> 0 . 5 , θ GLYPH<16> 0 . 03 , and ξ GLYPH<16> 0 . 005 . Suppose we want to control the number of infected individuals by modulating a (potentially expensive) prevention protocol u t . To model this setting, the cost function includes (1) a quadratic cost for infected individuals I t , and (2) a cost that is bilinear in the magnitude of prevention and the susceptible individuals:

where x t GLYPH<16> r S t , I t , R t s . Typically c 3 ¥ c 2 ¡ 0 to model the high cost of infection.

In Fig. 1, we compare GPC -Simplex against two baselines - (a) always executing u t GLYPH<16> r 1 , 0 s (i.e. full prevention), and (b) always executing u t GLYPH<16> r 0 , 1 s (i.e. no prevention) - for T GLYPH<16> 200 steps in the above model with no perturbations. We observe that GPC -Simplex suppresses the transmission rate via high prevention at the initial stage of the disease outbreak, then relaxes as the outbreak is effectively controlled. Moreover, GPC -Simplex outperforms both baselines in terms of cumulative cost. See Appendix F for additional experiments exhibiting the robustness of GPC -Simplex to perturbations (i.e. non-zero γ t 's) and different model parameters.

4.1 Controlling hospital flows: reproducing a study by [27]

We now turn to the recent work [27], which also studies a controlled SIR model. Similar to above, they considered a control that temporarily reduces the rate of contact within a population. In one scenario (inspired by the COVID-19 pandemic), they considered a cost function that penalizes

Cost Over Time,β=0.50,θ=0.03,E=0.005,C2=1,C=10

Cumulative Cost Over Time,β=0.50,θ=0.03,E=0.005,C2=1,C3=10

u(2) OverTime,β=0.50,θ=0.03,=0.005,C2=1,C=10

costs

cumulative costs

1.0

6

costs (full preventions)

costs (no preventions)

175

cumulative costs (full preventions)

cumulative costs (no preventions)

5

150

80

Cumulative cost

125

9'0

Cost

100

(z)n

E

75

0.4

2

50

1

25

0.2

0

0

25

50

75

Time Step

100

125

150

175

200

0

0

25

50

75

Time Step

100

125

150

175

200

0.0

0

25

50

75

Time Step

100

125

150

175

200

medical surges, i.e. when the number of infected exceeds a threshold y max determined by hospital capacities. Formally, they define the cost of a trajectory p x t , u t q T t GLYPH<16> 1 as

where W 0 is the principal branch of Lambert's W -function, and c 2 , c 3 are hyperparameters. The system parameters used by [27] are β GLYPH<16> 0 . 3 , θ GLYPH<16> 0 . 1 , ξ GLYPH<16> 0 . In the absence of noise and with a known cost function, [27] is able to compute the approximate solutions of the associated HamiltonJacobi-Bellman equations for various choices of c 2 , c 3 .

In Fig. 2, we show that GPC -Simplex (with a slightly modified instantaneous version of Eq. (13)) in fact matches the optimal solution analytically computed by [28]. See Appendix G for further experimental details, including the exact model parameters and cost function.

Statevs.time，c2=0.01,c3=100

Controlvs.time,c2=0.01,c3=100

x(1)(GPC-simplex)

x(1)

1.0

0.8

x(2) (GPC-simplex)

x(2)

0.8

0.6

ymax

(z)²n

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0

20

40

60

80

100

0

20

40

60

80

100

Time Step

Time Step

References

Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning , pages 111-119. PMLR, 2019.

Naman Agarwal, Elad Hazan, Anirudha Majumdar, and Karan Singh. A regret minimization approach to iterative learning control. In International Conference on Machine Learning , pages 100-109. PMLR, 2021.

Naman Agarwal, Elad Hazan, and Karan Singh. Logarithmic regret for online control. Advances in Neural Information Processing Systems , 32, 2019.

Linda JS Allen. Some discrete-time si, sir, and sis epidemic models. Mathematical biosciences , 124(1):83-105, 1994.

Rocío Balderrama, Javier Peressutti, Juan Pablo Pinasco, Federico Vazquez, and Constanza Sánchez de la Vega. Optimal control for a sir epidemic model with limited quarantine. Scientific Reports , 12(1):12583, 2022.

Ottar N Bjørnstad, Bärbel F Finkenstädt, and Bryan T Grenfell. Dynamics of measles epidemics: estimating scaling of transmission rates using a time series sir model. Ecological monographs , 72(2):169-184, 2002.

Fred Brauer, Carlos Castillo-Chavez, and Carlos Castillo-Chavez. Mathematical models in population biology and epidemiology , volume 2. Springer, 2012.

Tom Britton. Stochastic epidemic models: a survey. Mathematical biosciences , 225(1):24-35, 2010.

Yi-Cheng Chen, Ping-En Lu, Cheng-Shang Chang, and Tzu-Hsuan Liu. A time-dependent sir model for covid-19 with undetectable infected persons. Ieee transactions on network science and engineering , 7(4):3279-3294, 2020.

Ian Cooper, Argha Mondal, and Chris G Antonopoulos. A sir model assumption for the spread of covid-19 in different communities. Chaos, Solitons & Fractals , 139:110057, 2020.

Ross Cressman and Yi Tao. The replicator equation and other game dynamics. Proceedings of the National Academy of Sciences , 111(supplement_3):10810-10817, 2014.

Adil El-Alami Laaroussi, Mostafa Rachik, and Mohamed Elhia. An optimal control problem for a spatiotemporal sir model. International Journal of Dynamics and Control , 6:384-397, 2018.

Mohamed Elhia, Mostafa Rachik, and Elhabib Benlahmar. Optimal control of an sir model with delay in state and control variables. International scholarly research notices , 2013, 2013.

Herbert I Freedman. Deterministic mathematical models in population ecology. (No Title) , 1980.

Nicole M Gatto and Henry Schellhorn. Optimal control of the sir model in the presence of transmission and treatment uncertainty. Mathematical biosciences , 333:108539, 2021.

Noah Golowich, Elad Hazan, Zhou Lu, Dhruv Rohatgi, and Y. Jennifer Sun. Population control, 2024. https://github.com/jysun105/population_control .

EV Grigorieva, EN Khailov, and A Korobeinikov. Optimal control for a sir epidemic model with nonlinear incidence rate. Mathematical Modelling of Natural Phenomena , 11(4):89-104, 2016.

Elad Hazan. Introduction to online convex optimization, 2019.

Elad Hazan, Sham Kakade, and Karan Singh. The nonstochastic control problem. In Algorithmic Learning Theory , pages 408-421. PMLR, 2020.

Elad Hazan, Holden Lee, Karan Singh, Cyril Zhang, and Yi Zhang. Spectral filtering for general linear dynamical systems. Advances in Neural Information Processing Systems , 31, 2018.

Elad Hazan and Karan Singh. Introduction to online nonstochastic control. arXiv preprint arXiv:2211.09619 , 2022.

Elad Hazan, Karan Singh, and Cyril Zhang. Learning linear dynamical systems via spectral filtering. Advances in Neural Information Processing Systems , 30, 2017.

Josef Hofbauer and Karl Sigmund. Evolutionary games and population dynamics . Cambridge university press, 1998.

Chunyan Ji, Daqing Jiang, and Ningzhong Shi. The behavior of an sir epidemic model with stochastic perturbation. Stochastic analysis and applications , 30(5):755-773, 2012.

Sourabh Katoch, Sumit Singh Chauhan, and Vijay Kumar. A review on genetic algorithm: past, present, and future. Multimedia tools and applications , 80:8091-8126, 2021.

William Ogilvy Kermack and Anderson G McKendrick. A contribution to the mathematical theory of epidemics. Proceedings of the royal society of london. Series A, Containing papers of a mathematical and physical character , 115(772):700-721, 1927.

David I Ketcheson. Optimal control of an sir epidemic through finite-time non-pharmaceutical intervention. Journal of mathematical biology , 83(1):7, 2021.

David I Ketcheson. Sir-control-code. optimal control of an sir epidemic through finite-time non-pharmaceutical intervention, 2021.

Nikolay A Kudryashov, Mikhail A Chmykhov, and Michael Vigdorowitsch. Analytical features of the sir model and their applications to covid-19. Applied Mathematical Modelling , 90:466473, 2021.

Urszula Ledzewicz and Heinz Schättler. On optimal singular controls for a general sir-model with vaccination and treatment. In Conference Publications , volume 2011, pages 981-990. Conference Publications, 2011.

David A. Levin, Yuval Peres, and Elizabeth L. Wilmer. Markov chains and mixing times . American Mathematical Society, 2006.

Alfred J Lotka. Contribution to the theory of periodic reactions. The Journal of Physical Chemistry , 14(3):271-274, 2002.

Thomas Robert Malthus. An Essay on the Principle of Population Or a View of Its Past and Present Effects on Human Happiness, an Inquiry Into Our Prospects Respecting the Future Removal Or Mitigation of the Evils which it Occasions by Rev. TR Malthus . Reeves and Turner, 1872.

Emilio Molina and Alain Rapaport. An optimal feedback control that minimizes the epidemic peak in the sir model under a budget constraint. Automatica , 146:110596, 2022.

Arkadij Semenoviˇ c Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.

Boris Shulgin, Lewi Stone, and Zvia Agur. Pulse vaccination strategy in the sir epidemic model. Bulletin of mathematical biology , 60(6):1123-1148, 1998.

Max Simchowitz. Making non-stochastic control (almost) as easy as stochastic. Advances in Neural Information Processing Systems , 33:18318-18329, 2020.

Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR, 2020.

Mandavilli Srinivas and Lalit M Patnaik. Genetic algorithms: A survey. computer , 27(6):17-26, 1994.

Vito Volterra. Fluctuations in the abundance of a species considered mathematically. Nature , 118(2972):558-560, 1926.

Gul Zaman, Yong Han Kang, and Il Hyo Jung. Stability analysis and optimal vaccination of an sir epidemic model. BioSystems , 93(3):240-249, 2008.

Contents

A Additional preliminaries

For completeness, we recall the definition of a standard LDS [21].

Definition 9 (LDS) . Let d x , d u P N . A linear dynamical system (LDS) is described by a tuple L GLYPH<16> p A,B,x 1 , p w t q t P N , p c t q t P N q where A P R d x GLYPH<2> d x , B P R d x GLYPH<2> d u are the transition matrices; x 1 P R d x is the initial state; w t P R d x is the noise value at time t ; and c t : R d x GLYPH<2> R d u GLYPH<209> R is the cost function at time t . For each t ¥ 1 , given state x t P R d x and control u t P R d u at time t , the state at time t GLYPH<0> 1 is given by

and the instantaneous cost incurred at time t is given by c t p x t , u t q .

B Discussion on the observation model

Our main algorithm GPC -Simplex for online control of simplex LDSs assumes that for each t , the perturbation strength γ t is observed by the controller at the same time as it observes x t GLYPH<0> 1 (the algorithm does not require the entire sequence p γ t q t to be known in advance). In this appendix we discuss (a) why this is a crucial technical assumption for the algorithm, and (b) why it is a reasonable assumption in many natural population models.

First we explain why is it technically important for GPC -Simplex that the controller observes γ t . Recall that like the algorithm GPC from [1], GPC -Simplex is a disturbance-action controller , meaning that the control at time t is computed based on previous disturbances w t GLYPH<1> i . In the standard LDS model (Definition 9) studied by [1], it's clear that w t GLYPH<1> 1 can be computed from x t GLYPH<1> 1 , u t GLYPH<1> 1 , x t , using the fact that A,B are known. However, in the simplex LDS model, if γ t GLYPH<1> 1 is not directly observed, then in fact w t GLYPH<1> 1 may not be uniquely identifiable given x t GLYPH<1> 1 , u t GLYPH<1> 1 , x t . This is why GPC -Simplex requires observing the parameters γ t . It is an interesting open problem whether this assumption can be removed.

Second, we argue that in many practical applications, it is reasonable for γ t GLYPH<1> 1 to be observed along with the population state x t . The reason is that often the controller can observe not just the proportions of individuals of different categories in a population but also the total population size.

Formally, consider a population which has N t individuals at time t . Thus, if the distribution of the population across d categories is described by x t P ∆ d , then for each i P r d s there are N t p x t q i individuals in category i . Suppose that under control u t P I , this population evolves to a new distribution p 1 GLYPH<1> } u t } 1 q Ax t GLYPH<0> Bu t , but then the adversary adds n t new individuals to the population, whose distribution over categories is given by w t P ∆ d . Then if we write ¯ x t P R d ¥ 0 to denote the vector of counts of individuals in each category at time t , it holds that

where N t GLYPH<0> 1 GLYPH<16> N t GLYPH<0> n t is the total number of individuals at time t GLYPH<0> 1 , and we write γ t : GLYPH<16> n t {p N t GLYPH<0> n t q . Thus, the distribution of the population across the d categories at time t GLYPH<0> 1 is

which is exactly the update rule from Definition 3. Moreover, if the controller observes the total population counts N t , N t GLYPH<0> 1 in addition to x t , u t , x t GLYPH<0> 1 , then it may compute γ t GLYPH<16> p N t GLYPH<0> 1 GLYPH<1> N t q{ N t GLYPH<0> 1 as well as w t (using knowledge of A,B ), which is what we wanted to show.

C Proof of Theorem 7

In this section, we prove Theorem 7. We begin with an overview of this section that outlines the structure and the main idea behind the proof of Theorem 7.

Overview. GPC -Simplex (Algorithm 1) essentially runs mirror descent on the loss functions glyph[lscript] t p p, M r 1: H s q constructed in Line 9. In particular, the loss at time t measures the counterfactual cost of following the policy π p,M r 1: H s for the first t timesteps. Thus, the regret of GPC -Simplex against the comparator class K glyph[triangle] τ p L q (Definition 6) can be bounded by the following decomposition:

In more detail:

Approximation error of comparator class. Since GPC -Simplex is only optimizing over policies of the form π p,M r 1: H s for p p, M r 1: H s q P X d,H,a 0 ,α , we must show that every policy in the comparator class can be approximated by some policy π p,M r 1: H s . This is accomplished by Lemma 17.

Mismatch error of costs. The cost incurred by the mirror descent algorithm LazyMD at time t is glyph[lscript] t p p t , M r 1: H s t q , which is the counterfactual cost at time t had the current policy π p t ,M r 1: H s t been carried out from the beginning of the time. However, the cost actually incurred by the controller at time t is c t p x t , u t q , which is the cost incurred by following policy π p s ,M r 1: H s s at time t , for each s / t . Thus, there is a mismatch between the loss that GPC -Simplex is optimizing and the loss that GPC -Simplex needs to optimize. This mismatch can be bounded using the stability of mirror descent along with a mixing argument; see Lemma 21.

LazyMD regret. GPC -Simplex uses LazyMD as its subroutine for mirror descent. The regret of LazyMD can be bounded by standard guarantees; see Corollary 15.

C.1 Preliminaries on mirror descent

We begin with some preliminaries regarding mirror descent. Let X GLYPH<128> R d be a convex compact set, and let R : X GLYPH<209> R be a convex function. We consider the Lazy Mirror Descent algorithm LazyMD (also known as Following the Regularized Leader ) for online convex optimization on X . Given an offline optimization oracle over X , the function R , and a parameter η ¡ 0 , LazyMD chooses each iterate z t based on the historical loss functions glyph[lscript] s : X GLYPH<209> R (for s P r t GLYPH<1> 1 s ) as described in Algorithm 2.

Algorithm 2 LazyMD : Lazy Mirror Descent [35]

Require: Offline convex optimization oracle over set X GLYPH<128> R d ; convex regularization function R : X GLYPH<209> R ; step size η ¡ 0 ; loss functions glyph[lscript] 1 , . . . , glyph[lscript] T where glyph[lscript] t is revealed after iteration t .

1: for t ¥ 1 do

2: Compute and output the solution to the following convex optimization problem:

3: Receive loss function glyph[lscript] t : X GLYPH<209> R .

The following lemma bounds the regret of LazyMD against the single best z P X (in hindsight), for an appropriately chosen step size η .

Lemma 10 (Mirror descent) . Suppose that X GLYPH<128> R d is convex and compact, and let }GLYPH<4>} be a norm on R d . Let R : X GLYPH<209> R be a 1 -strongly convex function with respect to }GLYPH<4>} . Let L ¡ 0 , and let ρ : GLYPH<16> max z P X R p z q GLYPH<1> min z P X R p z q .

Fix an arbitrary sequence of loss functions glyph[lscript] t : X GLYPH<209> R which are each convex and L -Lipschitz with respect to } GLYPH<4> } . Then the iterates z t of LazyMD (Eq. (14) ) with an optimization oracle over X , regularizer R , step size η GLYPH<16> ? ρ {p L ? 2 T q , and loss functions glyph[lscript] 1 , . . . , glyph[lscript] T satisfy:

Moreover, for each t P r T GLYPH<1> 1 s , it holds that

Lemma 10 is essentially standard but we provide a proof for completeness.

Proof of Lemma 10. By [18, Theorem 5.2], it holds that

where } GLYPH<4> } GLYPH<14> : R d GLYPH<209> R is the dual norm of }GLYPH<4>} , defined by } y } GLYPH<14> : GLYPH<16> max } z }/ 1 x y, z y . Recall that a convex L -Lipschitz loss function glyph[lscript] t satisfies } ∇ glyph[lscript] t p z q} GLYPH<14> / L for all z P X . Thus, the above regret bound simplifies to

Substituting in η GLYPH<16> ? ρ L ? 2 T yields Eq. (15). To establish the movement bound Eq. (16), we argue as follows. Consider any y 1 , y 2 P R d and define, for i P t 1 , 2 u ,

The definition of w 2 implies that

where the second inequality is by 1 -strong convexity of R . Simplifying, we get

Symmetrically, the definition of w 1 together with strong convexity implies that

Adding the two above displays gives

where the second inequality uses convexity of R (which gives x ∇ R p w 1 q GLYPH<1> ∇ R p w 2 q , w 1 GLYPH<1> w 2 y ¥ 0 ). It follows that } w 1 GLYPH<1> w 2 } / } y 1 GLYPH<1> y 2 } GLYPH<14> . Setting y 1 : GLYPH<16> η GLYPH<176> t GLYPH<1> 1 s GLYPH<16> 1 ∇ glyph[lscript] s p z s q and y 2 : GLYPH<16> η GLYPH<176> t s GLYPH<16> 1 ∇ glyph[lscript] s p z s q , and recalling the definitions of z t , z t GLYPH<0> 1 from Eq. (14), we get

as desired.

We next apply Lemma 10 to the domain used in GPC -Simplex . Recall that, given d, H P N and real numbers 0 / a / b / 1 , we have defined X d,H,a,b : GLYPH<16> GLYPH<148> a 1 Pr a,b s ∆ d a 1 GLYPH<2> p S d a 1 q H .

Definition 11 (Entropy of a sub-distribution) . Let d P N . We define the function Ent : ∆ d / 1 GLYPH<209> R ¥ 0 by

where for any vector v P R d we write v c : GLYPH<16> 1 GLYPH<1> GLYPH<176> d j GLYPH<16> 1 v j P R .

Lemma 12. Let d P N and u, v P ∆ d / 1 . Then

That is, v GLYPH<222>GLYPH<209> GLYPH<1> Ent p v q is 1 -strongly convex on ∆ d / 1 with respect to }GLYPH<4>} 1 .

Proof. Let p be the probability mass function on r d GLYPH<0> 1 s with p i GLYPH<16> u i for all i P r d s , and let q be the probability mass function on r d GLYPH<0> 1 s with p i GLYPH<16> v i for all i P r d s . Then it can be checked that

where the first inequality is by Pinsker's inequality.

Definition 13 (Regularizer for mirror descent in GPC -Simplex ) . Let d, H P N and 0 / a / b / 1 . We define R d,H : X d,H,a,b GLYPH<209> R / 0 as follows (omitting the domain's dependence on a, b for notational simplicity):

Definition 14 (Norm for analysis of mirror descent in GPC -Simplex ) . Let d, H P N , and identify R d GLYPH<0> Hd 2 with R d GLYPH<2> p R d GLYPH<2> d q H . We define a norm }GLYPH<4>} d,H on R d GLYPH<0> Hd 2 as follows: for p P R d , M r 1: H s P p R d GLYPH<2> d q H ,

Corollary 15. Let d, H P N and 0 / a / b / 1 . Consider an arbitrary sequence of cost functions glyph[lscript] t : X d,H,a,b GLYPH<209> R which are convex and L -Lipschitz with respect to }GLYPH<4>} d,H . Then the iterates z t of LazyMD with η GLYPH<16> a 2 dH ln p d q{p L ? T q , regularizer R , and loss functions glyph[lscript] 1 , . . . , glyph[lscript] T satisfy the following regret guarantee:

Moreover, for β : GLYPH<16> ? 2 dH ln p d q ? T , for all t P r T GLYPH<1> 1 s , we have

Proof. Note that the set of p p, M r 1: H s q where p P R d and M r 1: H s P p R d GLYPH<2> d q H can be identified with R d GLYPH<0> Hd 2 . We apply Lemma 10 with X : GLYPH<16> X d,H,a,b , R GLYPH<16> R d,H , and the norm }GLYPH<4>} d,H . It is straightforward to check that X is convex and compact in R d GLYPH<0> Hd 2 . By Lemma 12, we have that R d,H is 1 -strongly convex with respect to the norm }GLYPH<4>} d,H . Moreover, note that

since 0 / Ent p v q / ln p d GLYPH<0> 1 q for all v P ∆ d / 1 . Thus, Lemma 10 implies the claimed bounds Eqs. (18) and (19), where to prove Eq. (19) we are using the fact that } C } 1 GLYPH<209> 1 GLYPH<16> max j Pr d s } C GLYPH<4> ,j } 1 / b GLYPH<176> j Pr d s } C GLYPH<4> ,j } 2 1 for all C P R d GLYPH<2> d .

C.2 Approximation of linear policies

Henceforth fix a simplex LDS L GLYPH<16> p A,B, I , x 1 , p γ t q t , p w t q t , p c t q t q on ∆ d , where I GLYPH<16> GLYPH<148> α Pr α,α s ∆ d α for some constants 0 / α / α / 1 .

Recall that any choice of parameters p p, M r 1: H s q P X d,H,a,b (for some hyperparameters H P N and α / a / b / α ) induces, via Eq. (8), a set of policies p π p,M r 1: H s s q s Pr T s . The policy π p,M r 1: H s s takes as input the disturbances w s GLYPH<1> 1 , . . . , w s GLYPH<1> H observed at the H time steps before step s , and outputs a control for step s . Recall that, in Algorithm 1, we used x t p p, M r 1: H s q , u t p p, M r 1: H s q to denote the state and control at time step t one would observe by playing the control π p,M r 1: H s s p w s GLYPH<1> 1: s GLYPH<1> H q at step s , for each 1 / s / t .

Formally, we have the following expressions for x t p p, M r 1: H s q , u t p p, M r 1: H s q :

Fact 16. For any p p, M r 1: H s q P X d,H,a,b and t P r T s , it holds that

where we have written, for t P r T s , i P N , α p,M r 1: H s t,i : GLYPH<16> -i GLYPH<1> 1 j GLYPH<16> 1 GLYPH<1> 1 GLYPH<1> GLYPH<15> GLYPH<15> GLYPH<15> π p,M r 1: H s t GLYPH<1> j GLYPH<15> GLYPH<15> GLYPH<15> 1 GLYPH<9> GLYPH<16> p 1 GLYPH<1> } p } 1 q i GLYPH<1> 1 .

Proof. This follows unrolling Eq. (6) with the controls u s : GLYPH<16> π p,M r 1: H s s p w s GLYPH<1> 1: s GLYPH<1> H q , and recalling the definitions in Eq. (7) and the conventions w 0 : GLYPH<16> x 1 , γ 0 GLYPH<16> 1 , and w t GLYPH<16> 0 for t GLYPH<160> 0 .

In a sense, Algorithm 1 performs online convex optimization over the set of such policies. Even if we can manage to show that doing so yields a good regret guarantee with respect to the class of policies t π p,M r 1: H s t : p p, M r 1: H s q P X d,H,a,b u for some choices of H,a,b , why should this imply a good regret guarantee with respect to the class K glyph[triangle] τ p L q of linear policies (see Definition 6)? Lemma 17 bridges this gap, showing that any policy in K glyph[triangle] τ p L q can be approximated by a policy of the form p π p,M r 1: H s s q s Pr T s .

Lemma 17 (Approximation) . Suppose that the cost functions c 1 , . . . , c T of L satisfy Assumption 1 with Lipschitz parameter L . Fix τ ¡ 0 , ε P p 0 , 1 q , and any K GLYPH<14> P S d / 1 such that t mix p A K GLYPH<14> q / τ . Write α GLYPH<14> : GLYPH<16> } K GLYPH<14> } 1 GLYPH<209> 1 . If H ¥ τ r log 2 p 2 LT 2 { ε q s , then there is some p p, M r 1: H s q P ∆ d α GLYPH<14> GLYPH<2> p S d α GLYPH<14> q H such that

where x t p K GLYPH<14> q , u t p K GLYPH<14> q denote the state and control that one would observe at time step t if one were to play according to the policy x GLYPH<222>GLYPH<209> K GLYPH<14> x at all time steps 1 / s / t .

Proof. For each t , if the controls u t are chosen to satisfy u t : GLYPH<16> K GLYPH<14> GLYPH<4> x t , then we have α t : GLYPH<16> } K GLYPH<14> } 1 GLYPH<209> 1 . Moreover, for 1 / t / T , we can write

where A K GLYPH<14> was defined in Eq. (5). By the assumption that t mix p A K GLYPH<14> q / τ , there is some unique p 1 P ∆ d such that that A K GLYPH<14> GLYPH<4> p 1 GLYPH<16> p 1 (see Definition 5). Moreover, by our bound on H and

Lemma 18, for any i ¡ H and q P ∆ d we have GLYPH<15> GLYPH<15> A i GLYPH<1> 1 K GLYPH<14> q GLYPH<1> p 1 GLYPH<15> GLYPH<15> 1 / p 1 { 2 q H { τ / ε {p 2 LT 2 q . Using that λ t, 0 GLYPH<16> GLYPH<176> t i GLYPH<16> H GLYPH<0> 1 λ t,i by the definition in Eq. (7),

For 1 / i / H , let us define M r i s : GLYPH<16> K GLYPH<14> A i GLYPH<1> 1 K GLYPH<14> P S d α GLYPH<14> and p : GLYPH<16> K GLYPH<14> GLYPH<4> p 1 P ∆ d α GLYPH<14> . Using Eqs. (21) and (24), we have that

where the final inequality uses Eq. (25).

Next, we may bound the difference in state vectors using Eq. (26), as follows: for any sequence of p u i q t i GLYPH<16> 1 with } u i } 1 GLYPH<16> α GLYPH<14> for all i , we can expand Eq. (6) to get

Thus, for any t P r T s , we have

By Eqs. (26) and (27) and Assumption 1, it follows that, for each t P r T s ,

which yields the claimed bound Eq. (22).

The following facts about distance to stationarity are well-known (see e.g. [31, Section 4.4]):

Lemma 18. Let X P S d have a unique stationary distribution π . Then the following inequalities hold for any c, t P N :

C.3 Bounding the memory mismatch error

In this section, we prove Lemma 21, which allows us to show that an algorithm with bounded aggregate loss with respect to the loss functions glyph[lscript] t defined on Line 9 of Algorithm 1 in fact has bounded aggregate cost with respect to the cost functions c t chosen by the adversary.

First, we introduce two useful lemmas on the mixing time of matrices (Definition 5).

Lemma 19. Let X P S d have a unique stationary distribution. Let Y P S d satisfy } X GLYPH<1> Y } 1 GLYPH<209> 1 / δ . Then for any t P N ,

Proof. For any v P ∆ d , we have } Xv GLYPH<1> Y v } 1 / δ . Ahybrid argument then yields that for any t ¥ 1 , } X t v GLYPH<1> Y t v } 1 / tδ . Then

where the first and last inequalities apply the first item of Lemma 18.

Lemma 20. Suppose that A,B P S d , K GLYPH<14> P S d / 1 satisfy t mix p A q ¡ 4 GLYPH<4> t mix p A K GLYPH<14> q . Then } K GLYPH<14> } 1 GLYPH<209> 1 ¡ 1 {p 96 GLYPH<4> t mix p A K GLYPH<14> qq .

Proof. Let us write τ : GLYPH<16> t mix p A K GLYPH<14> q and α GLYPH<14> : GLYPH<16> } K GLYPH<14> } 1 GLYPH<209> 1 , so that A K GLYPH<14> GLYPH<16> p 1 GLYPH<1> α GLYPH<14> q GLYPH<4> A GLYPH<0> BK GLYPH<14> . Suppose for the purpose of contradiction that α GLYPH<14> / 1 {p 96 τ q . We have that } A GLYPH<1> A K GLYPH<14> } 1 GLYPH<209> 1 / 2 α GLYPH<14> . By Lemma 18 and Definition 5, we have ¯ D A K GLYPH<14> p τ q / 2 D A K GLYPH<14> p τ q / 1 { 2 , so D A K GLYPH<14> p 4 τ q / ¯ D A K GLYPH<14> p 4 τ q / 1 { 16 . Using Lemma 19 and the assumption on α GLYPH<14> ,

meaning that t mix p A q / 4 τ

The last step is to bound the memory mismatch error.

Lemma 21 (Memory mismatch error) . Suppose that p c t q t satisfy Assumption 1 with Lipschitz parameter L . Let τ, β ¡ 0 , and suppose that K glyph[triangle] τ p L q is nonempty. Consider the execution of GPC -Simplex (Algorithm 1) on L with input τ . If the iterates p p t , M r 1: H s t q t Pr T s satisfy

then for each t P r T s , the loss function glyph[lscript] t computed at time step t satisfies

Proof. Recall that u t P ∆ d denotes the control chosen in step t of Algorithm 1. We write α t : GLYPH<16> } u t } 1 and, for i P r t s , α t,i : GLYPH<16> -i GLYPH<1> 1 j GLYPH<16> 1 p 1 GLYPH<1> α t GLYPH<1> j q . Note that α t GLYPH<16> } p t } 1 GLYPH<16> GLYPH<15> GLYPH<15> GLYPH<15> M r h s t GLYPH<15> GLYPH<15> GLYPH<15> 1 GLYPH<209> 1 for each h P r H s , by definition of X d,H,a 0 ,α .

Let us fix t P r T s , and write p : GLYPH<16> p t , M r 1: H s : GLYPH<16> M r 1: H s t . By Eq. (6), the state x t at step t of Algorithm 1 can be written as follows:

By assumption that K glyph[triangle] τ p L q is nonempty, there is some K GLYPH<14> P S d r α,α s satisfying t mix p A K GLYPH<14> q / τ . Let us write α GLYPH<14> : GLYPH<16> } K GLYPH<14> } 1 GLYPH<209> 1 , so that A K GLYPH<14> GLYPH<16> p 1 GLYPH<1> α GLYPH<14> q A GLYPH<0> BK GLYPH<14> . Moreover, recall we have written in Algorithm 1 that τ A : GLYPH<16> t mix p A q .

For 1 / i / t , define

Note that

Next, using Eq. (29) and Eq. (20), we have

The condition Eq. (28) together with the triangle inequality gives that } Bp t GLYPH<1> i GLYPH<1> Bp } 1 / iβ and GLYPH<15> GLYPH<15> GLYPH<15> BM r j s t GLYPH<1> i w t GLYPH<1> i GLYPH<1> j GLYPH<1> BM r j s w t GLYPH<1> i GLYPH<1> j GLYPH<15> GLYPH<15> GLYPH<15> 1 / iβ for all i, j ¥ 1 , as well as | α t GLYPH<1> i GLYPH<1> α t | / iβ for all i ¥ 1 . It follows that } v i GLYPH<1> v 1 i } 1 / iβ and | α t,i GLYPH<1> α p,M r 1: H s t,i | / i 2 β for all i ¥ 1 and that for any glyph[lscript] ¥ 1 ,

Using Eq. (32) and the fact that GLYPH<176> t i GLYPH<16> 1 α t,i } v i } 1 GLYPH<16> GLYPH<176> t i GLYPH<16> 1 α p,M r 1: H s t,i } v 1 i } 1 GLYPH<16> 1 , we see

We consider the following two cases:

Case 1: τ A / 4 τ . Write t 0 GLYPH<16> t τ A log 2 p 1 { β q u . Let the stationary distribution of A be denoted p GLYPH<14> P ∆ d . By Lemma 18, we have that for all i ¥ 1 , GLYPH<15> GLYPH<15> A i GLYPH<4> p GLYPH<1> p GLYPH<14> GLYPH<15> GLYPH<15> 1 / D A p i q / 1 { 2 t i { τ A u . Now, using Eq. (31), we may compute

GLYPH<15>

GLYPH<15>

for some universal constants C, C 1 . Above, the first inequality uses the triangle inequality, the second inequality uses Eq. (33), and the third inequality uses that } v i GLYPH<1> v 1 i } 1 / iβ, | α t,i GLYPH<1> α p,M r 1: H s t,i | / i 2 β , } v 1 i } 1 / 1 . The fourth inequality uses the bound GLYPH<176> t i GLYPH<16> 1 GLYPH<0> t 0 2 GLYPH<1> t i { τ A u / O p τ A β q / O p t 0 β q .

Case 2: τ A ¡ 4 τ . In this case, we claim that a 0 ¥ 1 {p 96 τ q . By choice of a 0 in Line 1 of Algorithm 1 and the fact that τ A ¡ 4 τ , it suffices to show that α ¥ 1 {p 96 τ q : to see this, note that τ A GLYPH<16> t mix p A q ¡ 4 τ ¥ 4 GLYPH<4> t mix p A K GLYPH<14> q , so Lemma 20 gives that } K GLYPH<14> } 1 GLYPH<209> 1 ¡ 1 {p 96 GLYPH<4> t mix p A K GLYPH<14> qq ¥ 1 {p 96 τ q . But } K GLYPH<14> } 1 GLYPH<209> 1 / α , and thus α ¡ 1 {p 96 τ q . This proves that a 0 ¥ 1 {p 96 τ q . Hence α i ¥ a 0 ¥ 1 {p 96 τ q , by definition of X d,H,a 0 ,α , for all i P r T s .

Write t 0 : GLYPH<16> t 200 τ GLYPH<4> log p 1 { β q u . Then for any i ¡ t 0 ,

Again using Eq. (31),

for some constants C, C 1 . Above, the first inequality uses Eq. (30); the second inequality uses the previously derived bounds | α t,i GLYPH<1> α p,M r 1: H s t,i | / i 2 β and } v i GLYPH<1> v 1 i } 1 / iβ ; and the final inequality uses that a 0 ¥ 1 {p 96 τ q .

In both cases, we have GLYPH<15> GLYPH<15> x t GLYPH<1> x t p p, M r 1: H s q GLYPH<15> GLYPH<15> 1 / C 1 τ 3 β log 3 p 1 { β q for some universal constant C 1 . By definition, the control u t chosen by Algorithm 1 at time step t is exactly u t GLYPH<16> u t p p, M r 1: H s q . Thus, using L -Lipschitzness of c t , we have

as desired.

C.4 Proof of Theorem 7

Before proving Theorem 7, we establish that the loss functions glyph[lscript] t used in GPC -Simplex are Lipschitz. Lemma 22. Let X P S d with τ : GLYPH<16> t mix p X q GLYPH<160>8 . Then for any i P N and v P R d with x ✶ , v y GLYPH<16> 0 , it holds that GLYPH<15> GLYPH<15> X i v GLYPH<15> GLYPH<15> 1 / 2 GLYPH<1> t i { τ u } v } 1 .

Proof. Fix v P R d with x ✶ , v y GLYPH<16> 0 . We can write v GLYPH<16> v GLYPH<0> GLYPH<1> v GLYPH<1> , where v GLYPH<0> , v GLYPH<1> P R d ¥ 0 are the non-negative and negative components of v respectively. We have } v GLYPH<0> } 1 GLYPH<16> } v GLYPH<1> } 1 GLYPH<16> 1 2 } v } 1 since x ✶ , v y GLYPH<16> 0 . Let u 1 : GLYPH<16> 2 v GLYPH<0> {} v } 1 and u 2 : GLYPH<16> 2 v GLYPH<1> {} v } 1 , so that u 1 , u 2 P ∆ d . By Lemma 18 and the definition of t mix p X q , we have

Lemma 23 (Lipschitzness of glyph[lscript] t ) . Let τ ¡ 0 , and suppose that K glyph[triangle] τ p L q is nonempty. For each t P r T s , the loss function glyph[lscript] t p p, M r 1: H s q GLYPH<16> c t p x t p p, M r 1: H s q , u t p p, M r 1: H s qq (as defined on Line 9 of Algorithm 1) is O p Lτ 2 q -Lipschitz with respect to the norm }GLYPH<4>} d,H in X d,H,a 0 ,α .

Proof. By L -Lipschitzness of c t with respect to }GLYPH<4>} 1 , it suffices to show that for any p p 1 , M r 1: H s 1 q , p p 2 , M r 1: H s 2 q P X d,H,a 0 ,α , we have

Fix p p 1 , M r 1: H s 1 q , p p 2 , M r 1: H s 2 q P X d,H,a 0 ,α , and write

Since ε / GLYPH<15> GLYPH<15> GLYPH<15> p p 1 , M r 1: H s 1 q GLYPH<1> p p 2 , M r 1: H s 2 q GLYPH<15> GLYPH<15> GLYPH<15> d,H , it suffices to show that Eqs. (36) and (37) hold with ε on the right-hand sides.

To verify Eq. (36) in this manner, we define, for b P t 1 , 2 u ,

Since λ t GLYPH<1> i, 0 ¯ λ t,i GLYPH<0> λ t,i GLYPH<0> GLYPH<176> H j GLYPH<16> 1 λ t,i GLYPH<0> j / 1 , we have } v i,b } 1 / 1 for each i P r t s , b P t 1 , 2 u . Moreover, } v i, 1 GLYPH<1> v i, 2 } 1 / p λ t GLYPH<1> i, 0 ¯ λ t,i GLYPH<0> GLYPH<176> H j GLYPH<16> 1 λ t,i GLYPH<0> j q GLYPH<4> ε / ε . Write σ 1 : GLYPH<16> } p 1 } 1 , σ 2 : GLYPH<16> } p 2 } 1 , so that | σ 1 GLYPH<1> σ 2 | / ε and |p 1 GLYPH<1> σ 1 q i GLYPH<1> p 1 GLYPH<1> σ 2 q i | / iε for all i ¥ 1 . Also note that for each b P t 1 , 2 u ,

where the final equality follows since γ 0 GLYPH<16> 1 . By Eq. (20), we have

We consider two cases, depending on the mixing time τ A : GLYPH<16> t mix p A q of A :

Case 1: τ A / 4 τ . Let the stationary distribution of A be denoted p GLYPH<14> P ∆ d . Then

for some constants C, C 1 . Above, the second equality uses Eq. (38), and the second inequality uses Lemma 22 together with the fact that A i GLYPH<1> 1 p GLYPH<14> GLYPH<16> p GLYPH<14> and

Case 2: τ A ¡ 4 τ . In this case, the assumption that K glyph[triangle] τ p L q is nonempty together with the choice of a 0 in Line 1 of Algorithm 1 and Lemma 20 gives that a 0 ¡ 1 {p 96 τ q . See Case 2 of the proof of Lemma 21 for more details of this argument, which uses the fact that τ A ¡ 96 τ .

✶ The final inequality uses the assumption that τ A / 4 τ .

for some constant C .

Thus, in both cases above, we have GLYPH<15> GLYPH<15> GLYPH<15> x t p p 1 , M r 1: H s 1 q GLYPH<1> x t p p 2 , M r 1: H s 2 q GLYPH<15> GLYPH<15> GLYPH<15> 1 / O p τε q , which verifies Eq. (36).

The proof of Eq. (37) is much simpler: we have

since λ t, 0 GLYPH<0> GLYPH<4> GLYPH<4> GLYPH<4> GLYPH<0> λ t,H GLYPH<16> 1 .

Proof of Theorem 7. Set β GLYPH<16> ? 2 dH ln d ? T , ε GLYPH<16> 1 { T , and K glyph[triangle] τ : GLYPH<16> K glyph[triangle] τ p L q . We will apply Corollary 15 to the sequence of iterates p p t , M r 1: H s t q produced in Algorithm 1, for the domain X d,H,a 0 ,α (i.e., a GLYPH<16> a 0 , b GLYPH<16> α ). Note that Lemma 23 gives that glyph[lscript] t is O p Lτ 2 q -Lipschitz, for each t P r T s . Thus Corollary 15 guarantees a regret bound (with respect to X d,H,a 0 ,α ) of O p Lτ 2 a dH ln p d q T q . Moreover, Eq. (19) of Corollary 15 ensures that the precondition Eq. (28) of Lemma 21 is satisfied. Thus, we may bound

where the first inequality uses Lemma 21 together with Eq. (19) of Corollary 15, and the second inequality uses Lemma 17 with glyph[epsilon1] GLYPH<16> 1 { T (by the theorem assumption, the inequality H ¥ τ r log 2 p 2 LT 2 { glyph[epsilon1] q s is indeed satisfied). Note that for the second inequality to hold, we also need that } K } 1 GLYPH<209> 1 ¥ a 0 for all K P K glyph[triangle] τ , which in particular requires (by Line 1) that } K } 1 GLYPH<209> 1 ¥ 1 {p 96 τ q if t mix p A q ¡ 4 τ . But if t mix p A q ¡ 4 τ , then for any K P K glyph[triangle] τ we have t mix p A q ¡ 4 GLYPH<4> t mix p A K q and hence } K } 1 GLYPH<209> 1 ¥ 1 {p 96 τ q by Lemma 20. Finally, the equality above uses the definition of glyph[lscript] t in Algorithm 1, and the final inequality uses the regret bound of Corollary 15. By our choice of β, ε , we see that the overall policy regret is ˜ O p Lτ 7 { 2 d 1 { 2 ? T q , as desired.

D Proof of Lower Bounds

In this section, we formally state and prove the regret lower bounds Theorem 1 and Theorem 8. The former states that the comparator class for online control of standard LDSs cannot be broadened to all marginally stable (time-invariant, linear) policies; the latter states that the mixing time assumption cannot be removed from the comparator class for online control of simplex LDSs. Both results hold even in constant dimension.

The basic idea is the same for both proofs: we construct two systems L 0 , L 1 which are identical until time T { 2 , but then at time T { 2 experience differing perturbations of constant magnitude. The costs are zero until time T { 2 , after which they penalize distance to a prescribed state (and can in fact be taken to be the same for both systems). The optimal strategy in the first T { 2 time steps therefore depends on which system the controller is in, but the controller does not observe this until time T { 2 , and hence will necessarily incur regret with respect to the optimal policy.

Formalizing this intuition requires two additional pieces: first, for both systems there must be a near-optimal time-invariant linear policy. This can be achieved by careful design of the dynamics, perturbations, and costs. Second, if the controller finds itself in a high-cost state at time T { 2 GLYPH<0> 1 , it must be unable to reach a low-cost state without incurring Ω p T q total cost along the way. In the standard LDS setting, we achieve this by setting the transition matrices A,B so that } B } GLYPH<16> O p 1 { T q (i.e. so constant-size controls have small effect on the state) and adding a penalty of | u t | to the cost for t ¡ T { 2 . In the simplex LDS setting, we achieve this by our choice of the valid constraint set I (which enforces that } u t } 1 GLYPH<16> O p 1 { T q for all t ).

See Fig. 3 for a pictorial explanation of the proof in the simplex LDS setting.

D.1 Proof of Theorem 1

In this section we give a formal statement and proof of Theorem 1. Recall the definition of an LDS (Definition 9). We define the class K κ p L q of policies that κ -marginally stabilize L below; it is equivalent to the class K κ,ρ p L q of policies that p κ, ρ q -strongly stabilize L (Definition 4) with ρ GLYPH<16> 0 .

Definition 24 (Marginal stabilization) . Amatrix M P R d GLYPH<2> d is κ -marginally stable if there is a matrix H P R d GLYPH<2> d so that GLYPH<15> GLYPH<15> H GLYPH<1> 1 MH GLYPH<15> GLYPH<15> / 1 and } M } , } H } , GLYPH<15> GLYPH<15> H GLYPH<1> 1 GLYPH<15> GLYPH<15> / κ . A matrix K P R d GLYPH<2> d is said to κ -marginally stabilize an LDS with transition matrices A,B P R d GLYPH<2> d if A GLYPH<0> BK is κ -marginally stable. For κ ¡ 0 and an LDS L on R d , we define K κ p L q to be the set of linear, time-invariant policies x GLYPH<222>GLYPH<209> Kx where K P R d GLYPH<2> d κ -marginally stabilizes L .

We also introduce a standard regularity assumption on cost functions: 15

Assumption 2. Let L ¡ 0 . We say that cost functions p c t q t , where c t : R d x GLYPH<2> R d u GLYPH<209> R , are L -regular if c t is convex and L -Lipschitz with respect to the Euclidean norm for all t .

Theorem 25 (Formal statement of Theorem 1) . Let Alg be any randomized algorithm for online control with the following guarantee:

1/2

T/2

1

0

T

𝑆1

𝑆2

𝑆3

Let d, T P N and κ ¡ 0 , and let L GLYPH<16> p A,B,x 1 , p w t q t , p c t q t q be an LDS with state space and control space R d ; L -regular cost functions p c t q t (Assumption 2); and perturbations p w t q t satisfying } w t } 2 / L for all t . Then the iterates p x t , u t q T t GLYPH<16> 1 produced by Alg with input p A,B,κ,T q on interaction with L satisfy

where p x L ,K t , u L ,K t q T t GLYPH<16> 1 are the iterates produced by following policy x GLYPH<222>GLYPH<209> Kx in system L for all t P r T s .

Then f p 1 , 1 , 1 , T q GLYPH<16> Ω p T q .

Remark 26. In the above theorem statement, if K κ p L q were replaced with K κ,ρ p L q , the class of linear time-invariant policies that p κ, ρ q -strongly stabilize L , then the main result of [1] would imply that in fact there is a (deterministic) algorithm GPC with regret at most poly p d, κ, L, ρ GLYPH<1> 1 qGLYPH<4> ? T log p T q on any LDS L satisfying the above conditions. Thus, Theorem 25 indeed provides a converse to [1].

We prove Theorem 25 by constructing a simple distribution over LDSs on which any algorithm must incur Ω p T q regret in expectation. Let β ¥ 2 be a constant that we will determine later, and fix T ¥ β . Recall that we denote an LDS on R d using the notation L GLYPH<16> p A,B,x 1 , p w t q t , p c t q t q , where A,B P R d GLYPH<2> d . We define two LDSs on R as follows:

where the (common) initial state is x 1 GLYPH<16> 1 , the (common) cost functions p c t q t are defined as

the perturbations of L 0 are w 0 t : GLYPH<16> 0 for all t , and the perturbations of L 1 are

For simplicity, we assume that T { 2 is an integer. Thus, at all times t GLYPH<24> T { 2 , the two systems have identical dynamics

but at time t GLYPH<16> T { 2 , system L 1 experiences a negative perturbation of magnitude 1 , whereas L 0 does not. The following lemma characterizes the performance of two time-invariant linear policies π 0 , π 1 for L 0 , L 1 respectively:

Lemma 27. Define π 0 , π 1 : R GLYPH<209> R by π 0 p x q GLYPH<16> x and π 1 p x q GLYPH<16> 0 . Then:

Policy π 0 is an element of K 1 p L 0 q , and the iterates p x L 0 ,π 0 t , u L 0 ,π 0 t q T t GLYPH<16> 1 produced by following π 0 in system L 0 satisfy

Policy π 1 is an element of K 1 p L 1 q , and the iterates p x L 1 ,π 1 t , u L 1 ,π 1 t q T t GLYPH<16> 1 produced by following π 1 in system L 1 satisfy

Proof. Note that π 0 , π 1 are both time-invariant linear policies. The inclusion π 0 P K 1 p L 0 q is immediate from the fact that L 0 has transitions A GLYPH<16> 1 , B GLYPH<16> GLYPH<1> β { T , and | A GLYPH<0> B | / 1 . Similarly, π 1 P K 1 p L 1 q because | A | / 1 . To bound the total cost of π 0 on L 0 , note that u L 0 ,π 0 t GLYPH<16> x L 0 ,π 0 t GLYPH<16> p 1 GLYPH<1> β { T q t GLYPH<1> 1 for all t P r T s . Hence,

Moreover, we have x L 1 ,π 1 t GLYPH<16> ✶ r t / T { 2 s and u L 1 ,π 1 t GLYPH<16> 0 for all t P r T s , from which it is clear that GLYPH<176> T t GLYPH<16> 1 c t p x L 1 ,π 1 t , u L 1 ,π 1 t q GLYPH<16> 0 .

Next, we show that the total cost of any trajectory p x t , u t q T t GLYPH<16> 1 can be lower bounded in terms of | x T { 2 GLYPH<0> 1 | in both L 0 and L 1 :

Lemma 28. Let Alg be any randomized algorithm for online control, and let b P t 0 , 1 u . The (random) trajectory p x t , u t q T t GLYPH<16> 1 produced by Alg in system L b satisfies the inequality

with probability 1 .

Proof. By definition of L 0 , L 1 , any valid trajectory in L b satisfies | u t | GLYPH<16> T β | x t GLYPH<0> 1 GLYPH<1> x t | for all T { 2 GLYPH<160> t GLYPH<160> T . We consider two cases:

If min T { 2 GLYPH<160> t / T | x t | ¥ 1 2 | x T { 2 GLYPH<0> 1 | , then

since β ¥ 2 .

If min T { 2 GLYPH<160> t / T | x t | GLYPH<160> 1 2 | x T { 2 GLYPH<0> 1 | , then

by the triangle inequality.

In both cases the claimed inequality holds.

We can now prove Theorem 25.

Proof of Theorem 25. Let b GLYPH<18> Unif pt 0 , 1 uq be an unbiased random bit, and let p x t , u t q T t GLYPH<16> 1 be the (random) trajectory produced by executing Alg on L b . On the one hand, by Eq. (41) applied to L 0 and L 1 , we have

where the first inequality uses the fact that the cost functions p c t q t are convex and 1 -Lipschitz and that | w 0 t | , | w 1 t | / 1 for all t P r T s ; and the second inequality is by Lemma 27. On the other hand, by Lemma 28, we have

where the key equality pGLYPH<14>q uses the fact that L 0 , L 1 are identical up until and including time T { 2 , and hence p x T { 2 , u T { 2 q is independent of b . Comparing Eq. (43) with Eq. (42) yields that

for any sufficiently large constant β .

D.2 Proof of Theorem 8

Definition 29. Let 0 / α / α / 1 and let I : GLYPH<16> GLYPH<148> α Pr α,α s ∆ d α . We define K p I q to be the set of linear, time-invariant policies x GLYPH<222>GLYPH<209> Kx where K P GLYPH<148> α Pr α,α s S d α .

Theorem 30 (Formal statement of Theorem 8) . Let Alg be any randomized algorithm for online control with the following guarantee:

Let d, T P N and I : GLYPH<16> GLYPH<148> α Pr 0 ,α s ∆ d α for some α P p 0 , 1 q . Let L GLYPH<16> p A,B, I , x 1 , p γ t q t , p w t q t , p c t q t q be a simplex LDS with state space ∆ d and cost functions p c t q t satisfying Assumption 1 with Lipschitz parameter L ¡ 0 . Then the iterates p x t , u t q T t GLYPH<16> 1 produced by Alg with input p A,B, I , T q on interaction with L satisfy

where p x L ,K t , u L ,K t q T t GLYPH<16> 1 are the iterates produced by following policy x GLYPH<222>GLYPH<209> Kx in system L for all t P r T s .

For any sufficiently large constant β , if we define α p T q : GLYPH<16> β { T , then f p 1 , 1 , α p T q , T q GLYPH<16> Ω p T q .

We define two simplex LDSs on ∆ 2 as follows:

where I 2 P R 2 GLYPH<2> 2 is the identity matrix, the (common) valid control set is I : GLYPH<16> GLYPH<148> α Pr 0 ,β { T s ∆ d α , the (common) initial state is x 1 GLYPH<16> p 0 , 1 q , the (common) cost functions p c t q t are defined as

the (common) perturbation strengths are γ t : GLYPH<16> 1 2 ✶ r t GLYPH<16> T { 2 s , and the perturbations of L 0 are w 0 t : GLYPH<16> p 1 { 2 , 1 { 2 q for all t whereas the perturbations of L 1 are w 1 t : GLYPH<16> p 1 , 0 q for all t . Thus, for both systems, the dynamics are described by

for all t GLYPH<24> T { 2 .

Lemma 31. Define π 0 , π 1 : ∆ 2 GLYPH<209> GLYPH<148> α Pr 0 , 1 s ∆ d by π 0 p x q : GLYPH<16> β T p 1 { 2 , 1 { 2 q and π 1 p x q : GLYPH<16> p 0 , 0 q . Then π 0 , π 1 P K p I q , and:

The iterates p x L 0 ,π 0 t , u L 0 ,π 0 t q T t GLYPH<16> 1 produced by following π 0 in system L 0 satisfy

The iterates p x L 1 ,π 1 t , u L 1 ,π 1 t q T t GLYPH<16> 1 produced by following π 1 in system L 1 satisfy

Proof. The fact that π 0 , π 1 P K p I q is immediate from Definition 29 and the choice of I . To bound the total cost of π 0 on L 0 , note that x L 0 ,π 0 t GLYPH<0> 1 p 2 q GLYPH<1> 1 { 2 GLYPH<16> p 1 GLYPH<1> β { T qp x t p 2 q GLYPH<1> 1 { 2 q for all t GLYPH<24> T { 2 , and x L 0 ,π 0 t GLYPH<0> 1 p 2 q GLYPH<1> 1 { 2 GLYPH<16> p 1 { 2 qp 1 GLYPH<1> β { T qp x t p 2 q GLYPH<1> 1 { 2 q for t GLYPH<16> T { 2 . Thus,

Moreover, we have x L 1 ,π 1 t GLYPH<16> p 0 , 1 q for all t / T { 2 and x L 1 ,π 1 t GLYPH<16> p 1 { 2 , 1 { 2 q for all t ¡ T { 2 , so indeed GLYPH<176> T t GLYPH<16> 1 c t p x L 1 ,π 1 t , u L 1 ,π 1 t q GLYPH<16> 0 as claimed.

Lemma 32. Let Alg be any randomized algorithm for online control, and let b P t 0 , 1 u . The (random) trajectory p x t , u t q T t GLYPH<16> 1 produced by Alg in system L b satisfies the inequality

Proof. By definition of L 0 , L 1 and the valid control set I , any valid trajectory in L b satisfies } x t GLYPH<1> x t GLYPH<0> 1 } 1 / 2 } u t } 1 / 2 β { T for all T { 2 GLYPH<160> t GLYPH<160> T . It follows that | x t GLYPH<0> 1 p 2 q GLYPH<1> 1 { 2 | ¥ | x t p 2 q GLYPH<1> 1 { 2 | GLYPH<1> 2 β { T for all such t , and hence

as claimed.

Proof of Theorem 30. Let b GLYPH<18> Unif pt 0 , 1 uq be an unbiased random bit, and let p x t , u t q T t GLYPH<16> 1 be the (random) trajectory produced by executing Alg on L b . On the one hand, by Eq. (44) applied to L 0 and L 1 , we have

where the first inequality uses the definition of I and the fact that the cost functions p c t q t are convex and 1 -Lipschitz per Assumption 1; and the second inequality is by Lemma 31. On the other hand, by Lemma 32, we have

where the key equality pGLYPH<14>q uses the fact that L 0 , L 1 are identical up until and including time T { 2 , and hence p x T { 2 , u T { 2 q is independent of b . Comparing Eq. (46) with Eq. (45) yields that

for any sufficiently large constant β .

E Implementation details

In this section we describe the version of GPC -Simplex (Algorithm 1) implemented for our experiments. First, the dynamical systems in our experiments are non-linear. The GPC -Simplex algorithm is still practical and applicable in such settings - concretely, any setting with update rule Eq. (9) - but of course several modifications/generalizations must be made:

The algorithm takes as input the function f describing the dynamics in Eq. (9), rather than transition matrices A,B . Accordingly, in Line 8, the expression p 1 GLYPH<1> } u t } 1 q Ax t GLYPH<0> Bu t

(which exactly corresponds to the noiseless update rule in a simplex LDS) is replaced by f p x t , u t q . Moreover, in Line 9, the hypothetical iterates x t p p, M r 1: H s q , u t p p, M r 1: H s q under policy π p,M r 1: H s are computed using the update rule f .

The algorithm directly takes as input a learning rate η for the mirror descent subroutine, rather than the mixing time bound τ . In our experiments, we always set η : GLYPH<16> a dH ln p H q{p 2 ? T q .

We always parametrize our systems so that the valid control set is the space of distributions ∆ d . Hence, the domain used for mirror descent is X d,H, 1 , 1 . Mirror descent is implemented by exponential weights updates with learning rate η and uniform initialization.

We remark that the above (natural) modifications to GPC -Simplex are analogous to the modifications to GPC made by [2] to perform online control for nonlinear systems.

F Experiments: Controlled SIR model

In this section, we provide additional experiments in the controlled SIR model. Specifically, in Appendix F.1 we provide experimental evaluations when there are perturbations to the system (i.e. γ t is not always 0 in Eq. (9)). In Appendix F.2 we vary the parameters of the SIR model.

F.1 Control in presence of perturbations

We experiment with the SIR system Eq. (11) with the following parameters:

We test the performance of our algorithm on p c 2 , c 3 q GLYPH<16> p 1 , 5 q . In addition, we add a perturbation sequence w t GLYPH<16> r 0 , 1 , 0 s , @ 1 / t / 200 . γ t GLYPH<18> 0 . 01 GLYPH<4> Ber p 0 . 2 q , @ 1 / t / 200 .

Fig. 4 shows comparison of the costs over T GLYPH<16> 200 time steps incurred by GPC-simplex to that of always executing u t GLYPH<16> r 1 , 0 s (full prevention) and that of always executing u t GLYPH<16> r 0 , 1 s (no prevention). In addition to cost, we plot the value of u t p 2 q over time, representing how relaxed prevention measure evolves over time according to GPC-simplex .

F.2 Alternative parameter settings

We experiment with two SIR systems with different set of parameters. The first uses the following parameters:

whereas the second uses the following parameters:

In both cases, the cost function is:

For both experiments, we test the performance of our algorithm on different choices of parameters for the cost function. In particular, we test the parameter tuples:

Figs. 5 and 6 show comparison of the costs over T GLYPH<16> 200 time steps incurred by GPC -Simplex to that of always executing u t GLYPH<16> r 1 , 0 s (full prevention) and that of always executing u t GLYPH<16> r 0 , 1 s (no prevention). Specifically, Fig. 5 uses the first set of parameters above, and Fig. 6 uses the second set. In addition to cost, we plot the value of u t p 2 q over time, representing how the effective transmission rate evolves over time according to GPC -Simplex .

We notice that our algorithm consistently outperforms the two baselines. No matter how we set the parameters, our algorithm will outperform the full-intervention baseline since its cumulative cost grows linearly with time. As c 3 gets larger, the gap between our algorithm and the no-intervention baseline becomes smaller, since the optimal policy with a high cost on control is basically playing no control.

and cost function given by:

Cost OverTime,β=0.50,θ=0.03,=0.005,C2=1,C3=5

u(2) Over Time,β=0.50,θ=0.03,=0.005,C2=1,C3=5

costs

1.0

3.0

costs(full preventions)

costs (no preventions)

0.8

2.5

2.0

0.6

(z)n

1.5

0.4

1.0

0.5 -

0.2

0.0

0.0-

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

Time Step

Time Step

Cost OverTime,β=0.50,θ=0.03,E=0.005,C2=1,C3=5

u(2) OverTime,β=0.50,θ=0.03,=0.005,C2=1,C3=5

1.0

3.0

costs

costs (full preventions)

costs (no preventions)

2.5

0.8

2.0

9°0

1.5

(z)n

0.4

1.0

0.5-

0.2

00

0

25

50

75

100

125

150

175

200

0

25

50

75

100

125

150

175

200

Time Step

Time Step

G Experiments: Controlling hospital flows

In this section, we provide more details regarding the setup and experiments in Section 4.1.

The continuous time dynamical system considered by [27] is the following: let S p t q , I p t q denote the susceptible and infected fraction of the population at time t , and let σ p t q denote the control at time t . The system has some initial state p S p 0 q , I p 0 qq in the set

reflecting the constraint that S p 0 q , I p 0 q represent disjoint proportions of a population, and the system evolves according to the differential equation

where γ ¡ 0 is some fixed model parameter, and the control σ p t q models a non-pharmaceutical intervention (NPI) inducing a time-dependent reproduction number σ p t q P r 0 , σ 0 s , where σ 0 is the base reproduction number in the absence of interventions. In most examples in [27], including the example of controlling hospital flows, the parameter settings σ 0 GLYPH<16> 3 and γ GLYPH<16> 0 . 1 are used. This means that the natural discretization of Eq. (48) is in fact equivalent to Eq. (11) with transmission rate β : GLYPH<16> γσ 0 GLYPH<16> 0 . 3 , recovery rate θ : GLYPH<16> γ GLYPH<16> 0 . 1 , loss-of-immunity rate ξ : GLYPH<16> 0 , no perturbations (i.e. γ t GLYPH<16> 0 for all t ), and control

at each time t .

Cost Over Time, β=0.50, θ=0.03, =0.005, C2=1, C=20

Cumulative Cost Over Time, β=0.50, 0=0.03, E=0.005, C2=1, C=20

u(2) Over Time,β=0.50, 0=0.03, =0.005, C2=1, C=20

costs

cumulative costs

1.0

12

(suoguanaud ing) sso

costs (no preventions)

350

cumulative costs (full preventions)

cumulative costs (no preventions)

10

00E

0.8

250

0.6

200

150

0.4

100

0.2

50

25

50

75

Time Step

100

125

150

175

200

25

50

75

Time Step

100

125

150

175

200

0.0七

25

50

75

Time Step

100

125

150

175

200

Cost Over Time, β=0.50, θ=0.03, E=0.005, C2=1, C=10

Cumulative Cost Over Time, β=0.50,θ=0.03,E=0.005, C=1,C=10

u(2) Over Time, β=0.50, θ=0.03, E=0.005, C2=1, C3=10

costs

costs (no preventions)

(suoquaad uny) sqso

175

cumulative costs

（suguaud） so enun

cumulative costs (no preventions)

1.0

150

0.8

0.6

100

75

0.4

50

25

0.2

25

50

75

Time Step

100

125

150

175

200

25

50

75

Time Step

100

125

150

175

200

0.0

25

50

75

Time Step

100

125

150

175

200

Cost Over Time,β=0.50, θ=0.03,=0.005, C2=1, C3=5

Cumulative Cost OverTime,β=0.50,θ=0.03,=0.005,C2=1,C=5

u(2) Over Time,β=0.50,θ=0.03,E=0.005, C=1, C=5

costs

cumulative costs

1.0

3.0

costs (full preventions)

costs (no preventions)

175

cumulative costs (full preventions)

cumulative costs (no preventions)

2.5

150

80

2.0

125

100

0.6

1.5

75

0.4

1.0

50

0.5

25

0.2

0.0

25

50

75

Time Step

100

125

150

175

200

25

50

75

Time Step

100

125

150

175

200

0.0

25

50

75

Time Step

100

125

150

175

200

Cost Over Time, β=0.50, θ=0.03, =0.005, C2=1, C3=1

Cumulative Cost OverTime,β=0.50,θ=0.03,=0.005,C=1,C=1

u(2) Over Time, β=0.50, 0=0.03, =0.005, C2=1, C=1

1.0

costs

175

cumulative costs

cumulative costs (full preventions)

1.0

0.8

costs (no preventions)

150

cumulative costs (no preventions)

0.9

0.6

503

125

0.8

Cumulative

100

0.4

75

0.6

50

0.2

0.5

25

0.4

0.0

25

50

75

100

125

150

175

200

25

50

75

100

125

150

175

200

25

50

75

100

125

150

175

200

Time Step

Time Step

Time Step

The goal in [27] is the following: given an initial state p S p 0 q , I p 0 qq along with a horizon length T ¡ 0 and the parameters listed above, choose an admissible control function σ : r 0 , T s GLYPH<209> r 0 , σ 0 s to minimize the loss

where L p S p t q , I p t q , σ p t qq is the instantaneous cost at time t , and the extra term S 8 p S p T q , I p T q , σ 0 q incentivizes the state of the system at time T to lead to a favorable long-term trajectory (in the absence of any interventions after time T ). In [27], the following formula for S 8 is given; see that paper for further discussion:

Cost Over Time, β=0.30, θ=0.05, =0.001, C2=1, C3=20

Cumulative Cost Over Time, β=0.30, θ=0.05, F=0.001, C2=1, C=20

u(2) Over Time, β=0.30,θ=0.05, =0.001, C2=1,C=20

costs (full preventions)

costs

175

cumulative costs

cumulative costs fulpreventions)

1.o

costs (no preventions)

cumulative costs (no preventions)

150

0.8

50

125

SO

100

(z)n

0.6

75

0.4

50

25

0.2

0.0

25

50

75

Time Step

100

125

150

175

200

25

50

75

Time Step

100

125

150

175

200

25

50

75

Time Step

100

125

150

175

200

Cost Over Time, β=0.30, 0=0.05,=0.001, C2=1, C=10

Cumulative Cost Over Time,β=0.30,θ=0.05,=0.001,C=1,C3=10

u(2) Over Time, β=0.30, θ=0.05, =0.001, C2=1, C3=10

3.0

costs (no preventions)

costs

(suoquaaud ng) sj5o

175

cumulative costs

cumulative costs (no preventions)

cumulative costs full preventions)

1.0

2.5

150

0.8

2.0

125

502

100

(z)n

9°0

1.5

75

1.0

0.4

50

0.5

25

0.2

0.0

25

50

75

100

125

150

175

200

25

50

75

100

125

150

175

200

25

OS

75

100

125

150

175

200

Time Step

Time Step

Time Step

Cost Over Time, β=0.30, θ=0.05, E=0.001, C2=1, C3=5

Cumulative Cost OverTime,β=0.30,θ=0.05,E=0.001,C=1,C=5

u(2) Over Time, β=0.30, θ=0.05, =0.001, C2=1, C3=5

1.6

costs

(suoguanaud ng) so

175

cumulative costs

cumulative costs (full preventions)

1.0

1.4

costs (no preventions)

150

cumulative costs (no preventions)

60

1.2

503

125

0.8

1.0

100

0.7

0.6-

75

0.6

0.4

50

0.5

0.2

25

0.4

00

25

50

75

Time Step

100

125

150

175

200

25

50

75

Time Step

100

125

150

175

200

0

25

50

75

Time Step

100

125

150

175

200

Cost Over Time, β=0.30,0=0.05,=0.001, C2=1,C3=1

Cumulative Cost Over Time,β=0.30,θ=0.05,E=0.001,C2=1,C=1

u(2) OverTime,β=0.30,θ=0.05,=0.001,C=1,C=1

(suoquanaud n) sjso)

costs

175

cumulative costs

cumulative costs (full preventions)

1.00

0.8

costs (no preventions)

cumulative costs (no preventions)

150

860

0.6

125

96°0

100

0.4

60

0.2

0.92

25

0.0

0.90

25

50

75

100

125

150

175

200

25

50

75

100

125

150

175

200

25

50

75

100

125

150

175

200

Time Step

Time Step

Time Step

The instantaneous cost is modeled by [27] as follows:

where c 2 , c 3 are some parameters determining the cost of preventing disease transmission and the cost of a medical surge (i.e. when the proportion of infected individuals exceeds y max ). Notice that the second term above will indeed be very small in magnitude unless I p t q exceeds y max .

Note that GPC -Simplex cannot directly handle end-of-trajectory losses such as the term S 8 p S p T q , I p T q , σ 0 q . Thus, in our evaluation of GPC -Simplex on this system, we instead incorporate S 8 into the instantaneous cost functions. Concretely, we use the following cost function at

time t :

Recall that we write x t GLYPH<16> p S t , I t , R t q and u t p 1 q GLYPH<16> 1 GLYPH<1> σ p t q{ σ 0 , so modulo the addition of S 8 to all times t GLYPH<160> T and the conversion from continuous time to discrete time, our loss is analogous to that of [27].

H Experiments: Controlled replicator dynamics

The replicator equation is a basic model in evolutionary game theory that describes how individuals in a population will update their strategies over time based on their payoffs from repeatedly playing a game with random opponents from the population [11]. The basic principle is that strategies (or traits) that perform better than average in a given environment will, over time, increase in frequency within the population, whereas strategies that perform worse than average will become less common.

Formally, consider a normal-form two-player game with d possible strategies and payoff matrix M P R d GLYPH<2> d . A population at time t is modelled by the proportion of individuals that currently favor each strategy, and thus can be summarized by a distribution x p t q P R d . The fitness of an individual playing strategy i P r d s in a population with strategy distribution x P ∆ d is defined to be

where e i P R d is the indicator vector for strategy i . That is, fitness M,x p i q is simply the expected payoff of playing strategy i against a random individual from the population. The replicator dynamics posit that the population's distribution over strategies x p t q will evolve according to the following differential equation:

It is straightforward to check that this differential equation preserves the invariant that x p t q is a distribution. This equation can induce various types of dynamics depending on the initialization and payoff matrix M : the distribution may converge to an equilibrium, or it may cycle, or it may even exhibit chaotic behavior [11]. In this study we focus on a simple (time-discretized) replicator equation - namely, the equation induced by a generalized Rock-Paper-Scissors game - when the payoffs may be controlled .

Controlled Rock-Paper-Scissors. The standard Rock-Paper-Scissors game has d GLYPH<16> 3 and payoff matrix

Consider a setting where the game is run by an external agent that is allowed to set the payoffs. For simplicity, we assume that the game remains zero-sum and the rewards sum to 1 , so the payoff matrix is now

for a control vector u P ∆ 3 . The discrete-time analogue of the replicator equation with this controlled payoff matrix M p u q is

where x t , u t P ∆ 3 are the population distribution and control at time t respectively, and η P p 0 , 1 q is the rate of evolution. Note that the term x t M p u t q x t does not need to appear in Eq. (50) because M p u t q is always zero-sum. Also, since η / 1 and all entries of M p u t q are at most 1 in magnitude, if x t is a distribution then x t GLYPH<0> 1 will remain a distribution. We omit noise in this study, so Eq. (50) is a special case of Eq. (9) with γ t GLYPH<16> 0 for all t .

0.10

0.08

0.06

Cost

0.04

0.02

0.00

0

Comparisonoflossesovertime

20

PopulationevolutionwithGPC-Simplex

1.0

Scissors (x[2])

Rock (x[1])

Paper (x[3])

0.8

Proportion

0.6

0.4

0.2

0.0

0

20

40

60

80

100

Time Step

40

60

GPC-Simplex costs

=Best-response costs

80

Default control costs

100

Time Step

Figure 7: Experimental results for dynamical system with horizon T GLYPH<16> 100 , uniform initial state, update rule Eq. (50) with η GLYPH<16> 1 { 4 , no perturbations, and time-invariant cost function c t p x t , u t q GLYPH<16> x 2 t 1 for all times t . GPC -Simplex was implemented as described in Appendix E. The Best Response controller at each time t picks the control u that minimizes c t GLYPH<1> 1 p f p x t , u q , u q . The default controller picks the uniform control u GLYPH<16> p 1 { 3 , 1 { 3 , 1 { 3 q .

Parameters and cost function. We define a (nonlinear) dynamical system with uniform initial state x 1 GLYPH<16> p 1 { 3 , 1 { 3 , 1 { 3 q , update rule Eq. (50) with η GLYPH<16> 1 { 4 , and T GLYPH<16> 100 timesteps. We consider the fixed cost function

which can be thought of as penalizing the strategy 'rock'.

Results. We compare GPC -Simplex (implemented as described in Appendix E) with a baseline control that simply uses the standard Rock-Paper-Scissors payoff matrix (up to scaling) induced by u GLYPH<16> p 1 { 3 , 1 { 3 , 1 { 3 q P ∆ 3 . As shown in Fig. 7a, GPC -Simplex (shown in blue) significantly outperforms this baseline (shown in green), learning to alter the payoff in such a way that the population tends to avoid the 'rock' strategy. The evolution of the dsitribution over time under GPC -Simplex is shown in Fig. 7b.

For completeness, we also compare GPC -Simplex against the 'Best Response' strategy (shown in dashed orange) that essentially performs 1 -step optimal control, using the fact that the cost function for this example is time-invariant. While both controllers eventually learn a good policy, Fig. 7a clearly shows that Best Response learns faster. However, it is strongly exploiting the time-invariance of the cost function, since in general, this algorithm computes the best response with respect to the previous cost function rather than the current cost function, which it does not observe until after playing a control. In Fig. 8, we consider a slightly modified system where the cost function includes a cost on the control with probability 1 { 2 . In this setting, we see that GPC -Simplex still eventually learns a good policy, whereas Best Response and the default control incur large costs through the trajectory. Best Response in particular suffers greatly due to the time-varying nature of the costs.

I Discussions

I.1 Broader impacts

Our work provides a robust algorithm with theoretical justifications for practical control problems that might be applicable to problems such as disease control. The experiments performed are preliminary. More careful empirical verification is necessary before our algorithm can be responsibly implemented in high-impact scenarios. Excluding the scenario of ill intention, we do not anticipate any negative social impact.

Comparisonofsliding-windowlossesovertime

0.6

Best-response costs

GPC-Simplex costs

Default control costs

0.5

0.4

Cost

0.3

0.2

0.1

0.0

0

25

50

75

100

125

150

175

200

TimeStep

I.2 Computational Resources for Experiments

The experiments in this work are simulations and relatively small-scaled. They were run on Google Colab with default compute resources. For each experiment, the time required to roll-out one trajectory using GPC -Simplex was less than 10 minutes.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The paper discussed the limitations of the work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For each theoretical result, the paper provides the full set of assumptions and a complete (and correct) proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper fully discloses all the information needed to reproduce the main experimental results of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to the anonymous repository containing our code. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not

including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experiments are primarily deterministic; the experiments with randomness are provided largely as proof-of-concept.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms to the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: we cite the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of controlling partially observable linear dynamical systems (LDS) with adversarial disturbances. Traditional methods for controlling LDS assume full observability, but this assumption does not hold in many real-world applications. The authors propose the GPC-PO-Simplex algorithm, which leverages mirror descent and approximate solutions to sub-problems to achieve control in partially observable settings. Theoretical guarantees are provided for the algorithm's performance, and empirical evaluations demonstrate its effectiveness in controlling population dynamics and other systems under adversarial conditions.\n\nSTRENGTHS:\nOriginality: \nThe paper introduces a novel approach to control partially observable LDS by combining mirror descent with a new parameterization method. This is a significant departure from traditional methods that assume full observability.\n\nQuality: \nThe theoretical foundations of the algorithm are robust, with detailed proofs and clear assumptions. The analysis includes regret bounds that highlight the algorithm's near-optimal performance under specific conditions.\n\nClarity: \nThe paper is well-structured and clearly written. The problem is well-motivated, and the methodology is presented in a step-by-step manner that is easy to follow.\n\nSignificance: \nThe ability to control systems with only partial observability has broad implications for various fields, including epidemiology, biology, and economics. The proposed method has the potential to be applied to a wide range of real-world problems.\n\nWEAKNESSES:\nIdealized Assumptions: \nThe algorithm assumes certain ideal conditions, such as known transition matrices and Gaussian noise, which may not hold in all real-world scenarios. Relaxing these assumptions could make the method more broadly applicable.\n\nLimited Generalization: \nWhile the algorithm performs well on the tested systems, its generalization to highly non-linear or more complex systems is not fully established. Further work is needed to test the algorithm's performance in a broader range of scenarios.\n\nSensitivity to Parameters: \nThe performance of the algorithm is sensitive to the choice of parameters. Providing more guidance on parameter selection or developing a method to adaptively choose parameters could improve practical usability.\n\nExperimental Scope: \nThe experimental validation is limited to simulated environments. More real-world experiments would strengthen the evidence for the algorithm's practical effectiveness.\n\nQUESTIONS:\n1.How does the algorithm perform in real-world applications with non-Gaussian noise and unknown transition matrices?\n2.Can the authors provide more detailed guidance or a method for adaptively selecting the algorithm's parameters?\n3.What are the computational requirements of the algorithm in large-scale systems? \n4. How robust is the algorithm to inaccuracies in the model assumptions, such as errors in the transition matrices or noise characteristics?\n5. How does the algorithm handle delays in control actions or observations, which are common in real-world systems?\n\nLIMITATIONS:\nThe authors have discussed some limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of actively controlling a class of dynamical systems that are used to model population dynamics, like the SIR model or replicator dynamics. Building on prior work on non-stochastic / adversarial control of linear dynamical systems, the authors propose a new class of dynamics, simplex LDS, and show how a generalization of disturbance action control policies can achieve sublinear regret with respect to the comparator class of linear controllers with bounded mixing time. In addition to their theoretical results, the authors carry out numerical simulations on nonlinear dynamical systems to investigate the benefits of their algorithm.\n\nSTRENGTHS:\nI really liked this paper. It does a nice job of taking recently developed tools from online linear control and extends them to a new, interesting, and well motivated, class of dynamical systems. I like how the paper shows both positive and negative results. I found the idea that there is a gap between prediction and control for marginally stable systems to be quite insightful, and the regret bounds for the simplex LDS systems to be quite neat. \n\nI also thought that the experiments were a very good complement to the theoretical results. It is always nice when experiments test algorithms in regimes where the theory may not exactly hold.\n\nWEAKNESSES:\nIt would be nice to include simple examples of simplex LDSs in the introduction when defining the model to provide the reader with more intuition. Some of the technical conditions are somewhat strange and hard to interpret, particularly the condition that the norm of the control vector is independent of the state in the comparator class. I think that the paper would be well served if the authors could provide further insights regarding why these assumptions are technically necessary, or under what conditions they might be justified. \n\nIn terms of experiments, it would also be nice if the authors experimented with higher dimensional systems. They would make the paper stronger, but I don’t feel they are strictly necessary.\n\nQUESTIONS:\nSee questions about regarding the comparator class of policies.\n\nAlso please make the figures and label text larger, they are a bit hard to read.\n\nLIMITATIONS:\nYes, these have been discussed to the extent necessary.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies online control problems for the simplex Linear Dynamical System model. They propose a gradient-based control algorithm GPC-Simplex and show its regret bound. Experiments are conducted to validate the performance of GPC-Simplex.

STRENGTHS:
The paper is in general well-written and smooth to follow. The proofs seem to be theoretically sound. The studied problem is of great interest and hard to tackle as it involves adversarial shocks and time-varying cost functions.

WEAKNESSES:
My main concern is the novelty of the proposed algorithm compared with the original GPC algorithm. Could the authors provide more justifications for this?

QUESTIONS:
It seems that the authors assume the cost functions are uniformly Lipschitz. What happens if each $c_t$ is $L_t$-Lipschitz, but there isn't an upper bound for $L_t$'s?

LIMITATIONS:
I don't see any limitations or potential negative societal impact of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the problem of population control in more practical settings such as the presence of adversarial noise and time-varying cost functions by proposing a robust methodology derived from online non-stochastic control theory.\n\nSTRENGTHS:\nThe paper provides regret guarantees against policies that do not strongly stabilize the simplex LDS by developing an algorithm that aims to achieve low regret against this policy class while being robust to adversarial shocks to the system and adaptable to time-varying cost functions. All assumptions were reasonably justified, and all theoretical results were adequately proved. The paper also provides promising empirical results by applying the proposed algorithm to controlled settings of the SIR model and the replicator dynamics from evolutionary game theory.\n\nWEAKNESSES:\n- Unclear motivations for the considered simplex LDS setting. A discussion on more general LDS settings would be helpful.\n\n- While an in-depth discussion on the assumption of gamma_t being observable was provided, there should be concrete examples where the controller could realistically observe the total counts of individuals of different types in a population. This is particularly important because the proposed algorithm and its guarantee are grounded on the simplex LDS model, which requires this assumption.\n\n- Missing formal analyses on time complexity. This could be important for larger-scaled experiments (d >> 3), which the paper did not explore.\n\nQUESTIONS:\nSee the above and below comments.\n\nLIMITATIONS:\nSee the above comments.\n\nHave the authors adequately addressed the limitations and potential negative societal impact of their work? If not, please include constructive suggestions for improvement. Authors should be rewarded rather than punished for being upfront about the limitations of their work and any potential negative societal impact.\n\n- Given the various assumptions made while proving the regret guarantee, a dedicated Limitations or Implications section delineating use cases and generalizability should be included.\n\n- Figure 1 and its texts are too small; the texts in Figure 2 as well.\n\n- The experiment on vanilla SIR model in the main text seems trivial and should be replaced with the replicator dynamics experiment in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}