SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This follows from using the definition of  $A_{\mathrm{finite},m}$  to bound  $\tilde{L}$  and  $H_2$  in Theorem 3.1. This improves existing results [Anava et al., 2015] by a factor of  $m^{1/4}$ . Our bound depends on the Lipschitz continuity constants as  $\sqrt{L\tilde{L}}$  whereas existing bounds depend as  $\tilde{L}$ , and  $\tilde{L}$  can be as large as  $\sqrt{m} L$  (Theorem 2.1). We defer the full proof to Appendix C and a detailed comparison with existing results to Appendix C.1. The following lower bound shows that this is tight in the worst-case.
Theorem 3.4. There exists an instance of the online convex optimization with finite memory problem with constant memory length  $m$ ,  $(\mathcal{X},\mathcal{H} = \mathcal{X}^m,A_{\mathrm{finite},m},B_{\mathrm{finite},m})$ , and there exist  $L$ -Lipschitz continuous loss functions  $\{f_t:\mathcal{H}\to \mathbb{R}\}_{t = 1}^T$  such that the regret of any algorithm  $\mathcal{A}$  satisfies
$$
R _ {T} (\mathcal {A}) \geq \Omega \left(m \sqrt {T L ^ {2}}\right).
$$
To the best of our knowledge, this is the first non-trivial lower bound for OCO with finite memory with an explicit dependence on the memory length  $m$ . Our construction involves three main steps, the first two of which are loosely based on Altschuler and Talwar [2018]. First, divide time into  $N = T / m$  blocks of size  $m$ . (For simplicity, assume  $T$  is a multiple of  $m$ .) Second, sample a random sign  $\epsilon_{n}$  for each block  $n \in [N]$ . Third, for  $t > m$  choose
$$
f_{t}(h_{t}) = \underbrace{\epsilon_{\left\lceil\frac{t}{m}\right\rceil}}_{(a)}\underbrace{Lm^{-\frac{1}{2}}}_{(b)}\underbrace{\left(x_{t - m + 1} + \cdots + x_{m\lfloor\frac{t}{m}\rfloor + 1}\right)}_{(c)},
$$
where term (a) is the random sign  $\epsilon_{n}$  sampled for the block  $n = \lceil t / m\rceil$  that  $t$  belongs to, term (b) is a scaling factor chosen while respecting the Lipschitz continuity constraint, and term (c) is a sum over a subset of past decisions. Two important features of this construction are: (i) a random sign is sampled for each block rather than each round; and (ii) the loss in round  $t$  depends on the history of decisions until and including the first round of the block that  $t$  belongs to. These exploit the fact that an algorithm cannot overwrite its history and penalize it for its past decisions even after it observes the random sign  $\epsilon_{n}$  for the current block. (See Fig. 1 for an illustration.) Existing lower bound proofs for OCO sample a random sign in each round and choose  $f_{t}(x_{t})\propto \epsilon_{t}x_{t}$ . A first attempt at extending this for the OCO with finite memory setting would be to choose  $f_{t}(h_{t})\propto \epsilon_{t}\sum_{k = 0}^{m - 1}x_{t - k}$ . However, in constraint to our approach, this does not exploit the fact that an algorithm cannot overwrite its history and does not suffice for obtaining a matching lower bound.
Figure 1: An illustration of the loss functions  $f_{t}$  for the OCO with finite memory lower bound.
Comparison of upper bound with prior work. The algorithmic ideas and analysis for our regret upper bound are influenced by Anava et al. [2015]. However, an important innovation in our work is the use of weighted norms in the case of linear sequence dynamics. This is a simple but powerful way of encoding prior knowledge about a problem, and allows us to derive non-trivial regret bounds in the case of unbounded-length histories. The technical complications that arise are captured in bounding the relevant quantities of interest, e.g., the Lipschitz constant  $\tilde{L}$ , the operator norm  $\|A^k\|$ , etc. Furthermore, using weighted norms even leads to improved regret bounds for some applications. Indeed, consider the application to online linear control with adversarial disturbances (Section 4.1). Our framework and upper bound applied to this problem (Theorem 4.1) improve upon the existing upper bound, which used a finite memory approximation. See Lemmas E.2 and E.6 for an illustration of the technical details involved when using weighted norms.
In this section we apply our framework to online linear control (Section 4.1) and online performative prediction (Section 4.2). We defer expanded details and proofs to Appendices E and F respectively.
4.1 Online Linear Control
Background. Online linear control (OLC) is the problem of controlling a system with linear dynamics, adversarial disturbances, and adversarial and convex losses. It combines aspects from control theory and online learning. We refer the reader to Agarwal et al. [2019b] for more details. Here, we introduce the basic mathematical setup of the problem.
Let  $S \subseteq \mathbb{R}^{d_s}$  and  $\mathcal{U} \subseteq \mathbb{R}^{d_u}$  denote the state and control spaces. Let  $s_t$  and  $u_t$  denote the state and control at time  $t$  with  $s_0$  being the initial state. The system evolves according to the linear dynamics  $s_{t+1} = Fs_t + Gu_t + w_t$ , where  $F \in \mathbb{R}^{d_s \times d_s}$ ,  $G \in \mathbb{R}^{d_s \times d_u}$  are matrices satisfying  $\|F\|_2, \|G\|_2 \leq \kappa$  and  $w_t \in \mathbb{R}^{d_s}$  is an adversarially chosen disturbance with  $\|w\|_2 \leq W$ . Without loss of generality, we assume that  $\kappa, W \geq 1$ ,  $d_s = d_u = d$ , and also define  $w_{-1} = s_0$ . For  $t = 0, \dots, T-1$ , let  $c_t: S \times U \to [0,1]$  be convex loss functions chosen by an oblivious adversary. The functions  $c_t$  satisfy the following Lipschitz condition: if  $\|s\|_2, \|u\|_2 \leq D_{\mathcal{X}}$ , then  $\|\nabla_s c_t(s,u)\|$ ,  $\|\nabla_u c_t(s,u)\| \leq L_0 D_{\mathcal{X}}$ . The goal in online linear control is to choose a sequence of policies that yield a sequence of controls  $u_t$  to minimize the regret  $R_T(\Pi) = \sum_{t=0}^{T-1} c_t(s_t, u_t) - \min_{\pi^* \in \Pi} \sum_{t=0}^{T-1} c_t (s_t^{\pi^*}, u_t^{\pi^*})$ , where  $s_t$  evolves according to linear dynamics stated above,  $\Pi$  denotes a controller class, and  $s_t^{\pi^*}, c_t^{\pi^*}$  denote the state and control at time  $t$  when the controls are chosen according to  $\pi^*$ .
A very simple controller class is constant input, i.e.,  $\Pi = \{\pi_u : \pi(s) = u \in \mathcal{U}\}$ . In this case, the history  $h_t$  can be represented by the finite-dimensional state  $s_t$ , and the operators can be set to  $A = F$  and  $B = G / \|G\|$ . However, like previous work [Agarwal et al., 2019b] we focus on the class of  $(\kappa, \rho)$ -strongly stable linear controllers,  $\mathcal{K}$ , where  $K \in \mathcal{K}$  satisfies  $F - GK = HLH^{-1}$  with  $\|K\|_2, \|H\|_2, \|H^{-1}\|_2 \leq \kappa$  and  $\|L\|_2 = \rho < 1$ . Given such a controller, the inputs are chosen as linear functions of the current state, i.e.,  $u_t = -K s_t$ . Unfortunately, parameterizing  $u_t$  directly with a linear controller as  $u_t = -K s_t$  leads to a non-convex problem because  $s_t$  is a non-linear function of  $K$ , e.g., if disturbances are 0, then  $s_t = (F - GK)^t s_0$ . An alternative parameterization is the disturbance-action controller (DAC).
Definition 4.1. Let  $K \in \mathcal{K}$  be fixed. The class of disturbance-action controllers (DACs)  $\mathcal{M}_K$  is  $\{(K, M): M = (M^{[s]})_{s=0}^{\infty}\}$ , where  $M^{[s]} \in \mathbb{R}^{d \times d}$  satisfies  $\|M^{[s]}\|_2 \leq \kappa^4\rho^s$ . The control in round  $k$  is chosen as  $u_k = -K s_k + \sum_{s=1}^{k+1} M^{[s]} w_{k-s}$ .
The class of such DACs has two important properties. First, it acts on the entire history of past disturbances. Consequently, given an arbitrary  $K \in \mathcal{K}$ , every  $K^{*} \in \mathcal{K}$  can be expressed as a DAC  $(K, M) \in \mathcal{M}_{K}$  with  $M = (M^{[1]}, \ldots, M^{[T + 1]}, 0, \ldots)$  [Agarwal et al., 2019a, Section 16.5]. That is,  $\mathcal{K} \subseteq \mathcal{M}_{K}$  and it suffices to compute regret against  $\mathcal{M}_{K}$  instead of  $\mathcal{K}$ . For the rest of this paper we fix  $K \in \mathcal{K}$  and denote  $\widetilde{F} = F - GK$ . Second, suppose  $M_{t} = (M_{t}^{[s]})_{s=0}^{\infty}$  is the parameter chosen in round  $t$  and the control  $u_{t}$  is chosen according to the DAC  $(K, M_{t})$ . Then,  $s_{t}$  and  $u_{t}$  are linear functions of the parameters, which implies that  $c_{t}$  is convex in the parameters. (See the next paragraph on "Formulation as OCO with Unbounded Memory" for a formula.) A similar parameterization was first considered for online linear control by Agarwal et al. [2019b] and is based on similar ideas in control theory, e.g., Youla [Youla et al., 1976, Kucera, 1975] and SLS [Wang et al., 2019, Anderson et al., 2019].
**Formulation as OCO with Unbounded Memory.** The first step is a change of variables with respect to the control inputs from linear controllers to DACs and the second is a corresponding change of variables for the state. Define the decision space  $\mathcal{X} = \{M = (M^{[s]}) : M^{[s]} \in \mathbb{R}^{d \times d}, \| M^{[s]} \|_2 \leq \kappa^4 \rho^s\}$ . Define the history space  $\mathcal{H}$  to be the set consisting of sequences  $h = (Y_k)$ , where  $Y_0 \in \mathcal{X}$  and  $Y_k = \widetilde{F}^{k-1} G X_k$  for  $X_k \in \mathcal{X}, k \geq 1$ . (Recall  $\widetilde{F} = F - G K$ .) Define weighted norms  $\| M \|_{\mathcal{X}}^2 = \sum_{s=1}^{\infty} \rho^{-s} \| M^{[s]} \|_F^2$  and  $\| h \|_{\mathcal{H}}^2 = \sum_{k=0}^{\infty} \xi_k^2 \| Y_k \|_{\mathcal{X}}^2$ , where the weights  $(\xi_k)$  are defined as  $\xi = (1,1,1, \rho^{-\frac{1}{2}}, \rho^{-1}, \rho^{-\frac{3}{2}}, \ldots)$ . Define the linear operators  $A : \mathcal{H} \to \mathcal{H}$  and  $B : \mathcal{W} \to \mathcal{H}$  as  $A((Y_0, Y_1, \ldots)) = (0, G Y_0, \widetilde{F} Y_1, \widetilde{F} Y_2, \ldots)$  and  $B(M) = (M, 0, 0, \ldots)$ . Note that the problem follows linear sequence dynamics with the  $\xi$ -weighted 2-norm (Definition 2.3). The weights in the norms on  $\mathcal{X}$  and  $\mathcal{H}$  increase exponentially. However, the norms  $\| M^{[s]} \|_F^2$  and  $\| \widetilde{F}^{k-1} G \|_F^2$  decrease exponentially as well: by definition of  $\mathcal{X}$  and the assumption on  $\widetilde{F} = F - G K$  for  $K \in \mathcal{K}$ . Leveraging these exponential decays to define exponentially increasing weights is crucial for deriving our regret bounds that are stronger than existing results.
Construct the functions  $f_{t} : \mathcal{H} \to \mathbb{R}$  that correspond to  $c_{t}(s_{t}, u_{t})$  as follows. Given a sequence of decisions  $(M_{0}, \ldots, M_{t})$ , the history at the end of round  $t$  is given by  $h_{t} = (M_{t}, GM_{t-1}, \widetilde{F}GM_{t-2}, \ldots, \widetilde{F}^{t-1}GM_{0}, 0, \ldots)$ . A simple inductive argument shows that the state and control in round  $t$  can be written as functions of  $h_{t}$  as  $s_{t} = \widetilde{F}^{t}s_{0} + \sum_{k=0}^{t-1}\sum_{s=1}^{k+1}\widetilde{F}^{t-k-1}GM_{k}^{[s]}w_{k-s} + w_{t-1}$  and  $u_{t} = -Ks_{t} + \sum_{s=1}^{t+1}M_{t}^{[s]}w_{t-s}$ . Define the functions  $f_{t} : \mathcal{H} \to \mathbb{R}$  by  $f_{t}(h) = c_{t}(s,u)$ , where  $s$  and  $u$  are the state and control determined by the history as above. Note that  $f_{t}$  is parameterized by the past disturbances. Since the state and control are linear functions of the history and  $c_{t}$  is convex, this implies that  $f_{t}$  is convex. Now, given the above formulation and the fact that the class of disturbance-action controllers is a superset of the class of  $(\kappa, \rho)$ -strongly-stable linear controllers, we have that the policy regret for the online linear control problem is at most the policy regret,  $\sum_{t=0}^{T-1}f_{t}(h_{t}) - \min_{M\in\mathcal{X}}\sum_{t=0}^{T-1}\tilde{f}_{t}(M)$ . The following is our main result for online linear control and it improves existing results [Agarwal et al., 2019b] by a factor of  $O(d\log(T)^{3.5}\kappa^{5}(1-\rho)^{-1})$ . See Appendix E.3 for a detailed comparison.
Theorem 4.1. Consider the online linear control problem as defined in Section 4.1. Suppose the decisions in round  $t$  are chosen using Algorithm 1. Then, the upper bound on the policy regret is
$$
O \left(L _ {0} W ^ {2} \sqrt {T} d ^ {\frac {1}{2}} \kappa^ {1 7} (1 - \rho) ^ {- 4. 5}\right). \tag {2}
$$
Comparison with prior and concurrent work. Existing works solve OLC (and its extensions) by making multiple finite memory approximations. First, they formulate the problem as OCO with finite memory. This requires bounding numerous error terms because the problem is inherently an OCO with unbounded memory problem. We bypass these error analysis steps entirely because the problem fits into our framework naturally. Second, existing works use the parameterization from Agarwal et al. [2019b] that only acts on a fixed, constant number of past disturbances. In particular, existing works use a "truncated" DAC policy that is a sequence of  $d \times d$  matrices of length  $2\kappa^4 (1 - \rho)^{-1}\log T$ . Our DAC policy acts on the entire history of disturbances and is a sequence of  $d \times d$  matrices of unbounded length. Yet, we capture the dimension of this infinite-dimensional space in a way that still improves the overall bound, including completely eliminating the dependence on  $\log T$ , and improving the dependence on  $d,\kappa$ , and  $(1 - \rho)$ . This improvement comes from our novel use of weighted norms on the history and decision spaces. These norms allow us to give tighter bounds on the relevant quantities in the regret upper bound, e.g.,  $\| A^{k}\|$  (Lemma E.2) and  $\tilde{L}$  (Lemma E.6).
In complementary concurrent work, Lin et al. [2022] focus on a more general online control problem. They improve regret bounds for this general version by a factor of  $\log T$  compared to existing reductions to OCO with finite memory. They do so by using that the impact of a past policy decays geometrically with time. On the other hand, the primary focus of our work is studying the complete dependence of present losses on the entire history in OCO. Applying our resulting OCO with unbounded memory framework to OLC, we improve upon existing results for OLC by removing all  $\log T$  factors and improving the dependence on  $d, \kappa$ , and  $(1 - \rho)$ .
4.2 Online Performative Prediction
Background. In many applications of machine learning the algorithm's decisions influence the data distribution, e.g., online labor markets [Anagnostopoulos et al., 2018, Horton, 2010], predictive policing [Lum and Isaac, 2016], on-street parking [Dowling et al., 2020, Pierce and Shoup, 2018], vehicle sharing markets [Banerjee et al., 2015], etc. Motivated by such applications, several works have studied the problem of performative prediction, which models the data distribution as a function of the decision-maker's decision [Perdomo et al., 2020, Mendler-Dünner et al., 2020, Miller et al., 2021, Brown et al., 2022, Ray et al., 2022, Jagadeesan et al., 2022]. Most of these works view the problem as a stochastic optimization problem; Jagadeesan et al. [2022] adopt a regret minimization perspective. We refer the reader to these citations for more details. As a natural extension to existing works, we introduce an online learning variant of performative prediction with geometric decay [Ray et al., 2022] that differs from the original formulations in a few key ways.
Let the decision set  $\mathcal{X} \subseteq \mathbb{R}^d$  be closed and convex with  $\|x\|_2 \leq D_{\mathcal{X}}$ . Let  $p_1$  denote the initial data distribution over the instance space  $\mathcal{Z}$ . In each round  $t \in [T]$ , the learner chooses a decision  $x_t \in \mathcal{X}$  and an oblivious adversary chooses a loss function  $l_t: \mathcal{X} \times \mathcal{Z} \to [0,1]$ , and then the learner suffers the loss  $L_t(x_t) = \mathbb{E}_{z \sim p_t}[l_t(x_t,z)]$ , where  $p_t = p_t(x_1, \ldots, x_t)$  is the data distribution in round  $t$ . The goal in our online learning setting is to minimize the difference between the algorithm's total loss and the total loss of the best fixed decision,  $\sum_{t=1}^{T} \mathbb{E}_{z \sim p_t}[l_t(x_t,z)] - \min_{x \in \mathcal{X}} \sum_{t=1}^{T} \mathbb{E}_{z \sim p_t(x)}[l_t(x,z)]$ , where  $p_t(x) = p_t(x, \ldots, x)$  is the data distribution in round  $t$  had  $x$  been chosen in all rounds so far. This measure is similar to performative regret [Jagadeesan et al., 2022] and is a natural generalization of performative optimality [Perdomo et al., 2020] for an online learning formulation.
We make the following assumptions. First, the loss functions  $l_{t}$  are convex and  $L_0$ -Lipschitz continuous. Second, the data distribution satisfies for all  $t \geq 1$ ,  $p_{t+1} = \rho p_t + (1 - \rho) \mathcal{D}(x_t)$ , where  $\rho \in (0,1)$  and  $\mathcal{D}(x_t)$  is a distribution over  $\mathcal{Z}$  that depends on the decision  $x_t$  [Ray et al., 2022]. Third,  $\mathcal{D}(x)$  is a location-scale distribution:  $z \sim \mathcal{D}(x)$  iff  $z \sim \xi + Fx$ , where  $F \in \mathbb{R}^{d \times d}$  satisfies  $\|F\|_2 < \infty$  and  $\xi$  is a random variable with mean  $\mu$  and covariance  $\Sigma$  [Ray et al., 2022].
Our problem formulation differs from existing work in the following ways. First, we adopt an online learning perspective on performative prediction with geometric decay, whereas Ray et al. [2022] adopt a stochastic optimization one. So, we assume that the loss functions  $l_{t}$  are adversarially chosen, whereas Ray et al. [2022] assume  $l_{t} = l$  are fixed. Second, we assume that the dynamics  $(\mathcal{D}$  and  $\rho$ ) are known (Assumption A1), whereas Ray et al. [2022] assume they are unknown and use samples from the data distribution. We believe that an appropriate extension of our framework that can deal with unknown linear operators  $A$  and  $B$  can be applied to this more difficult setting, and we leave this as future work. Third, even though Jagadeesan et al. [2022] also study an online learning variant
of performative prediction, they assume  $l_{t} = l$  are fixed and the data distribution depends only on the current decisions, whereas we assume the data distribution depends on the entire history of decisions.
Formulation as OCO with Unbounded Memory. Let  $\rho \in (0,1)$ . Let the decision space  $\mathcal{X} \subseteq \mathbb{R}^d$  be closed and convex with the 2-norm. Let the history space  $\mathcal{H}$  be the  $\ell^1$ -direct sum of countably infinite number of copies of  $\mathcal{X}$ . Define the linear operators  $A: \mathcal{H} \to \mathcal{H}$  and  $B: \mathcal{W} \to \mathcal{H}$  as  $A((y_0, y_1, \ldots)) = (0, \rho y_0, \rho y_1, \ldots)$  and  $B(x) = (x, 0, \ldots)$ . The problem is an OCO with  $\rho$ -discounted infinite memory problem and follows linear sequence dynamics with the 1-norm (Definition 2.3). Given a sequence of decisions  $(x_k)_{k=1}^t$ , the history is  $h_t = (x_t, \rho x_{t-1}, \ldots, \rho^{t-1}x_1, 0, \ldots)$  and the data distribution  $p_t = p_t(h_t)$  satisfies:  $z \sim p_t$  iff  $z \sim \sum_{k=1}^{t-1}(1 - \rho)\rho^{k-1}(\xi + Fx_{t-k}) + \rho^t p_1$ . This follows from the recursive definition of  $p_t$  and parametric assumption about  $\mathcal{D}(x)$ . Define the functions  $f_t: \mathcal{H} \to [0,1]$  by  $f_t(h_t) = \mathbb{E}_{z \sim p_t}[l_t(x_t, z)]$ . Now, the original goal of minimizing the difference between the algorithm's total loss and the total loss of the best fixed decision is equivalent to minimizing the policy regret. The following is our main result for online performative prediction.
Theorem 4.2. Consider the online performative prediction problem as defined in Section 4.2. Suppose the decisions in round  $t$  are chosen using Algorithm 1. Then, the upper bound on the policy regret is
$$
O \left(D _ {\mathcal {X}} L _ {0} \sqrt {T} \| F \| _ {2} (1 - \rho) ^ {- \frac {1}{2}} \rho^ {- 1}\right).
$$
In this paper we introduced a generalization of the OCO framework, "Online Convex Optimization with Unbounded Memory", that allows the loss in the current round to depend on the entire history of decisions until that point. We proved matching upper and lower bounds on the policy regret in terms of the time horizon, the  $p$ -effective memory capacity (a quantitative measure of the influence of past decisions on present losses), and other problem parameters (Theorems 3.1 and 3.2). As a special case, we proved the first non-trivial lower bound for OCO with finite memory (Theorem 3.4), which could be of independent interest, and also improved existing upper bounds (Theorem 3.3). We illustrated the power of our framework by bringing together the regret analysis of two seemingly disparate problems under the same umbrella: online linear control (Theorem 4.1), where we improve and simplify existing regret bounds, and online performative prediction (Theorem 4.2).
There are a number of directions for future research. A natural follow-up is to consider unknown dynamics (i.e., when the learner does not know the operators  $A$  and  $B$ ) and/or the case of bandit feedback (i.e., when the learner only observes  $f_{t}(h_{t})$ ). The extension to bandit feedback has been considered in the OCO and OCO with finite memory literature [Hazan and Li, 2016, Bubeck et al., 2021, Zhao et al., 2021, Gradu et al., 2020, Cassel and Koren, 2020]. It is tempting to think about a version where the history is a nonlinear, but decaying, function of the past decisions. The obvious challenge is that the nonlinearity would lead to non-convex losses. It is unclear how to deal with such issues, e.g., restricted classes of nonlinearities for which the OCO with unbounded memory perspective is still relevant [Zhang et al., 2015], different problem formulations such as online non-convex learning [Gao et al., 2018, Suggala and Netrapalli, 2020], etc.
There is a growing body of work on online linear control and its variants that rely on OCO with finite memory [Hazan et al., 2020, Agarwal et al., 2019c, Foster and Simchowitz, 2020, Cassel and Koren, 2020, Gradu et al., 2020, Li et al., 2021, Minasyan et al., 2021]. In this paper we showed how our framework can be used to improve and simplify regret bounds for the online linear control problem. Another direction for future work is to use our framework, perhaps with suitable extensions outlined above, to derive similar improvements for these other variants of online linear control.
We thank Sloan Nietert and Victor Sanches Portella for helpful discussions. We thank Wei-Yu Chan for pointing out an error in the proof of Lemma C.1. We also thank anonymous AISTATS and NeurIPS reviewers for their helpful comments on an older and the current version of the paper respectively. This research was partially supported by the NSERC Postgraduate Scholarships-Doctoral Fellowship 545847-2020, NSF awards CCF-2312774 and OAC-2311521, and a gift from Wayfair.
- Appendix A contains the proofs of the results from Section 2.
- Appendix B contains the proofs of existing results about FTRL for completeness.
- Appendix C contains the proofs of upper bounds on regret from Section 3.
- Appendix D contains the proofs of lower bounds on regret from Section 3.
- Appendix E contains the proofs of results about online linear control from Section 4.1.
- Appendix F contains the proofs of results about online performative prediction from Section 4.2.
- Appendix G discusses how to implement Algorithm 1 efficiently.
- Appendix H presents an algorithm for OCO with unbounded memory that provides the same upper bound on policy regret as Algorithm 1 while guaranteeing a small number of switches (Algorithm 2).
- Appendix I presents simulation experiments.
In this section prove Theorem 2.1. But first we prove a lemma that we use for proofs involving linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm (Definition 2.3). Recall that  $\| \cdot \|_{\mathcal{U}}$  denotes the norm associated with a space  $\mathcal{U}$  and the operator norm  $\| L\|$  for a linear operator  $L:\mathcal{U}\to \mathcal{V}$  is defined as  $\| L\| = \max_{u:\| u\|_{\mathcal{U}}\leq 1}\| Lu\|_{\mathcal{V}}$ .