SlowGuess/NeurIPS_2023_OCR huggingface dataset:
RaNAK Kumar
Ithaca, NY 14853
raunak@cs.cornell.edu
Sarah Dean
sdean@cornell.edu
Robert Kleinberg
rdk@cs.cornell.edu
Online convex optimization (OCO) is a widely used framework in online learning. In each round, the learner chooses a decision in a convex set and an adversary chooses a convex loss function, and then the learner suffers the loss associated with their current decision. However, in many applications the learner's loss depends not only on the current decision but on the entire history of decisions until that point. The OCO framework and its existing generalizations do not capture this, and they can only be applied to many settings of interest after a long series of approximation arguments. They also leave open the question of whether the dependence on memory is tight because there are no non-trivial lower bounds. In this work we introduce a generalization of the OCO framework, "Online Convex Optimization with Unbounded Memory", that captures long-term dependence on past decisions. We introduce the notion of  $p$ -effective memory capacity,  $H_{p}$ , that quantifies the maximum influence of past decisions on present losses. We prove an  $O(\sqrt{H_pT})$  upper bound on the policy regret and a matching (worst-case) lower bound. As a special case, we prove the first non-trivial lower bound for OCO with finite memory [Anava et al., 2015], which could be of independent interest, and also improve existing upper bounds. We demonstrate the broad applicability of our framework by using it to derive regret bounds, and to improve and simplify existing regret bound derivations, for a variety of online learning problems including online linear control and an online variant of performative prediction.
Numerous applications are characterized by multiple rounds of sequential interactions with an environment, e.g., prediction from expert advice [Littlestone and Warmuth, 1989, 1994], portfolio selection [Cover, 1991], routing [Awerbuch and Kleinberg, 2008], etc. One of the most popular frameworks for modelling such sequential decision-making problems is online convex optimization (OCO) [Zinkevich, 2003]. The OCO framework is as follows. In each round, the learner chooses a decision in a convex set and an adversary chooses a convex loss function, and then the learner suffers the loss associated with their current decision. The performance of an algorithm is measured by regret: the difference between the algorithm's total loss and that of the best fixed decision. We refer the reader to Shalev-Shwartz [2012], Hazan [2019], Orabona [2019] for surveys on this topic.
However, in many applications the loss of the learner depends not only on the current decisions but on the entire history of decisions until that point. For example, in online linear control [Agarwal et al., 2019b], in each round the learner chooses a "control policy" (i.e., decision), suffers a loss that is a function of the action taken by this policy and the current state of the system, and the system's state evolves according to linear dynamics. The current state depends on the entire history of actions and, therefore, the current loss depends not only on the current decision but the entire history of decisions. The OCO framework cannot capture such long-term dependence of the current loss on the past decisions and neither can existing generalizations that allow the loss to depend on a constant number of past decisions [Anava et al., 2015]. Although a series of approximation arguments can be used to apply finite memory generalizations of OCO to the online linear control problem, there is no OCO framework that captures the complete long-term dependence of current losses on past decisions. Furthermore, there are no non-trivial lower bounds for OCO in the memory setting, which leaves open the question whether the dependence on memory is tight.
Contributions. In this paper we introduce a generalization of the OCO framework, "Online Convex Optimization with Unbounded Memory" (Section 2), that allows the loss in the current round to depend on the entire history of decisions until that point. We introduce the notion of  $p$ -effective memory capacity,  $H_{p}$ , that quantifies the maximum influence of past decisions on present losses. We prove an  $O(\sqrt{H_pT})$  upper bound on the policy regret (Theorem 3.1) and a matching (worst-case) lower bound (Theorem 3.2). As a special case, we prove the first non-trivial lower bound for OCO with finite memory (Theorem 3.4), which could be of independent interest, and also improve existing upper bounds (Theorem 3.3). Our lower bound technique extends existing techniques developed for memoryless settings. We design novel adversarial loss functions that exploit the fact that an algorithm cannot overwrite its history. We illustrate the power of our framework by bringing together the regret analysis of two seemingly disparate problems under the same umbrella. First, we show how our framework improves and simplifies existing regret bounds for the online linear control problem [Agarwal et al., 2019b] in Theorem 4.1. Second, we show how our framework can be used to derive regret bounds for an online variant of performative prediction [Perdomo et al., 2020] in Theorem 4.2. This demonstrates the broad applicability of our framework for deriving regret bounds for a variety of online learning problems, particularly those that exhibit long-term dependence of current losses on past decisions.
Related work. The most closely related work to ours is the OCO with finite memory framework [Anava et al., 2015]. They consider a generalization of the OCO framework that allows the current loss to depend on a constant number of past decisions. There have been a number of follow-up works that extend the framework in a variety of other ways, such as non-stationarity [Zhao et al., 2022], incorporating switching costs [Shi et al., 2020], etc. However, none of these existing works go beyond a constant memory length and do not prove a non-trivial lower bound with a dependence on the memory length. In a different line of work, Bhatia and Sridharan [2020] consider a much more general online learning framework that goes beyond a constant memory length, but they only provide non-constructive upper bounds on regret. In contrast, our OCO with unbounded memory framework allows the current loss to depend on an unbounded number of past decisions, provides constructive upper bounds on regret, and lower bounds for a broad class of problems that includes OCO with finite memory with a general memory length  $m$ .
A different framework for sequential decision-making is multi-armed bandits [Bubeck and Cesà-Bianchi, 2012, Slivkins, 2019]. Qin et al. [2023] study a variant of contextual stochastic bandits where the current loss can depend on a sparse subset of all prior contexts. This setting differs from ours due to the feedback model, stochasticity, and decision space. Reinforcement learning [Sutton and Barto, 2018] is yet another popular framework for sequential decision-making that considers very general state-action models of feedback and dynamics. In reinforcement learning one typically measures regret with respect to the best state-action policy from some policy class, rather than the best fixed decision as in online learning and OCO. In the special case of linear control, policies can be reformulated as decisions while preserving convexity; we discuss this application in Section 4. Considering the general framework is an active area of research.
We defer discussion of related work for specific applications to Section 4.
The trivial lower bound refers to the  $\Omega (\sqrt{T})$  lower bound for OCO in the memoryless setting.
We begin with some motivation for the formalism used in our framework (Section 2.1). Many real-world applications involve controlling a physical dynamical system, for example, variable-speed wind turbines in wind energy electric power production [Boukhezzar and Siguerdidjane, 2010]. The typical solution for these problems has been to model them as offline control problems with linear time-invariant dynamics and use classical methods such as LQR and LQG [Boukhezzar and Siguerdidjane, 2010]. Instead of optimizing over the space of control inputs, the typical feedback control approach optimizes over the space of controllers, i.e., policies that choose a control input as a function of the system state. The standard controllers considered in the literature are linear controllers. Even when the losses are convex in the state and input, they are nonconvex in the linear controller. In the special case of quadratic losses in terms of the state and input, there is a closed-form solution for the optimal solution using the algebraic Riccati equations [Lancaster and Rodman, 1995]. But this does not hold for general convex losses resulting in convex reparameterizations such as Youla [Youla et al., 1976, Kucera, 1975] and SLS [Wang et al., 2019, Anderson et al., 2019]. The resulting parameterization represents an infinite dimensional system response and is characterized by a sequence of matrices. Recent work has studied an online approach for some of these control theory problems, where a sequence of controllers is chosen adaptively rather than choosing one offline [Abbasi-Yadtori and Szepesvári, 2011, Dean et al., 2018, Simchowitz and Foster, 2020, Agarwal et al., 2019b].
The takeaway from the above is that there are online learning problems in which (i) the current loss depends on the entire history of decisions; and (ii) the decision space can be more complicated than just a subset of  $\mathbb{R}^d$ , e.g., it can be an unbounded sequence of matrices. This motivates us to model the decision space as a Hilbert space and the history space as a Banach space in the formal problem setup below, and this subsumes the special cases of OCO and OCO with finite memory. This formalism not only lets us consider a wide range of spaces, such as  $\mathbb{R}^d$ , unbounded sequences of matrices, etc., but also lets us define appropriate norms on these spaces. This latter feature is crucial for deriving strong regret bounds for some applications such as online linear control. For this problem we derive improved regret bounds (Theorem 4.1) by defining weighted norms on the decision and history spaces, where the weights are chosen to leverage the problem structure.
Notation. We use  $\| \cdot \|_{\mathcal{U}}$  to denote the norm associated with a space  $\mathcal{U}$ . The operator norm for a linear  $L$  operator from space  $\mathcal{U} \to \mathcal{V}$  is defined as  $\| L \|_{\mathcal{U} \to \mathcal{V}} = \max_{u: \| u \|_{\mathcal{U}} \leq 1} \| Lu \|_{\mathcal{V}}$ . For convenience, sometimes we simply use  $\| \cdot \|$  when the meaning is clear from the context. For a finite-dimensional matrix we use  $\| \cdot \|_F$  and  $\| \cdot \|_2$  to denote its Frobenius norm and operator norm respectively.
Let the decision space  $\mathcal{X}$  be a closed and convex subset of a Hilbert space  $\mathcal{W}$  with norm  $\| \cdot \|_{\mathcal{X}}$  and the history space  $\mathcal{H}$  be a Banach space with norm  $\| \cdot \|_{\mathcal{H}}$ . Let  $A: \mathcal{H} \to \mathcal{H}$  and  $B: \mathcal{W} \to \mathcal{H}$  be linear operators. The game between the learner and an oblivious adversary proceeds as follows. Let  $T$  denote the time horizon and  $f_t: \mathcal{H} \to \mathbb{R}$  be loss functions chosen by the adversary. The initial history is  $h_0 = 0$ . In each round  $t \in [T]$ , the learner chooses  $x_t \in \mathcal{X}$ , the history is updated to  $h_t = Ah_{t-1} + Bx_t$ , and the learner suffers loss  $f_t(h_t)$ . An instance of an online convex optimization with unbounded memory problem is specified by the tuple  $(\mathcal{X}, \mathcal{H}, A, B)$ .
We use the notion of policy regret [Dekel et al., 2012] as the performance measure in our framework. The policy regret of a learner is the difference between its total loss and the total loss of a strategy that plays the best fixed decision in every round. The history after round  $t$  for a strategy that chooses  $x$  in every round is described by  $h_t = \sum_{k=0}^{t-1} A^k Bx$ , which motivates the following definition.
Definition 2.1. Given  $f_{t}:\mathcal{H}\to \mathbb{R}$ , the function  $\tilde{f}_t:\mathcal{X}\rightarrow \mathbb{R}$  is defined by  $\tilde{f}_t(x) = f_t(\sum_{k = 0}^{t - 1}A^k Bx)$ .
Definition 2.2 (Policy Regret). The policy regret of an algorithm  $\mathcal{A}$  is defined as  $R_{T}(\mathcal{A}) = \sum_{t=1}^{T} f_{t}(h_{t}) - \min_{x \in \mathcal{X}} \sum_{t=1}^{T} \tilde{f}_{t}(x)$ .
In many motivating examples such as online linear control (Section 4.1), the history at the end of a round is a sequence of linear transformations of past decisions. The following definition captures this formally and we leverage this structure to prove stronger regret bounds (Theorem 3.1).
Definition 2.3 (Linear Sequence Dynamics). Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . Let  $(\xi_k)_{k = 0}^{\infty}$  be a sequence of nonnegative real numbers satisfying  $\xi_0 = 1$ . We say that  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p\geq 1$  if
1.  $\mathcal{H}$  is the  $\xi$ -weighted  $\ell^p$ -direct sum of a finite or countably infinite number of copies of  $\mathcal{W}$ : every element  $y \in \mathcal{H}$  is a sequence  $y = (y_i)_{i \in \mathcal{I}}$ , where  $\mathcal{I} = \mathbb{N}$  or  $\mathcal{I} = \{0, \dots, n\}$  for some  $n \in \mathbb{N}$ , and  $\|y\|_{\mathcal{H}} = \left( \sum_{i \in \mathcal{I}} (\xi_i \| y_i \|)^p \right)^{1/p} < \infty$ .
2. We have  $A(y_0, y_1, \ldots) = (0, A_0y_0, A_1y_1, \ldots)$ , where  $A_i: \mathcal{W} \to \mathcal{W}$  are linear operators.
3. The operator  $B$  satisfies  $B(x) = (x,0,\ldots)$ .
Note that since the norm on  $\mathcal{H}$  depends on the weights  $\xi$ , the operator norm  $\|A^k\|$  also depends on  $\xi$ . If the weights are all equal to 1, then we simply say  $p$ -norm instead of  $\xi$ -weighted  $p$ -norm.
We make the following assumptions about the feedback model and the loss functions.
A1 The learner knows the operators  $A$  and  $B$ , and observes  $f_{t}$  at the end of each round  $t$ .
A2 The operator norm of  $B$  is at most 1, i.e.,  $\| B\| \leq 1$
A3 The functions  $f_{t}$  are convex.
A4 The functions  $f_{t}$  are  $L$ -Lipschitz continuous:  $\forall h, \tilde{h} \in \mathcal{H}$  and  $t \in [T]$ , we have  $|f_t(h) - f_t(\tilde{h})| \leq L \| h - \tilde{h} \|_{\mathcal{H}}$ .
Regarding Assumption A1, our results easily extend to the case where instead of observing  $f_{t}$ , the learner receives a gradient  $\nabla \tilde{f}_{t}(x_{t})$  from a gradient oracle, which can be implemented using knowledge of  $f_{t}$  and the dynamics  $A$  and  $B$ . Handling the cases when the operators  $A$  and  $B$  are unknown and/or the learner observes bandit feedback (i.e., only  $f_{t}(h_{t})$ ) are important problems and we leave them as future work. Note that our assumption that  $A$  and  $B$  are known is no more restrictive than in the existing literature on OCO with finite memory [Anava et al., 2015] where it is assumed that the learner knows the constant memory length. In fact, our assumption is more general because our framework not only captures constant memory length as a special case but allows for richer dynamics as we illustrate in Section 4. Assumption A2 is made for convenience, and it amounts to a rescaling of the problem. Assumption A3 can be replaced by the weaker assumption that  $\tilde{f}_{t}$  are convex (similar to the literature on OCO with finite memory [Anava et al., 2015]) and this is what we use in the rest of the paper.
Assumptions A1 and A4 imply that  $\tilde{f}_t$  are  $\tilde{L}$ -Lipschitz continuous for the following  $\tilde{L}$ .
Theorem 2.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . If  $f_{t}$  is  $L$ -Lipschitz continuous, then  $\tilde{f}_t$  is  $\tilde{L}$ -Lipschitz continuous for  $\tilde{L} \leq L\sum_{k=0}^{\infty}\|A^k\|$ . If  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p \geq 1$ , then  $\tilde{L} \leq L\left(\sum_{k=0}^{\infty}\|A^k\|^p\right)^{\frac{1}{p}}$ .
The proof follows from the definitions of  $\tilde{f}_t$  and  $\| \cdot \|_{\mathcal{H}}$ , and we defer it to Appendix A. The above bound is tighter than similar results in the literature on OCO with finite memory and online linear control. This theorem is a key ingredient, amongst others, in improving existing upper bounds on regret for OCO with finite memory (Theorem 3.3) and for online linear control (Theorem 4.1). Before presenting our final assumption we introduce the notion of  $p$ -effective memory capacity that quantifies the maximum influence of past decisions on present losses.
Definition 2.4 ( $p$ -Effective Memory Capacity). Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . For  $p\geq 1$ , the  $p$ -effective memory capacity is defined as
$$
H _ {p} (\mathcal {X}, \mathcal {H}, A, B) = \left(\sum_ {k = 0} ^ {\infty} k ^ {p} \| A ^ {k} \| ^ {p}\right) ^ {\frac {1}{p}}. \tag {1}
$$
When the meaning is clear from the context we simply use  $H_{p}$  instead. The  $p$ -effective memory capacity is an upper bound on the difference in histories for two sequences of decisions whose difference grows at most linearly with time. To see this, consider two sequences of decisions,  $(x_{k})$  and  $(\tilde{x}_{k})$ , whose elements differ by no more than  $k$  at time  $k$ :  $\| x_{k} - \tilde{x}_{k} \| \leq k$ . Then the histories generated by the two sequences have difference between bounded as  $\| h - \bar{h} \| = \| \sum_{k} A^{k} B(x_{k} - \tilde{x}_{k}) \| \leq \sum_{k} k \| A^{k} B \| \leq \sum_{k} k \| A^{k} \| = H_{1}$ , where the last inequality follows from Assumption A2. A similar bound holds with  $H_{p}$  instead when  $(\mathcal{X}, \mathcal{H}, A, B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm.
A5 The 1-effective memory capacity is finite, i.e.,  $H_{1} < \infty$ .
Since  $H_{p}$  is decreasing in  $p$ ,  $H_{1} < \infty$  implies  $H_{p} < \infty$  for all  $p \geq 1$ . For the case of linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm it suffices to make the weaker assumption that  $H_{p} < \infty$ . However, for simplicity of exposition, we assume that  $H_{1} < \infty$ .
2.3 Special Cases
OCO with Finite Memory. Consider the OCO with finite memory problem with constant memory length  $m$ . It can be specified in our framework by  $(\mathcal{X},\mathcal{H},A_{\mathrm{finite},m},B_{\mathrm{finite},m})$ , where  $\mathcal{H}$  is the  $\ell^2$ -direct sum of  $m$  copies of  $\mathcal{X}$ ,  $A_{\mathrm{finite},m}(x^{[m]},\ldots,x^{[1]}) = (0,x^{[m]},\ldots,x^{[2]}$ , and  $B_{\mathrm{finite},m}(x) = (x,0,\ldots,0)$ . Note that  $(\mathcal{X},\mathcal{H},A_{\mathrm{finite},m},B_{\mathrm{finite},m})$  follows linear sequence dynamics with the 2-norm. Our framework can even model an extension where the problem follows linear sequence dynamics with the  $p$ -norm for  $p \geq 1$  by simply defining  $\mathcal{H}$  to be the  $\ell^p$ -direct sum of  $m$  copies of  $\mathcal{X}$ .
OCO with  $\rho$ -discounted Infinite Memory. Our framework can also model OCO with infinite memory problems that are not modelled by existing OCO frameworks. Let  $\rho \in (0,1)$  be the discount factor and  $p \geq 1$ . An OCO with  $\rho$ -discounted infinite memory problem is specified by  $(\mathcal{X}, \mathcal{H}, A_{\mathrm{infinite},\rho}, B_{\mathrm{infinite},\rho})$ , where  $\mathcal{H}$  is the  $\ell^p$ -direct sum of countably many copies of  $\mathcal{X}$ ,  $A_{\mathrm{infinite},\rho}((y_0, y_1, \ldots)) = (0, \rho y_0, \rho y_1, \ldots)$ , and  $B_{\mathrm{infinite},\rho}(x) = (x, 0, \ldots)$ . Note that  $(\mathcal{X}, \mathcal{H}, A_{\mathrm{infinite},\rho}, B_{\mathrm{infinite},\rho})$  follows linear sequence dynamics with the  $p$ -norm. Due to space constraints we defer proofs of regret bounds for this problem to the appendix.
3 Regret Analysis
We present two algorithms for choosing the decisions  $x_{t}$ . Algorithm 1 uses follow-the-regularized-leader (FTRL) [Shalev-Shwartz and Singer, 2006, Abernethy et al., 2008] on the loss functions  $\tilde{f}_{t}$ . Due to space constraints, we discuss how to implement it efficiently in Appendix G and present simple simulation experiments in Appendix I. Algorithm 2, which we only present in Appendix H, combines FTRL with a mini-batching approach [Dekel et al., 2012, Altschuler and Talwar, 2018, Chen et al., 2020] to additionally guarantee that the decisions switch at most  $O\left(T\tilde{L}/LH_{1}\right)$  times. We defer the proofs of the following upper and lower bounds to Appendices C and D respectively.
Algorithm 1: FTRL
Input : Time horizon  $T$  , step size  $\eta$  ,  $\alpha$  -strongly-convex regularizer  $R:\mathcal{X}\to \mathbb{R}$
1 Initialize history  $h_0 = 0$
2 for  $t = 1,2,\ldots ,T$  do
3 Learner chooses  $x_{t}\in \arg \min_{x\in \mathcal{X}}\sum_{s = 1}^{t - 1}\tilde{f}_{s}(x) + \frac{R(x)}{\eta}.$
4 Set  $h_t = Ah_{t-1} + Bx_t$ .
5 Learner suffers loss  $f_{t}(h_{t})$  and observes  $f_{t}$ .
6 end
Theorem 3.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 1 with step-size  $\eta$  satisfies  $R_{T}(FTRL)\leq \frac{D}{\eta} +\eta \frac{T\tilde{L}^{2}}{\alpha} +\eta \frac{TTLH_{1}}{\alpha}$ . If
$$
\eta = \sqrt {\frac {\alpha D}{T \tilde {L} (L H _ {1} + \tilde {L})}}, t h e n
$$
$$
R _ {T} (F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} H _ {1}}\right).
$$
When  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm, then all of the above hold with  $H_{p}$  instead of  $H_{1}$ .
The proof of this theorem involves writing the regret as  $\sum_{t} f_{t}(h_{t}) - \tilde{f}_{t}(x_{t}) + \sum_{t} \tilde{f}_{t}(x_{t}) - \tilde{f}_{t}(x^{*})$ , and bounding the first term using the  $p$ -effective memory capacity and the second term using FTRL. We defer the full proof to Appendix C. The following lower bound shows that this is tight in the worst-case.
Theorem 3.2. There exists an instance of the online convex optimization with unbounded memory problem,  $(\mathcal{X},\mathcal{H},A,B)$ , that follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm and there exist  $L$ -Lipschitz continuous loss functions  $\{f_t:\mathcal{H}\to \mathbb{R}\}_{t = 1}^T$  such that the regret of any algorithm  $\mathcal{A}$  satisfies
$$
R _ {T} (\mathcal {A}) \geq \Omega \left(\sqrt {T L \tilde {L} H _ {p}}\right).
$$
The proof of this theorem follows from our lower bound for the special case of OCO with finite memory (Theorem 3.4), which we discuss in detail below. However, as we show in Appendix D, the lower bound holds for a much broader class of problems.
Specialization to OCO with finite memory. Now we show how our bounds specialize to the special case of OCO with finite memory (Section 2.3). Due to space constraints, we defer the specialization of the upper and lower bounds for OCO with  $\rho$ -discounted infinite memory (Section 2.3) to Appendix C and Appendix D respectively.
Theorem 3.3. Consider an online convex optimization with finite memory problem with constant memory length  $m$  specified by  $(\mathcal{X},\mathcal{H} = \mathcal{X}^m,A_{\mathrm{finite},m},B_{\mathrm{finite},m})$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 1 with step-size  $\eta = \sqrt{\frac{\alpha D}{T\tilde{L}(Lm^{\frac{3}{2}} + \tilde{L})}}$  satisfies
$$
R _ {T} (F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} m ^ {\frac {3}{2}}}\right) \leq O \left(m \sqrt {\frac {D}{\alpha} T L ^ {2}}\right).
$$