SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma A.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . If  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p\geq 1$ , then for all  $k\geq 1$
$$
\xi_ {k} \left\| A _ {k - 1} \cdot \cdot \cdot A _ {0} \right\| \leq \left\| A ^ {k} \right\|.
$$
Proof. Let  $x \in \mathcal{X}$  with  $\| x\|_{\mathcal{X}} = 1$ . We have
$$
\xi_ {k} \| A _ {k - 1} \dots A _ {0} x \| _ {\mathcal {X}} = \| A ^ {k} (x, 0, \dots) \| _ {\mathcal {H}} \leq \| A ^ {k} \| \| (x, 0, \dots) \| _ {\mathcal {H}} \leq \| A ^ {k} \|,
$$
where the last inequality follows because  $\| (x,0,\ldots)\|_{\mathcal{H}} = \xi_0\| x\|_{\mathcal{X}}$  and  $\xi_0 = 1$  by Definition 2.3.
Therefore,  $\| A_{k - 1}\cdot \cdot \cdot A_0\| \leq \| A^k\|$
Theorem 2.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . If  $f_{t}$  is L-Lipschitz continuous, then  $\tilde{f}_t$  is  $\tilde{L}$ -Lipschitz continuous for  $\tilde{L} \leq L\sum_{k=0}^{\infty}\|A^{k}\|$ . If  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p \geq 1$ , then  $\tilde{L} \leq L\left(\sum_{k=0}^{\infty}\|A^{k}\|^{p}\right)^{\frac{1}{p}}$ .
Proof. Let  $x, \tilde{x} \in \mathcal{X}$ . For the general case, we have
$$
\begin{array}{l} \left| \tilde {f} _ {t} (x) - \tilde {f} _ {t} (\tilde {x}) \right| = \left| f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B x\right) - f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B \tilde {x}\right) \right| \quad \text {b y} \\ \leq L \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B (x - \tilde {x}) \right\| _ {\mathcal {H}} \quad f _ {t} \text {i s} L \text {- L i p s c h i t z c o n t i n u o u s} \\ \leq L \sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| \| B \| \| x - \tilde {x} \| _ {\mathcal {X}} \\ \leq L \sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| \| x - \tilde {x} \| _ {\mathcal {X}} \quad \text {b y} \\ \leq L \sum_ {k = 0} ^ {\infty} \| A ^ {k} \| \| x - \tilde {x} \| _ {\mathcal {X}}. \\ \end{array}
$$
If  $(\mathcal{H},\mathcal{X},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p\geq 1$ , then we have
$$
\begin{array}{l} \left| \tilde {f} _ {t} (x) - \tilde {f} _ {t} (\tilde {x}) \right| = \left| f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B x\right) - f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B \tilde {x}\right) \right| \quad \text {b y} \\ \leq L \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B (x - \tilde {x}) \right\| _ {\mathcal {H}} \quad f _ {t} \text {i s} L \text {- L i p s c h i t z c o n t i n u o u s} \\ = L \left\| \left(0, A _ {0} (x - \tilde {x}), A _ {1} A _ {0} (x - \tilde {x}), \dots\right) \right\| \quad \text {b y} \\ = L \left(\sum_ {k = 0} ^ {t - 1} \xi_ {k} ^ {p} \| A _ {k - 1} \dots A _ {0} (x - \tilde {x}) \| ^ {p}\right) ^ {\frac {1}{p}} \quad \text {b y} \\ \leq L \left(\sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| ^ {p}\right) ^ {\frac {1}{p}} \| x - \tilde {x} \| _ {\mathcal {X}} \quad \text {b y} \\ \leq L \left(\sum_ {k = 0} ^ {\infty} \| A ^ {k} \| ^ {p}\right) ^ {\frac {1}{p}} \| x - \tilde {x} \| _ {\mathcal {X}}. \\ \end{array}
$$
B Standard Analysis of Follow-the-Regularized-Leader
In this section we state and prove some existing results about the follow-the-regularized-leader (FTRL) algorithm [Shalev-Shwartz and Singer, 2006, Abernethy et al., 2008]. These results are well known in the literature, but we prove them here for completeness and use them in the remainder of the paper. We use the below results for functions  $\tilde{f}_t$  with Lipschitz constants  $\tilde{L}$ . However, in this section we use a more general notation, denoting functions by  $g_t$  and their Lipschitz constant by  $L_g$ .
Consider the following setup for an online convex optimization (OCO) problem. Let  $T$  denote the time horizon. Let the decision space  $\mathcal{X}$  be a closed, convex subset of a Hilbert space and  $g_t: \mathcal{X} \to \mathbb{R}$  be loss functions chosen by an oblivious adversary. The functions  $g_t$  are convex and  $L_g$ -Lipschitz continuous. The game between the learner and the adversary proceeds as follows. In each round  $t \in [T]$ , the learner chooses  $x_t \in \mathcal{X}$  and the learner suffers loss  $g_t(x_t)$ . The goal of the learner is to minimize (static) regret,
$$
R _ {T} ^ {\text {s t a t i c}} = \sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - \min  _ {x \in \mathcal {X}} \sum_ {t = 1} ^ {T} g _ {t} (x). \tag {3}
$$
Let  $R:\mathcal{X}\to \mathbb{R}$  be an  $\alpha$ -strongly convex regularizer satisfying  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . The FTRL algorithm chooses iterates  $x_{t}$  as
$$
x _ {t} \in \underset {x \in \mathcal {X}} {\arg \min } \sum_ {s = 1} ^ {t - 1} g _ {s} (x) + \frac {R (x)}{\eta}, \tag {4}
$$
where  $\eta$  is a tunable parameter referred to as the step-size. In what follows, let  $g_0 = \frac{R}{\eta}$ . The analysis in this section closely follows Karlin [2017].
Lemma B.1. For all  $x\in \mathcal{X}$  FTRL (Eq.(4)) satisfies
$$
\sum_ {t = 0} ^ {T} g _ {t} (x) \geq \sum_ {t = 0} ^ {T} g _ {t} \left(x _ {t + 1}\right).
$$
Proof. We use proof by induction on  $T$ . The base case is  $T = 0$ . By definition,  $x_{1} \in \arg \min_{x \in \mathcal{X}} R(x)$ . Therefore,  $R(x) \geq R(x_{1})$  for all  $x \in \mathcal{X}$ . Recalling the notation  $g_{0} = \frac{R}{\eta}$  proves the base case. Now, assume that the lemma is true for  $T - 1$ . That is,
$$
\sum_ {t = 0} ^ {T - 1} g _ {t} (x) \geq \sum_ {t = 0} ^ {T - 1} g _ {t} \left(x _ {t + 1}\right).
$$
Let  $x \in \mathcal{X}$  be arbitrary. Since  $x_{T + 1} \in \arg \min_{x \in \mathcal{X}} \sum_{t = 0}^{T} g_t(x)$ , we have
$$
\begin{array}{l} \sum_ {t = 0} ^ {T} g _ {t} (x) \geq \sum_ {t = 0} ^ {T} g _ {t} \left(x _ {T + 1}\right) \\ = \sum_ {t = 0} ^ {T - 1} g _ {t} \left(x _ {T + 1}\right) + g _ {T} \left(x _ {T + 1}\right) \\ \geq \sum_ {t = 0} ^ {T - 1} g _ {t} \left(x _ {t + 1}\right) + g _ {T} \left(x _ {T + 1}\right) \quad \text {b y} \\ = \sum_ {t = 0} ^ {T} g _ {t} (x _ {t + 1}). \\ \end{array}
$$
Lemma B.2. For all  $x\in \mathcal{X}$  FTRL (Eq.4) satisfies
$$
\sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - \sum_ {t = 1} ^ {T} g _ {t} (x) \leq \frac {D}{\eta} + \sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - g _ {t} \left(x _ {t + 1}\right).
$$
$$
\sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - \sum_ {t = 1} ^ {T} g _ {t} (x) \leq \sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - \sum_ {t = 1} ^ {T} g _ {t} (x) + g _ {0} (x) - g _ {0} \left(x _ {1}\right)
$$
because  $x_{1} \in \arg \min_{x \in \mathcal{X}} g_{0}(x)$ . The proof of this lemma now follows by using the above inequality, Lemma B.1, the definition  $g_{0} = \frac{R}{\eta}$ , and the definition of  $D$ .
Theorem B.1. FTRL (Eq. (4)) satisfies
$$
\left\| x _ {t + 1} - x _ {t} \right\| _ {\mathcal {X}} \leq \eta \frac {L _ {g}}{\alpha} \quad a n d \quad R _ {T} ^ {\text {s t a t i c}} \leq \frac {D}{\eta} + \eta \frac {T L _ {g} ^ {2}}{\alpha}.
$$
Choosing  $\eta = \sqrt{\frac{\alpha D}{TL_g^2}}$  yields
$$
R _ {T} ^ {\text {s t a t i c}} \leq O \left(\sqrt {\frac {D}{\alpha} T L _ {g} ^ {2}}\right).
$$
Proof. Let  $x^{*} \in \arg \min_{x \in \mathcal{X}} \sum_{t=1}^{T} g_{t}(x)$ . Using Lemma B.2 we have
$$
\sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - \sum_ {t = 1} ^ {T} g _ {t} \left(x ^ {*}\right) \leq \frac {D}{\eta} + \sum_ {t = 1} ^ {T} g _ {t} \left(x _ {t}\right) - g _ {t} \left(x _ {t + 1}\right). \tag {5}
$$
We can bound the summands in the sum above as follows. Define  $G_{t}(x) = \sum_{s=0}^{t-1} g_{s}(x)$ . Then,  $x_{t} \in \arg \min_{x \in \mathcal{X}} G_{t}(x)$  and  $x_{t+1} \in \arg \min_{x \in \mathcal{X}} G_{t+1}(x)$ . Since  $\{g_{s}\}_{s=1}^{T}$  are convex,  $R$  is  $\alpha$ -strongly-convex, and  $g_{0} = \frac{R}{\eta}$ , we have that  $G_{t}$  is  $\frac{\alpha}{\eta}$ -strongly-convex. So,
$$
G _ {t} \left(x _ {t + 1}\right) \geq G _ {t} \left(x _ {t}\right) + \frac {\alpha}{2 \eta} \left\| x _ {t + 1} - x _ {t} \right\| _ {\mathcal {X}} ^ {2},
$$
$$
G _ {t + 1} \left(x _ {t}\right) \geq G _ {t + 1} \left(x _ {t + 1}\right) + \frac {\alpha}{2 \eta} \left\| x _ {t + 1} - x _ {t} \right\| _ {\mathcal {X}} ^ {2}.
$$
Adding the above two inequalities yields
$$
g _ {t} \left(x _ {t}\right) - g _ {t} \left(x _ {t + 1}\right) \geq \frac {\alpha}{\eta} \left\| x _ {t + 1} - x _ {t} \right\| _ {\mathcal {X}} ^ {2}. \tag {6}
$$
Since  $g_{t}$  are convex and  $L_{g}$ -Lipschitz continuous, we also have
$$
g _ {t} \left(x _ {t}\right) - g _ {t} \left(x _ {t + 1}\right) \leq L _ {g} \| x _ {t + 1} - x _ {t} \| _ {\mathcal {X}}. \tag {7}
$$
Combining Eqs. (6) and (7) we have
$$
\left\| x _ {t + 1} - x _ {t} \right\| _ {\mathcal {X}} \leq \eta \frac {L _ {g}}{\alpha}.
$$
This proves the first part of the theorem. Now, using this in Eq. (7) we have
$$
g _ {t} \left(x _ {t}\right) - g _ {t} \left(x _ {t + 1}\right) \leq \eta \frac {L _ {g} ^ {2}}{\alpha}. \tag {8}
$$
Finally, substituting this in Eq. (5) proves the second part of the theorem.
C Regret Analysis: Upper Bounds
First we prove a lemma that bounds the difference in the value of  $f_{t}$  evaluated at the actual history  $h_t$  and an idealized history that would have been obtained by playing  $x_{t}$  in all prior rounds.
Lemma C.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . If the decisions  $(x_{t})$  are generated by Algorithm 1, then
$$
\left| f _ {t} \left(h _ {t}\right) - \tilde {f} _ {t} \left(x _ {t}\right) \right| \leq \eta \frac {L \tilde {L} H _ {1}}{\alpha}
$$
for all rounds  $t$ . When  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p \geq 1$ , then
$$
\left| f _ {t} \left(h _ {t}\right) - \tilde {f} _ {t} \left(x _ {t}\right) \right| \leq \eta \frac {L \tilde {L} H _ {p}}{\alpha}
$$
for all rounds  $t$ .
$$
\left| f _ {t} \left(h _ {t}\right) - \tilde {f} _ {t} \left(x _ {t}\right) \right| = \left| f _ {t} \left(h _ {t}\right) - f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t}\right) \right|
$$
by Definition 2.1
$$
\leq L \left\| h _ {t} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\|
$$
by Assumption A4
$$
= L \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\|
$$
by definition of  $h_t$
$$
= L \underbrace {\left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B \left(x _ {t - k} - x _ {t}\right) \right\|}.
$$
First consider the general case where  $(\mathcal{X},\mathcal{H},A,B)$  does not necessarily follow linear sequence dynamics. We can bound the term (a) as
$$
\begin{array}{l} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B (x _ {t - k} - x _ {t}) \right\| \leq \sum_ {k = 0} ^ {t - 1} \left\| A ^ {k} B \right\| \| x _ {t} - x _ {t - k} \| \\ \leq \sum_ {k = 0} ^ {t - 1} \| A ^ {k} B \| k \eta \frac {\tilde {L}}{\alpha} \\ \leq \sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| k \eta \frac {\tilde {L}}{\alpha} \\ \leq \eta \frac {\tilde {L}}{\alpha} H _ {1}. \\ \end{array}
$$
by Theorem B.1
by Assumption A2
Plugging this into Eq. (9) completes the proof for the general case. Now consider the case when  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm. We can bound the term (a) as
$$
\begin{array}{l} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B \left(x _ {t - k} - x _ {t}\right) \right\| = \| \left(0, A _ {0} \left(x _ {t} - x _ {t - 1}\right), A _ {1} A _ {0} \left(x _ {t} - x _ {t - 2}\right), \dots\right) \| \quad \text {b y} \\ = \left(\sum_ {k = 0} ^ {t - 1} \xi_ {k} ^ {p} \| A _ {k - 1} \dots A _ {0} \left(x _ {t} - x _ {t - k}\right) \| ^ {p}\right) ^ {\frac {1}{p}} \quad \text {b y} \\ \leq \left(\sum_ {k = 0} ^ {t - 1} \xi_ {k} ^ {p} \| A _ {k - 1} \dots A _ {0} \| ^ {p} \| x _ {t} - x _ {t - k} \| ^ {p}\right) ^ {\frac {1}{p}} \\ \leq \left(\sum_ {k = 0} ^ {t - 1} \left\| A ^ {k} \right\| ^ {p} \| x _ {t} - x _ {t - k} \| ^ {p}\right) ^ {\frac {1}{p}} \quad \text {b y} \\ \leq \eta \frac {\tilde {L}}{\alpha} \left(\sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| ^ {p} k ^ {p}\right) ^ {\frac {1}{p}} \quad \text {b y} \\ \leq \eta \frac {\tilde {L}}{\alpha} H _ {p}. \\ \end{array}
$$
Plugging this into Eq. (9) completes the proof.
Now we restate and prove Theorem 3.1
Theorem 3.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 1 with step-size  $\eta$  satisfies  $R_{T}(FTRL)\leq \frac{D}{\eta} +\eta \frac{T\tilde{L}^{2}}{\alpha} +\eta \frac{TLLH_{1}}{\alpha}$ . If  $\eta = \sqrt{\frac{\alpha D}{T\tilde{L}(LH_1 + \tilde{L})}}$ , then
Proof. First consider the general case where  $(\mathcal{X},\mathcal{H},A,B)$  does not necessarily follow linear sequence dynamics. Let  $x^{*}\in \arg \min_{x\in \mathcal{X}}\sum_{t = 1}^{T}\tilde{f}_{t}(x)$ . Note that we can write the regret as
$$
\begin{array}{l} R _ {T} (\text {F T R L}) = \sum_ {t = 1} ^ {T} f _ {t} \left(h _ {t}\right) - \min  _ {x \in \mathcal {X}} \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x) \\ = \underbrace {\sum_ {t = 1} ^ {T} f _ {t} (h _ {t}) - \tilde {f} _ {t} (x _ {t})} _ {(a)} + \underbrace {\sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x _ {t}) - \tilde {f} _ {t} (x ^ {*})} _ {(b)}. \\ \end{array}
$$
We can bound term (a) using Lemma C.1 and term (b) using Theorem B.1. Therefore, we have
$$
\begin{array}{l} R_{T}(\text{FTRL}) = \underbrace{\sum_{t = 1}^{T}f_{t}(h_{t}) - \tilde{f}_{t}(x_{t})}_{(a)} + \underbrace{\sum_{t = 1}^{T}\tilde{f}_{t}(x_{t}) - \tilde{f}_{t}(x^{*})}_{(b)} \\ \leq \eta \frac {T L \tilde {L} H _ {1}}{\alpha} + \frac {D}{\eta} + \eta \frac {T \tilde {L} ^ {2}}{\alpha}. \\ \end{array}
$$
Choosing  $\eta = \sqrt{\frac{\alpha D}{T\tilde{L}(LH_1 + \tilde{L})}}$  yields
$$
R _ {T} (\mathrm {F T R L}) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} H _ {1}}\right),
$$
where we used the definition of  $p$ -effective memory capacity (Definition 2.4) and the bound on  $\tilde{L}$  (Theorem 2.1) to simplify the above expression. This completes the proof for the general case. The proof for when  $(\mathcal{X},\mathcal{H},A,B)$  follows linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm is the same as above, except we bound the term (a) above using Lemma C.1 for linear sequence dynamics.
Now we restate and prove Theorem 3.3.
Theorem 3.3. Consider an online convex optimization with finite memory problem with constant memory length  $m$  specified by  $(\mathcal{X},\mathcal{H} = \mathcal{X}^m,A_{\text{finite},m},B_{\text{finite},m})$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 1 with step-size  $\eta = \sqrt{\frac{\alpha D}{T\tilde{L}(Lm^{\frac{3}{2}} + \tilde{L})}}$  satisfies
The OCO with finite memory problem, as defined in the literature, follows linear sequence dynamics with the 2-norm. In this subsection we consider a more general version of the OCO with finite memory problem that follows linear sequence dynamics with the  $p$ -norm. We provide an upper bound on the policy regret for this more general formulation and the proof of Theorem 3.3 follows as a special case when  $p = 2$ .
Theorem C.1. Consider an online convex optimization with finite memory problem with constant memory length  $m$ ,  $(\mathcal{X},\mathcal{H} = \mathcal{X}^m,A_{\mathrm{finite},m},B_{\mathrm{finite},m})$ . Assume that the problem follows linear sequence dynamics with the  $p$ -norm for  $p \geq 1$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})| \leq D$  for all  $x, \tilde{x} \in \mathcal{X}$ . Algorithm 1 with step-size  $\eta$  satisfies
$$
R _ {T} (F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} m ^ {\frac {p + 1}{p}}}\right) \leq O \left(\sqrt {\frac {D}{\alpha} T L ^ {2} m ^ {\frac {p + 2}{p}}}\right).
$$
Proof. Using Theorem 3.1 it suffices to bound  $\tilde{L}$  and  $H_{p}$  for this problem. Note that  $\| A_{\mathrm{finite}}^k\| = 1$  if  $k\leq m$  and 0 otherwise. Using this we have
$$
H _ {p} = \left(\sum_ {k = 0} ^ {\infty} \left(k \| A _ {\text {f i n i t e}} ^ {k} \|\right) ^ {p}\right) ^ {\frac {1}{p}} = \left(\sum_ {k = 0} ^ {m} k ^ {p}\right) ^ {\frac {1}{p}} \leq O \left(m ^ {\frac {p + 1}{p}}\right).
$$
This proves the first inequality in the statement of the theorem. The second inequality follows from the above and Theorem 2.1, which states that
$$
\tilde {L} \leq L \left(\sum_ {k = 0} ^ {\infty} \| A _ {\text {f i n i t e}} ^ {k} \| ^ {p}\right) ^ {\frac {1}{p}} = L m ^ {\frac {1}{p}}.
$$
Finally, we provide an upper bound on the policy regret for the OCO with  $\rho$ -discounted infinite memory problem. For simplicity, we consider the case when the problem follows linear sequence dynamics with the 2-norm instead of a general  $p$ -norm.
Theorem C.2. Consider an online convex optimization with  $\rho$ -discounted infinite memory problem  $(\mathcal{X},\mathcal{H},A_{\mathrm{infinite}},\rho,B_{\mathrm{infinite}})$ . Suppose that the problem follows linear sequence dynamics with the 2-norm. Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 1 with step-size  $\eta$  satisfies
$$
R _ {T} (F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} (1 - \rho^ {2}) ^ {- \frac {3}{2}}}\right) \leq O \left(\sqrt {\frac {D}{\alpha} T L ^ {2} (1 - \rho^ {2}) ^ {- 2}}\right) \leq O \left(\sqrt {\frac {D}{\alpha} T L ^ {2} (1 - \rho) ^ {- 2}}\right).
$$
Proof. Using Theorem 3.1, it suffices to bound  $\tilde{L}$  and  $H_{p}$  for this problem. Recall that  $\| A_{\mathrm{infinite},\rho}^{k}\| = \rho^{k}$ . Using this we have
$$
H _ {2} = \left(\sum_ {k = 0} ^ {\infty} \left(k \| A _ {\text {f i n i t e}} ^ {k} \|\right) ^ {2}\right) ^ {\frac {1}{2}} = \left(\sum_ {k = 0} ^ {\infty} \left(k \rho^ {k}\right) ^ {2}\right) ^ {\frac {1}{2}} \leq (1 - \rho^ {2}) ^ {- \frac {3}{2}}.
$$
$$
\tilde {L} \leq L \left(\sum_ {k = 0} ^ {\infty} \| A _ {\text {i n f i n i t e}, \rho} ^ {k} \| ^ {2}\right) ^ {\frac {1}{2}} = L (1 - \rho^ {2}) ^ {- \frac {1}{2}}.
$$
The last inequality follows because  $1 - \rho^2 = (1 + \rho)(1 - \rho)$ , which implies that  $1 - \rho \leq 1 - \rho^2 \leq 2(1 - \rho)$  because  $\rho \in (0,1)$ .
C.1 Existing Regret Bound for OCO with Finite Memory
In this subsection we provide a detailed comparison of our upper bound on the policy regret for OCO with finite memory with that of Anava et al. [2015]. The material in this subsection comes from Appendix A.2 of their arXiv version or Appendix C.2 of their conference version.
The existing upper bound on regret is
$$
O \left(\sqrt {D T \lambda m ^ {\frac {3}{2}}}\right),
$$
where  $D = \max_{x, \tilde{x} \in \mathcal{X}} |R(x) - R(\tilde{x})|$ . Although the parameter  $\lambda$  is defined in terms of dual norms of the gradient of  $\tilde{f}_t$ , it is essentially the Lipschitz-continuity constant for  $\tilde{f}_t$ : for all  $x, \tilde{x} \in \mathcal{X}$ ,
$$
\left| \tilde {f} _ {t} (x) - \tilde {f} _ {t} (\tilde {x}) \right| \leq \sqrt {\lambda \alpha} \| x - \tilde {x} \|,
$$
where  $\alpha$  is the strong-convexity parameter of the regularizer  $R$  (or  $\sigma$  in the notation of Anava et al. [2015]). Therefore, the existing regret bound can be rewritten as
$$
O \left(\tilde {L} \sqrt {\frac {D}{\alpha} T m ^ {\frac {3}{2}}}\right).
$$
Our upper bound on the policy regret for OCO with finite memory Theorem 3.3 is
$$
O \left(\sqrt {\frac {D}{\alpha} L \tilde {L} T m ^ {\frac {3}{2}}}\right).
$$
Since  $\tilde{L} \leq \sqrt{m} L$  by Theorem 2.1, this leads to an improvement by a factor of  $m^{\frac{1}{4}}$ .
D Regret Analysis: Lower Bounds
We first restate Theorems 3.2 and 3.4.
Figure 2: An illustration of the loss functions  $f_{t}$  for the OCO with finite memory lower bound. Suppose  $T = 12$ ,  $m = 3$ ,  $L = 1$ , and  $p = 2$ . Time is divided into blocks of size  $m = 3$ . Consider round  $t = 5$ . The history is  $h_{5} = (x_{3}, x_{4}, x_{5})$ . The loss function  $f_{5}(h_{5})$  is a product of three terms: a random sign  $\epsilon_{2}$  sampled for the block that round 5 belongs to, namely, block 2; a scaling factor of  $m^{-\frac{1}{2}}$ ; a sum over the decisions in the history excluding those that were chosen after observing  $\epsilon_{2}$ , i.e., a sum over  $x_{3}$  and  $x_{4}$ , excluding  $x_{5}$ .
Theorem 3.2 follows from Theorem 3.4. However, the lower bound is true for a much broader class of problems as we show in this section. We first provide a lower bound for a more general formulation of the OCO with finite memory problem (Theorem D.1). The proof of Theorem 3.4 follows as a special case when  $p = 2$ . Then, we provide a lower bound for the OCO with  $\rho$ -discounted infinite memory problem (Theorem D.2).
The OCO with finite memory problem, as defined in the literature, follows linear sequence dynamics with the 2-norm. In this section we consider a more general version of the OCO with finite memory problem that follows linear sequence dynamics with the  $p$ -norm. We provide a lower bound on the policy regret for this more general formulation and the proof of Theorem 3.4 follows as a special case when  $p = 2$ .
Theorem D.1. For all  $p \geq 1$ , there exists an instance of the online convex optimization with finite memory problem with constant memory length  $m$ ,  $(\mathcal{X}, \mathcal{H} = \mathcal{X}^m, A_{\text{finite}, m}, B_{\text{finite}, m})$ , that follows linear sequence dynamics with the  $p$ -norm, and there exist  $L$ -Lipschitz continuous loss functions  $\{f_t : \mathcal{H} \to \mathbb{R}\}_{t=1}^T$  such that the regret of any algorithm  $\mathcal{A}$  satisfies
$$
R _ {T} (\mathcal {A}) \geq \Omega \left(\sqrt {T L ^ {2} m ^ {\frac {p + 2}{p}}}\right).
$$
Proof. Let  $\mathcal{X} = [-1,1]$  and consider an OCO with finite memory problem with constant memory length  $m$ ,  $(\mathcal{X},\mathcal{H} = \mathcal{X}^m,A_{\mathrm{finite},m},B_{\mathrm{finite},m})$ , that follows linear sequence dynamics with the  $p$ -norm. For simplicity, assume that  $T$  is a multiple of  $m$  (otherwise, the same proof works but with slightly more tedious bookkeeping) and that  $L = 1$  (otherwise, multiply the functions  $f_{t}$  defined below by  $L$ ).
Divide the  $T$  rounds into  $N = \frac{T}{m}$  blocks of  $m$  rounds each. Sample  $N$  independent Rademacher random variables  $\{\epsilon_1, \ldots, \epsilon_N\}$ , where each  $\epsilon_i$  is equal to  $\pm 1$  with probability  $\frac{1}{2}$ . Recall that  $h_t = (x_t, \ldots, x_{t-m+1})$ . Define the loss functions  $\{f_t\}_{t=1}^T$  as follows. (See Fig. 2 for an illustration.) If  $t \leq m$ , let  $f_t = 0$ . Otherwise, let
$$
\begin{array}{l} f _ {t} \left(h _ {t}\right) = \epsilon_ {\lceil \frac {t}{m} \rceil} m ^ {\frac {1 - p}{p}} \sum_ {k = 0} ^ {m - 1 - (t - m \lfloor \frac {t}{m} \rfloor - 1)} x _ {m \lfloor \frac {t}{m} \rfloor + 1 - k} \\ = \epsilon_ {\left\lceil \frac {t}{m} \right\rceil} m ^ {\frac {1 - p}{p}} \left(x _ {t - m + 1} + \dots + x _ {m \lfloor \frac {t}{m} \rfloor + 1}\right). \\ \end{array}
$$
In words, the loss in the first  $m$  rounds is equal to 0. Thereafter, in round  $t$  the loss is equal to a random sign  $\epsilon_{\lceil \frac{t}{m} \rceil}$ , which is fixed for that block, times a scaling factor, which is chosen according to the  $p$ -norm to ensure that the Lipschitz constant  $L$  is at most 1, times a sum of a subset of past decisions in the history  $h_t = (x_t, \ldots, x_{t-m+1})$ . This subset consists of all past decisions until and including the first decision of the current block, which is the decision in round  $m \left\lfloor \frac{t}{m} \right\rfloor + 1$ .