SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 2: Mini-Batch FTRL
Input: Time horizon  $T$ , step size  $\eta$ ,  $\alpha$ -strongly-convex regularizer  $R:\mathcal{X}\to \mathbb{R}$ , batch size  $S$ .  
1 Initialize history  $h_0 = 0$ .  
2 for  $t = 1,2,\ldots ,T$  do  
3 if  $t$  mod  $S = 1$  then  
4 Let  $N_{t} = \{1,\dots ,\lceil \frac{t}{S}\rceil \}$  denote the number of batches so far.  
5 For  $b\in N_t$ , let  $T_{b} = \{(b - 1)S + 1,\dots,bS\}$  denote the rounds in batch  $b$ .  
6 For  $b\in N_t$ , let  $g_{b} = \frac{1}{S}\sum_{s\in T_{b}}\tilde{f}_{s}$ . denote the average of the functions in batch  $b$ .  
7 Learner chooses  $x_{t}\in \arg \min_{x\in \mathcal{X}}\sum_{b\in N_{t}}g_{b}(x) + \frac{R(x)}{\eta}$ .  
8 end  
9 else  
10 Learner chooses  $x_{t} = x_{t - 1}$ .  
11 end  
12 Set  $h_t = Ah_{t - 1} + Bx_t$ .  
13 Learner suffers loss  $f_{t}(h_{t})$  and observes  $f_{t}$ .  
14 end
Theorem H.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 2 with batch size  $S$  and step-size  $\eta$  satisfies
$$
R _ {T} (M i n i - B a t c h F T R L) \leq \frac {S D}{\eta} + \eta \frac {T \tilde {L} ^ {2}}{\alpha} + \eta \frac {T L \tilde {L} H _ {1}}{S \alpha}.
$$
$$
I f \eta = \sqrt {\frac {\alpha S D}{T \tilde {L} \left(\frac {L H _ {1}}{S} + \tilde {L}\right)}}, t h e n
$$
$$
R _ {T} (M i n i - B a t c h F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T \left(L \tilde {L} H _ {1} + S \tilde {L} ^ {2}\right)}\right).
$$
Setting the batch size to be  $S = {}^{LH_1} / \tilde{L}$  we obtain the same upper bound on policy regret as Algorithm 1 while guaranteeing that the decisions  $x_{t}$  switch at most  ${}^{T}\tilde{L} / {}^{L}H_{1}$  times.
Corollary H.1. Consider an online convex optimization with unbounded memory problem specified by  $(\mathcal{X},\mathcal{H},A,B)$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$
for all  $x, \tilde{x} \in \mathcal{X}$ . Algorithm 2 with batch size  $S = \frac{LH_1}{\tilde{L}}$  and step-size  $\eta = \sqrt{\frac{\alpha SD}{T\tilde{L}\left(\frac{LH_1}{S} + \tilde{L}\right)}}$  satisfies
$$
R _ {T} (M i n i - B a t c h F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} H _ {1}}\right).
$$
Furthermore, the decisions  $x_{t}$  switch at most  $\frac{T\tilde{L}}{LH_1}$  times.
Intuitively, in the OCO with unbounded memory framework each decision  $x_{t}$  is penalized not just in round  $t$  but in future rounds as well. Therefore, instead of immediately changing the decision, it is prudent to stick to it for a while, collect more data, and then switch decisions. For the OCO with finite memory problem, the constant memory length  $m$  provides a natural measure of how long decisions penalized for and when one should switch decisions. In the general case, this is measured by the quantity  ${}^{LH_1 / \tilde{L}}$ . Note that this simplifies to  $m$  for OCO with finite memory for all  $p$ -norms.
Proof of Theorem H.1. For simplicity, assume that  $T$  is a multiple of  $S$ . Otherwise, the same proof works after replacing  $\frac{T}{S}$  with  $\lceil \frac{T}{S} \rceil$ . Let  $x^{*} \in \arg \min_{x \in \mathcal{X}} \sum_{t=1}^{T} \tilde{f}_{t}(x)$ . Note that we can write the regret as
$$
\begin{array}{l} R _ {T} (\text {M i n i - B a t c h F T R L}) = \sum_ {t = 1} ^ {T} f _ {t} \left(h _ {t}\right) - \min  _ {x \in \mathcal {X}} \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x) \\ = \underbrace {\sum_ {t = 1} ^ {T} f _ {t} (h _ {t}) - \tilde {f} _ {t} (x _ {t})} _ {(a)} + \underbrace {\sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x _ {t}) - \tilde {f} _ {t} (x ^ {*})} _ {(b)}. \\ \end{array}
$$
We can bound the term (b) using Theorem B.1 for mini-batches [Dekel et al., 2012, Altschuler and Talwar, 2018, Chen et al., 2020] by
$$
\frac {S D}{\eta} + \eta \frac {T \tilde {L} ^ {2}}{\alpha}.
$$
It remains to bound term (a). Let  $N = T / s$  denote the number of batches and  $T_{n} = \{(n - 1)S + 1,\dots ,nS\}$  denote the rounds in batch  $n\in [N]$ . We can write
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} f _ {t} \left(h _ {t}\right) - \tilde {f} _ {t} \left(x _ {t}\right) = \sum_ {t = 1} ^ {T} f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k}\right) - f _ {t} \left(\sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t}\right) \quad \text {b y} \\ \leq L \sum_ {t = 1} ^ {T} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\| \quad \text {b y} \\ \leq \frac {T}{S} L \underbrace {\sum_ {t \in T _ {N}} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\|} _ {(c)}, \\ \end{array}
$$
where the last inequality follows because of the following. Consider rounds  $t_1 = b_1S + r$  and  $t_2 = b_2S + r$  for  $b_1 < b_2$  and  $r \in [S]$ . Then,  $\| h_{t_1} - \sum_{k=0}^{t_1-1} A^k B x_{t_1} \| \leq \| h_{t_2} - \sum_{k=0}^{t_2-1} A^k B x_{t_2} \|$  because the latter sums over more terms in its history and decisions in consecutive batches have distance bounded above by  $\eta \tilde{L}/\alpha$  (Theorem B.1). Therefore, it suffices to show that term (c) is upper bounded by  $\eta \tilde{L} H_1/\alpha$ . We have
$$
\begin{array}{l} \sum_ {t \in T _ {N}} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\| \leq \sum_ {t \in T _ {N}} \sum_ {k = 0} ^ {t - 1} \left\| A ^ {k} B x _ {t - k} - A ^ {k} B x _ {t} \right\| \\ \leq \sum_ {t \in T _ {N}} \sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| \| B \| \| x _ {t - k} - x _ {t} \| \\ \leq \sum_ {t \in T _ {N}} \sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| \| x _ {t - k} - x _ {t} \| \quad \text {b y} \\ \end{array}
$$
Since the same decision  $x_{n}$  is chosen in all rounds of batch  $n$ , we can reindex and rewrite
$$
\begin{array}{l} \sum_ {t \in T _ {N}} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\| \leq \sum_ {t \in T _ {N}} \sum_ {k = 0} ^ {t - 1} \| A ^ {k} \| \| x _ {t - k} - x _ {t} \| \\ \leq \sum_ {o = 0} ^ {S - 1} \sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} \| A ^ {(N - n - 1) S + s + o} \| \| x _ {N} - x _ {n} \| \\ \leq \eta \frac {\tilde {L}}{\alpha} \sum_ {o = 0} ^ {S - 1} \sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} (N - n) \| A ^ {(N - n - 1) S + s + o} \| \\ = \eta \frac {\tilde {L}}{\alpha} \sum_ {o = 0} ^ {S - 1} \sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} n \| A ^ {(n - 1) S + s + o} \|, \\ \end{array}
$$
where the last inequality follows from bounding the distance between decision in consecutive batches Theorem B.1 and the triangle inequality. Expanding the triple sum yields
$$
\begin{array}{l} \sum_ {o = 0} ^ {S - 1} \sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} n \| A ^ {(n - 1) S + s + o} \| \\ \leq \| A \| + \dots + \| A ^ {S} \| + 2 \| A ^ {S + 1} \| + \dots + 2 \| A ^ {2 S} \| + 3 \| A ^ {2 S + 1} \| + \dots + 3 \| A ^ {3 S} \| + \dots \\ + \| A ^ {2} \| + \dots + \| A ^ {S + 1} \| + 2 \| A ^ {S + 2} \| + \dots + 2 \| A ^ {2 S + 1} \| + 3 \| A ^ {2 S + 2} \| + \dots + 3 \| A ^ {3 S + 1} \| + \dots \\ \end{array}
$$
$$
+ \| A ^ {S} \| + \dots + \| A ^ {2 S - 1} \| + 2 \| A ^ {2 S} \| + \dots + 2 \| A ^ {3 S - 1} \| + 3 \| A ^ {3 S} \| + \dots + 3 \| A ^ {4 S - 1} \| + \dots ,
$$
where each line above corresponds to a value of  $o \in \{0, \dots, S - 1\}$ . Adding up these terms yields  $H_{1}$ . This completes the proof.
Note that Theorem H.1 only provides an upper bound on the policy regret for the general case. Unlike Algorithm 1, it is unclear how to obtain a stronger bound depending on  $H_{p}$  for the case of linear sequence dynamics with the  $\xi$ -weighted  $p$ -norm for  $p > 1$ . The above proof can be specialized for this special case, similar to the proofs of Theorem 2.1 and Lemma C.1, to obtain
$$
\sum_ {t \in T _ {N}} \left\| \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t - k} - \sum_ {k = 0} ^ {t - 1} A ^ {k} B x _ {t} \right\| \leq \eta \frac {\tilde {L}}{\alpha} \sum_ {o = 0} ^ {S - 1} \left(\sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} \left(n \| A ^ {(n - 1) S + s + o} \|\right) ^ {p}\right) ^ {\frac {1}{p}}
$$
$$
\begin{array}{l} \sum_ {o = 0} ^ {S - 1} \left(\sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} \left(n \| A ^ {(n - 1) S + s + o} \|\right) ^ {p}\right) ^ {\frac {1}{p}} \\ \leq \left(\| A \| ^ {p} + \dots + \| A ^ {S} \| ^ {p} + 2 ^ {p} \| A ^ {S + 1} \| ^ {p} + \dots 2 ^ {p} \| A ^ {2 S} \| ^ {p} + 3 ^ {p} \| A ^ {2 S + 1} \| ^ {p} + \dots\right) ^ {\frac {1}{p}} \\ + \left(\| A ^ {2} \| ^ {p} + \dots + \| A ^ {S + 1} \| ^ {p} + 2 ^ {p} \| A ^ {S + 2} \| ^ {p} + \dots 2 ^ {p} \| A ^ {2 S + 1} \| ^ {p} + 3 ^ {p} \| A ^ {2 S + 2} \| ^ {p} + \dots\right) ^ {\frac {1}{p}} \\ \end{array}
$$
$$
\left(\| A ^ {S} \| ^ {p} + \dots + \| A ^ {2 S - 1} \| ^ {p} + 2 ^ {p} \| A ^ {2 S} \| ^ {p} + \dots 2 ^ {p} \| A ^ {3 S - 1} \| ^ {p} + 3 ^ {p} \| A ^ {3 S} \| ^ {p} + \dots\right) ^ {\frac {1}{p}}.
$$
The above expression cannot be easily simplified to  $O(H_{p})$ . However, for the special case of OCO with finite memory, which follows linear sequence dynamics with the 2-norm, we can do so by leveraging the special structure of the linear operator  $A_{\mathrm{finite},m}$ .
Theorem H.2. Consider an online convex optimization with finite memory problem with constant memory length  $m$  specified by  $(\mathcal{X},\mathcal{H} = \mathcal{X}^m,A_{\mathrm{finite},m},B_{\mathrm{finite},m})$ . Let the regularizer  $R:\mathcal{X}\to \mathbb{R}$  be  $\alpha$ -strongly-convex and satisfy  $|R(x) - R(\tilde{x})|\leq D$  for all  $x,\tilde{x}\in \mathcal{X}$ . Algorithm 2 with batch size  $m$
and step-size  $\eta = \sqrt{\frac{\alpha mD}{T\tilde{L}\left(Lm^{\frac{1}{2}} + \tilde{L}\right)}}$  satisfies
$$
R _ {T} (M i n i - B a t c h F T R L) \leq O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} m ^ {\frac {3}{2}}}\right) \leq O \left(m \sqrt {\frac {D}{\alpha} T L ^ {2}}\right).
$$
Furthermore, the decisions  $x_{t}$  switch at most  $\frac{T}{m}$  times.
Proof. Given the proof of Theorem H.1 and the above discussion, it suffices to show that
$$
\sum_ {o = 0} ^ {S - 1} \left(\sum_ {n = 1} ^ {N - 1} \sum_ {s = 1} ^ {S} \left(n \| A ^ {(n - 1) S + s + o} \|\right) ^ {2}\right) ^ {\frac {1}{2}} \leq H _ {2} = m ^ {\frac {3}{2}}.
$$
Recall that  $\| A_{\mathrm{finite}}^k \| = 1$  if  $k \leq m$  and 0 otherwise. Using this and  $S = m$ , we have that the above sum is at most  $\sqrt{m} + \sqrt{m - 1} + \dots + \sqrt{1} = O\left(m^{\frac{3}{2}}\right)$ . This completes the proof.
In this section we present some simple simulation experiments.2
Problem Setup. We consider the problem of online linear control with a constant input controller class  $\Pi = \{\pi_u : \pi(s) = u \in \mathcal{U}\}$ . Let  $T$  denote the time horizon. Let  $\mathcal{S} = \mathbb{R}^d$  and  $\mathcal{U} = \{u \in \mathbb{R}^d : \|u\|_2 \leq 1\}$  denote the state and control spaces. Let  $s_t$  and  $u_t$  denote the state and control at time  $t$  with  $s_0$  being the initial state. The system evolves according to linear dynamics  $s_{t+1} = F s_t + G u_t + w_t$ , where  $F, G \in \mathbb{R}^{d \times d}$  are system matrices and  $w_t \in \mathbb{R}^d$  is a disturbance. The loss function in round  $t$  is simply  $c_t(s_t, u_t) = c_t(s_t) = \sum_{j=1}^d s_{t,j}$ , where  $s_{t,j}$  denotes the  $j$ -th coordinate of  $s_t$ . The goal is to choose a sequence of control inputs  $u_0, \ldots, u_{T-1} \in \mathcal{U}$  to minimize the regret
$$
\sum_ {t = 0} ^ {T - 1} c _ {t} \left(s _ {t}, u _ {t}\right) - \min  _ {u \in \mathcal {U}} \sum_ {t = 0} ^ {T - 1} c _ {t} \left(s _ {t} ^ {u}, u\right),
$$
where  $s_t^u$  denotes the state in round  $t$  upon choosing control input  $u$  in each round. Note that the state in round  $t$  can be written as
$$
s _ {t} = \sum_ {k = 1} ^ {t} F ^ {k} G u _ {t - k} + \sum_ {k = 1} ^ {t} F ^ {k} w _ {t - k}.
$$
Therefore, we can formulate this problem as an OCO with unbounded memory problem by setting  $\mathcal{X} = \mathcal{U},\mathcal{H} = \{y\in \mathbb{R}^d:y = \sum_{k = 0}^{t}F^k Gu$  for some  $u\in \mathcal{U}$  and  $t\in \mathbb{N}\}$ ,  $A(h) = Fh,B(x) = Gx$ , and  $f_{t}(h_{t}) = c_{t}(\sum_{k = 1}^{t}F^{k}Gu_{t - k} + \sum_{k = 1}^{t}F^{k}w_{t - k})$ . Note that  $\mathcal{H},A$ , and  $B$  are all finite-dimensional.
Data. We set the time horizon  $T = 750$  and dimension  $d = 2$ . We sample the disturbances  $\{w_{t}\}$  from a standard normal distribution. We set the system matrix  $G$  to be the identity and the system matrix  $F$  to be a diagonal plus upper triangular matrix with the diagonal entries equal to  $\rho$  and the upper triangular entries equal to  $\alpha$ . We run simulations with various values of  $\rho$  and  $\alpha$ .
Implementation. We use the cvxpy library [Diamond and Boyd, 2016, Agrawal et al., 2018] for implementing Algorithm 1. We use step-sizes according to Theorems 3.1 and 3.3. We run the experiments on a standard laptop.
$^{2}$ https://github.com/raunakkmr/oco-with-memory-code.
Results. We compare the regret with respect to the optimal control input of OCO with unbounded memory and OCO with finite memory for various memory lengths  $m$  in Fig. 3 for  $\rho = 0.90$  and Fig. 4 for  $\rho = 0.95$ . There are a few important takeaways.
1. OCO with unbounded memory either performs as well as or better than OCO with finite memory, and it does so at comparable computational cost (Appendix G). In fact, the regret curve for OCO with unbounded memory reaches an asymptote whereas this is not the case for OCO with finite memory for a variety of memory lengths.
2. Knowledge of the spectral radius of  $F$ ,  $\rho$ , is not sufficient to tune the memory length  $m$  for OCO with finite memory. This is illustrated by comparing Figs. 3a to 3d. Even though small memory lengths perform well when the upper triangular value is small, they perform poorly when the upper triangular value is large. In contrast, OCO with unbounded memory performs well in all cases.
3. For a fixed memory length, OCO with unbounded memory eventually performs better than OCO with finite memory. This is illustrated by comparing Figs. 3a to 3d.
4. As we increase the memory length, the performance of OCO with finite memory eventually approaches that of OCO with unbounded memory. However, an advantage of OCO with unbounded memory is that it does not require tuning the memory length. For example, when  $\rho = 0.90$  and the upper triangular entry of  $F = 0.10$ , OCO with finite memory with  $m = 4$  performs comparably to  $m = 8$  and  $m = 16$  (Fig. 3c). However, when the upper triangular entry of  $F = 0.12$ , then it performs much worse (Fig. 3d). However, OCO with unbounded memory performs well in all cases without the need for tuning an additional hyperparameter in the form of memory length.
(b)
(d)
Figure 3: Regret plot for  $\rho = 0.90$ . The label 0C0-UM refers to formulating the problem as an OCO with unbounded memory problem. The 0C0-FM-m refers to formulating the problem as an OCO with finite memory problem with constant memory length  $m$ . The titles of the plots indicate the values of the dimension, the diagonal entries of  $F$ , and the upper triangular entries of  $F$ .
(e)
Figure 4: Regret plot for  $\rho = 0.95$ . The label 0C0-UM refers to formulating the problem as an OCO with unbounded memory problem. The 0C0-FM-m refers to formulating the problem as an OCO with finite memory problem with constant memory length  $m$ . The titles of the plots indicate the values of the dimension, the diagonal entries of  $F$ , and the upper triangular entries of  $F$ .