SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma E.6. The Lipschitz constant of  $f_{t}$  can be bounded above as
$$
L \leq O \left(L _ {0} D _ {\mathcal {X}} W \kappa (1 - \rho) ^ {- 1}\right),
$$
where  $D_{\mathcal{X}}$  is defined in Lemma E.5.
Proof. Let  $(M_0, \ldots, M_t)$  and  $(\widetilde{M}_0, \ldots, \widetilde{M}_t)$  be two sequences of decisions, where  $M_k$  and  $\widetilde{M}_k \in \mathcal{X}$ . Let  $h_t$  and  $\tilde{h}_t$  be the corresponding histories, and  $(s_t, u_t)$  and  $(\tilde{s}_t, \tilde{u}_t)$  be the corresponding state-control pairs at the end of round  $t$ . We have
$$
\begin{array}{l} \left| f _ {t} \left(h _ {t}\right) - f _ {t} \left(\tilde {h} _ {t}\right) \right| = \left| c _ {t} \left(s _ {t}, u _ {t}\right) - c _ {t} \left(\tilde {s} _ {t}, \tilde {u} _ {t}\right) \right| \\ \leq L _ {0} D _ {\mathcal {X}} \max  \left\{\| s _ {t} - \tilde {s} _ {t} \| _ {2}, \| u _ {t} - \tilde {u} _ {t} \| _ {2} \right\}, \\ \end{array}
$$
where the last inequality follows from our assumptions about the functions  $c_{t}$  and Lemma E.5. It suffices to bound the two norms on the right-hand side in terms of  $\| h_t - \tilde{h}_t\|_{\mathcal{H}}$ . For  $k = 0,\dots ,t - 1$
define  $Z_{k}^{[s]} = \widetilde{F}^{t - k - 1}G(M_{k}^{[s]} - \widetilde{M}_{k}^{[s]})$ . Using Eq. (14), we have
$$
\begin{array}{l} \| s _ {t} - \tilde {s} _ {t} \| _ {2} = \left\| \sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} Z _ {k} ^ {[ s ]} w _ {k - s} \right\| _ {2} \\ \leq \sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} \left\| Z _ {k} ^ {[ s ]} w _ {k - s} \right\| _ {2} \\ = \sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} \left\| \rho^ {- \frac {s}{2}} Z _ {k} ^ {[ s ]} \rho^ {\frac {s}{2}} w _ {k - s} \right\| _ {2} \\ \leq \sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} \left\| \rho^ {- \frac {s}{2}} Z _ {k} ^ {[ s ]} \right\| _ {2} \left\| \rho^ {\frac {s}{2}} w _ {k - s} \right\| _ {2} \\ = \sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} \xi_ {1 + t - 1 - k} \left\| \rho^ {- \frac {s}{2}} Z _ {k} ^ {[ s ]} \right\| _ {2} \xi_ {1 + t - 1 - k} ^ {- 1} \left\| \rho^ {\frac {s}{2}} w _ {k - s} \right\| _ {2} \\ \leq \sqrt {\sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} \xi_ {t - k} ^ {2} \left\| \rho^ {- \frac {s}{2}} Z _ {k} ^ {[ s ]} \right\| _ {2} ^ {2}} \sqrt {\sum_ {k = 0} ^ {t - 1} \sum_ {s = 1} ^ {k + 1} \xi_ {t - k} ^ {- 2} \left\| \rho^ {\frac {s}{2}} w _ {k - s} \right\| _ {2} ^ {2}} \tag {16} \\ = \underbrace {\sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {2} \sum_ {s = 1} ^ {k + 1} \left\| \rho^ {- \frac {s}{2}} Z _ {k} ^ {[ s ]} \right\| _ {2} ^ {2}}} _ {(a)} \underbrace {\sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {- 2} \sum_ {s = 1} ^ {k + 1} \left\| \rho^ {\frac {s}{2}} w _ {k - s} \right\| _ {2} ^ {2}} _ {(b)}, \\ \end{array}
$$
where Eq. (16) follows from the Cauchy-Schwarz inequality. The specific choice of weighted norms on  $\mathcal{X}$  and  $\mathcal{H}$  allow us to bound the terms (a) and (b) in terms of  $\| h_t - \tilde{h}_t\|_{\mathcal{H}}$ . We can bound the term (a) using the definition of  $Z_k^{[s]}$ ,  $\| \cdot \|_{\mathcal{X}}$ , and  $\| \cdot \|_{\mathcal{H}}$  as
$$
\begin{array}{l} \sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {2} \sum_ {s = 1} ^ {k + 1} \left\| \rho^ {- \frac {s}{2}} Z _ {k} ^ {[ s ]} \right\| _ {2} ^ {2}} = \sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {2} \sum_ {s = 1} ^ {k + 1} \rho^ {- s} \left\| \widetilde {F} ^ {t - k - 1} G (M _ {k} ^ {[ s ]} - \widetilde {M} _ {k} ^ {[ s ]}) \right\| _ {2} ^ {2}} \\ \leq \sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {2} \sum_ {s = 1} ^ {k + 1} \rho^ {- s} \left\| \widetilde {F} ^ {t - k - 1} G \left(M _ {k} ^ {[ s ]} - \widetilde {M} _ {k} ^ {[ s ]}\right) \right\| _ {F} ^ {2}} (17) \\ \leq \left\| h _ {t} - \tilde {h} _ {t} \right\| _ {\mathcal {H}}, (18) \\ \end{array}
$$
where Eq. (17) follows from part 1 of Lemma E.1 and Eq. (18) follows from the definitions of  $\| \cdot \|_{\mathcal{X}}$  and  $\| \cdot \|_{\mathcal{H}}$ . Using  $\| w_t\|_2 \leq W$  for all rounds  $t$ , we can bound the term (b) as
$$
\begin{array}{l} \sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {- 2} \sum_ {s = 1} ^ {k + 1} \left\| \rho^ {\frac {s}{2}} w _ {k - s} \right\| _ {2} ^ {2}} \leq W \sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {- 2} \sum_ {s = 1} ^ {k + 1} \rho^ {s}} \\ \leq W \sqrt {\sum_ {k = 0} ^ {t - 1} \xi_ {t - k} ^ {- 2} \frac {\rho (1 - \rho^ {k + 1})}{1 - \rho}} \\ \leq W (1 - \rho) ^ {- 1}, \tag {19} \\ \end{array}
$$
where Eq. (19) follows from the definition of  $(\xi_{k})$  (Eq. (13)). Substituting Eqs. (18) and (19) in Eq. (16), we have
$$
\left\| s _ {t} - \tilde {s} _ {t} \right\| _ {2} \leq W (1 - \rho) ^ {- 1} \left\| h _ {t} - \tilde {h} _ {t} \right\| _ {\mathcal {H}}.
$$
$$
\begin{array}{l} \left\| u _ {t} - \tilde {u} _ {t} \right\| = \left\| K \left(s _ {t} - \tilde {s} _ {t}\right) + \sum_ {s = 1} ^ {t + 1} \left(M _ {t} ^ {[ s ]} - \widetilde {M} _ {t} ^ {[ s ]}\right) w _ {t - s} \right\| _ {2} \\ \leq O \left(W \kappa (1 - \rho) ^ {- 1} \left\| h _ {t} - \tilde {h} _ {t} \right\| _ {\mathcal {H}}\right), \\ \end{array}
$$
where the last inequality follows from our assumption that  $\| K\| _2\leq \kappa$  and the above inequality for  $\| s_t - \tilde{s}_t\| _2$ . This completes the proof.
Lemma E.7. The Lipschitz constant of  $\tilde{f}_t$  can be bounded above as
$$
\tilde {L} \leq O \left(L _ {0} D _ {\mathcal {X}} W \kappa^ {5} (1 - \rho) ^ {- \frac {3}{2}}\right),
$$
Proof. Using Lemma E.2 that bounds  $\| A^k\|$ , we have
$$
\sqrt {\sum_ {k = 0} ^ {\infty} \left\| A ^ {k} \right\| ^ {2}} \leq O \left(\kappa^ {4} (1 - \rho) ^ {- \frac {1}{2}}\right).
$$
Using Theorem 2.1 that bounds  $\tilde{L}$  in terms of  $L$  and the above, we have
$$
\tilde {L} \leq O \left(L \kappa^ {4} (1 - \rho) ^ {- \frac {1}{2}}\right) \leq O \left(L _ {0} D _ {\mathcal {X}} W \kappa^ {5} (1 - \rho) ^ {- \frac {3}{2}}\right),
$$
where the last inequality follows from Lemma E.6.
Now we restate and prove Theorem 4.1.
Proof. Using Theorem 3.1 and the above lemmas, we can upper bound the policy regret of Algorithm 1 for the online linear control problem by
$$
\begin{array}{l} O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} H _ {2}}\right) \\ = O \left(\sqrt {d \kappa^ {8} (1 - \rho) ^ {- 1} T (L _ {0} W ^ {2} \kappa^ {9} (1 - \rho) ^ {- 3}) ^ {2} \kappa^ {4} (1 - \rho) ^ {- \frac {1}{2}} \kappa^ {4} (1 - \rho) ^ {- \frac {3}{2}}}\right) \\ = O \left(L _ {0} W ^ {2} \sqrt {T} d ^ {\frac {1}{2}} \kappa^ {1 7} (1 - \rho) ^ {- 4. 5}\right). \\ \end{array}
$$
E.3 Existing Regret Bound
The upper bound on policy regret for the online linear control problem in existing work is given in Agarwal et al. [2019b, Theorem 5.1]. The theorem statement only shows the dependence on  $\bar{L}, W,$  and  $T$ . The dependence on  $d, \kappa$ , and  $\rho$  can be found in the details of the proof. Below we give a detailed accounting of all of these terms in their regret bound.
To simplify notation let  $\gamma = 1 - \rho$ . Agarwal et al. [2019b] define
$$
H = \frac {\kappa^ {2}}{\gamma} \log (T) \quad \mathrm {a n d} \quad C = \frac {W (\kappa^ {2} + H \kappa_ {B} \kappa^ {2} a)}{\gamma (1 - \kappa^ {2} (1 - \gamma) ^ {H + 1})} + \frac {\kappa_ {B} \kappa^ {3} W}{\gamma}.
$$
The value of  $a$  is not specified in Theorem 5.1. However, from Theorem 5.3 and the definition of  $\mathcal{M}$  in Algorithm 1 their paper, we can infer that  $a = \kappa_B\kappa^3$ .
The final regret bound is obtained by summing Equations 5.1, 5.3, and 5.4. Given the definition of  $H$  above, we have that
$$
(1 - \gamma) ^ {H + 1} \leq \exp (- \kappa^ {2} \log T) = T ^ {- \kappa^ {2}}.
$$
So, the dominant term in the regret bound is Equation 5.4, which is
$$
O \left(L _ {0} W C d ^ {\frac {3}{2}} \kappa_ {B} ^ {2} \kappa^ {6} H ^ {2. 5} \gamma^ {- 1} \sqrt {T}\right).
$$
Substituting the values of  $H$  and  $C$  from above and collecting terms, we have that the upper bound on policy regret in existing work [Agarwal et al., 2019b, Theorem 5.1] is
$$
\begin{array}{l} O \left(L _ {0} W d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {2. 5} \kappa_ {B} ^ {2} \kappa^ {1 1} \gamma^ {- 3. 5} C\right) \\ = O \left(L _ {0} W d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {2. 5} \kappa_ {B} ^ {2} \kappa^ {1 1} \gamma^ {- 3. 5} \left(\frac {W (\kappa^ {2} + H \kappa_ {B} \kappa^ {2} a)}{\gamma (1 - \kappa^ {2} (1 - \gamma) ^ {H + 1})} + \frac {\kappa_ {B} \kappa^ {3} W}{\gamma}\right)\right) \\ = O \left(L _ {0} W d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {2. 5} \kappa_ {B} ^ {2} \kappa^ {1 1} \gamma^ {- 3. 5} \left(\frac {W \kappa^ {2}}{\gamma (1 - \kappa^ {2} (1 - \gamma) ^ {H + 1})} + \frac {W \kappa_ {B} ^ {2} \kappa^ {7} \log (T)}{\gamma^ {2} (1 - \kappa^ {2} (1 - \gamma) ^ {H + 1})} + \frac {\kappa_ {B} \kappa^ {3} W}{\gamma}\right)\right) \\ = O \left(L _ {0} W ^ {2} d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {2. 5} \kappa_ {B} ^ {2} \kappa^ {1 3} \gamma^ {- 4. 5} (1 - \kappa^ {2} (1 - \gamma) ^ {H + 1}) ^ {- 1}\right) \\ + O \left(L _ {0} W ^ {2} d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {3. 5} \kappa_ {B} ^ {4} \kappa^ {1 8} \gamma^ {- 5. 5} \left(1 - \kappa^ {2} (1 - \gamma) ^ {H + 1}\right) ^ {- 1}\right) \\ + O \left(L _ {0} W ^ {2} d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {2. 5} \kappa_ {B} ^ {3} \kappa^ {1 4} \gamma^ {- 4. 5}\right) \\ = O \left(L _ {0} W ^ {2} d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {3. 5} \kappa_ {B} ^ {4} \kappa^ {1 8} \gamma^ {- 5. 5}\right). \\ \end{array}
$$
Above we used that  $\lim_{T\to \infty}(1 - \kappa^2 (1 - \gamma)^{H + 1})^{-1} = 1$  to simplify the expressions. Therefore, the upper bound on policy regret for the online linear control problem in existing work is
$$
O \left(L _ {0} W ^ {2} d ^ {\frac {3}{2}} \sqrt {T} \log (T) ^ {3. 5} \kappa_ {B} ^ {4} \kappa^ {1 8} \gamma^ {- 5. 5}\right). \tag {20}
$$
F Online Performative Prediction
Before formulating the online performative prediction problem in our OCO with unbounded memory framework, we state the definition of 1-Wasserstein distance that we use in our regret analysis. Informally, the 1-Wasserstein distance is a measure of the distance between two probability measures.
Definition F.1 (1-Wasserstein Distance). Let  $(\mathcal{Z},d)$  be a metric space. Let  $\mathbb{P}(\mathcal{Z})$  denote the set of Radon probability measures  $\nu$  on  $\mathcal{Z}$  with finite first moment. That is, there exists  $z^{\prime}\in \mathcal{Z}$  such that  $\mathbb{E}_{z\sim \nu}[d(z,z^{\prime})] < \infty$ . The 1-Wasserstein distance between two probability measures  $\nu ,\nu^{\prime}\in \mathbb{P}(\mathcal{Z})$  is defined as
$$
W _ {1} (\nu , \nu^ {\prime}) = \sup  \left\{\mathbb {E} _ {z \sim \nu} [ f (z) ] - \mathbb {E} _ {z \sim \nu^ {\prime}} [ f (z) ] \right\},
$$
where the supremum is taken over all 1-Lipschitz continuous functions  $f:\mathcal{Z}\to \mathbb{R}$
F.1 Formulation as OCO with Unbounded Memory
Now we formulate the online performative prediction problem in our framework by defining the decision space  $\mathcal{X}$ , the history space  $\mathcal{H}$ , and the linear operators  $A: \mathcal{H} \to \mathcal{H}$  and  $B: \mathcal{W} \to \mathcal{H}$ . Then, we define the functions  $f_t: \mathcal{H} \to \mathbb{R}$  in terms of  $l_t$  and finally, prove an upper bound on the policy regret. For notational convenience, let  $(y_k)$  denote the sequence  $(y_0, y_1, \ldots)$ .
Let  $\rho \in (0,1)$ . Let the decision space  $\mathcal{X} \subseteq \mathbb{R}^d$  be closed and convex with  $\| \cdot \|_{\mathcal{X}} = \| \cdot \|_2$ . Let the history space  $\mathcal{H}$  be the  $\ell^1$ -direct sum of countably infinite number of copies of  $\mathcal{X}$ . Define the linear operators  $A: \mathcal{H} \to \mathcal{H}$  and  $B: \mathcal{X} \to \mathcal{H}$  as
$$
A \left(\left(y _ {0}, y _ {1}, \dots\right)\right) = \left(0, \rho y _ {0}, \rho y _ {1}, \dots\right) \quad \text {a n d} \quad B (x) = \left(x, 0, \dots\right).
$$
Note that the problem is an OCO with  $\rho$ -discounted infinite memory problem and follows linear sequence dynamics with the 1-norm (Definition 2.3).
Given a sequence of decisions  $(x_{k})_{k = 1}^{t}$ , the history is  $h_t = (x_t,\rho x_{t - 1},\ldots ,\rho^{t - 1}x_1,0,\ldots)$  and the data distribution  $p_t = p_t(h_t)$  satisfies:
$$
z \sim p _ {t} \text {i f f} z \sim \sum_ {k = 1} ^ {t - 1} (1 - \rho) \rho^ {k - 1} \left(\xi + F x _ {t - k}\right) + \rho^ {t} p _ {1}. \tag {21}
$$
This follows from the recursive definition of  $p_t$  and parametric assumption about  $\mathcal{D}(x)$ . Define the functions  $f_t: \mathcal{H} \to [0,1]$  by
$$
f _ {t} (h _ {t}) = \mathbb {E} _ {z \sim p _ {t}} [ l _ {t} (x _ {t}, z) ].
$$
With the above formulation and definition of  $f_{t}$ , the original goal of minimizing the difference between the algorithm's total loss and the total loss of the best fixed decision is equivalent to minimizing the policy regret,
$$
\sum_ {t = 1} ^ {T} f _ {t} (h _ {t}) - \min _ {x \in \mathcal {X}} \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x).
$$
F.2 Regret Analysis
Lemma F.1. The operator norm  $\| A^s\|$  is bounded above as
$$
\left\| A ^ {s} \right\| \leq O \left(\rho^ {s}\right).
$$
Proof. Recall the definition of  $\mathcal{H}$  and  $\| \cdot \|_{\mathcal{H}}$ . Let
$$
\left(y _ {0}, y _ {1}, \dots\right) = \left(x _ {0}, \rho x _ {1}, \rho^ {2} x _ {2}, \dots\right)
$$
$$
\sum_ {k = 0} ^ {\infty} \left\| y _ {k} \right\| = 1.
$$
From the definition of the operator  $A$ , we have
$$
A ^ {s} \left(\left(y _ {0}, y _ {1}, \dots\right)\right) = \left(0, \dots , 0, \rho^ {s} x _ {0}, \rho^ {s + 1} x _ {1}, \dots\right).
$$
Now we bound  $\| A^s\|$  as follows. By definition of  $A^s$  and  $\| \cdot \|_{\mathcal{H}}$ , we have
$$
\| A ^ {s} ((y _ {0}, y _ {1}, \dots)) \| = \sum_ {k = 0} ^ {\infty} \rho^ {s + k} \| x _ {k} \| = \rho^ {s} \sum_ {k = 0} ^ {\infty} \rho^ {k} \| x _ {k} \| = \rho^ {s} \sum_ {k = 0} ^ {\infty} \| y _ {k} \| = \rho^ {s}.
$$
Lemma F.2. The 1-effective memory capacity is bounded above as
$$
H _ {2} \leq O \left((1 - \rho) ^ {- 2}\right).
$$
Proof. Using Lemma F.1 to bound  $\| A^k \|$ , we have
$$
H _ {1} = \sum_ {k = 0} ^ {\infty} k \| A ^ {k} \| = \sum_ {k = 0} ^ {\infty} k \rho^ {k} \leq O \left((1 - \rho) ^ {- 2}\right).
$$
Lemma F.3. Suppose  $R: \mathcal{X} \to \mathbb{R}$  is defined by  $R(x) = \frac{1}{2} \| x\|_{\mathcal{X}}^2$ . Then, it is 1-strongly-convex and  $D = \max_{x,\tilde{x}\in \mathcal{X}}|R(x) - R(\tilde{x})|\leq D_{\mathcal{X}}^2$ .
Proof. Note that  $R$  is 1-strongly-convex by definition. By the assumption that  $\| x\|_{\mathcal{X}}\leq D_{\mathcal{X}}$  for all  $x\in \mathcal{X}$ , we have that  $D\leq D_{\mathcal{X}}^{2}$ .
Lemma F.4. The Lipschitz constant of  $f_{t}$  can be bounded above as
$$
L \leq O \left(L _ {0} \frac {1 - \rho}{\rho} \| F \| _ {2}\right).
$$
Proof. Let  $(x_{1},\ldots ,x_{t})$  and  $(\tilde{x}_1,\dots ,\tilde{x}_t)$  be two sequences of decisions, where  $x_{k},\tilde{x}_{k}\in \mathcal{X}$ . Let  $h_t$  and  $\tilde{h}_t$  be the corresponding histories, and  $p_t$  and  $\tilde{p}_t$  be the corresponding distributions at the end of round  $t$ . We have
$$
\begin{array}{l} \left| f _ {t} \left(h _ {t}\right) - f _ {t} \left(\tilde {h} _ {t}\right) \right| \\ = \left| \mathbb {E} _ {z \sim p _ {t}} \left[ l _ {t} \left(x _ {t}, z\right) \right] - \mathbb {E} _ {z \sim \tilde {p} _ {t}} \left[ l _ {t} \left(\tilde {x} _ {t}, z\right) \right] \right| \\ = \left| \mathbb {E} _ {z \sim p _ {t}} \left[ l _ {t} \left(x _ {t}, z\right) \right] - \mathbb {E} _ {z \sim p _ {t}} \left[ l _ {t} \left(\tilde {x} _ {t}, z\right) \right] + \mathbb {E} _ {z \sim p _ {t}} \left[ l _ {t} \left(\tilde {x} _ {t}, z\right) \right] - \mathbb {E} _ {z \sim \tilde {p} _ {t}} \left[ l _ {t} \left(\tilde {x} _ {t}, z\right) \right] \right| \\ \leq L _ {0} \left\| x _ {t} - \tilde {x} _ {t} \right\| _ {2} + L _ {0} W _ {1} \left(p _ {t}, \tilde {p} _ {t}\right), \\ \end{array}
$$
where the last inequality follows from the assumptions about the functions  $l_{t}$  and the definition of the Wasserstein distance  $W_{1}$ . By definition of  $p_t$  (Eq. (21)), we have
$$
\begin{array}{l} W _ {1} \left(p _ {t}, \tilde {p} _ {t}\right) \leq \sum_ {k = 1} ^ {t - 1} \frac {1 - \rho}{\rho} \rho^ {k} \| F \| _ {2} \| x _ {t - k} - \tilde {x} _ {t - k} \| _ {2} \\ \leq \frac {1 - \rho}{\rho} \| F \| _ {2} \| h _ {t} - \tilde {h} _ {t} \| _ {\mathcal {H}}, \\ \end{array}
$$
where the last inequality follows from the definition of  $\| \cdot \|_{\mathcal{H}}$ . Therefore,  $L \leq L_0 \frac{1 - \rho}{\rho} \| F \|_2$ .
Lemma F.5. The Lipschitz constant of  $f_{t}$  can be bounded above as
$$
\tilde {L} \leq O \left(L _ {0} \frac {1}{\rho} \| F \| _ {2}\right).
$$
Proof. Using Lemma F.1 that bounds  $\| A^k\|$ , we have
$$
\sum_ {k = 0} ^ {\infty} \left\| A ^ {k} \right\| = (1 - \rho) ^ {- 1}.
$$
$$
\tilde {L} \leq O (L (1 - \rho) ^ {- 1}) = O \left(L _ {0} \frac {1}{\rho} \| F \| _ {2}\right),
$$
where the last equality follows from Lemma F.4.
Now we restate and prove Theorem 4.2.
Proof. Using Theorem 3.1 and the above lemmas, we can upper bound the policy regret of Algorithm 1 for the online performative prediction problem by
$$
O \left(\sqrt {\frac {D}{\alpha} T L \tilde {L} H _ {1}}\right) = O \left(D _ {\mathcal {X}} L _ {0} \| F \| _ {2} (1 - \rho) ^ {- \frac {1}{2}} \rho^ {- 1} \sqrt {T}\right).
$$
We note that the upper bound can be improved by defining a weighted norm on  $\mathcal{H}$  similar to the approach in Appendix E. However, here we present the looser analysis for simplicity of exposition.
G Implementation Details for Algorithm 1
In this section we discuss how to implement Algorithm 1 efficiently.
Dimensionality of  $\mathcal{X}$ . First, note that the decisions  $x\in \mathcal{X}$  could be high-dimensional, e.g., an unbounded sequence of matrices as in the online linear control problem, but this is external to our framework and is application dependent. Our framework can be applied to  $\mathcal{X}$  or to a lower-dimensional decision space  $\mathcal{X}'$ . However, the choice of  $\mathcal{X}'$  and analyzing the difference
$$
\min  _ {x ^ {\prime} \in \mathcal {X} ^ {\prime}} \sum_ {t = 1} ^ {T} \tilde {f} _ {t} \left(x ^ {\prime}\right) - \min  _ {x \in \mathcal {X}} \sum_ {t = 1} ^ {T} \tilde {f} _ {t} (x)
$$
is application dependent. For example, for the online linear control problem one could consider a restricted class of disturbance-action controllers that operate on a constant number of past disturbances as opposed to all the past disturbances, and then analyze the difference between these two policy classes. See, for example, Agarwal et al. [2019b, Lemma 5.2].
Computational cost of each iteration of Algorithm 1. Now we discuss how to implement each iteration of Algorithm 1 efficiently. We are interested in the computational cost of computing the decision  $x_{t+1}$  as a function of  $t$ . (Given the above discussion about the dimensionality of  $\mathcal{X}$ , we ignore the fact that the dimensionality of the decisions themselves could depend on  $t$ .) Therefore, for the purposes of this section we (i) use  $O(\cdot)$  notation to hide absolute constants and problem parameters excluding  $t$  and  $T$ ; (ii) invoke the operators  $A$  and  $B$  by calling oracles  $\mathcal{O}_A(\cdot)$  and  $\mathcal{O}_B(\cdot)$ ; and (iii) evaluate the functions  $f_t$  by calling oracles  $\mathcal{O}_f(t, \cdot)$ . Recall from Assumption A1 that we assume the learner knows the operators  $A$  and  $B$ , and observes  $f_t$  at the end of each round  $t$ . So, the oracles  $\mathcal{O}_A, \mathcal{O}_B$ , and  $\mathcal{O}_f$  are readily available.
Algorithm 1 chooses the decision  $x_{t + 1}$  as
$$
x _ {t + 1} \in \underset {x \in \mathcal {X}} {\arg \min} \sum_ {s = 1} ^ {t} \tilde {f} _ {s} (x) + \frac {R (x)}{\eta} = \underset {x \in \mathcal {X}} {\arg \min} \underbrace {\sum_ {s = 1} ^ {t} f _ {s} \left(\sum_ {k = 0} ^ {s - 1} A ^ {k} B x\right)} _ {= F _ {t} (x)} + \frac {R (x)}{\eta}.
$$
Since  $F_{t}(x)$  is a sum of  $f_{1},\ldots ,f_{t}$ , evaluating  $F_{t}(x)$  requires  $\Theta (t)$  oracle calls to  $\mathcal{O}_f$ . However, this issue is present in FTRL for OCO and OCO with finite memory as well and is not specific to our framework. To deal with this issue, one could consider mini-batching algorithms [Dekel et al., 2012, Altschuler and Talwar, 2018, Chen et al., 2020] such as Algorithm 2.
A naive implementation to evaluate  $F_{t}(x)$  could require  $O(t^{3})$  oracle calls to  $\mathcal{O}_A$ : for each  $s \in [t]$ , constructing the argument  $\sum_{k=0}^{s-1} A^k Bx$  for  $f_s$  could require  $k$  oracle calls to  $\mathcal{O}_A$  to compute  $A^k Bx$  for a total of  $O(s^2)$  oracle calls. However,  $F_{t}(x)$  can be evaluated with just  $O(t)$  oracle calls to  $\mathcal{O}_A$  by constructing the arguments incrementally. For  $t \geq 0$ , define  $\Gamma_t : \mathcal{X} \to \mathcal{H}$  as
$$
\Gamma_ {0} (x) = B x
$$
$$
\Gamma_ {t} (x) = A \left(\Gamma_ {t - 1} (x)\right) \quad \text {f o r} t \geq 1.
$$
Note that  $\Gamma_t(Bx) = A^t Bx$ . Also, for  $t \geq 1$ , define  $\Phi_t: \mathcal{X} \to \mathcal{H}$  as
$$
\Phi_ {1} (x) = \Gamma_ {0} (x)
$$
$$
\Phi_ {t} (x) = \Phi_ {t - 1} (x) + \Gamma_ {t - 1} (x) \quad \text {f o r} t \geq 2.
$$
Note that  $\Phi_s(x) = \sum_{k=0}^{s-1} A^k Bx$  is the argument for  $f_s$ . These can be constructed incrementally as follows.
1. Construct  $\Gamma_0(x)$  using one oracle call to  $\mathcal{O}_B$ .
2. For  $s = 1$
(a) Construct  $\Phi_1(x) = \Gamma_0(x)$
(b) Construct  $\Gamma_1(x)$  from  $\Gamma_0(x)$  using one oracle call to  $\mathcal{O}_A$ .
3. For  $s \geq 2$ ,
(a) Construct  $\Phi_s(x)$  by adding  $\Phi_{s - 1}(x)$  and  $\Gamma_{s - 1}(x)$ . This can be done in  $O(1)$  time. Recall from our earlier discussion that  $O(\cdot)$  hides absolute constants and problem parameters excluding  $t$  and  $T$ .
(b) Construct  $\Gamma_s(x)$  from  $\Gamma_{s - 1}(x)$  using one oracle call to  $\mathcal{O}_A$
By incrementally constructing  $\Phi_s(x)$  as above, we can evaluate  $F_t(x)$  in  $O(t)$  time with  $O(1)$  oracle calls to  $\mathcal{O}_B$ ,  $O(t)$  oracle calls to  $\mathcal{O}_A$ , and  $O(t)$  oracle calls to  $\mathcal{O}_f$ .
Memory usage of Algorithm 1. We end with a brief discussion of the memory usage of Algorithm 1. We are interested in the memory usage of computing the decision  $x_{t + 1}$  as a function of  $t$ . (Given the discussion about the dimensionality of  $\mathcal{X}$  at the start of this section, we ignore the fact that the dimensionality of the decisions themselves could depend on  $t$ .) For each  $t \in [T]$ , the memory usage could be as low as  $O(1)$  (if, for example,  $\mathcal{X} \subseteq \mathbb{R}^d$ , and  $A, B \in \mathbb{R}^{d \times d}$ , which implies that  $\Phi_t(x)$  is a  $d$ -dimensional vector) or as high as  $O(t)$  (if, for example,  $\Phi_t(x)$  is a  $t$ -length sequence of  $d$ -dimensional vectors). However, the memory usage is already  $\Omega(t)$  to store the functions  $f_1, \ldots, f_t$ . Therefore, Algorithm 1 only incurs a constant factor overhead.
H An Algorithm with A Low Number of Switches: Mini-Batch FTRL
In this section we present an algorithm (Algorithm 2) for OCO with unbounded memory that provides the same upper bound on policy regret as Algorithm 1 while guaranteeing a small number of switches. Algorithm 2 combines FTRL on the functions  $\tilde{f}_t$  with a mini-batching approach. First, it divides rounds into batches of size  $S$ , where  $S$  is a parameter. Second, at the start of batch  $b \in \{1, \dots, \lceil T / s \rceil\}$ , it performs FTRL on the functions  $\{g_1, \dots, g_b\}$ , where  $g_i$  is the average of the functions  $\tilde{f}_t$  in batch  $i$ . Then, it uses this decision for the entirety of the current batch. By design, Algorithm 2 switches decisions at most  $O(T / s)$  times. This algorithm is inspired by similar algorithms for online learning and OCO [Dekel et al., 2012, Altschuler and Talwar, 2018, Chen et al., 2020].