SlowGuess/NeurIPS_2023_OCR huggingface dataset:
almost surely. In particular,  $\mathbb{E}_{t - 1}[(Z^t (f^*))\mid x_{1:H}^t ]\geq 0$
Proof of Lemma C.8. Recall  $Z^t (f) = -\frac{1}{\lambda}\log \mathbb{E}_{g\sim q^t (\cdot |f)}\exp (-\lambda \delta^t (g,f))$  , thus we have
$$
\begin{array}{l} - \lambda \mathbb {E} _ {t - 1} \left[ Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] = \mathbb {E} _ {t - 1} \left[ \log \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp \left(- \lambda \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right] \\ \leq \log \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp \left(- \lambda \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right] (Jensen) \\ \leq \log \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp \left(- \lambda (1 - 2 \lambda) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}\right) \mid x _ {1: H} ^ {t} \right] (Lemma C.6) \\ \leq \log \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \left(1 - \frac {1}{2} \lambda (1 - 2 \lambda) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}\right) \mid x _ {1: H} ^ {t} \right] (Lemma C.1) \\ \leq - \frac {\lambda}{4} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] = - \frac {\lambda}{4} \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}. \\ \end{array}
$$
Recalling that  $Z^t(f) = -\frac{1}{\lambda} \log \mathbb{E}_{g \sim q^t(\cdot|f)} \exp(-\lambda \delta^t(g, f))$  and  $\delta^t(f) = \mathbb{E}_{g \sim q^t(\cdot|f)} \delta^t(g, f)$ , we obtain the following relationship via Jensen's inequality and second order expansion of the exponential function.
Lemma C.9. For any  $t \leq T$  and realization of  $S^{t-1}$ , we have that as long as  $\lambda < 1/8$ ,  $Z^t(f) \leq \delta^t(f)$  and  $|Z^t(f)| \leq \frac{9}{8} \mathbb{E}_{g \sim q^t(\cdot|f)} |\delta^t(g, f)|$  for all  $g, f \in \mathcal{G}$ .
Proof of Lemma C.9. By Jensen's inequality,
$$
Z ^ {t} (f) = - \frac {1}{\lambda} \log \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp (- \lambda \delta^ {t} (g, f)) \leq \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \delta^ {t} (g, f) = \delta^ {t} (f).
$$
On the other hand, applying Lemma C.2, we have
$$
\begin{array}{l} - \lambda Z ^ {t} (f) = \log \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp (- \lambda \delta^ {t} (g, f)) \\ \leq \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \left(- \lambda \delta^ {t} (g, f) + \lambda^ {2} \delta^ {t} (g, f) ^ {2}\right). \\ \end{array}
$$
$$
| Z ^ {t} (f) | \leq | \delta^ {t} (f) | + \lambda \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \delta^ {t} (g, f) ^ {2} \leq \frac {9}{8} \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} | \delta^ {t} (g, f) |.
$$
Another important fact we will use is that  $f^{*}$  satisfies  $\delta^t (f^*,f^*) = 0$ . Furthermore, for any  $f\in \mathcal{G}$ , the pair  $\mathcal{T}f$ ,  $f$  always has  $\delta^t (\mathcal{T}f,f) = 0$  for all  $t\in [T]$ .
Lemma C.10. For any  $t \leq T$ , under Assumption C.1 we have  $\delta^t(f^*, f^*) = 0$ . In addition, if Assumption C.2 is satisfied, then for any  $f$ ,  $\delta^t(\mathcal{T}f, f) = 0$ .
Proof of Lemma C.10. By the definition of  $f^{*}$ ,
$$
\delta^ {t} \left(f ^ {*}, f ^ {*}\right) = \frac {1}{H} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, f ^ {*}\right) - \ell_ {h, 2} \left(y _ {h} ^ {t}, f ^ {*}\right)\right) ^ {2} - \left(\mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f ^ {*}\right) \mid x _ {h} ^ {t} \right] - \ell_ {h, 2} \left(y _ {h} ^ {t}, f ^ {*}\right)\right) ^ {2} = 0.
$$
Likewise, by the definition of  $\mathcal{T}f$
$$
\delta^ {t} (\mathcal {T} f, f) = \frac {1}{H} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, \mathcal {T} f\right) - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) ^ {2} - \left(\mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right] - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) ^ {2} = 0.
$$
C.3.3 Proof of Lemma C.4
In this section, we prove Lemma C.4, which gives a guarantee for the outer exponential weights update used within Algorithm 5.
Proof of Lemma C.4. The definition of the exponential weights update (in particular, (23)) implies that
$$
\begin{array}{l} - \eta \cdot \min  _ {f} \left(\sum_ {t = 1} ^ {T} \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f)\right) - \log | \mathcal {G} | \\ \leq \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {f \sim p ^ {t}} \exp \left(- \eta \left(\beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f)\right)\right)\right) \\ \leq \frac {1}{3} \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {f \sim p ^ {t}} \exp \left(- 3 \eta \beta t ^ {t} (f)\right)\right) + \frac {1}{3} \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {f \sim p ^ {t}} \exp \left(- 3 \eta \delta^ {t} (f, f)\right)\right) \\ + \frac {1}{3} \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {f \sim p ^ {t}} \exp \left(3 \eta Z ^ {t} (f)\right)\right), \\ \end{array}
$$
where the final inequality holds due to the fact that  $\mathbb{E}[XYZ] \leq \sqrt[3]{\mathbb{E}[X^3]\mathbb{E}[Y^3]\mathbb{E}[Z^3]}$  for positive random variables  $X, Y, Z$  (which in turn can be shown via a repeated application of Hölder's inequality). We further have
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \log \left(\mathbb {E} _ {f \sim p ^ {t}} \exp \left(- 3 \eta \delta^ {t} (f, f)\right)\right) \mid x _ {1: H} ^ {t} \right] \\ \leq \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \exp \left(- 3 \eta \delta^ {t} (f, f)\right) \mid x _ {1: H} ^ {t} \right]\right) (Jensen) \\ \leq \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \exp (- \eta (1 - 6 \eta) \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2}) \mid x _ {1: H} ^ {t} \right]\right) \quad (L e m m a C. 6) \\ \leq \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} (1 - \eta (1 - 6 \eta) \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2}) \mid x _ {1: H} ^ {t} \right]\right) (Lemma C.1) \\ \leq - \frac {\eta}{2} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right]. \\ \end{array}
$$
Meanwhile, using Lemma C.2, we get
$$
\log (\mathbb {E} _ {t - 1} [ \mathbb {E} _ {f \sim p ^ {t}} \exp (- 3 \eta \beta \iota^ {t} (f)) | x _ {1: H} ^ {t} ]) \leq \mathbb {E} _ {t - 1} \big [ \mathbb {E} _ {f \sim p ^ {t}} (- 3 \eta \beta \iota^ {t} (f) + 9 \eta^ {2} \beta^ {2} (\iota^ {t} (f)) ^ {2}) | x _ {1: H} ^ {t} \big ].
$$
Moreover, under Assumption C.1, the benchmark term is negative by
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \min  _ {f} \left(\sum_ {t = 1} ^ {T} \delta^ {t} (f, f) - Z ^ {t} (f) + \beta \iota^ {t} (f)\right) \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \sum_ {t = 1} ^ {T} \delta^ {t} \left(f ^ {*}, f ^ {*}\right) - Z ^ {t} \left(f ^ {*}\right) + \beta \iota^ {t} \left(f ^ {*}\right) \mid x _ {1: H} ^ {t} \right] \\ = - \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ Z ^ {t} \left(f ^ {*}\right) \mid x _ {1: H} ^ {t} \right] \quad (\text {L e m m a C . 1 0 a n d t h e d e f i n i t i o n o f} \iota^ {t}) \\ \leq 0. \tag {Lemma C.8} \\ \end{array}
$$
Furthermore, the log-exponential term is—up to a constant—bounded by its first order expansion:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \log (\mathbb {E} _ {t - 1} [ \mathbb {E} _ {f \sim p _ {t}} \exp (3 \eta Z ^ {t} (f)) \mid x _ {1: H} ^ {t} ]) \\ \leq \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p _ {t}} \exp \left(3 \eta \delta^ {t} (f)\right) \mid x _ {1: H} ^ {t} \right]\right) (Lemma C.9) \\ = \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \exp \left(3 \eta \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right]\right) \\ \leq \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp \left(3 \eta \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right]\right) \quad \text {(b y J e n s e n)} \\ \leq \sum_ {t = 1} ^ {T} \log \left(\mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp \left(3 \eta (1 + 6 \eta) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}\right) \mid x _ {1: H} ^ {t} \right]\right) (Lemma C.6) \\ \leq \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} 6 \eta (1 + 6 \eta) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] (Lemma C.1) \\ \leq 32 \eta \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right]. (Lemma C.8) \\ \end{array}
$$
By using the fact that  $|\iota^t(f)| \leq 1$  and combining the above displays, we obtain that
$$
\begin{array}{l} - \log | \mathcal {G} | \leq - \frac {\eta}{6} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ + \frac {3 2 \eta}{3} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \\ + 3 \eta^ {2} \beta^ {2} - \mathbb {E} _ {t - 1} [ \mathbb {E} _ {f \sim p ^ {t}} [ \eta \beta \iota^ {t} (f) ] ]. \\ \end{array}
$$
Rearranging yields the desired statement.
C.3.4 Proof of Lemma C.5
We state three technical lemmas, Lemmas C.11 to C.13, which are proven in subsequent subsections, then prove Lemma C.5 as a consequence.
Lemma C.11. Almost surely with respect to the draw of  $S^T$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t} (\cdot | S ^ {t - 1})} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \\ \leq \frac {1}{\lambda} \underbrace {\sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {p ^ {t}} \left[ \left(e ^ {- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f , f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \right] \mid x _ {1 : H} ^ {t} \right]} _ {\text {T e r m I}} \\ - \underbrace {\sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {p ^ {t}} \left[ \left(e ^ {- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f , f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) Z ^ {t} (f) \right] \mid x _ {1 : H} ^ {t} \right]} _ {\text {T e r m I I}} + \frac {1}{\lambda} \log | \mathcal {G} |. \\ \end{array}
$$
Lemma C.12. Under Assumption C.2, for any  $0 < \delta < 1$ , with probability at least  $1 - \delta$  over the draw of  $S^T$ , we have for all  $t \leq T$ ,
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left(e ^ {- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \mid x _ {1: H} ^ {t} \right] \\ \leq 32 \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(Z ^ {t} (f) + \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2}\right) \log \left(\left| \mathcal {G} \right| T / \delta\right) \right] \mid x _ {1: H} ^ {t} \right] \\ + \left(2 4 \eta^ {2} \beta^ {2} + 4 \eta \beta\right) \log (| \mathcal {G} | T / \delta). \\ \end{array}
$$
Lemma C.13. For any  $t \leq T$ , the following bound holds almost surely with respect to the draw of  $S^t$ :
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(e ^ {- \eta \left[ \beta_ {\iota} ^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) Z ^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] \\ \leq \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ 1 5 \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 2 8 0 Z ^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] + 4 \eta \beta^ {2}. \\ \end{array}
$$
Proof of Lemma C.5. By Lemma C.11, we have
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \leq \frac {1}{\lambda} \cdot (\text {T e r m I}) - (\text {T e r m I I}) + \frac {1}{\lambda} \log | \mathcal {G} |.
$$
Thus under Assumption C.2, using the bound in Lemma C.12 for Term I and using the bound in Lemma C.13 for Term II, we have with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \\ \leq \frac {1}{\lambda} \cdot \sum_ {t = 1} ^ {T} 3 2 \eta \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(Z ^ {t} (f) + \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2}\right) \log (| \mathcal {G} | T / \delta) \right] \mid x _ {1: H} ^ {t} \right] \\ + (2 4 \eta^ {2} \beta^ {2} + 4 \eta \beta) \log (| \mathcal {G} | T / \delta) T \\ + \eta \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ 1 5 | \mathcal {E} (f, f, x _ {1: H} ^ {t}) | ^ {2} + 2 8 0 Z ^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] \\ + 4 \eta \beta^ {2} T + \frac {1}{\lambda} \log | \mathcal {G} |. \\ \end{array}
$$
Reorganizing the terms, we have
$$
\begin{array}{l} \left(1 - \frac {3 2 \eta}{\lambda} \log (| \mathcal {G} | T / \delta) - 2 8 0 \eta\right) \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \\ \leq \left(\frac {3 2 \eta}{\lambda} \log (| \mathcal {G} | T / \delta) + 1 5 \eta\right) \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ + (2 4 \eta^ {2} \beta^ {2} + 4 \eta \beta) \log (| \mathcal {G} | T / \delta) T + 4 \eta \beta^ {2} T + \frac {1}{\lambda} \log | \mathcal {G} |. \\ \end{array}
$$
Using the assumed bound on  $\eta$  and  $\lambda$  gives
$$
\begin{array}{l} \frac {1}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} [ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} ] \leq \frac {1}{2 ^ {8}} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \right] \mid x _ {1: H} ^ {t} \right] \\ + (2 4 \eta^ {2} \beta^ {2} + 4 \eta \beta) \log (| \mathcal {G} | T / \delta) T + 4 \eta \beta^ {2} T + 8 \log | \mathcal {G} |. \\ \end{array}
$$
C.3.5 Proof of Lemma C.11 (Error Decomposition)
Proof of Lemma C.11. Recalling the relationships  $q^{t + 1}(g \mid f) = q^t(g \mid f) \cdot e^{-\lambda[\delta^t(g, f) - Z^t(f)]}$  and  $p^{t + 1}(f) = p^t(f) \cdot e^{-\eta[\beta t^t(f) + \delta^t(f, f) - Z^t(f) - Z^t]}$ , the proof begins with the following manipulation:
$$
\begin{array}{l} \mathbb {E} _ {f \sim p ^ {t + 1} (\cdot | S ^ {t})} \left[ \log \frac {1}{q ^ {t + 1} (\mathcal {T} f \mid f)} \right] - \mathbb {E} _ {f \sim p ^ {t} (\cdot | S ^ {t - 1})} \left[ \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \right] \\ = \mathbb {E} _ {p ^ {t + 1} - p _ {t}} \left[ \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \right] + \mathbb {E} _ {p ^ {t + 1} - p _ {t}} \left[ \log \frac {q ^ {t} (\mathcal {T} f \mid f)}{q ^ {t + 1} (\mathcal {T} f \mid f)} \right] \\ + \mathbb {E} _ {p _ {t}} \left[ \log \frac {q ^ {t} (\mathcal {T} f \mid f)}{q ^ {t + 1} (\mathcal {T} f \mid f)} \right] \\ = \mathbb {E} _ {p ^ {t}} \left[ \left(e ^ {- \eta \left[ \beta_ {\iota} ^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \right] \\ + \mathbb {E} _ {p _ {t}} \left[ \left(e ^ {- \eta \left[ \beta_ {\iota} ^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) \cdot \lambda \cdot \left(\delta^ {t} (\mathcal {T} f, f) - Z ^ {t} (f)\right) \right] \\ + \lambda \cdot \mathbb {E} _ {p _ {t}} [ \delta^ {t} (\mathcal {T} f, f) - Z ^ {t} (f) ]. \\ \end{array}
$$
Thus, taking expectation with respect to  $\mathbb{E}_{t-1}[\cdot]$ , dividing by  $\lambda$ , rearranging, summing over  $t$  on both sides, and taking advantage of telescoping, we obtain
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t} (\cdot | S ^ {t - 1})} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \\ \leq \frac {1}{\lambda} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {p ^ {t}} \left[ \left(e ^ {- \eta [ \beta t ^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} ]} - 1\right) \log \frac {1}{q ^ {t} (\mathcal {T} f \mid f)} \right] \mid x _ {1: H} ^ {t} \right] \\ + \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {p ^ {t}} \left[ \left(e ^ {- \eta \left[ \beta t ^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1\right) \cdot (\delta^ {t} (\mathcal {T} f, f) - Z ^ {t} (f)) \right] \mid x _ {1: H} ^ {t} \right] \\ + \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {p _ {t}} \left[ \delta^ {t} (\mathcal {T} f, f) \right] \mid x _ {1: H} ^ {t} \right] \\ + \frac {1}{\lambda} \left(\mathbb {E} _ {0} \left[ \mathbb {E} _ {f \sim p ^ {1}} \log \frac {1}{q _ {1} (\mathcal {T} f \mid f)} \mid x _ {1: H} ^ {1} \right] - \mathbb {E} _ {T} \left[ \mathbb {E} _ {f \sim p ^ {T + 1}} \log \frac {1}{q _ {T + 1} (\mathcal {T} f \mid f)} \mid x _ {1: H} ^ {T + 1} \right]\right). \\ \end{array}
$$
Finally, we use that  $\delta^t (\mathcal{T}f,f) = 0$ ,  $\log \frac{1}{q_{T + 1}(\mathcal{T}f|f)}\geq 0$ , and  $\log \frac{1}{q_1(\mathcal{T}f|f)} = \log |\mathcal{G}|.$
C.3.6 Proof of Lemma C.12 (Bound on Term I)
Toward proving Lemma C.12, we state and prove a series of technical lemmas, Lemmas C.14 to C.17.
Lemma C.14. Under Assumption C.2, for any  $0 < \delta < 1$ , the event  $A^t$  defined below holds simultaneously for all  $t \leq T$  with probability at least  $1 - \delta$  over the draw of  $S^T$ :
$$
A ^ {t} = \left\{\sup  _ {f \in \mathcal {G}} \log \frac {1}{q ^ {t + 1} (\mathcal {T} f \mid f)} \leq 2 \log (| \mathcal {G} | T / \delta) \right\}.
$$
Proof of Lemma C.14. Recalling that  $q^{t + 1}(g \mid f) = \frac{\exp\left(-\lambda \sum_{s = 1}^{t} \delta^s(g, f)\right)}{\sum_{g' \in \mathcal{G}} \exp\left(-\lambda \sum_{s = 1}^{t} \delta^s(g', f)\right)}$ , we have
$$
\begin{array}{l} \log \left(\mathbb {E} _ {S ^ {t}} \left[ \sup  _ {f} \frac {1}{q ^ {t + 1} (\mathcal {T} f \mid f)} \right]\right) = \log \left(\mathbb {E} _ {S ^ {t}} \left[ \sup  _ {f} \sum_ {g} \exp \left(- \lambda \sum_ {s = 1} ^ {t} \delta^ {s} (g, f)\right) \right]\right) (Lemma C.10) \\ \leq \log \left(\mathbb {E} _ {S ^ {t}} \left[ \sum_ {f, g} \exp \left(- \lambda \sum_ {s = 1} ^ {t} \delta^ {s} (g, f)\right) \right]\right) \\ = \log \left(\sum_ {f, g} \mathbb {E} _ {S ^ {t}} \left[ \exp \left(- \lambda \sum_ {s = 1} ^ {t} \delta^ {s} (g, f)\right) \right]\right) \\ \leq 2 \log | \mathcal {G} |, (Lemma C.6) \\ \end{array}
$$
where the final inequality uses  $0 < \lambda < 1/2$  as well as the fact that for any fixed choice of  $f, g$ , conditioned on  $x_{1:H}^s$ ,  $s \leq t$ , the random variables  $\delta^s(g, f)$  (for  $s \leq t$ ) are independent. Then, by Markov's inequality and the union bound, we have the desired result.
Lemma C.15. For any  $t \leq T$ ,  $0 \leq \lambda \leq 1/8$ ,  $f$ , almost surely with respect to the draw of  $S^t$  and  $x_{1:H}^t$ , we have
$$
\mathbb {E} _ {t - 1} \left[ \left| Z ^ {t} (f) \right| ^ {2} \mid x _ {1: H} ^ {t} \right] \leq 4 0 \cdot \mathbb {E} _ {t - 1} \left[ Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right].
$$
Proof of Lemma C.15. Using Lemma C.9, we get
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ | Z ^ {t} (f) | ^ {2} \mid x _ {1: H} ^ {t} \right] \leq 2 \mathbb {E} _ {t - 1} \left[ \left(\mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} | \delta^ {t} (g, f) |\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ \leq 2 \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \left| \delta^ {t} (g, f) \right| ^ {2} \mid x _ {1: H} ^ {t} \right] \quad \text {(b y J e n s e n)} \\ \leq 1 0 \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] \quad (Lemma C.7) \\ \leq 4 0 \mathbb {E} _ {t - 1} \left[ Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right]. (Lemma C.8) \\ \end{array}
$$
Lemma C.16. For any  $t \leq T$ , almost surely with respect to the draw of  $S^t$  and  $x_{1:H}^t$ , we have
$$
\mathbb {E} _ {t - 1} \left[ | Z ^ {t} | ^ {2} \mid x _ {1: H} ^ {t} \right] \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t} (\cdot | S ^ {t - 1})} [ 1 0 \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 8 0 Z ^ {t} (f) ] \mid x _ {1: H} ^ {t} \right] + 3 \beta^ {2}.
$$
Proof of Lemma C.16. Using Jensen's inequality, we have
$$
\begin{array}{l} | Z ^ {t} | = \left| - \frac {1}{\eta} \log \left(\mathbb {E} _ {f \sim p ^ {t}} \exp \left(- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) \right]\right)\right) \right| \\ \leq \mathbb {E} _ {f \sim p ^ {t}} [ | \beta t ^ {t} (f) | + | \delta^ {t} (f, f) | + | Z ^ {t} (f) | ]. \\ \end{array}
$$
Thus by Lemma C.7 and Lemma C.15, we have
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ | Z ^ {t} | ^ {2} \mid x _ {1: H} ^ {t} \right] \leq 3 \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ | \beta \iota^ {t} (f) | ^ {2} + | \delta^ {t} (f, f) | ^ {2} + | Z ^ {t} (f) | ^ {2} \right] \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ 1 0 \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 8 0 Z ^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] + 3 \beta^ {2}. \\ \end{array}
$$
Lemma C.17. For any  $0 \leq \eta \leq 1/240$ ,  $f$ , almost surely with respect to the draw of  $S^t$  and  $x_{1:H}^t$ , we have
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ e ^ {- \eta \left[ \beta_ {\iota} ^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1 \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ 1 6 \eta Z ^ {t} (f) + 4 \eta \mathbb {E} _ {f ^ {\prime} \sim p ^ {t} (\cdot | S ^ {t - 1})} \mathcal {E} \left(f ^ {\prime}, f ^ {\prime}, x _ {1: H} ^ {t}\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ + 1 2 \eta^ {2} \beta^ {2} + 2 \eta \beta . \\ \end{array}
$$
Proof of Lemma C.17. Jensen's inequality gives that
$$
Z ^ {t} \leq \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} \left[ \beta \iota^ {t} \left(f ^ {\prime}\right) + \delta^ {t} \left(f ^ {\prime}, f ^ {\prime}\right) - Z ^ {t} \left(f ^ {\prime}\right) \right]
$$
$$
Z ^ {t} (f) \leq \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} [ \delta^ {t} (g, f) ] = \delta^ {t} (f).
$$
We may then write
$$
\begin{array}{l} \mathbb {E} _ {t - 1} \left[ e ^ {- \eta \left[ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} \right]} - 1 \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} _ {t - 1} \left[ e ^ {- \eta \beta \iota^ {t} (f) - \eta \delta^ {t} (f, f) + \eta \delta^ {t} (f) + \eta \mathbb {E} _ {f ^ {\prime} \sim_ {p} t} [ \beta \iota^ {t} (f ^ {\prime}) + \delta^ {t} (f ^ {\prime}, f ^ {\prime}) - Z ^ {t} (f ^ {\prime}) ]} - 1 \mid x _ {1: H} ^ {t} \right] \\ \leq \frac {1}{6} \mathbb {E} _ {t - 1} \Big [ \mathbb {E} _ {f ^ {\prime} \sim p ^ {t}} \Big [ e ^ {- 6 \eta \beta_ {\iota} t (f)} + e ^ {- 6 \eta \delta^ {t} (f, f)} + e ^ {6 \eta \delta^ {t} (f)} + e ^ {6 \eta \beta_ {\iota} t (f ^ {\prime})} + e ^ {6 \eta \delta^ {t} (f ^ {\prime}, f ^ {\prime})} + e ^ {- 6 \eta Z ^ {t} (f ^ {\prime})} - 6 \Big ] \mid x _ {1: H} ^ {t} \Big ], \\ \end{array}
$$
where the second equality uses Jensen's inequality to pull the  $\mathbb{E}_{f'\sim p^t}[\cdot ]$  outside of the exponential and then the fact that  $a_1\dots a_6\leq \frac{1}{6}\sum_{i = 1}^{6}a_i^6$  for real numbers  $a_1,\ldots ,a_6$ . We control the six terms on the right hand side separately as follows:
- Term (a): Using Lemma C.2 we get
$$
\mathbb {E} _ {t - 1} \left[ e ^ {- 6 \eta \beta \iota^ {t} (f)} \mid x _ {1: H} ^ {t} \right] \leq 1 + 6 \eta \beta + 3 6 \eta^ {2} \beta^ {2}.
$$
- Term (b): Using Lemma C.6 we get
$$
\mathbb {E} _ {t - 1} \left[ e ^ {- 6 \eta \delta^ {t} (f, f)} \mid x _ {1: H} ^ {t} \right] \leq 1.
$$
- Term (c):