SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Lemma C.2. Let  $f(x) = e^{-x} - 1 + x - x^2$ . We have  $f''(x) = e^{-x} - 2 < 0$  for  $x \geq -1/8$ . Thus,  $f'$  is monotonically decreasing on  $x \geq -1/8$ , so for  $x \in [-1/8, 0]$ ,  $f'(x) \geq f'(0) = 0$ . Hence,  $f(x)$  is non-decreasing on  $x \in [-1/8, 0]$ . Furthermore,  $f'(x) = -e^{-x} + 1 - 2x \leq 0$  for  $x \geq 0$ . Thus  $f(x)$  obtains maximum value at  $x = 0$ , and  $f(x) \leq f(0) = 0$ .
C.2 Basic Online Learning Results
In this section we state a technical lemma regarding the performance of the exponential weights algorithm for online learning. Let  $\mathcal{G}$  be an abstract set of hypotheses. We consider the following online learning process.
- Learner predicts a (random) hypothesis  $g^{t} \in G$ .
- Nature reveals  $\ell^t \in \mathcal{L} \coloneqq (\mathcal{G} \to \mathbb{R})$  and learner suffers loss  $\ell^t(g^t)$ .
We define regret to the class  $\mathcal{G}$  via
$$
\operatorname {R e g} _ {\mathrm {O L}} = \sum_ {t = 1} ^ {T} \mathbb {E} _ {g ^ {t} \sim \mu^ {t}} \left[ \ell^ {t} \left(g ^ {t}\right) \right] - \inf  _ {g \in \mathcal {G}} \sum_ {t = 1} ^ {T} \ell^ {t} (g), \tag {17}
$$
where  $\mu^t\in \Delta (\mathcal{G})$  is the learner's randomization distribution for step  $t$
Lemma C.3. Consider the exponential weights update method with learning rate  $\eta > 0$ , which sets
$$
\mu^ {t} (g) \propto \exp \left(- \eta \sum_ {i <   t} \ell^ {i} (g)\right).
$$
For any sequence of non-negative loss functions  $\ell^1, \ldots, \ell^T$ , this algorithm satisfies
$$
\operatorname {R e g} _ {\mathrm {O L}} \leq \frac {\eta}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {g ^ {t} \sim \mu^ {t}} \left[ \left(\ell^ {t} \left(g ^ {t}\right)\right) ^ {2} \right] + \frac {\log | \mathcal {G} |}{\eta}. \tag {18}
$$
In addition, for any sequence of loss functions  $\ell^1,\ldots ,\ell^T$  with  $\ell^t (g)\in [-L,L]$  for all  $g\in \mathcal{G}$ , if  $\eta \leq (2L)^{-1}$ , then
$$
\operatorname {R e g} _ {\mathrm {O L}} \leq 2 \eta \sum_ {t = 1} ^ {T} \mathbb {E} _ {g ^ {t} \sim \mu^ {t}} \left[ \left(\ell^ {t} \left(g ^ {t}\right) - \mathbb {E} _ {g ^ {\prime} \sim \mu^ {t}} \left[ \ell^ {t} \left(g ^ {\prime}\right) \right]\right) ^ {2} \right] + \frac {\log | \mathcal {G} |}{\eta} \leq 4 \eta \sum_ {t = 1} ^ {T} \mathbb {E} _ {g ^ {t} \sim \mu^ {t}} \left[ \left(\ell^ {t} \left(g ^ {t}\right)\right) ^ {2} \right] + \frac {\log | \mathcal {G} |}{\eta} \tag {19}
$$
Proof of Lemma C.3. A standard telescoping argument combined with the fact that  $-\inf_{g\in \mathcal{G}}\sum_{t = 1}^{T}\ell^{t}(g)\leq \frac{1}{\eta}\log \left(\exp \left(\sum_{g\in \mathcal{G}} - \eta \sum_{t = 1}^{T}\ell^{t}(g)\right)\right)$  (e.g., Cesa-Bianchi and Lugosi [4]) gives that for any choice  $\eta >0$  and any sequence of loss functions, exponential weights has
$$
\begin{array}{l} \operatorname {R e g} _ {\mathrm {O L}} \leq \sum_ {t = 1} ^ {T} \mathbb {E} _ {g \sim \mu^ {t}} [ \ell^ {t} (g) ] + \frac {1}{\eta} \sum_ {t = 1} ^ {T} \log \left(\sum_ {g \in \mathcal {G}} \mu^ {t} (g) \exp (- \eta \ell^ {t} (g))\right) + \frac {\log | \mathcal {G} |}{\eta} (20) \\ = \frac {1}{\eta} \sum_ {t = 1} ^ {T} \log \left(\sum_ {g \in \mathcal {G}} \mu^ {t} (g) \exp \left(- \eta \left(\ell^ {t} (g) - \mathbb {E} _ {g ^ {\prime} \sim \mu^ {t}} \left[ \ell^ {t} \left(g ^ {\prime}\right) \right]\right)\right)\right) + \frac {\log | \mathcal {G} |}{\eta}. (21) \\ \end{array}
$$
We first prove (18). Using that  $\log(x) \leq x - 1$  for  $x \geq 0$  and  $\exp(-x) \leq 1 - x + \frac{x^2}{2}$  for  $x \geq 0$ , we have
$$
\log \left(\sum_ {g \in \mathcal {G}} \mu^ {t} (g) \exp (- \eta \ell^ {t} (g))\right) \leq - \eta \mathbb {E} _ {g \sim \mu^ {t}} [ \ell^ {t} (g) ] + \frac {\eta^ {2}}{2} \mathbb {E} _ {g \sim \mu^ {t}} \left[ (\ell^ {t} (g)) ^ {2} \right],
$$
$$
\mathbf {R e g} _ {\mathrm {O L}} \leq \frac {\eta}{2} \sum_ {t = 1} ^ {T} \mathbb {E} _ {g \sim \mu^ {t}} \left[ (\ell^ {t} (g)) ^ {2} \right] + \frac {\log | \mathcal {G} |}{\eta}.
$$
To prove (19), we use that  $\log(x) \leq x - 1$  for  $x \geq 0$  and  $\exp(-x) \leq 1 - x + 2x^2$  whenever  $|x| \leq 1$  to get
$$
\log \left(\sum_ {g \in \mathcal {G}} \mu^ {t} (g) \exp (- \eta (\ell^ {t} (g) - \mathbb {E} _ {g ^ {\prime} \sim \mu^ {t}} [ \ell^ {t} (g ^ {\prime}) ]))\right) \leq 2 \eta^ {2} \mathbb {E} _ {g \sim \mu^ {t}} \left[ (\ell^ {t} (g) - \mathbb {E} _ {g ^ {\prime} \sim \mu^ {t}} [ \ell^ {t} (g ^ {\prime}) ]) ^ {2} \right],
$$
$$
\mathbf {R e g} _ {\mathrm {O L}} \leq 2 \eta \sum_ {t = 1} ^ {T} \mathbb {E} _ {g \sim \mu^ {t}} \left[ \left(\ell^ {t} (g) - \mathbb {E} _ {g ^ {\prime} \sim \mu^ {t}} [ \ell^ {t} (g ^ {\prime}) ]\right) ^ {2} \right] + \frac {\log | \mathcal {G} |}{\eta}.
$$
C.3 Online Learning with Completeness
In this section, we give guarantees for an online learning algorithm Algorithm 5, which generalizes the two-timescale exponential weights algorithm (Algorithm 4) of Agarwal and Zhang [2]. We describe and analyze the algorithm in a general online learning framework, which abstracts away the core problem solved by Algorithm 4: value function estimation using a Bellman complete value function class.
Let  $\mathcal{G}$  be an abstract set of hypotheses. We consider and online learning process parameterized by a positive integer  $H$  and  $\alpha \in [0,1]$ .
- There are  $2H + 1$  outcome spaces,  $\{\mathcal{X}_h\}_{h\in [H]}$ ,  $\{\mathcal{Y}_h\}_{h\in [H]}$ , and  $\mathcal{W}$ .
- There are  $H$  unknown probability kernels  $\{\mathcal{K}_h:\mathcal{X}_h\to \mathcal{Y}_h\}_{h\in [H]}$
- There are  $2H + 1$  known loss functions  $\{\ell_{h,1}:\mathcal{X}_h\times \mathcal{G}\to [0,1]\}_{h\in [H]}$ ,  $\{\ell_{h,2}:\mathcal{Y}_h\times \mathcal{G}\to [0,1]\}_{h\in H}$ , and  $l_{3}:\mathcal{W}\times \mathcal{G}\to [0,1]$ .
Define  $S^0 = \varnothing$ . We consider the following process. For  $t = 1, \dots, T$ :
- Learner predicts a (randomized) hypothesis  $f^t \in \mathcal{G}$ .
- Nature reveals  $\{x_h^t, y_h^t\}_{h \in [H]}$  and  $w^t$ . The outcomes  $\{x_h^t\}_{h \in [H]}$  can be chosen adaptively, but they are mutually independent given  $S^{t-1}$ . The outcome  $w^t$  can be chosen adaptively based on  $S^{t-1}$  and  $\{x_h^t\}_{h \in [H]}$ . Each outcome  $y_h^t \sim \mathcal{K}_h(x_h^t)$  is drawn independently for each  $h \in [H]$ .
- The history is updated via  $S^t \gets S^{t-1} \bigcup \{x_h^t, y_h^t\}_{h \in [H]} \bigcup \{w^t\}$ .
- The learner suffers loss
$$
\frac {1}{H} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, f ^ {t}\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f ^ {t}\right) \mid x _ {h} ^ {t} \right]\right) ^ {2} + \alpha l _ {3} \left(w ^ {t}, f ^ {t}\right). \tag {22}
$$
The learner's goal is to minimize a form of regret for the cumulative loss given in (22). This loss function reflects two objective. The first objective involves the  $H$  losses  $\{\ell_{1,h}\}_{h\in [H]}$  with corresponding outcomes  $\{x_{h}\}_{h\in [H]}$ , as well as the  $H$  losses  $\{\ell_{2,h}\}_{h\in [H]}$  tied to outcomes  $\{y_{h}\}_{h\in [H]}$  which are generated stochastically based on  $\{x_{h}\}_{h\in [H]}$ . The primary objective is to minimize the primary error
$$
\frac {1}{H} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, f ^ {t}\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f ^ {t}\right) \mid x _ {h} ^ {t} \right]\right) ^ {2}
$$
The secondary objective is to minimize  $\sum_{t=1}^{T} l_3(w^t, f^t)$ , and the ultimate goal is to minimize a weighted sum of the two objectives.
This online learning setup, adapted from Agarwal and Zhang [2], generalizes the reinforcement learning setting in which Algorithm 4 operates. The adaptively chosen outcome  $x_h^t$  corresponds to the state-action pair at the  $h$ -th step,  $(s_h^t, a_h^t)$ , with the policy at time  $t$  chosen in an adaptive, potentially adversarial fashion. The conditionally stochastic outcome  $y_h^t$  corresponds to the reward and the next state  $(r_h^t, s_{h+1}^t)$ , which is sampled independently from the MDP's reward distribution and transition distribution at step  $h$ , and is conditionally independent given  $(s_h^t, a_h^t)$ . The learner's objective in the RL framework is to predict a value function  $f^t = Q^t$  that minimizes the squared Bellman error, realized by selecting the losses  $\ell_{1,h}(x_h^t, f^t) = Q_h^t(s_h^t, a_h^t)$  and  $\ell_{2,h}(y_h^t, f^t) = r_h^t + \max_a Q_{h+1}^t(s_{h+1}^t, a)$ . The secondary objective is to predict the value function optimistically, with  $\ell_3(w^t, f^t) = -\max_a Q(s_1^t, a)$  and  $w^t = s_1^t$ .
Algorithm 5 Two-Timescale Exponential Weights (adapted from Agarwal and Zhang [2])
1: Initialize  $S_0 \gets \varnothing$ .
2: for  $t = 1,2,\ldots ,T$  do
3: For all  $f, g \in \mathcal{G}$ , define
$$
\Delta_ {h} ^ {t} (g, f) := \ell_ {h, 1} \left(x _ {h} ^ {t}, g\right) - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right),
$$
$$
q ^ {t} (g \mid f) := q ^ {t} (g \mid f, S ^ {t - 1}) \propto \exp \left(- \lambda \cdot \frac {1}{H} \sum_ {s = 1} ^ {t - 1} \sum_ {h = 1} ^ {H} \Delta_ {h} ^ {s} (g, f) ^ {2}\right),
$$
$$
L ^ {t} (f) := \frac {1}{H} \sum_ {h = 1} ^ {H} \Delta_ {h} ^ {t} (f, f) ^ {2} + \frac {1}{\lambda} \log \left(\mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \left[ \exp \left(- \lambda \cdot \frac {1}{H} \sum_ {h = 1} ^ {H} \Delta_ {h} ^ {t} (g, f) ^ {2}\right) \right]\right),
$$
$$
p ^ {t} (f) := p ^ {t} (f \mid S ^ {t - 1}) \propto \exp \left(- \eta \sum_ {s = 1} ^ {t - 1} \left(\beta l _ {3} \left(w ^ {s}, f\right) + L ^ {s} (f)\right)\right).
$$
4: Sample and predict  $f^t \sim p^t$ .
5: Observe  $\{x_h^t, y_h^t\}_{h \in [H]}, w^t$ . and update  $S_t \gets S_{t-1} \bigcup \{x_h^t, y_h^t\}_{h \in [H]} \bigcup \{w^t\}$ .
To analyze Algorithm 5, we make a generalized realizability assumption and a generalized completeness assumption; these assumptions abstract away the notions of realizability and completeness in RL.
Assumption C.1 (Realizability for online learning). There exists  $f^{*} \in \mathcal{G}$  such that for all  $h \in [H]$  and  $x_{h} \in \mathcal{X}_{h}$ , we have
$$
\ell_ {h, 1} \left(x _ {h}, f ^ {*}\right) = \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f ^ {*}\right) \mid x _ {h} \right].
$$
Assumption C.2 (Completeness for online learning). For any  $f \in \mathcal{G}$ , there exists  $g \in \mathcal{G}$  such that for all  $h \in [H]$  and  $x_h \in \mathcal{X}_h$ , we have
$$
\ell_ {h, 1} \left(x _ {h}, g\right) = \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} \right].
$$
For any  $f \in \mathcal{G}$ , we denote the corresponding  $g \in \mathcal{G}$  satisfying this property by  $g = \mathcal{T}f$ .
For functions  $g$  and  $f$  and outcome  $x_{h}$ , define
$$
\mathcal {E} _ {h} (g, f, x _ {h}) := \ell_ {h, 1} \left(x _ {h}, g\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} \right];
$$
this quantity generalizes the notion of Bellman error for reinforcement learning. Recalling that  $\Delta_h^t (f,g)\coloneqq \ell_{h,1}(x_h^t,f) - \ell_{h,2}(y_h^t,g)$ , it follows immediately
$$
\mathbb {E} \left[ \Delta_ {h} ^ {t} (g, f) \mid x _ {h} ^ {t} \right] = \mathbb {E} \left[ \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, g\right) - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) \mid x _ {h} ^ {t} \right] = \mathcal {E} _ {h} (g, f, x _ {h} ^ {t}).
$$
In addition, let us define
$$
\mathcal {E} (g, f, x _ {1: H}) ^ {2} := \frac {1}{H} \sum_ {h = 1} ^ {H} \left(\mathcal {E} _ {h} (g, f, x _ {h})\right) ^ {2},
$$
$$
\iota^ {t} (f) := l _ {3} \left(w ^ {t}, f\right) - l _ {3} \left(w ^ {t}, f ^ {*}\right).
$$
The following result is the main theorem concerning the performance of Algorithm 5.
Theorem C.1. Let  $\lambda = 1/8$ ,  $\eta < 1/(2^{16}(\log(|\mathcal{G}|T/\delta) + 1))$ , and  $0 < \beta < 1$ . Under Assumption C.1 and Assumption C.2, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \frac {1}{H} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, f\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right]\right) ^ {2} + 1 2 \beta l _ {3} \left(w ^ {t}, f\right) \right] \mid x _ {h} ^ {t} \right] \\ - \left(\frac {1}{H} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, f ^ {*}\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f ^ {*}\right) \mid x _ {h} ^ {t} \right]\right) ^ {2} + 1 2 \beta l _ {3} \left(w ^ {t}, f ^ {*}\right)\right) \\ = \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 1 2 \beta \iota^ {t} (f) \right] \mid x _ {1: H} ^ {t} \right] \\ \leq 2 ^ {1 6} \left(\eta \beta \log \left(| \mathcal {G} | T / \delta\right) T + \log \left(| \mathcal {G} |\right) / \eta + \beta^ {2} T\right). \\ \end{array}
$$
C.3.1 Proof of Theorem C.1
For our analysis, it will be useful to consider the following offset version of the loss:
$$
\delta_ {h} ^ {t} (g, f) := \Delta_ {h} ^ {t} (g, f) ^ {2} - (\mathbb {E} [ \ell_ {h, 2} (y _ {h}, f) \mid x _ {h} ^ {t} ] - \ell_ {h, 2} (y _ {h} ^ {t}, f)) ^ {2}.
$$
We use  $x_{1:H}$  as a shorthand for  $\{x_h\}_{h\in [H]}$  and further define
$$
\begin{array}{l} \delta^ {t} (g, f) := \frac {1}{H} \sum_ {h = 1} ^ {H} \delta_ {h} ^ {t} (g, f), \\ \delta^ {t} (f) := \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \delta^ {t} (g, f), \\ Z ^ {t} (f) := - \frac {1}{\lambda} \log \mathbb {E} _ {g \sim q ^ {t} (\cdot | f)} \exp (- \lambda \delta^ {t} (g, f)), \\ Z ^ {t} = - \frac {1}{\eta} \log \mathbb {E} _ {f \sim p ^ {t}} \exp \left(- \eta \cdot \left[ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) \right]\right). \\ \end{array}
$$
Recall from Algorithm 5 that
Thus, we can verify the following relationships:
$$
q ^ {t} (g \mid f) = \frac {\exp \left(- \lambda \sum_ {s = 1} ^ {t - 1} \delta^ {s} (g , f)\right)}{\sum_ {g ^ {\prime} \in \mathcal {G}} \exp \left(- \lambda \sum_ {s = 1} ^ {t - 1} \delta^ {s} (g ^ {\prime} , f)\right)},
$$
$$
p ^ {t} (f) = \frac {\exp \left(- \eta \sum_ {s = 1} ^ {t - 1} \left[ \beta \iota^ {s} (f) + \delta^ {s} (f , f) - Z ^ {s} (f) \right]\right)}{\sum_ {f ^ {\prime} \in \mathcal {G}} \exp \left(- \eta \sum_ {s = 1} ^ {t - 1} \left[ \beta \iota^ {s} \left(f ^ {\prime}\right) + \delta^ {s} \left(f ^ {\prime} , f ^ {\prime}\right) - Z ^ {s} \left(f ^ {\prime}\right) \right]\right)}, \tag {23}
$$
$$
q ^ {t + 1} (g \mid f) = q ^ {t} (g \mid f) \cdot e ^ {- \lambda [ \delta^ {t} (g, f) - Z ^ {t} (f) ]},
$$
$$
p ^ {t + 1} (f) = p ^ {t} (f) \cdot e ^ {- \eta [ \beta \iota^ {t} (f) + \delta^ {t} (f, f) - Z ^ {t} (f) - Z ^ {t} ]}.
$$
In what follows, we use  $\mathbb{E}_{t - 1}[\cdot ]$  to abbreviate  $\mathbb{E}[\cdot |\mathfrak{F}^{t - 1}]$  , where  $\mathfrak{F}^{t - 1}\coloneqq \sigma (S^{t - 1})$
Proof of Theorem C.1. Under Assumption C.1,  $(\ell_{h,1}(x_h^t,f^*) - \mathbb{E}[\ell_{h,2}(y_h,f^*)\mid x_h^t])^2 = 0$ . Thus, the first equality holds by definition as
$$
\begin{array}{l} \frac {1}{H} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ (\ell_ {h, 1} (x _ {h} ^ {t}, f) - \mathbb {E} [ \ell_ {h, 2} (y _ {h}, f) \mid x _ {h} ^ {t} ]) ^ {2} + 1 2 \beta l _ {3} (w ^ {t}, f) \right] \mid x _ {h} ^ {t} \right] \\ - \left(\frac {1}{H} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, f ^ {*}\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f ^ {*}\right) \mid x _ {h} ^ {t} \right]\right) ^ {2} + 1 2 \beta l _ {3} \left(w ^ {t}, f ^ {*}\right)\right) \\ = \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left(\mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 1 2 \beta \iota^ {t} (f)\right) \mid x _ {1: H} ^ {t} \right]. \\ \end{array}
$$
To prove the result, we appeal to two technical lemmas, Lemma C.4 and Lemma C.5, stated below. Plugging the bound from Lemma C.5 into Lemma C.4, we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] + 6 \beta \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \iota (f) \mid x _ {1: H} ^ {t} \right] \\ \leq 64 \left(\frac {1}{1 2 8} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left[ \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \right] \mid x _ {1: H} ^ {t} \right] + (4 8 \eta^ {2} \beta^ {2} + 8 \eta \beta) \log (| \mathcal {G} | T / \delta) T + 8 \eta \beta^ {2} T + 1 6 \log | \mathcal {G} |\right) \\ + \frac {6}{\eta} \log | \mathcal {G} | + 1 8 \beta^ {2} T. \\ \end{array}
$$
Rearranging, we obtain
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \left(\mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} + 1 2 \beta \iota^ {t} (f)\right) \mid x _ {1: H} ^ {t} \right] \leq 2 ^ {1 6} (\eta \beta \log (| \mathcal {G} | T / \delta) T + \log (| \mathcal {G} |) / \eta + \beta^ {2} T).
$$
We now state the technical lemmas, Lemma C.4 and Lemma C.5, used in the proof above.
Lemma C.4. Under Assumption C.1, for any  $0 \leq \eta \leq 1/24$  and  $0 < \beta < 1$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathcal {E} (f, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right] + 6 \beta \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \iota^ {t} (f) \mid x _ {1: H} ^ {t} \right] \\ \leq 6 4 \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] + 1 8 \eta \beta^ {2} T + \frac {6}{\eta} \log | \mathcal {G} |. \\ \end{array}
$$
Lemma C.5. Let  $\lambda = 1/8$  and  $\eta < 1/(2^{16}(\log(|\mathcal{G}|T/\delta) + 1))$ . Under Assumption C.2, for any  $0 < \delta < 1$ , with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \leq \frac {1}{1 2 8} \sum_ {t = 1} ^ {T} \mathbb {E} _ {t - 1} \left[ \mathbb {E} _ {f \sim p ^ {t}} \mathcal {E} \left(f, f, x _ {1: H} ^ {t}\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ + \left(4 8 \eta^ {2} \beta^ {2} + 8 \eta \beta\right) \log (| \mathcal {G} | T / \delta) T \\ + 8 \eta \beta^ {2} T + 1 6 \log | \mathcal {G} |. \\ \end{array}
$$
The remainder of the proof is organized as follows.
- Appendix C.3.2 presents basic technical lemmas.
- Appendix C.3.3 presents the proof of Lemma C.4.
- Appendix C.3.4 presents the proof of Lemma C.5.
- Appendices C.3.6 and C.3.7 contain additional technical lemmas used in the proof of Lemma C.5.
C.3.2 Basic properties
In this section, we present basic technical results that will be used within the proof of Lemma C.5 and Lemma C.4.
In this section, we will present some basic properties of  $\delta^t (g,f)$ ,  $\mathcal{E}(g,f,x_{1:H}^t)^2$  and  $Z^{t}(f)$  for any  $t\in [T],g,f\in \mathcal{G},x_{1:H}^{t}\in \bigcup_{h\in [H]}\mathcal{X}_{h}$ . These properties are mainly due to a sub-Gaussian term in the definition of  $\delta^t (g,f)$ . Recall
$$
\begin{array}{l} \delta^ {t} (g, f) = \frac {1}{H} \sum_ {h = 1} ^ {H} \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, g\right) - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) ^ {2} - \left(\mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right] - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) ^ {2} \\ = \frac {1}{H} \sum_ {h = 1} ^ {H} \left(\left(\ell_ {h, 1} \left(x _ {h} ^ {t}, g\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right]\right) ^ {2} \right. \\ + 2 \left(\mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right] - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) \left(\ell_ {h, 1} \left(x _ {h} ^ {t}, g\right) - \mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right]\right)) \\ = \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} + \frac {1}{H} \sum_ {h = 1} ^ {H} 2 (\mathbb {E} [ \ell_ {h, 2} (y _ {h}, f) \mid x _ {h} ^ {t} ] - \ell_ {h, 2} (y _ {h} ^ {t}, f)) \mathcal {E} _ {h} (g, f, x _ {h} ^ {t}). \\ \end{array}
$$
Note that the quantity  $\mathbb{E}[\ell_{h,2}(y_h,f)\mid x_h^t ] - \ell_{h,2}(y_h^t,f)$ , conditioned on  $x_{h}^{t}$ , is a mean 0 random variable bounded by  $[-1,1]$ . Thus, we have
$$
\mathbb {E} \left[ \delta^ {t} (g, f) \mid x _ {1: H} ^ {t} \right] = \mathbb {E} \left[ \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right]. \tag {24}
$$
Furthermore, since  $\mathbb{E}[\ell_{h,2}(y_h,f)\mid x_h^t ] - \ell_{h,2}(y_h^t,f)$  is sub-Gaussian with variance proxy  $1 / 2$  by Hoeffding's inequality, we have the following three lemmas.
Lemma C.6. For any  $t \leq T$ ,  $g, f \in \mathcal{G}$  and  $c \in \mathbb{R}$ , we have
$$
\mathbb {E} \left[ \exp \left(- c \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right] \leq \mathbb {E} \left[ \exp \left(- c (1 - 2 c) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}\right) \mid x _ {1: H} ^ {t} \right].
$$
In addition, for any  $0 < c < \frac{1}{2}$ , we have  $\mathbb{E}[\exp (-c\delta^t (g,f))\mid x_{1:H}^t ]\leq 1$
Proof of Lemma C.6. By the 1/2-sub-Gaussianity of  $\mathbb{E}[\ell_{h,2}(y_h,f)\mid x_h^t ] - \ell_{h,2}(y_h^t,f)$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(- c \delta^ {t} (g, f)\right) \mid x _ {1: H} ^ {t} \right] \\ = \mathbb {E} \left[ \exp (- c \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}) \mid x _ {1: H} ^ {t} \right] \mathbb {E} \left[ \exp \left(- \frac {2 c}{H} \sum_ {h = 1} ^ {H} (\mathbb {E} [ \ell_ {h, 2} (y _ {h}, f) \mid x _ {h} ^ {t} ] - \ell_ {h, 2} (y _ {h} ^ {t}, f)) \mathcal {E} _ {h} (g, f, x _ {h} ^ {t})\right) \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} \left[ \exp \left(- c (1 - 2 c / H) \mathcal {E} \left(g, f, x _ {1: H} ^ {t}\right) ^ {2}\right) \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} \left[ \exp \left(- c (1 - 2 c) \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2}\right) \mid x _ {1: H} ^ {t} \right]. \\ \end{array}
$$
Lemma C.7. For any  $t \leq T, g, f$  and  $x_{1:H}^t$ , we have  $\mathbb{E}\big[\delta^t(g,f)^2 \mid x_{1:H}^t\big] \leq 5\mathbb{E}\big[\mathcal{E}(g,f,x_{1:H}^t)^2 \mid x_{1:H}^t\big]$ .
Proof of Lemma C.7. The result follows by writing
$$
\begin{array}{l} \mathbb {E} \left[ \delta^ {t} (g, f) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ = \mathbb {E} \left[ \left(\mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} + \frac {1}{H} \sum_ {h = 1} ^ {H} 2 \left(\mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right] - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) \mathcal {E} _ {h} (g, f, x _ {h} ^ {t})\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ = \mathbb {E} \left[ \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {4} + \frac {4}{H ^ {2}} \left(\sum_ {h = 1} ^ {H} (\mathbb {E} [ \ell_ {h, 2} (y _ {h}, f) \mid x _ {h} ^ {t} ] - \ell_ {h, 2} (y _ {h} ^ {t}, f)) \mathcal {E} _ {h} (g, f, x _ {h} ^ {t})\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ \leq \mathbb {E} \left[ \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} + \frac {4}{H ^ {2}} \sum_ {h = 1} ^ {H} \left(\left(\mathbb {E} \left[ \ell_ {h, 2} \left(y _ {h}, f\right) \mid x _ {h} ^ {t} \right] - \ell_ {h, 2} \left(y _ {h} ^ {t}, f\right)\right) \mathcal {E} _ {h} \left(g, f, x _ {h} ^ {t}\right)\right) ^ {2} \mid x _ {1: H} ^ {t} \right] \\ \leq 5 \mathbb {E} \left[ \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \mid x _ {1: H} ^ {t} \right], \\ \end{array}
$$
where the first equality is by definition, the second equality is by expanding the terms and notice that the cross terms have zero mean, the first inequality is obtained using the fact that  $|\mathcal{E}_h(g,f,x_h)|\leq 1$  and that the cross terms of the expansion of the second term have zero mean and the final inequality is by  $|\mathbb{E}[\ell_{h,2}(y_h,f)\mid x_h^t ] - \ell_{h,2}(y_h^t,f)|\leq 1.$
Lemma C.8. For all  $t \leq T$ , as long as  $0 \leq \lambda \leq 1/8$ , we have that for all  $f$ ,
$$
4 \mathbb {E} _ {t - 1} \left[ Z ^ {t} (f) \mid x _ {1: H} ^ {t} \right] \geq \mathbb {E} _ {g \sim q ^ {t} (\cdot | f, S ^ {t - 1})} \left[ \mathcal {E} (g, f, x _ {1: H} ^ {t}) ^ {2} \right] \geq 0
$$