SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Proposition 1. We first provide an excess risk bound for the model  $\widehat{\mathbf{w}}_t$  at each time  $t$  and then combine them for  $T$  rounds. For each time  $t\in [T]$ , let  $\widetilde{R}_t(\mathbf{w}) = \mathbb{E}_{(\mathbf{x},y)\sim S_0}\left[r_t^* (\mathbf{x})\ell (\mathbf{w}^\top \mathbf{x},y)\right]$  be the risk built upon the true density ratio function  $r_t^*$  and  $\widehat{R}_t(\mathbf{w}) = \mathbb{E}_{(\mathbf{x},y)\sim S_0}\left[\widehat{r}_t(\mathbf{x})\ell (\mathbf{w}^\top \mathbf{x},y)\right]$  be that built upon the empirical density ratio function  $\widehat{r}_t$ . We can decompose the excess risk of the trained model  $\widehat{\mathbf{w}}_t$  as
$$
R _ {t} (\widehat {\mathbf {w}} _ {t}) - R _ {t} (\mathbf {w} _ {t} ^ {*}) = \underbrace {R _ {t} (\widehat {\mathbf {w}} _ {t}) - \widetilde {R} _ {t} (\widehat {\mathbf {w}} _ {t})} _ {\text {t e r m (a)}} + \underbrace {\widetilde {R} _ {t} (\widehat {\mathbf {w}} _ {t}) - \widetilde {R} _ {t} (\mathbf {w} _ {t} ^ {*})} _ {\text {t e r m (b)}} + \underbrace {\widetilde {R} _ {t} (\mathbf {w} _ {t} ^ {*}) - R _ {t} (\mathbf {w} _ {t} ^ {*})} _ {\text {t e r m (c)}}.
$$
For term (a), by a standard generalization error analysis as shown in Lemma 2, with probability at least  $1 - \delta / (2T)$ , the gap between  $R_{t}(\widehat{\mathbf{w}}_{t})$  and  $\widetilde{R}_{t}(\widehat{\mathbf{w}}_{t})$  is bounded by
$$
\mathbf {t e r m} (\mathbf {a}) = R _ {t} \left(\widehat {\mathbf {w}} _ {t}\right) - \widetilde {R} _ {t} \left(\widehat {\mathbf {w}} _ {t}\right) \leq \frac {2 B G D R}{\sqrt {N _ {0}}} + 4 B L \sqrt {\frac {2 \ln \left((8 T) / \delta\right)}{N _ {0}}}.
$$
Then, we proceed to bound term (b).
$$
\begin{array}{l} \operatorname {t e r m} (\mathbf {b}) = \widetilde {R} _ {t} \left(\widehat {\mathbf {w}} _ {t}\right) - \widetilde {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) \\ = \widetilde {R} _ {t} (\widehat {\mathbf {w}} _ {t}) - \widehat {R} _ {t} (\widehat {\mathbf {w}} _ {t}) + \widehat {R} _ {t} (\widehat {\mathbf {w}} _ {t}) - \widehat {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) + \widehat {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) - \widetilde {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) \\ \leq \widetilde {R} _ {t} (\widehat {\mathbf {w}} _ {t}) - \widehat {R} _ {t} (\widehat {\mathbf {w}} _ {t}) + \widehat {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) - \widetilde {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) \\ = \mathbb {E} _ {(\mathbf {x}, y) \sim S _ {0}} \left[ \left(r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x})\right) \ell \left(\widehat {\mathbf {w}} _ {t} ^ {\top} \mathbf {x}, y\right) \right] + \mathbb {E} _ {(\mathbf {x}, y) \sim S _ {0}} \left[ \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) \ell \left(\left(\mathbf {w} _ {t} ^ {*}\right) ^ {\top} \mathbf {x}, y\right) \right] \\ \leq 2 L \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \left| r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) \right| \right], \\ \end{array}
$$
where the first inequality is due to the optimality of  $\widehat{\mathbf{w}}_t$  and the second inequality is by the condition  $\max_{\mathbf{x} \in \mathcal{W}, y \in \mathcal{Y}, \mathbf{w} \in \mathcal{W}} |\ell(\mathbf{w}^\top \mathbf{x}, y)| \leq L$ .
Finally, as for term (c), since  $\mathbf{w}_t^*$  is independent of  $S_0$ , a direct application of the Hoeffding's inequality [65, Lemma B.6] implies,
$$
\operatorname {t e r m} (\mathbf {c}) = \widetilde {R} _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) - R _ {t} \left(\mathbf {w} _ {t} ^ {*}\right) \leq B L \sqrt {\frac {2 \ln ((8 T) / \delta)}{N _ {0}}}
$$
with probability at least  $1 - \delta /(2T)$ . Combining the upper bounds for term (a), term (b) and term (c) and taking the summation over  $T$  rounds, we get
$$
\Re_ {T} (\{\widehat {\mathbf {w}} _ {t} \} _ {t = 1} ^ {T}) \leq \frac {2 B G D R T}{\sqrt {N _ {0}}} + 5 B L T \sqrt {\frac {2 \ln ((8 T) / \delta)}{N _ {0}}} + 2 L \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ | r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) | ]
$$
with probability at least  $1 - \delta$ , which completes the proof.
C.2 Proof of Proposition 2
Proof of Proposition 2. We begin with converting the estimation error to the squared formulation,
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} [ | r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) | ] \leq \sum_ {t = 1} ^ {T} \sqrt {\mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} [ | r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) | ^ {2} ]}
$$
$$
\leq \sqrt {T \left(\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} \left[ \left| r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) \right| ^ {2} \right]\right)}, \tag {12}
$$
where the first inequality is by Jensen's inequality and the last inequality is due to the Cauchy-Schwarz inequality. Then, we can bound the squared density ratio estimation error as
$$
\begin{array}{l} \frac {\mu}{2} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} \left[ | \widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) | ^ {2} \right] \leq \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} \left[ \psi (r _ {t} ^ {*} (\mathbf {x})) - \psi (\widehat {r} _ {t} (\mathbf {x})) - \partial \psi (\widehat {r} _ {t} (\mathbf {x})) (r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x})) \right] \\ = \operatorname {E B} _ {\psi} \left(r _ {t} ^ {*} \| \widehat {r} _ {t}\right) = L _ {t} ^ {\psi} \left(\widehat {r} _ {t}\right) - L _ {t} ^ {\psi} \left(r _ {t} ^ {*}\right), \tag {13} \\ \end{array}
$$
where the first inequality is due to the strong convexity of the function  $\psi$ . The last equality is by the definition of the loss function  $L_{t}^{\psi}$ . Combining (12) and (13), we have
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} [ | \widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) | ] \leq \sqrt {\frac {2 T}{\mu} \left(\sum_ {t = 1} ^ {T} L _ {t} ^ {\psi} (\widehat {r} _ {t}) - \sum_ {t = 1} ^ {T} L _ {t} ^ {\psi} (r _ {t} ^ {*})\right)}.
$$
C.3 Proof of Theorem 1
Proof of Theorem 1. Following the same arguments in obtaining (12), the empirical cumulative estimation error of the density ratio estimators can be bounded as
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0} (\mathbf {x})} \left[ \left| r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) \right| \right] \leq \sqrt {T \left(\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0} (\mathbf {x})} \left[ \left| r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) \right| ^ {2} \right]\right)}. \tag {14}
$$
Then, we proceed to bound the squared cumulative estimation error by the regret of the loss function  $\widetilde{L}_t^\psi (r)$  defined in (5). For notation simplicity, we define
$$
f _ {1} (z) = \partial \psi (z) z - \psi (z) \quad \text {a n d} \quad f _ {2} (z) = \partial \psi (z).
$$
In such a case, the loss function  $\widetilde{L}_t^\psi (r) = \mathbb{E}_{\mathbf{x}\sim S_0}[f_1(r(\mathbf{x}))] - \mathbb{E}_{\mathbf{x}\sim \mathcal{D}_t(\mathbf{x})}[f_2(r(\mathbf{x}))]$  and the regret over  $\widetilde{L}_t^\psi$  can be written as
$$
\widetilde {L} _ {t} ^ {\psi} \left(\widehat {r} _ {t}\right) - \widetilde {L} _ {t} ^ {\psi} \left(r _ {t} ^ {*}\right) = \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {1} \left(\widehat {r} _ {t} (\mathbf {x})\right) - f _ {1} \left(r _ {t} ^ {*} (\mathbf {x})\right) \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t}} \left[ f _ {2} \left(\widehat {r} _ {t} (\mathbf {x})\right) - f _ {2} \left(r _ {t} ^ {*} (\mathbf {x})\right) \right]. \tag {15}
$$
For the first term of R.H.S. of (15), by the Taylor's Theorem, for each  $\mathbf{x} \in \mathcal{X}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {1} \left(\widehat {r} _ {t} (\mathbf {x})\right) - f _ {1} \left(r _ {t} ^ {*} (\mathbf {x})\right) \right] \\ = \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \partial f _ {1} (r _ {t} ^ {*} (\mathbf {x})) (\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) + \frac {\partial^ {2} f _ {1} (\xi_ {\mathbf {x}})}{2} (\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] \\ \geq \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \partial^ {2} \psi \left(r _ {t} ^ {*} (\mathbf {x})\right) r _ {t} ^ {*} (\mathbf {x}) \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) \right] + \frac {\mu}{2} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right]. \tag {16} \\ \end{array}
$$
In above  $\xi_{\mathbf{x}} \in \mathrm{dom} \psi$  is a certain point on the line collecting  $\widehat{r}_t(\mathbf{x})$  and  $r_t^*(\mathbf{x})$ . The second inequality is due to the definition of  $f_1$  such that,
$$
\partial^ {2} f _ {1} \left(\xi_ {\mathbf {x}}\right) = \partial^ {2} \psi \left(\xi_ {\mathbf {x}}\right) - \partial^ {3} \psi \left(\xi_ {\mathbf {x}}\right) \xi_ {\mathbf {x}} \geq \mu ,
$$
where the inequality holds due to the  $\mu$ -strong convexity of  $\psi(z)$  and the condition  $\xi_{\mathbf{x}}\partial^{3}\psi(\xi_{\mathbf{x}}) \leq 0$  for all  $\xi_{\mathbf{x}} \in \mathrm{dom}\psi$ .
Then, we proceed to bound the second term of R.H.S. of (15),
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t}} \left[ f _ {2} \left(\widehat {r} _ {t} (\mathbf {x})\right) - f _ {2} \left(r _ {t} ^ {*} (\mathbf {x})\right) \right] \\ = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t}} \left[ \partial f _ {2} \left(r _ {t} ^ {*} (\mathbf {x})\right) \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) \right] + \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t}} \left[ \frac {\partial^ {2} f _ {2} \left(\xi_ {\mathbf {x}}\right)}{2} \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right] \\ \leq \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t}} \left[ \partial^ {2} \psi \left(r _ {t} ^ {*} (\mathbf {x})\right) \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) \right] = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ \partial^ {2} \psi \left(r _ {t} ^ {*} (\mathbf {x})\right) r _ {t} ^ {*} (\mathbf {x}) \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) \right], \tag {17} \\ \end{array}
$$
In above  $\xi_{\mathbf{x}} \in \operatorname{dom} \psi$  is a certain point on the line connecting  $\widehat{r_t}(\mathbf{x})$  and  $r_t^*(\mathbf{x})$ . The first inequality is due to the definition of  $f_2$  and the condition on  $\psi$ , such that  $\partial^2 f_2(\xi_{\mathbf{x}}) = \partial^3 \psi(\xi_{\mathbf{x}}) \leq 0$ . Plugging (16) and (17) into (15), and rearranging the terms, we arrive
$$
\frac {\mu}{2} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right] \leq \widetilde {L} _ {t} ^ {\psi} \left(\widehat {r} _ {t}\right) - \widetilde {L} _ {t} ^ {\psi} \left(r _ {t} ^ {*}\right) + U _ {t}, \tag {18}
$$
where the term  $U_{t}$  is defined as
$$
U _ {t} = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ \partial^ {2} \psi \big (r _ {t} ^ {*} (\mathbf {x}) \big) r _ {t} ^ {*} (\mathbf {x}) \big (\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) \big) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \partial^ {2} \psi \big (r _ {t} ^ {*} (\mathbf {x}) \big) r _ {t} ^ {*} (\mathbf {x}) \big (\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) \big) \right].
$$
It remains to analyze the generalization gap  $U_{t}$  for every iteration. The main challenge is that the learned model  $\widehat{r_t}$  depends on the initial dataset  $S_0$  and the generalization error is related to the complexity of the hypothesis of the density ratio estimator. In the following lemma, we use the covering number [65, Chapter 27] to measure the complexity of the hypothesis space. It is possible to provide an upper bound of  $U_{t}$  with other advanced complexity measure.
Lemma 1. Let  $\mathcal{H}_r \triangleq \{r : \mathcal{X} \to \mathbb{R} \mid \| r \|_{\infty} \leq B_r'\}$  be a hypothesis space of the density ratio function whose covering number is  $N(\mathcal{H}_r, \varepsilon, \| \cdot \|_{\infty})$ . Then, for all model  $r \in \mathcal{H}_r$  and  $\delta \in (0,1)$ , we have the following upper bound
$$
\begin{array}{l} U _ {t} = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ \partial^ {2} \psi \big (r _ {t} ^ {*} (\mathbf {x}) \big) r _ {t} ^ {*} (\mathbf {x}) \big (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) \big) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \partial^ {2} \psi \big (r _ {t} ^ {*} (\mathbf {x}) \big) r _ {t} ^ {*} (\mathbf {x}) \big (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) \big) \right] \\ \leq \frac {C _ {r} (\mu) \log \left(2 N (\mathcal {H} _ {r} , \varepsilon , \| \cdot \| _ {\infty}) / \delta\right)}{N _ {0}} + \frac {\mu}{4} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + 2 A _ {r} \varepsilon + \frac {\mu \varepsilon^ {2}}{4}, \\ \end{array}
$$
with probability at least  $1 - \delta$  for any  $\varepsilon > 0$ , where  $C_r(\mu) = 4A_rB_r / 3 + 16A_r^2 / \mu + 3\mu B_r^2 = \mathcal{O}(\max \{\mu, 1 / \mu\})$  with constants  $A_r = |\max_{\mathbf{x} \in \mathcal{X}} \partial^2 \psi(r_t^*(\mathbf{x})) r_t^*(\mathbf{x})|$ , and  $B_r = \max \{B, B_r'\}$ .
Combining (18) and Lemma 1 with the hypothesis space  $\mathcal{H}_r = \mathcal{H}_{\theta} \triangleq \{\mathbf{x} \mapsto h(\mathbf{x}, \theta) \mid \theta \in \Theta\}$  and rearranging the terms yields,
$$
\frac {\mu}{4} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right] \leq \widetilde {L} _ {t} ^ {\psi} (\widehat {r} _ {t}) - \widetilde {L} _ {t} ^ {\psi} (r _ {t} ^ {*}) + \frac {C _ {r} (\mu) \log \left(2 N \left(\mathcal {H} _ {\theta} , \varepsilon , \| \cdot \| _ {\infty}\right) / \delta\right)}{N _ {0}} + 2 A _ {r} \varepsilon + \frac {\mu \varepsilon^ {2}}{4}.
$$
Further choosing  $\varepsilon = 1 / T$  and taking the summation of the  $T$  iterations, we have
$$
\begin{array}{l} \frac {\mu}{4} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right] \\ \leq \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} ^ {\psi} (\widehat {r} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} ^ {\psi} \left(r _ {t} ^ {*}\right) + \frac {C _ {r} (\mu) T \log \left(2 T N \left(\mathcal {H} _ {\theta} , ^ {1} / T , \| \cdot \| _ {\infty}\right) / \delta\right)}{N _ {0}} + D _ {r} \tag {19} \\ \end{array}
$$
where  $D_r = 2A_r + \mu/(4T) = \mathcal{O}(1)$  is a constant. Then, we proceed to bound the covering number of the hypothesis space  $\mathcal{H}_{\theta}$ . Let  $\theta, \theta' \in \Theta$  be the corresponding parameters for the two density ratio function  $r, r' \in \mathcal{H}_{\theta}$ . Then, we can show that for any  $\| \theta - \theta' \|_2 \leq \varepsilon$ , we have
$$
\| r - r ^ {\prime} \| _ {\infty} = \max  _ {\mathbf {x} \in \mathcal {X}} | h (\mathbf {x}, \theta) - h (\mathbf {x}, \theta^ {\prime}) | \leq G _ {h} \| \theta - \theta^ {\prime} \| _ {2},
$$
where  $G_{h} = \max_{\mathbf{x}\in \mathcal{X},\theta \in \Theta}\| \nabla h(\mathbf{x},\theta)\|_{2}$  is the Lipschitz continuity constant. Then, we can check that the covering number of  $\mathcal{H}_{\theta}$  in terms of  $\| \cdot \|_{\infty}$  can be bounded by that of  $\Theta$  in terms of  $\| \cdot \| _2$  as
$$
N \left(\mathcal {H} _ {\theta}, ^ {1} / T, \| \cdot \| _ {\infty}\right) \leq N \left(\Theta , ^ {1} / \left(G _ {h} T\right), \| \cdot \| _ {2}\right) \leq \left(3 S G _ {h} T\right) ^ {d}. \tag {20}
$$
In the above, the last inequality holds because the parameter space  $\Theta$  is essentially a  $L_{2}$ -ball with radius  $S$ , whose covering number is bounded by  $(3S / \varepsilon)^{d}$  [66]. Then, plugging (20) into (19), we arrive
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \left(\widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right] \leq \frac {4}{\mu} \sum_ {t = 1} ^ {T} \left(\widetilde {L} _ {t} ^ {\psi} \left(\widehat {r} _ {t}\right) - \widetilde {L} _ {t} ^ {\psi} \left(r _ {t} ^ {*}\right)\right) + \frac {4 d T C _ {r} (\mu) \log \left(6 S G _ {h} T / \delta\right)}{\mu N _ {0}} + \frac {4 D _ {r}}{\mu}.
$$
Then, we complete the proof by plugging this inequality into (14),
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0} (\mathbf {x})} [ | \widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) | ]
$$
$$
\begin{array}{l} \leq \sqrt {\frac {4 T}{\mu} \left(\sum_ {t = 1} ^ {T} L _ {t} ^ {\psi} (\hat {r} _ {t}) - \sum_ {t = 1} ^ {T} L _ {t} ^ {\psi} (r _ {t} ^ {*})\right) + \frac {4 d T ^ {2} C _ {r} (\mu) \log (6 S G _ {h} T / \delta)}{\mu N _ {0}} + \frac {4 T D _ {r}}{\mu}} \\ = \sqrt {\frac {4 T}{\mu} \max \left\{\sum_ {t = 1} ^ {T} L _ {t} ^ {\psi} (\widehat {r} _ {t}) - \sum_ {t = 1} ^ {T} L _ {t} ^ {\psi} (r _ {t} ^ {*}), 0 \right\}} + \mathcal {O} \left(\max \left\{1, \frac {1}{\mu} \right\} \cdot \frac {T \sqrt {d} \log (T / \delta)}{\sqrt {N _ {0}}}\right). \\ \end{array}
$$
C.4 Useful Lemmas
Lemma 2. Let  $\widetilde{R}_t(\mathbf{w}) = \mathbb{E}_{(\mathbf{x},y)\sim S_0}\left[r_t^* (\mathbf{x})\ell (\mathbf{w}^\top \mathbf{x},y)\right]$ . With probability at least  $1 - \delta$  over the drawn of  $S_{0}$ , all predictions  $\mathbf{w}\in \mathcal{W}$  satisfy
$$
\left| R _ {t} (\mathbf {w}) - \widetilde {R} _ {t} (\mathbf {w}) \right| \leq \frac {2 B G D R}{\sqrt {N _ {0}}} + 4 B L \sqrt {\frac {2 \ln (4 / \delta)}{N _ {0}}}.
$$
Proof of Lemma 2. Since the risk function  $\widetilde{R}_t(\mathbf{w})$  is bounded by  $BL$  for all  $\mathbf{w} \in \mathcal{W}$ , the standard analysis on generalization bound with Rademacher complexity (e.g. [65, Theorem 26.5]) shows that
$$
\left| R _ {t} (\mathbf {w}) - \widetilde {R} _ {t} (\mathbf {w}) \right| \leq 2 \widehat {\Re} _ {S _ {0}} (\mathcal {L} _ {t}) + 4 B L \sqrt {\frac {2 \ln (4 / \delta)}{N _ {0}}}
$$
for any  $\mathbf{w} \in \mathcal{W}$ . In above, the function space  $\mathcal{L}_t$  is defined as  $\mathcal{L}_t = \{(x,y) \mapsto r_t^*(x) \ell(w^\top x, y) | w \in \mathcal{W}\}$  and  $\widehat{\Re}_{S_0}(\mathcal{L}_t) = \frac{1}{N_0} \mathbb{E}_{\sigma \sim \{\pm 1\}^{N_0}} \left[ \sup_{\mathbf{w} \in \mathcal{W}} \sum_{n=1}^{N_0} \sigma_i r_t(\mathbf{x}_n) \ell(w^\top \mathbf{x}_n, y_n) \right]$  is the empirical Rademacher complexity of  $\mathcal{L}_t$ .
Note that the function  $\phi_n: \mathbb{R} \to \mathbb{R}$  defined as  $\phi_n(z) = r_t^*(\mathbf{x}_n)\ell(z, y_n)$  is  $BG$ -Lipschitz, such that  $|\phi_n(\mathbf{w}_1^\top \mathbf{x}_n) - \phi_n(\mathbf{w}_2^\top \mathbf{x}_n)| \leq BG|(\mathbf{w}_1 - \mathbf{w}_2)^\top \mathbf{x}_n|$  for any  $\mathbf{w}_1, \mathbf{w}_2 \in \mathcal{W}$ . Then, according to the Talagrand's lemma [65, Lemma 26.9], we have
$$
\widehat {\mathfrak {R}} _ {S _ {0}} (\mathcal {L} _ {t}) \leq B G \widehat {\mathfrak {R}} _ {S _ {0}} (\widetilde {\mathcal {W}}),
$$
where  $\widetilde{\mathcal{W}} = \{\mathbf{x} \mapsto \mathbf{w}^\top \mathbf{x} \mid \mathbf{w} \in \mathcal{W}\}$  is a hypothesis space for the linear functions. By [65, Lemma 26.10], the Rademacher complexity of the linear function class is bounded by
$$
\widehat {\mathfrak {R}} _ {S _ {0}} (\widetilde {\mathcal {W}}) \leq \frac {D \max  _ {\mathbf {x} _ {n} \in S _ {0}} \| \mathbf {x} _ {n} \| _ {2}}{\sqrt {N _ {0}}} \leq \frac {D R}{\sqrt {N _ {0}}}.
$$
We complete the proof by combining the above displayed inequalities.
Proof of Lemma 1. We begin with the analysis for the generalization gap for a fixed model  $r \in \mathcal{H}_r$ . For notation simplicity, we denote by
$$
f _ {r} (\mathbf {x}) = \partial^ {2} \psi \left(r _ {t} ^ {*} (\mathbf {x})\right) r _ {t} ^ {*} (\mathbf {x}) \left(r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right),
$$
and let the random variable  $Z_{i} = \mathbb{E}_{\mathbf{x}\sim \mathcal{D}_{0}}[f_{r}(\mathbf{x})] - f_{r}(\mathbf{x}_{i})$  for any  $\mathbf{x}_i\in S_0$ . Since  $\mathbf{x}_i$  is i.i.d. sampled from  $\mathcal{D}_0$  and  $f_{r}$  is independent of  $S_0$ , we have  $\mathbb{E}[Z_i] = 0$  and  $|Z_{i}| \leq 2A_{r}B_{r}$  with the constants  $A_{r} = \max_{\mathbf{x}\in \mathcal{X}}|\partial^{2}\psi (r_{t}^{*}(\mathbf{x}))r_{t}^{*}(\mathbf{x})|$  and  $B_{r} = \max \{B,\max_{\mathbf{x}\in \mathcal{X},r\in \mathcal{H}_{r}}|r(\mathbf{x})|\}$ . Besides, the variance of  $Z_{i}$  is bounded by
$$
\mathbb {E} [ Z _ {i} ^ {2} ] = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} [ (f _ {r} (\mathbf {x})) ^ {2} ] - (\mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} [ f _ {r} (\mathbf {x}) ]) ^ {2} \leq \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ (f _ {r} (\mathbf {x})) ^ {2} \right],
$$
Then, by the Bernstein's inequality [65, Lemma B.9], with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim D _ {0}} \left[ f _ {r} (\mathbf {x}) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {r} (\mathbf {x}) \right] \leq \frac {4 A _ {r} B _ {r} \log (1 / \delta)}{3 N _ {0}} + \sqrt {\frac {2 \log (1 / \delta) \mathbb {E} _ {\mathcal {D} _ {0}} [ (f _ {r} (\mathbf {x})) ^ {2} ])}{N _ {0}}} \\ \leq \left(\frac {4 A _ {r} B _ {r}}{3} + \frac {1 6 A _ {r} ^ {2}}{\mu}\right) \frac {\log (1 / \delta)}{N _ {0}} + \frac {\mu}{8 A _ {r} ^ {2}} \mathbb {E} _ {\mathcal {D} _ {0}} \left[ \left(f _ {r} (\mathbf {x})\right) ^ {2} \right] \\ \end{array}
$$
$$
\leq \left(\frac {4 A _ {r} B _ {r}}{3} + \frac {1 6 A _ {r} ^ {2}}{\mu}\right) \frac {\log (1 / \delta)}{N _ {0}} + \frac {\mu}{8} \mathbb {E} _ {\mathcal {D} _ {0}} \left[ \left(r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right], \tag {21}
$$
where the first inequality is due to the AM-GM inequality. The second inequality is due to the fact that  $|f_r(\mathbf{x})| \leq |\partial^2 \psi(r_t^*(\mathbf{x})) r_t^*(\mathbf{x})| \cdot |r(\mathbf{x}) - r_t^*(\mathbf{x})| \leq A_r |r(\mathbf{x}) - r_t^*(\mathbf{x})|$  for any  $\mathbf{x} \in \mathcal{X}$ .
Then, since  $r$  is a fixed model in  $\mathcal{H}_r$ , we can further bound  $\mathbb{E}_{\mathcal{D}_0}\left[(r(\mathbf{x}) - r_t^* (\mathbf{x}))^2\right]$  by  $\mathbb{E}_{S_0}\left[(r(\mathbf{x}) - r_t^* (\mathbf{x}))^2\right]$  by applying the Bernstein's inequality to the random variable  $Y_{i} = (r(\mathbf{x}_{i}) - r_{t}^{*}(\mathbf{x}_{i}))^{2} / (4B_{r}^{2})$ . Specifically, after checking  $\mathbb{E}[Y_i] = \mathbb{E}_{\mathcal{D}_0}\left[(r(\mathbf{x}) - r_t^* (\mathbf{x}))^2\right] / (4B_r^2)$  and  $Y_{i}\in [0,1]$ , a direct application of [65, Lemma B.10] implies
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D} _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] \\ \leq \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + \frac {1 6 B _ {r} ^ {2} \log (1 / \delta)}{N _ {0}} + \sqrt {\frac {8 B _ {r} ^ {2} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] \log (1 / \delta)}{N _ {0}}} \\ \leq \frac {2 4 B _ {r} ^ {2} \log (1 / \delta)}{N _ {0}} + \mathbb {E} _ {S _ {0}} \left[ \left(r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right], \tag {22} \\ \end{array}
$$
where the last inequality is due to the AM-GM inequality. Plugging (22) into (21), for a fixed  $r \in \mathcal{H}_r$ , we have