SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+} | r} \left[ g \left(\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}, r\right) \right] \leq \sqrt {2 \beta_ {2} ^ {2} I \left(\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+} \mid R = r\right)}.
$$
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left| \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}, R} \left[ g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}, R) \right] \right| \leq \frac {\beta_ {2}}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {R} \sqrt {2 I ^ {R} (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+})},
$$
Proof. Let  $h(x) = \frac{e^x - x - 1}{x^2}$  be the Bernstein function. Similar to the proof of Theorem 3.1, we let  $g(\tilde{w}^{+},\tilde{z}_{i}^{+}) = \mathbb{E}_{\widetilde{W}_{i}^{-}|\tilde{w}^{+}}\left[\ell (\widetilde{W}_{i}^{-},\tilde{z}_{i}^{+})\right] - \ell (\tilde{w}^{+},\tilde{z}_{i}^{+})$ . We have already known that  $\mathbb{E}_{\widetilde{W}^{+},\widetilde{Z}_{i}^{+^{\prime}}} \left[ g(\widetilde{W}^{+},\widetilde{Z}_{i}^{+^{\prime}})\right] = 0$  and  $\left|g(\widetilde{W}^{+},\widetilde{Z}_{i}^{+^{\prime}})\right| \leq \gamma_{1}$ . By Lemma A.5,
$$
\begin{array}{l} \log \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime}} \left[ e ^ {t g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime})} \right] \leq h (\gamma_ {1} t) t ^ {2} \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime}} \left[ \left(\mathbb {E} _ {\widetilde {W} _ {i} ^ {-} | \widetilde {W} ^ {+}} \left[ \ell (\widetilde {W} _ {i} ^ {-}, \widetilde {Z} _ {i} ^ {+ \prime}) \right] - \ell (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+ \prime})\right) ^ {2} \right] \\ \leq h \left(\gamma_ {1} t\right) t ^ {2} \gamma_ {2} ^ {2}, \\ \end{array}
$$
where the second inequality is by the definition of  $\gamma_{2}$ -SCH-B stability.
Plugging the above into Eq. (17),
$$
\begin{array}{l} \mathbb {E} _ {\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}} \left[ g (\widetilde {W} ^ {+}, \widetilde {Z} _ {i} ^ {+}) \right] \leq \inf  _ {t > 0} \frac {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+}) + \log \mathbb {E} _ {\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime}} \left[ e ^ {t g (\widetilde {W} ^ {+} , \widetilde {Z} _ {i} ^ {+ \prime})} \right]}{t} \\ \leq \inf _ {t > 0} \frac {I (\widetilde {W} ^ {+} ; \widetilde {Z} _ {i} ^ {+})}{t} + h (\gamma_ {1} t) t \gamma_ {2} ^ {2}. \\ \end{array}
$$
Usually we have  $\gamma_2^2\leq \gamma_1^2\leq \gamma_1$  , we let  $t = 1 / \gamma_{1}$  , then
$$
h (\gamma_ {1} t) t \gamma_ {2} ^ {2} = \frac {h (1) \gamma_ {2} ^ {2}}{\gamma_ {1}} \approx 0. 7 2 \frac {\gamma_ {2} ^ {2}}{\gamma_ {1}}.
$$
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {\gamma_ {1}}{n} \sum_ {i = 1} ^ {n} I (\widetilde {W} ^ {+}; \widetilde {Z} _ {i} ^ {+}) + \frac {0 . 7 2 \gamma_ {2} ^ {2}}{\gamma_ {1}}.
$$
Proof. We now prove the first bound. Let  $g(\tilde{w}_i, \hat{z}_i, u_i) = (-1)^{u_i} \left( \ell(\tilde{w}_i^-, \hat{z}_i) - \ell(\tilde{w}_i^+, \hat{z}_i) \right)$ . By Lemma A.2, we have
$$
\mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} \mid \tilde {w} _ {i}} \left[ g \left(\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i}\right) \right] \leq \inf  _ {t > 0} \frac {I \left(\widehat {Z} _ {i} ; U _ {i} \mid \widetilde {W} _ {i} = \tilde {w} _ {i}\right) + \log \mathbb {E} _ {\widehat {Z} _ {i} , U _ {i} ^ {\prime} \mid \tilde {w} _ {i}} \left[ e ^ {t g \left(\tilde {w} _ {i} , \widehat {Z} _ {i} , U _ {i} ^ {\prime}\right)} \right]}{t}. \tag {19}
$$
Since  $U_{i}^{\prime} \perp \widehat{Z}_{i}$ , we have  $\mathbb{E}_{U_i'}[g(\tilde{w}_i, \hat{z}_i, U_i')] = \mathbb{E}_{U_i'}\left[-1^{U_i'}\left(\ell(\tilde{w}_i^{-}, \hat{z}_i) - \ell(\tilde{w}_i^{+}, \hat{z}_i)\right)\right] = 0$  for any  $\tilde{w}_i$  and  $\hat{z}_i$ . Ergo,
$$
\mathbb {E} _ {\widehat {Z} _ {i} | \tilde {w} _ {i}} \left[ \mathbb {E} _ {U _ {i} ^ {\prime}} \left[ g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i} ^ {\prime}) \right] \right] = 0.
$$
By the definition of  $\Delta_1(\tilde{w}_i)$
$$
\left| g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i} ^ {\prime}) \right| = \left| \ell (\tilde {w} _ {i} ^ {-}, \widehat {Z} _ {i}) - \ell (\tilde {w} _ {i} ^ {+}, \widehat {Z} _ {i}) \right| \leq \sup _ {z _ {i} \in \mathcal {Z} _ {\tilde {w} _ {i}}} \left| \ell (\tilde {w} _ {i} ^ {-}, z _ {i}) - \ell (\tilde {w} _ {i} ^ {+}, z _ {i}) \right| \leq \Delta_ {1} (\tilde {w} _ {i}).
$$
Thus,  $g(\tilde{w}_i, \widehat{Z}_i, U_i')$  is a zero-mean random variable bounded in  $[-\Delta_1(\tilde{w}_i), \Delta_1(\tilde{w}_i)]$  for a fixed  $\tilde{w}_i$ . By Lemma A.3, we have
$$
\mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} ^ {\prime} | \bar {w} _ {i}} \left[ e ^ {t g (\bar {w}, \widehat {Z} _ {i}, U _ {i} ^ {\prime})} \right] \leq e ^ {\frac {t ^ {2} \Delta_ {1} (\bar {w} _ {i}) ^ {2}}{2}}.
$$
Plugging the above into Eq. (19),
$$
\begin{array}{l} \mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} \mid \tilde {w} _ {i}} \left[ g \left(\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i}\right) \right] \leq \inf  _ {t > 0} \frac {I \left(\widehat {Z} _ {i} ; U _ {i} \mid \widetilde {W} _ {i} = \tilde {w} _ {i}\right) + \frac {t ^ {2} \Delta_ {1} \left(\tilde {w} _ {i}\right) ^ {2}}{2}}{t} \tag {20} \\ = \Delta_ {1} (\tilde {w} _ {i}) \sqrt {2 I (\widehat {Z} _ {i} ; U _ {i} | \widetilde {W} _ {i} = \tilde {w} _ {i})}. \\ \end{array}
$$
Recall Eq. (14) in Lemma A.1 and by Jensen's inequality for the absolute function, the first bound is obtained:
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta_ {1} \left(\widetilde {W} _ {i}\right) \sqrt {2 I \widetilde {W} _ {i} \left(\widehat {Z} _ {i} ; U _ {i}\right)} \right]. \tag {21}
$$
To prove the second bound, we return to Eq. (20), and take expectation over  $\widetilde{W}_i$  first. By Jensen's inequality,
$$
\begin{array}{l} \mathbb {E} _ {\widehat {Z} _ {i}, U _ {i}, \widetilde {W} _ {i}} \left[ g (\tilde {W} _ {i}, \widehat {Z} _ {i}, U _ {i}) \right] \leq \inf  _ {t > 0} \frac {I \left(\widehat {Z} _ {i} ; U _ {i} \mid \widetilde {W} _ {i}\right) + \frac {t ^ {2} \mathbb {E} _ {\bar {W} _ {i}} \left[ \Delta \left(\widetilde {W} _ {i}\right) ^ {2} \right]}{2}}{t} \tag {22} \\ = \sqrt {2 \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta (\widetilde {W} _ {i}) ^ {2} \right] I (\widehat {Z} _ {i} ; U _ {i} | \widetilde {W} _ {i})}. \\ \end{array}
$$
Therefore, we have the second bound as below
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \sqrt {2 \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta \left(\widetilde {W} _ {i}\right) ^ {2} \right] I \left(\widehat {Z} _ {i} ; U _ {i} \mid \widetilde {W} _ {i}\right)}. \tag {23}
$$
For the second part of Theorem 4.1, notice that it's valid to let  $\gamma_3 = \mathbb{E}_{\widetilde{W}_i}\left[\Delta (\widetilde{W}_i)\right]$ , then recall Eq. (21),
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta (\widetilde {W} _ {i}) \sqrt {2 I ^ {\widetilde {W} _ {i}} (\widehat {Z} _ {i} ; U _ {i})} \right] \leq \frac {\sqrt {2} \gamma_ {3}}{n} \sum_ {i = 1} ^ {n} \sqrt {\sup _ {\tilde {w} _ {i} \in (\mathcal {W} _ {s}) _ {s \in \mathcal {Z} ^ {n}} ^ {2}} I ^ {\tilde {w} _ {i}} (\widehat {Z} _ {i} ; U _ {i})}.
$$
Proof. The proof is similar to [18, Theorem 2.1]. By the chain rule,
$$
I \left(\widehat {Z} _ {i}; U _ {i}, \widetilde {W} _ {i}\right) = I \left(\widehat {Z} _ {i}; U _ {i} \mid \widetilde {W} _ {i}\right) + I \left(\widehat {Z} _ {i}; \widetilde {W} _ {i}\right). \tag {24}
$$
Since  $H(\widehat{Z}_i|U_i, \widetilde{W}_i) = H(\widehat{Z}_i|W_i, U_i, \widetilde{W}_i) = H(\widehat{Z}_i|W_i)$ , we have  $I(\widehat{Z}_i; U_i, \widetilde{W}_i) = H(\widehat{Z}_i) - H(\widehat{Z}_i|U_i, \widetilde{W}_i) = H(\widehat{Z}_i) - H(\widehat{Z}_i|W_i) = I(\widehat{Z}_i; W_i)$ . Thus,  $I(\widehat{Z}_i; U_i, \widetilde{W}_i) = I(\widehat{Z}_i; W_i)$ . Recall Eq. (24) and by the non-negativity of mutual information, we have  $I(\widehat{Z}_i; U_i | \widetilde{W}_i) \leq I(W_i; \widehat{Z}_i)$ . Note that  $I(W_i; \widehat{Z}_i) = I(\widetilde{W}_i^+; \widetilde{Z}_i^+) = I(W; Z_i)$ . This completes the proof.
D.3 Proof of Theorem 4.3
Proof. We first return to Eq. (19) in the previous proof, and we have already known that  $g(\tilde{w}_i,\widehat{Z}_i,U_i')$  is a zero-mean random variable bounded in  $[-\Delta_1(\tilde{w}_i),\Delta_1(\tilde{w}_i)]$  for a fixed  $\tilde{w}_i$ .
By Lemma A.5, we have
$$
\begin{array}{l} \log \mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} ^ {\prime} | \tilde {w} _ {i}} \left[ e ^ {t g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i} ^ {\prime})} \right] \leq h \left(\Delta_ {1} (\tilde {w} _ {i}) t\right) t ^ {2} \mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} ^ {\prime} | \tilde {w} _ {i}} \left[ g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i} ^ {\prime}) ^ {2} \right] \\ = h \left(\Delta_ {1} (\tilde {w} _ {i}) t\right) t ^ {2} \mathbb {E} _ {\widehat {Z} _ {i} | \tilde {w} _ {i}} \left[ \left(\ell (\tilde {w} _ {i} ^ {-}, \widehat {Z} _ {i}) - \ell (\tilde {w} _ {i} ^ {+}, \widehat {Z} _ {i})\right) ^ {2} \right]. \\ \end{array}
$$
$$
\mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} | \tilde {w} _ {i}} \left[ g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i}) \right] \leq \inf  _ {t > 0} \frac {I \left(\widehat {Z} _ {i} ; U _ {i} \mid \widetilde {W} _ {i} = \tilde {w} _ {i}\right)}{t} + h \left(\Delta_ {1} (\tilde {w} _ {i}) t\right) t \mathbb {E} _ {\widehat {Z} _ {i} | \tilde {w} _ {i}} \left[ \left(\ell \left(\tilde {w} _ {i} ^ {-}, \widehat {Z} _ {i}\right) - \ell \left(\tilde {w} _ {i} ^ {+}, \widehat {Z} _ {i}\right)\right) ^ {2} \right]. \tag {25}
$$
Let  $t = \frac{1}{\Delta_1(\tilde{w}_i)}$ , we have
$$
\mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} | \tilde {w} _ {i}} \left[ g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i}) \right] \leq \Delta_ {1} (\tilde {w} _ {i}) I (\widehat {Z} _ {i}; U _ {i} | \widetilde {W _ {i}} = \tilde {w} _ {i}) + 0. 7 2 \frac {\mathbb {E} _ {\widehat {Z} _ {i} | \tilde {w} _ {i}} \left[ \left(\ell (\tilde {w} _ {i} ^ {-} , \widehat {Z} _ {i}) - \ell (\tilde {w} _ {i} ^ {+} , \widehat {Z} _ {i})\right) ^ {2} \right]}{\Delta_ {1} (\tilde {w} _ {i})}.
$$
Let  $\Lambda (\tilde{w}_i) = \mathbb{E}_{\widehat{Z}_i|\tilde{w}_i}\left[\left(\ell (\tilde{w}_i^-, \widehat{Z}_i) - \ell (\tilde{w}_i^+, \widehat{Z}_i)\right)^2\right] / \Delta_1(\tilde{w}_i)^2$  then
$$
\left| \mathcal {E} _ {\mu} (\mathcal {A}) \right| \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {\widetilde {W} _ {i}} \left[ \Delta_ {1} \left(\widetilde {W} _ {i}\right) \left(I ^ {\widetilde {W} _ {i}} \left(\widehat {Z} _ {i}; U _ {i}\right) + 0. 7 2 \Lambda \left(\widetilde {W} _ {i}\right)\right) \right].
$$
For the second part, if  $\mathcal{A}$  is further  $\beta_{2}$ -uniform stable, recall Eq. (25) and by the non-decreasing property of  $h$ , we have
$$
\mathbb {E} _ {\widehat {Z} _ {i}, U _ {i} | \tilde {w} _ {i}} \left[ g (\tilde {w} _ {i}, \widehat {Z} _ {i}, U _ {i}) \right] \leq \inf  _ {t > 0} \frac {I (\widehat {Z} _ {i} ; U _ {i} | \widetilde {W} _ {i} = \tilde {w} _ {i})}{t} + h (\beta_ {2} t) t \mathbb {E} _ {\widehat {Z} _ {i} | \tilde {w} _ {i}} \left[ \left(\ell (\tilde {w} _ {i} ^ {-}, \widehat {Z} _ {i}) - \ell (\tilde {w} _ {i} ^ {+}, \widehat {Z} _ {i})\right) ^ {2} \right].
$$
Let  $t = \frac{1}{\beta_2}$  and taking expectation over  $\widetilde{W_i}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\widehat {Z} _ {i}, U _ {i}, \widetilde {W} _ {i}} \left[ g (\widetilde {W} _ {i}, \widehat {Z} _ {i}, U _ {i}) \right] \leq \beta_ {2} I (\widehat {Z} _ {i}; U _ {i} | \widetilde {W} _ {i}) + 0. 7 2 \frac {\mathbb {E} _ {\widehat {Z} _ {i} , \widetilde {W} _ {i}} \left[ \left(\ell (\widetilde {W} _ {i} ^ {-} , \widehat {Z} _ {i}) - \ell (\widetilde {W} _ {i} ^ {+} , \widehat {Z} _ {i})\right) ^ {2} \right]}{\beta_ {2}} \\ = \beta_ {2} I (\widehat {Z} _ {i}; U _ {i} | \widetilde {W _ {i}}) + 0. 7 2 \frac {\gamma_ {4} ^ {2}}{\beta_ {2}}, \\ \end{array}
$$
where the equality is by the definition of  $\gamma_4$ -SCH-D stability.
$$
| \mathcal {E} _ {\mu} (\mathcal {A}) | \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \beta_ {2} I (\widehat {Z} _ {i}; U _ {i} | \widetilde {W _ {i}}) + 0. 7 2 \frac {\gamma_ {4} ^ {2}}{\beta_ {2}}.
$$
D.4 Proof of Theorem 4.4
We present a stronger version of Theorem 4.4.
Theorem D.1. Under the same conditions in Theorem 4.1, and we further assume that  $\mathcal{A}$  is  $\gamma_{2}$ -SCH-B stable and symmetric with respect to  $S$ , i.e. it does not depend on the order of the elements in the training sample. Let  $\bar{\Delta}_{1}(\widetilde{W}) = \frac{1}{n}\sum_{i=1}^{n}\Delta_{1}(\widetilde{W}_{i})^{2}$ , we have
$$
\mathbb {E} _ {W, S} \left[ (L _ {S} (W) - L _ {\mu} (W)) ^ {2} \right] \leq \frac {6}{n} \mathbb {E} _ {\widetilde {W}} \left[ \bar {\Delta} _ {1} (\widetilde {W}) \left(I ^ {\widetilde {W}} (E; U) + \frac {\log 3}{2}\right) \right] + \frac {1}{n} + 4 \gamma_ {2} ^ {2}.
$$
Then Theorem 4.4 is a corollary of Theorem D.1.
Proof of Theorem 4.4. For  $\beta_{2}$ -uniform stable algorithm, by  $\bar{\Delta}_{1}(\widetilde{W})\leq \beta_{2}^{2}$  and  $\gamma_2^2\leq \beta_2^2$  we have
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ (L _ {S} (W) - L _ {\mu} (W)) ^ {2} \right] \leq \frac {6 \beta_ {2} ^ {2}}{n} \left(I (E; U | \widetilde {W}) + \frac {\log 3}{2}\right) + \frac {1}{n} + 4 \beta_ {2} ^ {2} \\ = 4 \beta_ {2} ^ {2} \left(\frac {1 . 5 I (E ; U | \widetilde {W}) + 0 . 8 2}{n} + 1\right) + \frac {1}{n}. \\ \end{array}
$$
Before we prove Theorem D.1, we need to first obtain the following lemma.
Lemma D.1. Under the same conditions in Theorem 4.1, let  $\bar{\Delta}_1(\widetilde{W}) = \frac{1}{n}\sum_{i=1}^{n}\Delta_1(\widetilde{W}_i)^2$ , we have
$$
\mathbb {E} _ {W, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i}, Z _ {i}) \right] - L _ {S} (W)\right) ^ {2} \right] \leq \frac {3}{n} \mathbb {E} _ {\widetilde {W}} \left[ \bar {\Delta} _ {1} (\widetilde {W}) \left(I ^ {\widetilde {W}} (E; U) + \frac {\log 3}{2}\right) \right].
$$
Proof of Lemma D.1. Here we borrow some proof techniques used in [58, Thm. 2].
Let  $g(\tilde{w}, \hat{z}_i, u_i) = (-1)^{u_i} \left( \ell(\tilde{w}_i^-, \hat{z}_i) - \ell(\tilde{w}_i^+, \hat{z}_i) \right)$  and let  $G \sim \mathcal{N}(0, 1)$  be an independent standard Gaussian random variable. Let  $f = t \cdot \left( \frac{1}{n} \sum_{i=1}^{n} g \right)^2$  in Lemma A.2, then
$$
\begin{array}{l} \mathbb {E} _ {E, U | \tilde {w}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g (\tilde {w}, \widehat {Z} _ {i}, U _ {i})\right) ^ {2} \right] \leq \inf  _ {t > 0} \frac {I (E ; U | \widetilde {W} = \tilde {w}) + \log \mathbb {E} _ {E , U ^ {\prime}} | \tilde {w} \left[ e ^ {t \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g (\tilde {w} , \widehat {Z} _ {i} , U _ {i} ^ {\prime})\right) ^ {2}} \right]}{t} \\ = \inf  _ {t > 0} \frac {I (E ; U | \widetilde {W} = \tilde {w}) + \log \mathbb {E} _ {E , U ^ {\prime} | \tilde {w}} \left[ \mathbb {E} _ {G} \left[ e ^ {\frac {G \sqrt {2 t}}{n} \sum_ {i = 1} ^ {n} g (\tilde {w} , \hat {Z} _ {i} , U _ {i} ^ {\prime})} \right] \right]}{t} (26) \\ = \inf  _ {t > 0} \frac {I (E ; U | \widetilde {W} = \tilde {w}) + \log \mathbb {E} _ {G , E | \tilde {w}} \left[ \prod_ {i = 1} ^ {n} \mathbb {E} _ {U _ {i} ^ {\prime}} \left[ e ^ {\frac {G \sqrt {2 t}}{n} g (\tilde {w} , \widehat {Z} _ {i} , U _ {i} ^ {\prime})} \right] \right]}{t} \\ \leq \inf  _ {t > 0} \frac {I (E ; U | \widetilde {W} = \tilde {w}) + \log \mathbb {E} _ {G} \left[ e ^ {\frac {G ^ {2} t \sum_ {i = 1} ^ {n} \Delta_ {1} (\tilde {w} _ {i}) ^ {2}}{n ^ {2}}} \right]}{t} (27) \\ \leq \inf  _ {t \in \left(0, \frac {n ^ {2}}{2 \sum_ {i = 1} ^ {n} \Delta_ {1} (\tilde {w} _ {i}) ^ {2}}\right)} \frac {I (E ; U | \widetilde {W} = \tilde {w}) + \log \left(1 / \sqrt {1 - \frac {2 t \sum_ {i = 1} ^ {n} \Delta_ {1} (\tilde {w} _ {i}) ^ {2}}{n ^ {2}}}\right)}{t} (28) \\ = \inf_{t\in \left(0,\frac{n^{2}}{2\sum_{i = 1}^{n}\Delta_{1}(\tilde{w}_{i})^{2}}\right)}\frac{I(E;U|\widetilde{W} = \tilde{w}) - \frac{1}{2}\log\left(1 - \frac{2t\sum_{i = 1}^{n}\Delta_{1}(\tilde{w}_{i})^{2}}{n^{2}}\right)}{t}, \\ \end{array}
$$
where Eq. (26) is by the moment generating function of Gaussian distribution:  $\mathbb{E}_G\left[e^{\lambda G}\right] = e^{\frac{\lambda^2}{2}}$  for all  $\lambda \in \mathbb{R}$ , Eq. (27) is by Lemma A.3 and Eq. (28) is by the moment generating function of chi-squared distribution:  $\mathbb{E}_G\left[e^{\lambda G^2}\right] \leq \frac{1}{\sqrt{1 - 2\lambda}}$  for  $\lambda < \frac{1}{2}$ .
Let  $t = \frac{n^2}{3\sum_{i=1}^{n}\Delta_1(\tilde{w}_i)^2}$  be substituted to the last equation above, we have
$$
\mathbb {E} _ {E, U | \tilde {w}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g (\tilde {w}, \widehat {Z} _ {i}, U _ {i})\right) ^ {2} \right] \leq \frac {3}{n ^ {2}} \sum_ {i = 1} ^ {n} \Delta_ {1} (\tilde {w} _ {i}) ^ {2} \left(I (E; U | \widetilde {W} = \tilde {w}) + \frac {\log 3}{2}\right). \tag {29}
$$
Let  $\bar{\Delta}_{1}(\tilde{w}) = \frac{1}{n}\sum_{i=1}^{n}\Delta_{1}(\tilde{w}_{i})^{2}$ , and taking expectation over  $\widetilde{W}$  for both sides,
$$
\mathbb {E} _ {E, U, \widetilde {W}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g (\widetilde {W}, \widehat {Z} _ {i}, U _ {i})\right) ^ {2} \right] \leq \frac {3}{n} \mathbb {E} _ {\widetilde {W}} \left[ \bar {\Delta} _ {1} (\widetilde {W}) \left(I ^ {\widetilde {W}} (E; U) + \frac {\log 3}{2}\right) \right]. \tag {30}
$$
Applying Jensen's inequality to the square function, we have
$$
\mathbb {E} _ {W, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i}, Z _ {i}) \right] - L _ {S} (W)\right) ^ {2} \right] \leq \mathbb {E} _ {E, U, \widetilde {W}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} g (\widetilde {W}, \widehat {Z} _ {i}, U _ {i})\right) ^ {2} \right].
$$
Combining Eq. (30) with the inequality above will conclude the proof.
We are now in a position to prove Theorem D.1.
Proof of Theorem D.1.
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ (L _ {S} (W) - L _ {\mu} (W)) ^ {2} \right] \\ = \mathbb {E} _ {W, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(W, Z _ {i}\right) - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell \left(W, Z ^ {\prime}\right) \right]\right) ^ {2} \right] \\ = \mathbb {E} _ {W, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell (W, Z _ {i}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i}, Z _ {i}) \right] + \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i}, Z _ {i}) \right] - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell (W, Z ^ {\prime}) \right]\right) ^ {2} \right] \\ \leq 2 \underbrace {\mathbb {E} _ {W , S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell (W , Z _ {i}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i} , Z _ {i}) \right]\right) ^ {2} \right]} _ {B _ {1}} \\ + 2 \underbrace {\mathbb {E} _ {W , S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i} , Z _ {i}) \right] - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell (W , Z ^ {\prime}) \right]\right) ^ {2} \right]} _ {B _ {2}}, \\ \end{array}
$$
where the last inequality is by  $(x + y)^2 \leq 2x^2 + 2y^2$ . Notice that  $B_{1}$  can be bounded by using
Lemma D.1. We now focus on  $B_{2}$ . Since  $\mathbb{E}_{Z'}[\ell(W, Z')] = \frac{1}{n} \sum_{i=1}^{n} \mathbb{E}_{Z'}[\ell(W, Z')]$ , we have
$$
\begin{array}{l} B _ {2} = \mathbb {E} _ {W, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} [ \ell (W ^ {i}, Z _ {i}) ] - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {Z ^ {\prime}} [ \ell (W, Z ^ {\prime}) - \mathbb {E} _ {W ^ {i} | W} [ \ell (W ^ {i}, Z ^ {\prime}) ] + \mathbb {E} _ {W ^ {i} | W} [ \ell (W ^ {i}, Z ^ {\prime}) ] ]\right) ^ {2} \right] \\ = \mathbb {E} _ {W, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell \left(W ^ {i}, Z _ {i}\right) - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell \left(W ^ {i}, Z ^ {\prime}\right) \right] \right] - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {Z ^ {\prime}} \left[ \ell \left(W, Z ^ {\prime}\right) - \mathbb {E} _ {W ^ {i} | W} \left[ \ell \left(W ^ {i}, Z ^ {\prime}\right) \right] \right]\right) ^ {2} \right] \\ \leq 2 \underbrace {\mathbb {E} _ {W , S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i} , Z _ {i}) - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell (W ^ {i} , Z ^ {\prime}) \right] \right]\right) ^ {2} \right]} _ {B _ {3}} \\ + 2 \underbrace {\mathbb {E} _ {W} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {Z ^ {\prime}} \left[ \ell (W , Z ^ {\prime}) - \mathbb {E} _ {W ^ {i} | W} \left[ \ell (W ^ {i} , Z ^ {\prime}) \right] \right]\right) ^ {2} \right]} _ {B _ {4}}. \\ \end{array}
$$
For  $B_{3}$ , we apply Jensen's inequality to move the expectation over  $W^{i}$  outside of the square function,
$$
\begin{array}{l} B _ {3} \leq \mathbb {E} _ {W ^ {1}, W ^ {2}, \dots , W ^ {n}, S} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(W ^ {i}, Z _ {i}\right) - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell \left(W ^ {i}, Z ^ {\prime}\right) \right]\right) ^ {2} \right] \\ = \mathbb {E} _ {S ^ {\prime}, S, R} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell (\mathcal {A} (S ^ {i}, R), Z _ {i}) - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell (\mathcal {A} (S ^ {i}, R), Z ^ {\prime}) \right]\right) ^ {2} \right]. \\ \end{array}
$$
Notice that  $S^{\prime}$ ,  $S$  and  $R$  are all independent with each other (so  $W^{i}$ ,  $Z_{i}$  and  $Z_{i}^{\prime}$  are also independent with each other). If we further let  $\mathcal{A}$  be symmetric, namely  $W^{i}$  does not depend on  $i$ , then the inequality above is equivalent to
$$
\begin{array}{l} B _ {3} \leq \mathbb {E} _ {W, S ^ {\prime}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell \left(W, Z _ {i} ^ {\prime}\right) - \mathbb {E} _ {Z ^ {\prime}} \left[ \ell \left(W, Z ^ {\prime}\right) \right]\right) ^ {2} \right] \\ = \mathbb {E} _ {W} \left[ \mathbb {E} _ {S ^ {\prime}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell (W, Z _ {i} ^ {\prime}) - \mathbb {E} _ {Z ^ {\prime}} [ \ell (W, Z ^ {\prime}) ]\right) ^ {2} \right] \right]. \\ \end{array}
$$
Hence, the inner expectation in the RHS above is just the variance of the sample mean of  $n$  i.i.d bounded random variables. Recall that  $\ell(\cdot, \cdot) \in [0,1]$ , thereby
$$
B _ {3} \leq \mathbb {E} _ {W} \left[ \frac {\operatorname {V a r} _ {Z ^ {\prime}} \left(\ell \left(W , Z ^ {\prime}\right)\right)}{n} \right] \leq \frac {1}{4 n},
$$
where the second inequality is by Lemma A.4.
Then, for  $B_4$ , we also apply Jensen's inequality to the square function, and by the definition of  $\gamma_{2}$ -SCH-B stability, we have
$$
\begin{array}{l} B _ {4} \leq \mathbb {E} _ {W, Z ^ {\prime}} \left[ \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \ell (W, Z ^ {\prime}) - \mathbb {E} _ {W ^ {i} | W} \left[ \ell \left(W ^ {i}, Z ^ {\prime}\right) \right]\right) ^ {2} \right] \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} _ {W, Z ^ {\prime}} \left[ \left(\ell (W, Z ^ {\prime}) - \mathbb {E} _ {W ^ {i} | W} \left[ \ell \left(W ^ {i}, Z ^ {\prime}\right) \right]\right) ^ {2} \right] \leq \gamma_ {2} ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {W, S} \left[ \left(L _ {S} (W) - L _ {\mu} (W)\right) ^ {2} \right] \leq 2 B _ {1} + 2 B _ {2} \\ \leq 2 B _ {1} + 4 B _ {3} + 4 B _ {4} \\ \leq 2 B _ {1} + \frac {1}{n} + 4 \gamma_ {2} \\ \leq \frac {6}{n} \mathbb {E} _ {\widetilde {W}} \left[ \bar {\Delta} _ {1} (\widetilde {W}) \left(I ^ {\widetilde {W}} (E; U) + \frac {\log 3}{2}\right) \right] + \frac {1}{n} + 4 \gamma_ {2} ^ {2}, \\ \end{array}
$$
where the last inequality is by Lemma D.1. This completes the proof.
D.5 Proof of Theorem 4.5
Proof. Let  $g(\tilde{z}_i^+, w_i, \bar{w}_i, u_i) = (-1)^{u_i} \left( \ell(\bar{w}_i, \tilde{z}_i^+) - \ell(w_i, \tilde{z}_i^+) \right)$ . Again, by Lemma A.2, we have