SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$L_{2}$ -Uniform Stability of Randomized Learning Algorithms: Sharper Generalization Bounds and Confidence Boosting
Xiao-Tong Yuan
School of Intelligence Science and Technology
Suzhou, 215163, China
xtyuan1980@gmail.com
Ping Li
VecML Inc. www vecml.com
Levue, WA 98004, USA
pingli98@gmail.com
Exponential generalization bounds with near-optimal rates have recently been established for uniformly stable algorithms (Feldman and Vondrák, 2019; Bousquet et al., 2020). We seek to extend these best known high probability bounds from deterministic learning algorithms to the regime of randomized learning. One simple approach for achieving this goal is to define the stability for the expectation over the algorithm's randomness, which may result in sharper parameter but only leads to guarantees regarding the on-average generalization error. Another natural option is to consider the stability conditioned on the algorithm's randomness, which is way more stringent but may lead to generalization with high probability jointly over the randomness of sample and algorithm. The present paper addresses such a tension between these two alternatives and makes progress towards relaxing it inside a classic framework of confidence-boosting. To this end, we first introduce a novel concept of  $L_{2}$ -uniform stability that holds uniformly over data but in second-moment over the algorithm's randomness. Then as a core contribution of this work, we prove a strong exponential bound on the first-moment of generalization error under the notion of  $L_{2}$ -uniform stability. As an interesting consequence of the bound, we show that a bagging-based meta algorithm leads to near-optimal generalization with high probability jointly over the randomness of data and algorithm. We further substantialize these generic results to stochastic gradient descent (SGD) to derive sharper exponential bounds for convex or non-convex optimization with natural time-decaying learning rates, which have not been possible to prove with the existing stability-based generalization guarantees.
In many statistical learning problems, we are interested in designing a randomized algorithm  $A: \mathcal{Z}^N \times \mathcal{R} \mapsto \mathcal{W}$  that maps a training data sample  $S = \{Z_i\}_{i \in [N]} \in \mathcal{Z}^N$  with an algorithm's random parameter  $\xi \in \mathcal{R}$  to a model  $A(S, \xi) \in \mathcal{W}$ . Here  $\mathcal{Z}$  and  $\mathcal{R}$  are some measurable sets, and  $\mathcal{W}$  is a closed subset of an Euclidean space. The ultimate goal is to find a suitable algorithm such that the following population risk evaluated at the model should be as small as possible:
$$
R (A (S, \xi)) := \mathbb {E} _ {Z} [ \ell (A (S, \xi); Z) ],
$$
where  $Z \in \mathcal{Z}$  and  $\ell : \mathcal{W} \times \mathcal{Z} \mapsto \mathbb{R}^+$  is a non-negative bounded loss function whose value  $\ell(w; z)$  measures the loss evaluated at  $z$  with parameter  $w$ . It is generally the case that the underlying data distribution is unknown, and in this case the data points  $Z_i$  are usually assumed to be independent.
Then, a natural alternative measurement that mimics the computationally intractable population risk is the empirical risk given by
$$
R _ {S} (A (S, \xi)) := \mathbb {E} _ {Z \sim \mathrm {U n i f} (S)} [ \ell (A (S, \xi); Z) ] = \frac {1}{N} \sum_ {i = 1} ^ {N} \ell (A (S, \xi); Z _ {i}).
$$
The bound on the difference between the population and empirical risks is of central interest in understanding the generalization performance of a learning algorithm. In particular, we hope to derive a suitable law of large numbers, i.e., a sample size vanishing rate  $b_{N}$  such that the generalization bound  $|R_{S}(A(S,\xi)) - R(A(S,\xi))| \lesssim b_{N}$  holds with high probability over the randomness of  $S$  and hopefully the randomness of  $\xi$  as well. Let  $R^{*} \coloneqq \min_{w \in \mathcal{W}} R(w)$  be the optimal value of the population risk. Conditioned on  $S$ , suppose that  $A(S,\xi)$  is an almost minimizer of the empirical risk  $R_{S}$  such that  $R_{S}(A(S,\xi)) - \min_{w \in \mathcal{W}} R_{S}(w) \leq \varepsilon$ , then the generalization bound immediately implies an excess risk bound  $R(A(S,\xi)) - R^{*} \lesssim b_{N} + \frac{1}{\sqrt{N}} + \varepsilon$  based on the standard risk decomposition and Hoeffding's inequality. Therefore, generalization guarantees also play a crucial role in understanding the stochastic optimization performance of a learning algorithm.
A powerful proxy for analyzing the generalization bounds is the stability of learning algorithms to changes in the training dataset. Since the seminal work of Bousquet and Elisseeff (2002), stability has been extensively demonstrated to beget dimension-independent generalization bounds for deterministic learning algorithms (Mukherjee et al., 2006; Shalev-Shwartz et al., 2010), as well as for randomized learning algorithms such as bagging and SGD (Elisseeff et al., 2005; Hardt et al., 2016). So far, the best known results about generalization bounds are offered by approaches based on the notion of uniform stability (Feldman and Vondrak, 2018, 2019; Bousquet et al., 2020; Klochkov and Zhivotovskiy, 2021) which is independent to the underlying distribution of data. For randomized algorithms, the definition of uniform stability can be extended in two natural ways by respectively considering 1) the stability averaged over the algorithm's randomness (Hardt et al., 2016) and 2) the stability conditioned on the algorithm's randomness (Feldman and Vondrak, 2019). The former is simpler to show but typically leads to on-average generalization bounds, while the latter is relatively more stringent but may yield deviation bounds given that the conditional stability holds with high probability over the algorithm's randomness. Between these two extreme cases, however, the generalization behavior of randomized learning algorithm still remains largely under explored.
To address the above mentioned theoretical gap between the current lines of results, we explore the opportunities of deriving exponential generalization bounds for randomized learning algorithms beyond the notions of on-average stability and conditional stability. A concrete working example of our study is the widely used stochastic gradient descent (SGD) algorithm that carries out the following recursion for all  $t \geq 1$  with learning rate  $\eta_t > 0$ :
$$
w _ {t} := \Pi_ {\mathcal {W}} \left(w _ {t - 1} - \eta_ {t} \nabla_ {w} \ell \left(w _ {t - 1}; Z _ {i _ {t}}\right)\right), \tag {1}
$$
where  $i_t \in [N]$  is a random index of data under with or without replacement sampling, and  $\Pi_{\mathcal{W}}$  is the Euclidean projection operator associated with  $\mathcal{W}$ . The in-expectation generalization of SGD has been studied under on-average stability (Hardt et al., 2016; Zhou et al., 2022; Lei and Ying, 2020), while the exponential bounds have recently been established given that the stability holds with high probability over the sampling path of SGD (Feldman and Vondrák, 2019; Bassily et al., 2020).
1.1 Prior results
Let us start by briefly reviewing some state-of-the-art exponential generalization bounds under the notion of uniform stability and its randomized variants. We denote by  $S \doteq \tilde{S}$  if a pair of data sets  $S$  and  $\tilde{S}$  differ in a single element. A randomized learning algorithm  $A$  is said to have on-average  $\gamma_{N}$ -uniform stability (Elisseeff et al., 2005) if it satisfies the following uniform bound:
$$
\sup  _ {S \doteq \tilde {S}, Z \in \mathcal {Z}} \left| \mathbb {E} _ {\xi} \left[ \ell (A (S, \xi); Z) - \ell (A (\tilde {S}, \xi); Z) \right] \right| \leq \gamma_ {N}. \tag {2}
$$
This definition is equivalent to the concept of uniform stability defined for the expectation of loss  $\mathbb{E}_{\xi}[\ell (A(S,\xi);Z)]$ . Suppose that the loss function is bounded in the interval  $[0,M]$ . Then essentially it has been shown in Feldman and Vondrak (2019) that for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over  $S$ , the on-average generalization error is upper bounded by
$$
\left| \mathbb {E} _ {\xi} \left[ R (A (S, \xi)) - R _ {S} (A (S, \xi)) \right] \right| \lesssim \gamma_ {N} \log (N) \log \left(\frac {N}{\delta}\right) + M \sqrt {\frac {\log (1 / \delta)}{N}}. \tag {3}
$$
Bousquet et al. (2020) later derived a slightly improved exponential bound that implies
$$
\left| \mathbb {E} _ {\xi} \left[ R (A (S, \xi)) - R _ {S} (A (S, \xi)) \right] \right| \lesssim \gamma_ {N} \log (N) \log \left(\frac {1}{\delta}\right) + M \sqrt {\frac {\log (1 / \delta)}{N}}. \tag {4}
$$
These bounds are near-tight (up to logarithmic factors) in the sense of an  $\mathcal{O}\left(\gamma_N\log \left(\frac{1}{\delta}\right) + \sqrt{\frac{\log(1 / \delta)}{N}}\right)$  lower deviation bound on sum of random functions with  $\gamma_{N}$ -uniform stability (Bousquet et al., 2020, Proposition 9). Concerning the excess risk bound, Klochkov and Zhivotovsky (2021) essentially derived the following result using the sample-splitting techniques of Bousquet et al. (2020):
$$
\mathbb {E} _ {\xi} \left[ R (A (S, \xi)) \right] - R ^ {*} \lesssim \Delta_ {\mathrm {o p t}} + \mathbb {E} \left[ \Delta_ {\mathrm {o p t}} \right] + \gamma_ {N} \log (N) \log \left(\frac {1}{\delta}\right) + \frac {(M + B) \log (1 / \delta)}{N}, \tag {5}
$$
where  $\Delta_{\mathrm{opt}}\coloneqq \mathbb{E}_{\xi}\left[R_S(A(S,\xi))\right] - \min_{w\in \mathcal{W}}R_S(w)$  represents the in-expectation empirical risk suboptimality, and  $B$  is the constant of the generalized Bernstein condition (Koltchinskii, 2006). While sharp in the dependence on sample size, one common limitation of the above uniform stability implied generalization and risk bounds lies in that these high-probability results only hold in expectation with respect to  $\xi$ , the internal randomness of algorithm.
Alternatively, consider that  $A$  has  $\gamma_N$ -uniform stability with probability at least  $1 - \delta'$  for some  $\delta' \in (0, 1)$  over the random draw of  $\xi$ , i.e.,
$$
\mathbb {P} \left\{\sup  _ {S \dot {=} \tilde {S}, Z \in \mathcal {Z}} | \ell (A (S, \xi); Z) - \ell (A (\tilde {S}, \xi); Z) | \leq \gamma_ {N} \right\} \geq 1 - \delta^ {\prime}. \tag {6}
$$
Suppose that the randomness of  $A$  is independent of the training set  $S$ . Then the bound of Bousquet et al. (2020) naturally implies that with probability at least  $1 - \delta - \delta'$  over  $S$  and  $\xi$ ,
$$
\left| R (A (S, \xi)) - R _ {S} (A (S, \xi)) \right| \lesssim \gamma_ {N} \log (N) \log \left(\frac {1}{\delta}\right) + M \sqrt {\frac {\log (1 / \delta)}{N}}. \tag {7}
$$
This is by far the best known generalization bound of randomized stable algorithms that hold with high probability jointly over the randomness of data and algorithm. The result, however, relies heavily on the high-probability uniform stability as expressed in (6). For the SGD recursion (1) with fixed learning rate  $\eta_t \equiv \eta$ , it is possible to show that  $\gamma_N \lesssim \eta \sqrt{T} + \frac{\eta T}{N}$  and  $\delta' = N \exp(-\frac{N}{2})$  in (6) (Bassily et al., 2020). For SGD with time decaying learning rates, which has been widely studied in theory (Harvey et al., 2019; Rakhlin et al., 2012) and applied in practice for training popular deep nets such as ResNet and DenseNet (Bengio et al., 2017), it is not clear if the condition in (6) is still valid for  $\gamma_N$  and  $\delta'$  of interest. Madden et al. (2020) indeed have established a high-probability uniform stability bound for minibatch SGD with learning rates  $\eta_t \lesssim \frac{1}{Nt}$ . However, such a fairly conservative choice of learning rates tends to impair the empirical minimization performance of SGD and thus is of limited interest from the perspective of risk minimization.
More specially for randomized learning methods such as bagging (Breiman, 1996) and SGD, the randomness of algorithm can be precisely characterized by a vector of i.i.d. parameters  $\xi = \{i_1,\dots,i_t\}$  which are independent on data  $S$ . In such cases, assume additionally that  $A(S,\xi)$  has uniform stability with respect to  $\xi$  conditioned on  $S$ , i.e.,  $\sup_{\xi \dot{=} \tilde{\xi}}|\ell (A(S,\xi)) - \ell (A(S,\tilde{\xi}))|\leq \rho_T$ . Then the following exponential bound has been derived by Elisseeff et al. (2005):
$$
\left| R (A (S)) - R _ {S} (A (S)) \right| \lesssim \gamma_ {N} + \left(\frac {1 + N \gamma_ {N}}{\sqrt {N}} + \sqrt {T} \rho_ {T}\right) \sqrt {\log \left(\frac {1}{\delta}\right)}. \tag {8}
$$
Provided that  $\gamma_N \lesssim \frac{1}{N}$  and  $\rho_T \lesssim \frac{1}{T}$ , the above bound shows that the generalization bound scales as  $\mathcal{O}\left(\frac{1}{\sqrt{N}} + \frac{1}{\sqrt{T}}\right)$  with high probability. However, the rate of the above bound is sub-optimal and will show no guarantee on convergence if  $\gamma_N \gtrsim \frac{1}{\sqrt{N}}$  and/or  $\rho_T \gtrsim \frac{1}{\sqrt{T}}$ . As an example, for non-convex SGD with learning rate  $\eta_t = O\left(\frac{1}{t}\right)$ , it can be shown that  $\gamma_N \lesssim \frac{\sqrt{T}}{N}$  and  $\rho_T$  scales as large as  $\mathcal{O}(1)$ .
Open problem. So far, it still remains open if the exponential generalization bounds for deterministic uniformly stable algorithms might be extended to randomized learning algorithms under the variants of uniform stability tighter than the on-average version (2) but less restrictive than the high-probability
version (6). Particularly, we are interested in the following notion of  $L_{2}$ -uniform stability (as formally introduced in Definition 1) with parameter  $\gamma_{\mathrm{L}_2,N}$ :
$$
\sup  _ {S \div \tilde {S}, Z \in \mathcal {Z}} \mathbb {E} _ {\xi} \left[ \left(\ell (A (S, \xi); Z) - \ell (A (\tilde {S}, \xi); Z)\right) ^ {2} \right] \leq \gamma_ {\mathrm {L} _ {2}, N} ^ {2}, \tag {9}
$$
which represents a second-moment variant of the uniform stability for randomized learning algorithms. For example, as we will shortly show in Section 4 that SGD with practical time-decaying learning rates has  $L_{2}$ -uniform stability with favorable parameters. The main goal of the present work is to derive sharper exponential generalization bounds for randomized learning algorithms under the notion of  $L_{2}$ -uniform stability.
1.2 Overview of our contribution
The fundamental contribution of this work is a near-optimal first-moment generalization error bound for  $L_{2}$ -uniformly stable algorithms, which is summarized in Theorem 1 and highlighted below:
$$
\mathbb {E} _ {\xi} \left[ | R (A (S, \xi)) - R _ {S} (A (S, \xi)) | \right] \lesssim \gamma_ {\mathrm {L} _ {2}, N} \log (N) \log \left(\frac {1}{\delta}\right) + M \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
While our first-moment bound above has an identical convergence rate to that of the on-average bound in (4), the former is stronger in the sense that the expectation is taken outside the generalization gap and thus implies the latter where the expectation is taken inside. The key ingredients of our analysis are a set of fine-grained concentration inequalities for randomized functions (Proposition 1) and sums of randomized functions (Proposition 2), which respectively generalize the classic bounded-difference inequalities and a prior result of Bousquet et al. (2020) under the considered  $L_{2}$ -uniform bounded difference conditions. These generalized concentration inequalities and their proof arguments are novel to our knowledge and should be of independent interests in analyzing randomized functions.
As an important consequence of our main result, we reveal that a bagging-based meta procedure (see Algorithm 1) can be used to boost the confidence of generalization for  $L_{2}$ -uniformly stable algorithms. More specifically, in the presented bagging procedure we independently run a randomized algorithm  $A$  multiple  $K$  times over a fraction of the training set to obtain  $K$  solutions. Then we evaluate the validation error of these candidate solutions over a holdout training subset, and output the solution that has the smallest training-validation gap. Our result in Theorem 2 shows that for any confidence level  $\delta \in (0,1)$ , setting  $K \asymp \log \left(\frac{1}{\delta}\right)$  yields a near-optimal generalization bound for the selected solution that holds with high probability jointly over the randomness of data and algorithm.
We have substantialized our results to SGD with smooth (Corollary 1) or non-smooth (Corollary 2) convex losses, and smooth non-convex losses (Corollary 3) as well. For an instance, our result in Corollary 1 shows that when invoked to SGD with smooth convex loss and learning rates  $\eta_t = \mathcal{O}\left(\frac{1}{\sqrt{t}}\right)$ , the generalization bound of the output of Algorithm 1 may scale as  $\mathcal{O}\left(\log(N)\log\left(\frac{1}{\delta}\right)\sqrt{\frac{\log(T)}{N} + \frac{\sqrt{T}}{N}}\right)$ .
To compare with the  $\mathcal{O}\left(\frac{\sqrt{T}}{N}\right)$  in-expectation bound of smooth convex SGD (Hardt et al., 2016), our bound above for the boosted SGD is comparable in convergence rate while it holds with high probability jointly over the randomness of data and sampling path.
2  $L_{2}$ -Uniform Stability and Generalization
2.1 Notation and definitions
Let us introduce some notation to be used in our analysis. We abbreviate  $[N] := \{1, \dots, N\}$ . Recall that  $S = \{Z_i\}_{i \in [N]}$  is a set of i.i.d. training data. Denote by  $S' = \{Z_i'\}_{i \in [N]}$  an independent copy of  $S$  and we write  $S^{(i)} = \{Z_1, \dots, Z_{i-1}, Z_i', Z_{i+1}, \dots, Z_N\}$ . For a real-valued random variable  $Y$ , its  $L_q$ -norm for  $q \geq 1$  is given by  $\|Y\|_q = (\mathbb{E}[|Y|^q])^{1/q}$ . By definition it can be verified that  $\forall q \geq 2$ ,
$$
\left\| Y \right\| _ {q} ^ {2} = \left(\mathbb {E} [ | Y | ^ {q} ]\right) ^ {2 / q} = \left(\mathbb {E} [ | Y ^ {2} | ^ {q / 2} ]\right) ^ {2 / q} = \left\| Y ^ {2} \right\| _ {q / 2}. \tag {10}
$$
Let  $h: \mathcal{Z}^N \mapsto \mathbb{R}$  be some measurable function and consider the random variable  $h(S) = h(Z_1, \ldots, Z_N)$ . For  $h(S)$  and any index set  $I \subseteq [N]$ , we define the following abbreviations:
$$
h \left(S _ {I}\right) := \mathbb {E} \left[ h (S) \mid S _ {I} \right], \quad \| h \| _ {q} \left(S _ {I}\right) := \left(\mathbb {E} \left[ | h (S) | ^ {q} \mid S _ {I} \right]\right) ^ {1 / q}.
$$
We say a function  $f$  to be  $G$ -Lipschitz continuous over  $\mathcal{W}$  if  $|f(w) - f(\tilde{w})| \leq G\| w - \tilde{w}\|$  for all  $w, \tilde{w} \in \mathcal{W}$ , and it is  $L$ -smooth if  $\| \nabla f(w) - \nabla f(\tilde{w})\| \leq L\| w - \tilde{w}\|$ . For a pair of functions  $f, f' \geq 0$ , we use  $f \lesssim f'$  (or  $f' \gtrsim f$ ) to denote  $f \leq c f'$  for some universal constant  $c > 0$ .
In the following definition, we formally introduce the concept of  $L_{2}$ -uniform stability for randomized learning algorithms to be investigated in this work.
Definition 1 ( $L_{2}$ -Uniform stability of randomized learning algorithms). We say a randomized learning algorithm  $A: \mathcal{Z}^{N} \times \mathcal{R} \mapsto \mathcal{W}$  to have  $L_{2}$ -uniform stability with parameter  $\gamma_{L_{2}, N} \geq 0$  if
$$
\sup  _ {S, Z _ {i} ^ {\prime}, Z} \mathbb {E} _ {\xi} \left[ \left(\ell (A (S, \xi); Z) - \ell (A (S ^ {(i)}, \xi); Z)\right) ^ {2} \right] \leq \gamma_ {L _ {2}, N} ^ {2}.
$$
Remark 1. By definition the  $L_{2}$ -uniform stability has a second-moment dependence on the internal randomness of algorithm conditioned on data, while it is invariant to the data distribution. This justifies the name of such a notion of mixed algorithmic stability.
Remark 2. On one hand, by Jensen's inequality the  $L_{2}$ -uniform stability implies the on-average uniform stability defined in (2). On the other hand, the second-order form of  $L_{2}$ -uniform stability is by definition weaker than the high-probability uniform stability in (6). If the algorithm's randomness  $\xi$  can be expressed as a set of i.i.d. random bits, then the  $L_{2}$ -uniform stability is also weaker than the conditional uniform stability conditioned on data  $S$  (Elisseeff et al., 2005).
Throughout this paper, we assume for simplicity that the output models  $A(S^{(i)},\xi)$  and  $A(S,\xi)$  share the same internal random bit  $\xi$  which is invariant to data. With similar analysis techniques, it is indeed possible to extend Definition 1 and our main results to the general setting where the randomness of algorithm is allowed to be dependent on data, such as in posterior sampling for Bayesian learning.
2.2 Concentration inequalities for randomized functions
We begin by establishing in the following result a group of first- and second-order concentration inequalities (in moments) for randomized functions of independent random variables.
Proposition 1. Let  $S = \{Z_1, Z_2, \dots, Z_N\}$  be a set of independent random variables valued in  $\mathcal{Z}$  and  $\xi$  be a random variable valued in  $\mathcal{R}$ . Let  $g: \mathcal{Z}^N \times \mathcal{R} \mapsto \mathbb{R}$  be a measurable function that satisfies the following  $L_2$ -bounded-difference condition:
$$
\sup _ {S, Z _ {i} ^ {\prime}} \mathbb {E} _ {\xi} \left[ \left(g (S, \xi) - g (S ^ {(i)}, \xi)\right) ^ {2} \right] \leq \beta^ {2}.
$$
Then for any  $q \geq 2$ ,
$$
\left\| \mathbb {E} _ {\xi} \left[ \left| g (S, \xi) - \mathbb {E} _ {S} [ g (S, \xi) ] \right| \right] \right\| _ {q} \leq 3 \beta \sqrt {N q}, \tag {11}
$$
$$
\left\| \mathbb {E} _ {\xi} \left[ (g (S, \xi) - \mathbb {E} _ {S} [ g (S, \xi) ]) ^ {2} \right] \right\| _ {q} \leq 6 8 N \beta^ {2} q. \tag {12}
$$
Proof in sketch. Let us consider  $h(S) \coloneqq \mathbb{E}_{\xi} \left[ \| g(S, \xi) - \mathbb{E}_S[g(S, \xi)] \| \right]$ . The given  $L_2$ -bounded-difference condition implies that  $h(S)$  has bounded-difference property. Then the desired first-order bound in (11) can be obtained by respectively invoking a moment Efron-Stein inequality (Boucheron et al., 2005, Theorem 2) to upper bound  $\| h(S) - \mathbb{E}[h(S)] \|_q$  and a slightly modified Efron-Stein inequality to bound the mean  $\mathbb{E}[h(S)]$ . To prove the second-order concentration bound, we consider the function  $h'(S) \coloneqq \mathbb{E}_{\xi} \left[ (g(S, \xi) - \mathbb{E}_S[g(S, \xi)])^2 \right]$ , which can be shown to be weakly self-bounding (see Definition 2) under the  $L_2$ -bounded-difference condition. Then the desired bound (12) can be derived by applying the upper tail bound of Boucheron et al. (2005, Theorem 6.19) and lower tail bound of Klochkov and Zhivotovsky (2021, Proposition 3.1) for weakly self-bounding functions. See Appendix A.2 for a detailed proof of this result.
The moment bound in (11) extends the McDiarmid's (bounded difference) inequality (McDiarmid et al., 1989) to randomized functions with the  $L_{2}$ -bounded-difference property. The second-order concentration bound in (12) is crucial for proving the moment bound of sums in Proposition 2, as it can be used to sharply control some second-order components involved in the arguments. These generic inequalities are expected to be of independent interests for understanding the first-/second-order concentration behavior of randomized functions.
2.3 A moment inequality for sums of randomized functions
As a key intermediate result, we further establish in the following proposition a moment concentration inequality for sums of randomized functions that satisfy the  $L_{2}$ -bounded-difference condition. This result extends the moment bound for sums of functions (Bousquet et al., 2020, Theorem 4) to sums of randomized functions.
Proposition 2. Let  $S = \{Z_1, Z_2, \dots, Z_N\}$  be a set of independent random variables valued in  $\mathcal{Z}$  and  $\xi$  be a random variable valued in  $\mathcal{R}$ . Let  $g_1, \dots, g_N$  be a set of measurable functions  $g_i : \mathcal{Z}^N \times \mathcal{R} \mapsto \mathbb{R}$  that satisfy the following conditions for any  $i \in [N]$ :