SlowGuess/NeurIPS_2023_OCR huggingface dataset:
& Department of Computer Science
Technion - Israel Institute of Technology;
Hilla Schefler
hillas@campus.technion.ac.il
We show that many definitions of stability found in the learning theory literature are equivalent to one another. We distinguish between two families of definitions of stability: distribution-dependent and distribution-independent Bayesian stability. Within each family, we establish equivalences between various definitions, encompassing approximate differential privacy, pure differential privacy, replicability, global stability, perfect generalization, TV stability, mutual information stability, KL-divergence stability, and Rényi-divergence stability. Along the way, we prove boosting results that enable the amplification of the stability of a learning rule. This work is a step towards a more systematic taxonomy of stability notions in learning theory, which can promote clarity and an improved understanding of an array of stability concepts that have emerged in recent years.
Algorithmic stability is a major theme in learning theory, where seminal results have firmly established its close relationship with generalization. Recent research has further highlighted the intricate interplay between stability and additional properties of interest beyond statistical generalization. These properties encompass privacy [DMNS06], fairness [HKRR18], replicability  $\mathrm{[BGH^{+}23}$ , ILPS22], adaptive data analysis  $\mathrm{[DFH^{+}15b}$ ,  $\mathrm{DFH^{+}15a]}$ , and mistake bounds in online learning [ALMM19, BLM20].
This progress has come with a proliferation of formal definitions of stability, including pure and approximate Differential Privacy [DMNS06,  $\mathrm{DKM}^{+}06$ ], Perfect Generalization  $[\mathrm{CLN}^{+}16]$ , Global Stability [BLM20], KL-Stability [McA99], TV-Stability [KKMV23],  $f$ -Divergence Stability [EGI20], Rényi Divergence Stability [EGI20], and Mutual Information Stability [XR17,  $\mathrm{BMN}^{+}18$ ], as well as related combinatorial quantities such as the Littlestone dimension [Lit87] and the clique dimension [AMSY23].
It is natural to wonder to what extent these various and sundry notions of stability actually differ from one another. The type of equivalence we consider between definitions of stability is as follows.
Definition A and Definition B are weakly equivalent if for every hypothesis class  $\mathcal{H}$  the following holds:
$\mathcal{H}$  has a PAC learning rule that is stable according to Definition  $A$ $\iff \quad \mathcal{H}$  has a PAC learning rule that is stable according to Definition  $B$
This type of equivalence is weak because it does not imply that a learning rule satisfying one definition also satisfies the other.
Recent results show that many stability notions appearing in the literature are in fact weakly equivalent. The work of  $\left[\mathrm{BGH}^{+}23\right]$  has shown sample efficient reductions between approximate differential privacy, replicability, and perfect generalization. Combined with the work of  $\left[\mathrm{ABL}^{+}22, \mathrm{ILPS}22, \mathrm{KKMV}23, \mathrm{MM}22\right]$ , a rich web of equivalences is being uncovered between approximate differential privacy and other definitions of algorithmic stability (see Fig. 1).
In this paper we extend the study of equivalences between notions of stability, and make it more systematic. Our starting point is the following observation: many of the definitions mentioned above belong to a broad family of definitions of stability, which we informally call Bayesian definitions of stability. Definitions in this family roughly take the following form: a learning rule  $A$  is considered stable if the quantity
$$
d \Big (A (S), \mathcal {P} \Big)
$$
is small enough, where:
-  $d$  is a measure of dissimilarity between distributions.
-  $\mathcal{P}$  is a specific prior distribution over hypotheses;
-  $A(S)$  is the posterior distribution, i.e., the distribution of hypotheses generated by the learning rule  $A$  when applied to the input sample  $S$ .
Namely, a Bayesian definition of stability is parameterized by a choice of  $d$ , a choice of  $\mathcal{P}$ , and a specification of how small the dissimilarity is required to be.
Remark 1.1. To understand our choice of the name Bayesian stability, recall that the terms prior and posterior come from Bayesian statistics. In Bayesian statistics the analyst has some prior distribution over possible hypothesis before conducting the analysis, and chooses a posterior distribution over hypotheses when the analysis is complete. Bayesian stability is defined in terms of the dissimilarity between these two distributions.
A central insight of this paper is that there exists a meaningful distinction between two types of Bayesian definitions, based on whether the choice of the prior  $\mathcal{P}$  depends on the population distribution  $\mathcal{D}$ :
- Distribution-independent (DI) stability. These are Bayesian definitions of stability in which  $\mathcal{P}$  is some fixed prior that depends only on the class  $\mathcal{H}$  and the learning rule  $A$ , and does not depend on the population distribution  $\mathcal{D}$ . Namely, they take the form:
$\exists$  prior  $\mathcal{P} \forall$  population  $\mathcal{D} \forall m \in \mathbb{N}$ :  $d(A(S), \mathcal{P})$  is small,
where  $S\sim \mathcal{D}^m$
- Distribution-dependent (DD) stability. Here, the prior may depend also on  $\mathcal{D}$ , so each population distribution  $\mathcal{D}$  might have a different prior. Namely:
$\forall$  population  $\mathcal{D}\exists$  prior  $\mathcal{P}_{\mathcal{D}}\forall m\in \mathbb{N}:d(A(S),\mathcal{P}_{\mathcal{D}})$  is small.
A substantial body of literature has investigated the interconnections among distribution-dependent definitions. In Theorem 1.4, we provide a comprehensive summary of the established equivalences. A
<sup>1</sup>An example for an application in the context of generalization is the classic PAC Bayes Theorem. The theorem assures that for every population distribution and any given prior  $\mathcal{P}$ , the difference between the population error of an algorithm  $A$  and the empirical error of  $A$  is bounded by  $\tilde{O}\left(\frac{\sqrt{\mathrm{KL}(A(S),\mathcal{P})}}{m}\right)$ , where  $m$  is the size of the input sample  $S$ , and the KL divergence is the "measure of dissimilarity" between the prior and the posterior. See e.g. Theorem 3.2.
natural question arises as to whether a similar web of equivalences exists for distribution-independent definitions. Our principal contribution is to affirm that, indeed, such a network exists. Identifying such equivalences is a step towards creating a comprehensive taxonomy of stability definitions.
Our first main contribution is an equivalence between distribution-independent definitions of stability.
Theorem (Informal Version of Theorem 2.1). The following definitions of stability are weakly equivalent:
1. Pure Differential Privacy; (Definition 3.5)
2. Distribution-Independent KL-Stability; (Definition 3.6)
3. Distribution-Independent One-Way Pure Perfect Generalization; (Definition 3.7)
4. Distribution-Independent  $\mathsf{D}_{\alpha}$  -Stability for  $\alpha \in (1,\infty)$  (Definition 3.6)
Where  $\mathsf{D}_{\alpha}$  is the Renyi divergence of order  $\alpha$ . Furthermore, a hypothesis class  $\mathcal{H}$  has a PAC learning rule that is stable according to one of these definitions if and only if  $\mathcal{H}$  has finite fractional clique dimension (See Appendix B.1).
Remark 1.2. Observe that  $DI$  KL-stability is equivalent to  $DI \, \mathsf{D}_1$ -stability, and  $DI$  one-way pure perfect generalization is equivalent to  $DI \, \mathsf{D}_{\infty}$ -stability. Therefore, The above theorem can be viewed as stating a weak equivalence between pure differential privacy and  $\mathsf{D}_{\alpha}$ -stability for  $\alpha \in [1,\infty]$ .
Remark 1.3. In this paper we focus purely on the information-theoretic aspects of learning under stability constraints, and therefore we consider learning rules that are mathematical functions, and disregard considerations of computability and computational complexity.
Table 1 summarizes the distribution-independent definitions discussed in Theorem 2.1. All the definitions in each row are weakly equivalent.
Table 1: Distribution-independent Bayesian definitions of stability.
Name	Dissimilarity	Definition
KL-Stability	PS[KL(A(S) || P) ≤ o(m)] ≥ 1 - o(1)	3.6
Dα-Stability	PS[Dα(A(S) || P) ≤ o(m)] ≥ 1 - o(1)	3.6
Pure Perfect Generalization	PS[∀O : A(S)(O) ≤ eo(m)P(O)] ≥ 1 - o(1)	3.7
One example for how the equivalence results can help build bridges between different stability notions in the literature is the connection between pure differential privacy and the PAC-Bayes theorem. Both of these are fundamental ideas that have been extensively studied. Theorem 2.1 states that a hypothesis class admits a pure differentially private PAC learner if and only if it admits a distribution independent KL-stable PAC learner. This is an interesting and non-trivial connection between two well studied notions. As a concrete example of this connection, recall that thresholds over the real line cannot be learned by a differentially private learner [ALMM19]. Hence, by Theorem 2.1, there does not exist a PAC learner for thresholds that is KL-stable. Another example is half-spaces with margins in  $\mathbb{R}^d$ . Half-spaces with margins are differentially private learnable [?], therefore there exists a PAC learner for half-spaces with margins that is KL-stable.
Our second main contribution is a boosting result for weak learners that have bounded KL-divergence with respect to a distribution-independent prior. Our result demonstrates that distribution-independent KL-stability is boostable. It is interesting to see that one can simultaneously boost both the stability and the learning parameters of an algorithm.
Theorem (Informal Version of Theorem 2.2). Let  $\mathcal{H}$  be a hypothesis class. If there exists a weak learner  $A$  for  $\mathcal{H}$ , and there exists a prior distribution  $\mathcal{P}$  such that the expectation of  $\mathrm{KL}(A(S)\parallel \mathcal{P})$  is bounded, then there exists a KL-stable PAC learner that admits a logarithmic divergence bound.
The proof of Theorem 2.2 relies on connections between boosting of PAC learners and online learning with expert advice.
Figure 1: A summary of equivalences between distribution-dependent definitions of stability (Theorem 1.4). A solid black arrow from  $A$  to  $B$  means that definition  $A$  weakly implies definition  $B$ . A dashed blue arrow from  $A$  to  $B$  means that  $A$  weakly implies  $B$  only if the domain  $\mathcal{X}$  is countable. A dotted red arrow from  $A$  to  $B$  means that  $A$  weakly implies  $B$  only if the domain  $\mathcal{X}$  is finite. A double brown arrow from  $A$  to  $B$  means that every learning rule that satisfies definition  $A$  also satisfies definition  $B$ .
Edge key:
1. Lemma C.6
2. Lemma C.13 ([PNG22])
3. Lemma C.11
4. Lemma C.8 ([LM20])
5. Corollary 2, [ALMM19]
6. Theorem 10, [BLM20]
7. Theorem 17, [BLM20]
8. Corollary 3.13,  $[\mathrm{BGH}^{+}23]$
9. Lemma 3.14,  $\left[\mathrm{BGH}^{+}23\right]$
10. Theorem 3.17,  $\left[\mathrm{BGH}^{+}23\right]$
11. Theorem 3.19. [BGH+23]
12. Theorem 3.1,  $[\mathrm{BGH}^{+}23]$
13. Theorem 1, [KKMV23]
14. Theorem 2, [KKMV23]
15. Theorem 4, [KKMV23]
16. Theorem 3, [KKMV23]
17. Theorems 2.2, 2.7 [AMSY23]
Lastly, after conducting an extensive review of the literature, we have compiled a comprehensive network of equivalence results for distribution-dependent definitions of stability. This network is presented in Theorem 1.4, Figure 1, and Table 2.
Theorem 1.4 (Distribution-Dependent Equivalences;  $\mathrm{[ABL^{+}22}$  ILPS22, MM22, PNG22,  $\mathrm{BGH^{+}23}$  KKMV23]). The following definitions of stability are weakly equivalent with respect to an arbitrary hypothesis class  $\mathcal{H}$ :
1.Approximate Differential Privacy; (Definition 3.5)
2. Distribution-Dependent KL-Stability; (Definition 3.6)
3. Mutual-Information Stability; (Definition 3.12)
4. Global Stability. (Definition 3.11)
If the domain is countable then the following are also weakly equivalent to the above:
5. Distribution-Dependent TV-Stability; (Definition 3.13)
6. Replicability. (Definition 3.8)
If the domain is finite then the following are also weakly equivalent to the above:
7. One-Way Perfect Generalization; (Definition 3.7)
8. Max Information. (Definition 3.14)
Furthermore, for any hypothesis class  $\mathcal{H}$ , the following conditions are equivalent:
-  $\mathcal{H}$  has a PAC learning rule that is stable according to one of the definitions 1 to 6 (and the cardinality of the domain is as described above);
-  $\mathcal{H}$  has finite Littlestone dimension; (Definition C.3)
-  $\mathcal{H}$  has finite clique dimension. (Definition C.5)
We emphasize that Theorem 1.4 is a summary of existing results, and is not a new result. We believe that our compilation serves as a valuable resource, and that stating these results here in a unified framework helps to convey the conceptual message of this paper. Namely, the fact that a large number of disparate results can neatly be organized based on our notions of distribution-dependent and distribution-independent definitions of stability is a valuable observation that can help researchers make sense of the stability landscape.
Table 2: Distribution-dependent Bayesian definitions of stability.
Name	Dissimilarity	Definition	References
KL-Stability	PS[KL(A(S) || P_D) ≤ o(m)] ≥ 1 - o(1)	3.6	[McA99]
TV-Stability	ES[TV(A(S), P_D)] ≤ o(1)	3.13	[KKMV23]
MI-Stability	ES[KL(A(S) || P_D)] ≤ o(m)	3.12	[XR17, BMN+18]
Perfect Generalization	PS[∀O : A(S)(O) ≤ e^εP_D(O) + δ] ≥ 1 - o(1)	3.7	[CLN+16]
Global Stability	PS,h~P_D[A(S) = h] ≥ η	3.11	[BLM20]
Replicability	PR~R[PS,h_r~P_D,r[A(S;r) = hr] ≥ η] ≥ η	3.10	[BGH+23, ILPS22]
The literature on stability is vast. Stability has been studied in the context of optimization, statistical estimation, regularization (e.g., [Tik43] and [Phi62]), the bias-variance tradeoff, algorithmic stability (e.g., [BE02]; see bibliography in Section 13.6 of [SB14]), bagging [Bre96], online learning and optimization and bandit algorithms (e.g., [Han58]; see bibliography in Section 28.6 of [LS20]), and other topics.
There are numerous definitions of stability, including pure and approximate Differential Privacy [DMNS06,  $\mathrm{DKM}^{+}06$ ], Perfect Generalization  $[\mathrm{CLN}^{+}16]$ , Global Stability [BLM20], KL-Stability [McA99], TV-Stability [KKMV23],  $f$ -Divergence Stability [EGI20], Rényi Divergence Stability [EGI20], and Mutual Information Stability [XR17,  $\mathrm{BMN}^{+}18$ ].
Our work is most directly related to the recent publication by Bun et al.  $\mathrm{[BGH^{+}23]}$ . They established connections and separations between replicability, approximate differential privacy, max-information and perfect generalization for a broad class of statistical tasks. The reductions they present are sample-efficient, and nearly all are computationally efficient and apply to a general outcome space. Their results are central to the understanding of equivalences between notions of stability as laid out in the current paper.
A concurrent work by Kalavasis et al. [KKMV23] showed that TV-stability, replicability and approximate differential privacy are equivalent; this holds for general statistical tasks on countable domains, and for PAC learning on any domain. They also provide a statistical amplification and TV-stability boosting algorithm for PAC learning on countable domains.
Additionally, recent works [AUZ23, HKMN23] have shown an equivalence between differential privacy and robustness for estimation tasks.
Theorem 2.2 is a boosting result. Boosting has been a central topic of study in computational learning theory since its inception in the 1990s by Schapire [?] and Freund [Fre95]. The best-known boosting algorithm is AdaBoost [FS97], which has been extensively studied. Boosting also has rich connections with other topics such as game theory, online learning, and convex optimization (see [SF12], Chapter 10 in [SB14], and Chapter 7 in [MRT18]).
This section presents the complete versions of Theorems 1.4 and 2.2. We provide a concise overview of the key ideas and techniques employed in the proofs. All proofs appear in the appendices.
Please refer to Section 3 for a complete overview of preliminaries, including all technical terms and definitions.
2.1 Equivalences between DI Bayesian Notions of Stability
The following theorem, which is one of the main results of this paper, shows the equivalence between different distribution-independent definitions. The content of Theorem 2.1 is summarized in Table 1.
Theorem 2.1 (Distribution-Independent Equivalences). Let  $\mathcal{H}$  be a hypothesis class. The following is equivalent.
1. There exists a learning rule that PAC learns  $\mathcal{H}$  and satisfied pure differential privacy (Definition 3.5).
2.  $\mathcal{H}$  has finite fractional clique dimension.
3. For every  $\alpha \in [1,\infty ]$ , there exists a learning rule that PAC learns  $\mathcal{H}$  and satisfies distribution-independent  $\mathrm{D}_{\alpha}$ -stability (Definition 3.6).
4. For every  $\alpha \in [1,\infty ]$ , there exists a distribution-independent  $\mathsf{D}_{\alpha}$ -stable PAC learner  $A$  for  $\mathcal{H}$ , that satisfies the following:
(i)  $A$  is interpolating almost surely. Namely, for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ ,  $\mathbb{P}_{S\sim \mathcal{D}^m}[\mathrm{L}_S(A(S)) = 0] = 1$ .
(ii) A admits a divergence bound of  $f(m) = O(\log m)$ , with confidence  $\beta(m) \equiv 0$ . I.e., for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ ,  $\mathsf{D}_{\alpha}(A(S) \| \mathcal{P}) \leq O(\log m)$  with probability 1, where  $S \sim \mathcal{D}^m$  and  $\mathcal{P}$  is a prior distribution independent of  $\mathcal{D}$ .
(iii) For every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , the expected population loss of  $A$  with respect to  $\mathcal{D}$  satisfies  $\mathbb{E}_{S\sim \mathcal{D}^m}[\mathrm{L}_\mathcal{D}(A(S))] \leq O\left(\sqrt{m^{-1}\log m}\right)$ .
In particular, plugging  $\alpha = 1$  in Item (ii) implies KL-stability with divergence bound of  $f(m) = O(\log m)$  and confidence  $\beta(m) \equiv 0$ . Plugging  $\alpha = \infty$  implies distribution-independent one-way  $\varepsilon$ -pure perfect generalization, with  $\varepsilon(m) \leq O(\log m)$  and confidence  $\beta(m) \equiv 0$ .
2.1.1 Proof Idea for Theorem 2.1
We prove the following chain of implications:
$$
\text {P u r e} \mathrm {D P} \xrightarrow {(1)} \mathrm {D} _ {\infty} - \text {S t a b i l i t y} \xrightarrow {(2)} \mathrm {D} _ {\alpha} - \text {S t a b i l i t y} \forall \alpha \in [ 1, \infty ] \xrightarrow {(3)} \text {P u r e} \mathrm {D P}.
$$
Pure DP  $\Rightarrow$  D $_{\infty}$ -Stability. The first step towards proving implication (1) is to define a suitable prior distribution  $\mathcal{P}$  over hypotheses. The key tool we used in order to define  $\mathcal{P}$  is the characterization of pure DP via the fractional clique dimension [AMSY23]. In a nutshell, [AMSY23] proved that (i) a class  $\mathcal{H}$  is pure DP learnable if and only if the fractional clique dimension of  $\mathcal{H}$  is finite; (ii) the fractional clique dimension is finite if and only if there exists a polynomial  $q(m)$  and a distribution over hypothesis  $\mathcal{P}_m$ , such that for every realizable sample  $S$  of size  $m$ , we have
$$
\mathbb {P} _ {h \sim \mathcal {P} _ {m}} [ \mathrm {L} _ {S} (h) = 0 ] \geq \frac {1}{q (m)}. \tag {1}
$$
(For more details please refer to Appendix B.1.) Now, the desired prior distribution  $\mathcal{P}$  is defined to be a mixture of all the  $\mathcal{P}_m$ 's.
The next step in the proof is to define a learning rule  $A$ : (i) sample hypotheses from the prior  $\mathcal{P}$ ; (ii) return the first hypothesis  $h$  that is consistent with the input sample  $S$  (i.e.  $\mathrm{L}_S(h) = 0$ ).  $A$  is well-defined since with high probability it will stop and return a hypothesis after  $\approx q(m)$  re-samples from  $\mathcal{P}$ . Since the posterior  $A(S)$  is supported on  $\{h : \mathrm{L}_S(h) = 0\}$ , a simple calculation which follows from Equation (1) shows that for every realizable distribution  $\mathcal{D}$ ,  $\mathrm{D}_{\infty}(A(S) \| \mathcal{P}) \leq \log(q(m))$  almost surly where  $S \sim \mathcal{D}^m$ .
Finally, since for  $\alpha \in [1,\infty]$  the Rényi divergence  $\mathsf{D}_{\alpha}(\mathcal{Q}_1\parallel \mathcal{Q}_2)$  is non-decreasing in  $\alpha$  (see Lemma A.1), we conclude that  $\mathsf{KL}(A(S)\parallel \mathcal{P})\leq O(\log m)$ , hence by PAC-Bayes theorem  $A$  generalizes.
$\mathsf{D}_{\infty}$  -Stability  $\Rightarrow \mathsf{D}_{\alpha}$  -Stability  $\forall \alpha \in [1,\infty ]$  . This implication is immediate since the Renyi divergence  $\mathsf{D}_{\alpha}(\mathcal{Q}_1\parallel \mathcal{Q}_2)$  is non-decreasing in  $\alpha$
$\mathsf{D}_{\alpha}$ -Stability  $\forall \alpha \in [1,\infty] \Rightarrow \mathbf{PureDP}$ . In fact, it suffices to assume KL-stability. We prove that the promised prior  $\mathcal{P}$  satisfies that for every realizable sample  $S$  of size  $m$ , we have  $\mathbb{P}_{h\sim \mathcal{P}}[\mathrm{L}_S(h) = 0] \geq \frac{1}{\mathrm{poly}(m)}$ , and conclude that  $\mathcal{H}$  is pure DP learnable. Given a realizable sample  $S$  of size  $m$ , we uniformly sample  $\approx m\log m$  examples from  $S$  and feed the new sample  $S'$  to the promised KL-stable learner  $A$ . By noting that if  $\mathsf{KL}(A(S') \parallel \mathcal{P})$  is small, one can lower bound the probability of an event according to  $\mathcal{P}$  by its probability according to  $A(S')$ . The proof then follows by applying a standard concentration argument.
2.2 Stability Boosting
We prove a boosting result for weak learners with bounded KL with respect to a distribution-independent prior. We show that every learner with bounded KL that slightly beats random guessing can be amplified to a learner with logarithmic KL and expected loss of  $O(\sqrt{m^{-1}\log m})$ .
Theorem 2.2 (Boosting Weak Learners with Bounded KL). Let  $\mathcal{X}$  be a set, let  $\mathcal{H} \subseteq \{0,1\}^{\mathcal{X}}$  be a hypothesis class, and let  $A$  be a learning rule. Assume there exists  $k \in \mathbb{N}$  and  $\gamma > 0$  such that
$$
\forall \mathcal {D} \in \operatorname {R e a l i z a b l e} (\mathcal {H}): \mathbb {E} _ {S \sim \mathcal {D} ^ {k}} [ \mathrm {L} _ {\mathcal {D}} (A (S)) ] \leq \frac {1}{2} - \gamma , \tag {2}
$$
and there exists  $\mathcal{P} \in \Delta\big(\{0,1\}^{\mathcal{X}}\big)$  and  $b \geq 0$  such that
$$
\forall \mathcal {D} \in \operatorname {R e a l i z a b l e} (\mathcal {H}): \mathbb {E} _ {S \sim \mathcal {D} ^ {k}} [ \mathrm {K L} (A (S) \| \mathcal {P}) ] \leq b. \tag {3}
$$
Then, there exists an interpolating learning rule  $A^{\star}$  that PAC learns  $\mathcal{H}$  with logarithmic KL-stability. More explicitly, there exists a prior distribution  $\mathcal{P}^{\star} \in \Delta\left(\{0,1\}^{\mathcal{X}}\right)$  and function  $b^{\star}$  and  $\varepsilon^{\star}$  that depend on  $\gamma$  and  $b$  such that
$$
\forall \mathcal {D} \in \operatorname {R e a l i z a b l e} (\mathcal {H}) \forall m \in \mathbb {N}:
$$
$$
\mathbb {P} _ {S \sim \mathcal {D} ^ {m}} [ \mathrm {K L} (A ^ {\star} (S) \| \mathcal {P} ^ {\star}) \leq b ^ {\star} (m) = O (\log (m)) ] = 1, \tag {4}
$$
$$
\mathbb {E} _ {S \sim \mathcal {D} ^ {m}} \left[ \mathrm {L} _ {\mathcal {D}} \left(A ^ {\star} (S)\right) \right] \leq \varepsilon^ {\star} (m) = O \left(\sqrt {\frac {\log (m)}{m}}\right). \tag {5}
$$
2.2.1 Proof Idea for Theorem 2.2