SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pasin Manurangsi  
Google Research  
Bangkok, Thailand  
pasin@google.com
Raghu Meka  
UCLA  
Los Angeles, CA, US  
raghum@cs.ucla.edu
Chiyuan Zhang  
Google Research  
Mountain View, CA, US  
chiyuan@google.com
Previous work on user-level differential privacy (DP) [GKM21,  $\mathrm{BGH}^{+}23$ ] obtained generic algorithms that work for various learning tasks. However, their focus was on the example-rich regime, where the users have so many examples that each user could themselves solve the problem. In this work we consider the example-scarce regime, where each user has only a few examples, and obtain the following results:
- For approximate-DP, we give a generic transformation of any item-level DP algorithm to a user-level DP algorithm. Roughly speaking, the latter gives a (multiplicative) savings of  $O_{\varepsilon, \delta}(\sqrt{m})$  in terms of the number of users required for achieving the same utility, where  $m$  is the number of examples per user. This algorithm, while recovering most known bounds for specific problems, also gives new bounds, e.g., for PAC learning.
- For pure-DP, we present a simple technique for adapting the exponential mechanism [MT07] to the user-level setting. This gives new bounds for a variety of tasks, such as private PAC learning, hypothesis selection, and distribution learning. For some of these problems, we show that our bounds are near-optimal.
Differential privacy (DP) [DMNS06,  $\mathrm{DKM}^{+}06$ ] has become a widely popular notion of privacy quantifying a model's leakage of personal user information. It has seen a variety of industrial and governmental deployments including [Gre16, App17, DKY17, Abo18, RE19, TM20]. Many fundamental private machine learning algorithms in literature have been devised under the (implicit) assumption that each user only contributes a single example to the training dataset, which we will refer to as item-level  $DP$ . In practice, each user often provides multiple examples to the training data. In this user-level  $DP$  setting, the output of the algorithm is required to remain (approximately) statistically indistinguishable even when all the items belonging to a single user are substituted (formal definitions below). Given such a strong (and preferable) privacy requirement, it is highly non-trivial for the algorithm to take advantage of the increased number of examples. Several recent studies have explored this question and showed—for a wide variety of tasks—that, while challenging, this is possible [LSY+20, LSA+21, GKM21, ILPS22, BGH+23].
To discuss further, let us define some notation. We use  $n$  to denote the number of users and  $m$  to denote the number of examples per user. We use  $\mathcal{Z}$  to denote the universe of possible samples. We write  $\pmb{x} = (x_{1,1},\dots,x_{1,m},x_{2,1},\dots,x_{n,m})\in \mathcal{Z}^{nm}$  to denote the input to our algorithm, and
$\pmb{x}_i \coloneqq (x_{i,1}, \dots, x_{i,m})$  for  $i \in [n]$  to denote the  $i$ th user's examples. Furthermore, we write  $\pmb{x}_{-i}$  to denote the input with the  $i$ th user's data  $\pmb{x}_i$  removed. Similarly, for  $S \subseteq [n]$ , we write  $\pmb{x}_{-S}$  to denote the input with all the examples of users in  $S$  removed.
Definition 1 (User-Level Neighbors). Two inputs  $\pmb{x}$ ,  $\pmb{x}'$  are user-level neighbors, denoted by  $\pmb{x} \asymp \pmb{x}'$ , iff  $\pmb{x}_{-i} = \pmb{x}_{-i}'$  for some  $i \in [n]$ .
Definition 2 (User-Level and Item-Level DP). For  $\varepsilon, \delta > 0$ , we say that a randomized algorithm $^1$ $\mathbb{A} : \mathcal{Z} \to \mathcal{O}$  is  $(\varepsilon, \delta)$ -user-level DP iff, for any  $\pmb{x} \asymp \pmb{x}'$  and any  $S \subseteq \mathcal{O}$ , we have  $\operatorname*{Pr}[\mathbb{A}(\pmb{x}) \in S] \leq e^{\varepsilon} \operatorname*{Pr}[\mathbb{A}(\pmb{x}') \in S] + \delta$ . When  $m = 1$ , we say that  $\mathbb{A}$  is  $(\varepsilon, \delta)$ -item-level DP.
The case  $\delta = 0$  is referred to as pure-  $DP$ , whereas the case  $\delta > 0$  is referred to as approximate-  $DP$ .
To formulate statistical tasks studied in our work, we consider the setting where there is an unknown distribution  $\mathcal{D}$  over  $\mathcal{Z}$  and the input  $\pmb{x} \sim \mathcal{D}^{nm}$  consists of  $nm$  i.i.d. samples drawn from  $\mathcal{D}$ . A task  $\mathfrak{T}$  (including the desired accuracy) is defined by  $\Psi_{\mathfrak{T}}(\mathcal{D})$ , which is the set of "correct" answers. For a parameter  $\gamma$ , we say that the algorithm  $\mathbb{A}$  is  $\gamma$ -useful if  $\operatorname*{Pr}_{\pmb{x} \sim \mathcal{D}^{nm}}[\mathbb{A}(\pmb{x}) \in \Psi_{\mathfrak{T}(\mathcal{D})}] \geq \gamma$  for all valid  $\mathcal{D}$ . Furthermore, we say that  $nm$  is the sample complexity of the algorithm, whereas  $n$  is its user complexity. For  $m \in \mathbb{N}$  and  $\varepsilon, \delta > 0$ , let  $n_m^{\mathfrak{T}}(\varepsilon, \delta; \gamma)$  denote the smallest user complexity of any  $(\varepsilon, \delta)$ -user-level DP algorithm that is  $\gamma$ -useful for  $\mathfrak{T}$ . When  $\gamma$  is not stated, it is assumed to be  $2/3$ .
Generic Algorithms Based on Stability. So far, there have been two main themes of research on user-level DP learning. The first aims to provide generic algorithms that work with many tasks. Ghazi et al. [GKM21] observed that any pseudo-globally stable (aka reproducible [ILPS22]) algorithm can be turned into a user-level DP algorithm. Roughly speaking, pseudo-global stability requires that the algorithm, given a random input dataset and a random string, returns a canonical output—which may depend on the random string—with a large probability (e.g., 0.5). They then show that the current best generic PAC learning item-level DP algorithms from [GGKM21] (which are based on yet another notion of stability) can be made into pseudo-globally stable algorithms. This transforms the aforementioned item-level DP algorithms into user-level DP algorithms.
In a recent breakthrough, Bun et al.  $\mathrm{[BGH^{+}23]}$  significantly expanded this transformation by showing that any item-level DP algorithm can be compiled into a pseudo-globally stable algorithm—albeit with some overhead in the sample complexity. Combining with [GKM21], they then get a generic transformation of any item-level DP algorithm to a user-level DP algorithm. Such results are of (roughly) the following form: If there is an item-level DP algorithm for some task with sample complexity  $n$ , then there is a user-level DP algorithm with user complexity  $O(\log(1/\delta)/\varepsilon)$  as long as  $m \geq \widetilde{\Omega}_{\varepsilon,\delta}(n^2)$ . In other words, if each user has sufficiently many samples, the algorithm needs very few users to learn under the user-level DP constraint. However, the requirement on the number of examples per user can be prohibitive: due to the nature of the reduction, it requires each user to have enough samples to learn by themselves. This leads to the following natural question: $^2$
Is there a generic transformation from item-level to user-level DP algorithms for small  $m$ ?
Algorithms for Specific Tasks & the  $\sqrt{m}$  Savings. Meanwhile, borrowing a page from its item-level DP counterpart, another active research theme has been to study specific tasks and provide (tight) upper and lower bounds on their sample complexity for user-level DP. Problems such as mean estimation, stochastic convex optimization (SCO), and discrete distribution learning are well-understood under user-level DP (for approximate-DP); see, e.g., [LSY+20, LSA+21, NME22, GKK+23]. A pattern has emerged through this line of studies: (the privacy-dependent part of) the user complexity in the user-level DP setting is often roughly  $1 / \sqrt{m}$  smaller than that of the item-level DP setting. For example, for the task of discrete distribution learning on domain  $[k]$  up to total variation distance of  $\alpha > 0$  (denoted by DDk( $\alpha$ )), the user complexity for the user-level DP is [DHS15, ASZ21, NME22]
$$
n _ {m} ^ {\mathrm {D D} k (\alpha)} (\varepsilon , \delta) = \tilde {\Theta} _ {\delta} \left(\frac {k}{\varepsilon \alpha \sqrt {m}} + \frac {k}{\alpha^ {2} m}\right).
$$
<sup>1</sup>For simplicity, we will only consider the case where the output space  $\mathcal{O}$  is finite throughout this work to avoid measure-theoretic issues. This assumption is often without loss of generality since even the output in continuous problems can be discretized in such a way that the "additional error" is negligible.
2Of course, there is a generic transformation where each user throws away all but one example; here, we are looking for one that can reduce the user complexity compared to the item-level setting.
We can see that the first privacy-dependent term decreases by a factor of  $\sqrt{m}$  whereas the latter privacy-independent term decreases by a factor of  $m$ . A similar phenomenon also occurs in the other problems discussed above. As such, it is intriguing to understand this question further:
Is there a common explanation for these  $\sqrt{m}$  saving factors for the privacy-dependent term?
Approximate-DP. We answer both questions by showing a generic way to transform any item-level DP algorithm into a user-level DP algorithm such that the latter saves roughly  $\sqrt{m}$  in terms of the number of users required. Since the formal expression is quite involved, we state below a simplified version, which assumes that the sample complexity can be written as the sum of two terms, the privacy-independent term (i.e.,  $n_{\mathrm{stat}}^{\mathfrak{T}}$ ) and the privacy-dependent term that grows linearly in  $1 / \varepsilon$  and polylogarithmically in  $1 / \delta$  (i.e.,  $\left(\frac{\log(1 / \delta)^{O(1)}}{\varepsilon}\right)\cdot n_{\mathrm{priv}}^{\mathfrak{T}}$ ). This is the most common form of sample complexity of DP learning discovered so far in the literature—in particular, this covers all the tasks we have discussed previously. The corollary below shows that the privacy-dependent term decreases by a factor of roughly  $\sqrt{m}$  whereas the privacy-independent term decreases by a factor of  $m$ . The full version of the statement can be found in Theorem 10.
Corollary 3 (Main Theorem-Simplified). Let  $\mathfrak{T}$  be any task and  $\gamma >0$  be a parameter. Suppose that for any sufficiently small  $\varepsilon, \delta >0$ , there is an  $(\varepsilon, \delta)$ -item-level DP algorithm that is  $\gamma$ -useful for  $\mathfrak{T}$  with sample complexity  $\widetilde{O}\left(\frac{\log(1 / \delta)^c}{\varepsilon} \cdot n_{\text{priv}}^{\mathfrak{T}} + n_{\text{stat}}^{\mathfrak{T}}\right)$  for some constant  $c$ . Then, for any sufficiently small  $\varepsilon, \delta >0$ , there exists an  $(\varepsilon, \delta)$ -user-level DP algorithm that is  $(\gamma - o(1))$ -useful for  $\mathfrak{T}$  and has user complexity
$$
\widetilde {O} \left(\frac {1}{\sqrt {m}} \cdot \frac {\log (1 / \delta) ^ {c + 3 / 2}}{\varepsilon^ {2}} \cdot n _ {p r i v} ^ {\mathfrak {T}} + \frac {1}{m} \cdot n _ {s t a t} ^ {\mathfrak {T}} + \frac {\log (1 / \delta)}{\varepsilon}\right).
$$
Our result provides a powerful and generic tool in learning with user-level DP, as it can translate any item-level DP bound to user-level DP bound. Specifically, up to polylogarithmic terms and the dependency on  $\varepsilon$ , the corollary above recovers all the aforementioned user complexity bounds for mean estimation, SCO, and discrete distribution learning. In addition, it also presents new sample complexity bounds for the moderate  $m$  regime for PAC learning.
While our result is very general, it—perhaps inevitably—comes at a cost: the algorithm is computationally inefficient and the dependence on  $\varepsilon$  (and  $\delta$ ) is not tight. We discuss this—along with other open questions—in more detail in Section 5.
Pure-DP. Surprisingly, the pure-DP setting is less explored compared to approximate-DP in the context of learning with user-level DP; the sample complexity of several of the problems mentioned above (e.g., discrete distribution learning and SCO) is well-understood under approximate-DP but not so under pure-DP. While we do not provide a generic transformation from item-level DP algorithms to user-level DP algorithms for pure-DP, we give simple modifications of the popular pure-DP exponential mechanism [MT07] that allow it to be used in the user-level DP setting. Consequently, we obtain improved bounds for several problems. Due to space constraints, in the main body of the paper, we will only discuss PAC learning. Results for other problems, such as hypothesis testing and distribution learning can be found in Appendix E.
In PAC learning,  $\mathcal{Z} = \mathcal{X}\times \{0,1\}$  and there is a concept class  $\mathcal{C}\subseteq \{0,1\}^{\mathcal{X}}$ . An error of  $c:\mathcal{Z}\to$ $\{0,1\}$  w.r.t.  $\mathcal{D}$  is defined as  $\mathrm{err}_{\mathcal{D}}(c)\coloneqq \operatorname*{Pr}_{(x,y)\sim \mathcal{D}}[c(x)\neq y]$ . The task PAC  $(\mathcal{C};\alpha)$  is to, for any distribution  $\mathcal{D}$  that has zero error on some concept  $c^{*}\in \mathcal{C}$  (aka realizable by  $\mathcal{C}$ ), output  $c:\mathcal{Z}\rightarrow \{0,1\}$  such that  $\mathrm{err}_{\mathcal{D}}(c)\leq \alpha$ . We give a tight characterization of the user complexity for PAC learning<sup>3</sup>:
Theorem 4. Let  $\mathcal{C}$  be any concept class with probabilistic representation dimension  $d$  (i.e.  $\mathrm{PRDim}(\mathcal{C}) = d$ ). Then, for any sufficiently small  $\alpha, \varepsilon > 0$  and for all  $m \in \mathbb{N}$ , we have
$$
n _ {m} ^ {\mathrm {P A C} (\mathcal {C}; \alpha)} (\varepsilon , \delta = 0) = \tilde {\Theta} \left(\frac {d}{\varepsilon} + \frac {d}{\varepsilon \alpha m}\right).
$$
Previous work [GKM21,  $\mathrm{BGH^{+}23}$  ] achieves the tight sample complexity only for large  $m\left(\gg d^2\right)$ .
$^{3}$ PRDim is defined in Definition 20.
1.1.1 Technical Overview
In this section, we give a rough overview of our proofs. We will be intentionally vague; all definitions and results will be formalized later in the paper.
Approximate-DP. At a high-level, our proof proceeds roughly as follows. First, we show that any  $(\varepsilon, \delta)$ -item-level DP  $\mathbb{A}$  with high probability satisfies a local version of user-level DP for which we only compare $^4$ $\mathbb{A}(\boldsymbol{x}_{-i})$  for different  $i$ , but with  $\varepsilon$  that is increased by a factor of roughly  $\sqrt{m}$  (see Definition 16 and Theorem 17). If this were to hold not only for  $\boldsymbol{x}_{-i}$  but also for all user-level neighbors  $\boldsymbol{x}'$  of  $\boldsymbol{x}$ , then we would have been done since the  $(\varepsilon, \delta)$ -item-level DP algorithm would have also yielded  $(\varepsilon \sqrt{m}, \delta)$ -user-level DP (which would then imply a result in favor of Corollary 3). However, this is not true in general and is where the second ingredient of our algorithm comes in: We use a propose-test-release style algorithm [DL09] to check whether the input is close to a "bad" input for which the aforementioned local condition does not hold. If so, then we output  $\perp$ ; otherwise, we run the item-level DP algorithm. Note that the test passes w.h.p. and thus we get the desired utility.
This second step is similar to recent work of Kohli and Laskowski [KL21] and Ghazi et al.  $\mathrm{[GKK^{+}23]}$  who used a similar algorithm for additive noise mechanisms (namely Laplace and Gaussian mechanisms). The main difference is that instead of testing for local sensitivity of the function as in [KL21,  $\mathrm{GKK^{+}23}$  ], we directly test for the privacy loss of the algorithm A.
As for the first step, we prove this via a connection to a notion of generalization called sample perfect generalization  $\mathrm{[CLN^{+}16]}$  (see Definition 12). Roughly speaking, this notion measures the privacy loss when we run an  $(\varepsilon ,\delta)$ -item-level DP algorithm on two  $\pmb{x},\pmb{x}^{\prime}$  drawn independently from  $D^n$ . We show that w.h.p.  $\mathbb{A}(\pmb {x})\approx_{\varepsilon ',\delta '}\mathbb{A}(\pmb{x}^{\prime})$  for  $\varepsilon^{\prime} = O(\sqrt{n\log(1 / \delta)}\cdot \varepsilon)$ . Although ostensibly unrelated to the local version of user-level DP that we discussed above, it turns out that these are indeed related: if we view the algorithm that fixes examples of all but one user, then applying the sample perfect generalization bound on just the input of the last user (which consists of only  $m$  samples) shows that the privacy loss when changing the last user is  $O(\sqrt{m\log(1 / \delta)}\cdot \varepsilon)$  w.h.p. It turns out that we can turn this argument into a formal bound for our local notion of user-level DP (see Section 3.2.1.)
We remark that our results on sample perfect generalization also resolve an open question of Cummings et al.  $\mathrm{[CLN^{+}16]}$  ; we defer this discussion to Appendix C.1.
Finally, we also note that, while Bun et al.  $\mathrm{[BGH^{+}23]}$  also uses the sample perfect generalization notion, their conversion from item-level DP algorithms to user-level DP algorithms require constructing pseudo-globally stable algorithms with sample complexity  $m$ . This is impossible when  $m$  is small, which is the reason why their reduction works only in the example-rich setting.
Pure-DP. Recall that the exponential mechanism [MT07] works as follows. Suppose there is a set  $\mathcal{H}$  of candidates we would like to select from, together with the scoring functions  $(\mathrm{scr}_H)_{H\in \mathcal{H}}$  where  $\mathrm{scr}_H:\mathcal{Z}^n\to \mathbb{R}$  has sensitivity at most  $\Delta$ . The algorithm then outputs  $H$  with probability proportional to  $\exp \left(-\varepsilon \cdot \mathrm{scr}_H(\pmb {x}) / 2\Delta\right)$ . The error guarantee of the algorithm scales with the sensitivity  $\Delta$  (see Theorem 9.) It is often the case that  $\mathrm{scr}_H$  depends on the sum of individual terms involving each item. Our approach is to clip the contribution from each user so that the sensitivity is small. We show that selecting the clipping threshold appropriately is sufficient to obtain new user-level pure-DP (and in some cases optimal) algorithms.
Let  $[n] = \{1, \dots, n\}$ , let  $[x]_{+} = \max(0, x)$ , and let  $\widetilde{O}(f)$  denote  $O(f \log^c f)$  for some constant  $c$ . Let  $\operatorname{supp}(A)$  denote the support of distribution  $A$ . We recall some standard tools from DP [Vad17].
For two distributions  $A, B$ , let  $d_{\varepsilon}(A \parallel B) = \sum_{x \in \operatorname{supp}(A)} [A(x) - e^{\varepsilon} B(x)]_{+}$  denote the  $e^{\varepsilon}$ -hockey stick divergence. For brevity, we write  $A \approx_{\varepsilon, \delta} B$  to denote  $d_{\varepsilon}(A \parallel B) \leq \delta$  and  $d_{\varepsilon}(B \parallel A) \leq \delta$ .
Lemma 5 ("Triangle Inequality"). If  $A \approx_{\varepsilon', \delta'} B$  and  $B \approx_{\varepsilon, \delta} C$ , then  $A \approx_{\varepsilon + \varepsilon', e^{\varepsilon \delta'} + e^{\varepsilon' \delta}} C$ .
Lemma 6 (Group Privacy). Let  $\mathbb{A}$  be any  $(\varepsilon, \delta)$ -item-level DP algorithm and  $\pmb{x}, \pmb{x}'$  be  $k$ -neighbors $^5$ , then  $\mathbb{A}(\pmb{x}) \approx_{\varepsilon', \delta'} \mathbb{A}(\pmb{x}')$  where  $\varepsilon' = k\varepsilon$ ,  $\delta' = \frac{e^{k\varepsilon} - 1}{e^{\varepsilon} - 1}\delta \leq ke^{k\varepsilon}\delta$ .
4 In Theorem 17, we need to compare  $\mathbb{A}(\pmb{x}_{-S})$  for  $S$ 's of small sizes as well.
$^{5}$ I.e., there exists  $\pmb{x} = \pmb{x}_0, \pmb{x}_1, \dots, \pmb{x}_k = \pmb{x}'$  such that  $\pmb{x}_{i-1}, \pmb{x}_i$  are item-level neighbors for all  $i \in [k]$ .
Theorem 7 (Amplification-by-Subsampling [BBG18]). Let  $n, n' \in \mathbb{N}$  be such that  $n' \geq n$ . For any  $(\varepsilon, \delta)$ -item-level DP algorithm  $\mathbb{A}$  with sample complexity  $n$ , let  $\mathbb{A}'$  denote the algorithm with sample complexity  $n'$  that randomly chooses  $n$  out of the  $n'$  input samples and then runs  $\mathbb{A}$  on the subsample. Then  $\mathbb{A}'$  is  $(\varepsilon', \delta')$ -DP where  $\varepsilon' = \ln(1 + \eta(e^{\varepsilon} - 1))$ ,  $\delta' = \eta\delta$  for  $\eta = n/n'$ .
We also need the definition of the shifted and truncated discrete Laplace distribution:
Definition 8 (Shifted Truncated Discrete Laplace Distribution). For any  $\varepsilon, \delta > 0$ , let  $\kappa = \kappa(\varepsilon, \delta) := 1 + \lceil \ln(1/\delta)/\varepsilon \rceil$  and let  $\mathsf{TDLap}(\varepsilon, \delta)$  be the distribution supported on  $\{0, \dots, 2\kappa\}$  with probability mass function at  $x$  being proportional to  $\exp(-\varepsilon \cdot |x - \kappa|)$ .
Exponential Mechanism (EM). In the (generalized) selection problem (SELECT), we are given a candidate set  $\mathcal{H}$  and scoring functions  $\mathrm{scr}_H$  for all  $H\in \mathcal{H}$  whose sensitivities are at most  $\Delta$ . We say that an algorithm  $\mathbb{A}$  is  $(\alpha ,\beta)$ -accurate iff  $\operatorname*{Pr}_{H\sim \mathbb{A}(\boldsymbol {x})}[\mathrm{scr}_H(\boldsymbol {x})\leq \min_{H'\in \mathcal{H}}\mathrm{scr}_{H'}(\boldsymbol {x}) + \alpha ]\geq 1 - \beta$ .
Theorem 9 ([MT07]). There is an  $\varepsilon$ -DP ( $O(\Delta \cdot \log(|\mathcal{H}| / \beta) / \varepsilon), \beta)$ -accurate algorithm for SELECT.
3 Approximate-DP
In this section, we prove our main result on approximate-DP user-level learning, which is stated in Theorem 10 below. Throughout the paper we say that a parameter is "sufficiently small" if it is at most an implicit absolute constant.
Theorem 10 (Main Theorem). Let  $\mathfrak{T}$  be any task and  $\gamma > 0$  be a parameter. Then, for any sufficiently small  $\varepsilon, \delta > 0$  and  $m \in \mathbb{N}$ , there exists  $\varepsilon' = \frac{\varepsilon^2}{\log(1/\delta)\sqrt{m\log(m/\delta)}}$  and  $\delta' = \Theta\left(\delta \cdot \frac{\varepsilon}{m\log(1/\delta)}\right)$  such that
$$
n _ {m} ^ {\mathfrak {T}} (\varepsilon , \delta ; \gamma - o (1)) \leq \widetilde {O} \left(\frac {\log (1 / \delta)}{\varepsilon} + \frac {1}{m} \cdot n _ {1} ^ {\mathfrak {T}} \left(\varepsilon^ {\prime}, \delta^ {\prime}; \gamma\right)\right).
$$
Note that Corollary 3 follows directly from Theorem 10 by plugging in the expression  $n_1^{\mathfrak{T}}(\varepsilon', \delta'; \gamma) = \widetilde{O}\left(\frac{\log(1 / \delta')^c}{\varepsilon'} \cdot n_{\mathrm{priv}}^{\mathfrak{T}} + n_{\mathrm{stat}}^{\mathfrak{T}}\right)$  into the bound.
While it might be somewhat challenging to interpret the bound in Theorem 10, given that the  $\varepsilon$  values on the left and the right hand sides are not the same, a simple subsampling argument (similar to one in  $[\mathrm{BGH}^{+}23])$  allows us to make the LHS and RHS have the same  $\varepsilon$ . In this case, we have that the user complexity is reduced by a factor of roughly  $1 / \sqrt{m}$ , as stated below. We remark that the  $\delta$  values are still different on the two sides; however, if we assume that the dependence on  $1 / \delta$  is only polylogarithmic, this results in at most a small gap of at most polylogarithmic factor in  $1 / \delta$ .
Theorem 11 (Main Theorem-Same  $\varepsilon$  Version). Let  $\mathfrak{T}$  be any task and  $\gamma >0$  be a parameter. Then, for any sufficiently small  $\varepsilon ,\delta >0$  , and  $m\in \mathbb{N}$  , there exists  $\delta^{\prime} = \Theta \left(\delta \cdot \frac{\varepsilon}{m\log(1 / \delta)}\right)$  such that
$$
n _ {m} ^ {\mathfrak {T}} (\varepsilon , \delta ; \gamma - o (1)) \leq \widetilde {O} \left(\frac {\log (1 / \delta)}{\varepsilon} + \frac {1}{\sqrt {m}} \cdot \frac {\log (1 / \delta) ^ {1 . 5}}{\varepsilon} \cdot n _ {1} ^ {\mathfrak {T}} (\varepsilon , \delta^ {\prime}; \gamma)\right).
$$