SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Claim 2.9 (Privacy amplification by sub-sampling [Kasiviswanathan et al., 2011]). Let  $\mathcal{A}$  be an  $(\varepsilon', \delta')$ -differentially private algorithm operating on a database of size  $n$ . Let  $\varepsilon \leq 1$  and let  $t = \frac{n}{\varepsilon} (3 + \exp(\varepsilon'))$ . Construct an algorithm  $\mathcal{B}$  operating the database  $D = (z_i)_{i=1}^t$ . Algorithm  $\mathcal{B}$  randomly selects a subset  $J \subseteq \{1, 2, \ldots, t\}$  of size  $n$ , and executes  $\mathcal{A}$  on  $D_J = (z_i)_{i \in J}$ . Then  $\mathcal{B}$  is  $\left(\varepsilon, \frac{4\varepsilon}{3 + \exp(\varepsilon')} \delta'\right)$ -differentially private.
2.2 Preliminaries from PAC learning
A concept class  $C$  over data domain  $X$  is a set of predicates  $c: X \to \{0,1\}$  (called concepts) which label points of the domain  $X$  by either 0 or 1. A learner  $\mathcal{A}$  for concept class  $C$  is given  $n$  examples sampled i.i.d. from an unknown probability distribution  $\mathcal{D}$  over the data domain  $X$  and labeled according to an unknown target concept  $c \in C$ . The learner should output a hypothesis  $h: X \to [0,1]$  that approximates  $c$  for the distribution  $\mathcal{D}$ . More formally,
Definition 2.10 (generalization error). The generalization error of a hypothesis  $h: X \to [0,1]$  with respect to concept  $c$  and distribution  $\mathcal{D}$  is defined as  $\text{error}_{\mathcal{D}}(c,h) = \text{Exp}_{x \sim \mathcal{D}}[|h(x) - c(x)|]$ .
Definition 2.11 (PAC learning [Valiant, 1984]). Let  $C$  be a concept class over a domain  $X$ . Algorithm  $\mathcal{A}$  is an  $(\alpha, \beta, n)$ -PAC learner for  $C$  if for all  $c \in C$  and all distributions  $\mathcal{D}$  on  $X$ ,
$$
\Pr \left[ \left(x _ {1}, \dots , x _ {n}\right) \sim \mathcal {D} ^ {n}; h \sim \mathcal {A} \left(\left(x _ {1}, c \left(x _ {1}\right)\right), \dots , \left(x _ {n}, c \left(x _ {n}\right)\right); \text {e r r o r} _ {\mathcal {D}} (c, h) \leq \alpha \right] \geq 1 - \beta , \right.
$$
where the probability is over the sampling of  $(x_{1},\ldots ,x_{n})$  from  $\mathcal{D}$  and the coin tosses of  $\mathcal{A}$ . The parameter  $n$  is the sample complexity of  $\mathcal{A}$ .
See Appendix A for additional preliminaries on PAC learning.
2.3 Preliminaties from private learning
Definition 2.12 (private PAC learning [Kasiviswanathan et al., 2011]). Algorithm  $\mathcal{A}$  is a  $(\alpha, \beta, \epsilon, \delta, n)$ -private PAC learner if (i)  $\mathcal{A}$  is an  $(\alpha, \beta, n)$ -PAC learner and (ii)  $\mathcal{A}$  is  $(\epsilon, \delta)$ -differentially private.
Kasiviswanathan et al. [2011] provided a generic private learner with  $O(\log(|X|))$  labeled samples. $^{7}$  Beimel et al. [2013a] introduced the representation dimension and showed that any concept class  $C$  can be privately learned with  $\Theta(\mathrm{RepDim}(C))$  samples. For the sample complexity of  $(\epsilon, \delta)$ -differentially private learning of threshold functions over domain  $X$ , Bun et al. [2015] gave a lower bound of  $\Omega(\log^{*}|X|)$ . Recently, Cohen et al. [2022] gave a (nearly) matching upper bound of  $\tilde{O}(\log^{*}|X|)$ .
3 Towards private everlasting prediction
In this work, we extend private prediction to answering any sequence of prediction queries – unlimited in length. Our main goal is to present a generic private everlasting predictor with low training sample complexity  $|S|$ .
Definition 3.1 (everlasting prediction). Let  $\mathcal{A}$  be an algorithm with the following properties:
1. Algorithm  $\mathcal{A}$  receives as input  $n$  labeled examples  $S = \{(x_{i},y_{i})\}_{i = 1}^{n}\in (X\times \{0,1\})^{n}$  and selects a hypothesis  $h_0:X\to \{0,1\}$ .
2. For round  $r \in \mathbb{N}$ , algorithm  $\mathcal{A}$  gets a query, which is an unlabeled element  $x_{n + r} \in X$ , outputs  $h_{r - 1}(x_{n + r})$  and selects a hypothesis  $h_r: X \to \{0, 1\}$ .
We say that  $\mathcal{A}$  is an  $(\alpha, \beta, n)$ -everlasting predictor for a concept class  $C$  over a domain  $X$  if the following holds for every concept  $c \in C$  and for every distribution  $\mathcal{D}$  over  $X$ . If  $x_1, x_2, \ldots$  are sampled i.i.d. from  $\mathcal{D}$ , and the labels of the  $n$  initial samples  $S$  are correct, i.e.,  $y_i = c(x_i)$  for  $i \in [n]$ , then  $\operatorname*{Pr}[\exists r \geq 0 \text{ s.t. } \operatorname*{error}_{\mathcal{D}}(c, h_r) > \alpha] \leq \beta$ , where the probability is over the sampling of  $x_1, x_2, \ldots$  from  $\mathcal{D}$  and the randomness of  $\mathcal{A}$ .
Applying the Dwork-Feldman notion of a private prediction interface to everlasting predictors we get:
Definition 3.2. An algorithm  $\mathcal{A}$  is an  $(\alpha, \beta, \epsilon, \delta, n)$ -everlasting differentially private prediction interface if (i)  $\mathcal{A}$  is a  $(\epsilon, \delta)$ -differentially private prediction interface  $M$  (as in Definition 1.1), and (ii)  $\mathcal{A}$  is an  $(\alpha, \beta, n)$ -everlasting predictor.
As a warmup, consider an  $(\alpha, \beta, \epsilon, \delta, n)$ -everlasting differentially private prediction interface  $\mathcal{A}$  for concept class  $C$  over (finite) domain  $X$  (as in Definition 3.2 above). Assume that  $\mathcal{A}$  does not vary its hypotheses, i.e. (in the language of Definition 3.1)  $h_r = h_0$  for all  $r > 0$ . Note that a computationally unlimited adversarial querying algorithm can recover the hypothesis  $h_0$  by issuing all queries  $x \in X$ . Hence, in using  $\mathcal{A}$  indefinitely we lose any potential benefits to sample complexity of restricting access to  $h_0$  to being black-box and getting to the point where the lower-bounds on  $n$  from private learning apply. A consequence of this simple observation is that a generic private everlasting predictor should answer all prediction queries with a single hypothesis - it should modify its hypothesis over time as it processes new queries.
We now take this observation a step further, showing that a private everlasting predictor that answers prediction queries solely based on its training sample  $S$  (and randomness, but not on the queries) is subject to the same sample complexity lower bounds as private learners.
<sup>7</sup>We omit the dependency on  $\epsilon, \delta, \alpha, \beta$  in this brief review.
Formally,  $\mathcal{A}$  can be thought of as two mechanisms  $\mathcal{A} = (M_0,M_1)$  where  $M_0$  is  $(\epsilon ,\delta)$ -differentially private. (i) On input a labeled training sample  $S$  mechanism  $M_0$  computes a hypothesis  $h_0$ . (ii) On a query  $x\in X$  mechanism  $M_{1}$  replies  $h_0(x)$ .
Consider an  $(\alpha, \beta < 1/8, \epsilon, \delta, n)$ -everlasting differentially private prediction interface  $\mathcal{A}$  for concept class  $C$  over (finite) domain  $X$  that upon receiving the training set  $S \in (X \times \{0,1\})^n$  selects an infinite sequence of hypotheses  $\{h_r\}_{r \geq 0}$  where  $h_r : X \to \{0,1\}$ . Formally, we can think of  $\mathcal{A}$  as composed of three mechanisms  $\mathcal{A} = (M_0, M_1, M_2)$  where  $M_0$  is  $(\epsilon, \delta)$ -differentially private:
- On input a labeled training sample  $S \in (X \times \{0,1\})^n$  mechanism  $M_0$  computes an initial state and an initial hypothesis  $(\sigma_0, h_0) = M_0(S)$ .
- On a query  $x_{n + r}$  mechanism  $M_1$  produces an answer  $M_1(x_{n + r}) = h_i(x_{n + r})$  and mechanism  $M_2$  updates the hypothesis-state pair  $(h_{r + 1},\sigma_{r + 1}) = M_2(\sigma_r)$ .
Note that as  $M_0$  and  $M_2$  do not receive the sequence  $\{x_{n + r}\}_{r\geq 0}$  as input, the sequence  $\{h_r\}_{r\geq 0}$  depends solely on  $S$ . Furthermore as  $M_1$  and  $M_2$  post-process the outcome of  $M_0$ , i.e., the sequence of queries and predictions  $\{(x_r,h_r(x_r))\}_{r\geq 0}$  preserves  $(\varepsilon ,\delta)$ -differential privacy with respect to the training set  $S$ . In Appendix B we prove:
Theorem 3.3. A can be transformed into a  $(O(\alpha), O(\beta), \epsilon, \delta, O(n\log(1/\beta))$ -private PAC learner for  $C$ .
4 Private everlasting prediction - definition
Theorem 3.3 requires us to seek private predictors whose prediction relies on more information than what is provided by the initial labeled sample. Possibilities include requiring the input of additional labeled or unlabeled examples during the lifetime of the predictor, while protecting the privacy of these examples.
We choose to rely on the queries for updating the predictor's internal state. This introduces a potential privacy risk for these queries as sensitive information about a query may be leaked in the predictions following it. Furthermore, we need take into account that a privacy attacker may choose their queries adversarially and adaptively.
Definition 4.1 (private everlasting black-box prediction). An algorithm  $\mathcal{A}$  is an  $(\alpha, \beta, \varepsilon, \delta, n)$ -private everlasting black-box predictor for a concept class  $C$  if
1. Prediction:  $\mathcal{A}$  is an  $(\alpha, \beta, n)$ -everlasting predictor for  $C$  (as in Definition 3.1).
2. Privacy: For every adversary  $\mathcal{B}$  and every  $t\geq 1$  , the random variables  $\mathrm{View}_{\mathcal{B},t}^{0}$  and  $\mathrm{View}_{\mathcal{B},t}^{1}$  (defined in Figure 1) are  $(\varepsilon ,\delta)$  -indistinguishable.
5 Tools from prior works
We briefly describe tools from prior works that we use in our construction. See Appendix C for a more detailed account.
Algorithm LabelBoost [Beimel et al., 2021]: Algorithm LabelBoost takes as input a partially labeled database  $S \circ T \in (X \times \{0,1,\perp\})^*$  (where the first portion of the database,  $S$ , contains examples by some concept  $c \in C$ ) and outputs a similar database where both  $S$  and  $T$  are (re)labeled by a concept  $h \in C$  such that  $error_{\mathcal{D}}(c,h)$  is bounded. We use the following lemmata from Beimel et al. [2021]:
Lemma 5.1 (privacy of Algorithm LabelBoost). Let  $\mathcal{A}$  be an  $(\epsilon, \delta)$ -differentially private algorithm operating on labeled databases. Construct an algorithm  $\mathcal{B}$  that on input a partially labeled database  $S \circ T \in (X \times \{0, 1, \bot\})^*$  applies  $\mathcal{A}$  on the outcome of LabelBoost(S  $\circ T$ ). Then,  $\mathcal{B}$  is  $(\epsilon + 3, 4\epsilon \delta)$ -differentially private.
Lemma 5.2 (Utility of Algorithm LabelBoost). Fix  $\alpha$  and  $\beta$ , and let  $S \circ T$  be s.t.  $S$  is labeled by some target concept  $c \in C$ , and s.t.  $|T| \leq \frac{\beta}{e} \mathrm{VC}(C) \exp \left( \frac{\alpha |S|}{2 \mathrm{VC}(C)} \right) - |S|$ . Consider the execution of LabelBoost on  $S \circ T$ , and let  $h$  denote the hypothesis chosen by LabelBoost to relabel  $S \circ T$ . With probability at least  $(1 - \beta)$  we have that  $\mathrm{error}_S(h) \leq \alpha$ .
Parameters:  $b \in \{0,1\}$ ,  $t \in \mathbb{N}$ .
Training Phase:
1. The adversary  $\mathcal{B}$  chooses two sets of  $n$  labeled elements  $(x_1^0, y_1^0), \ldots, (x_n^0, y_n^0)$  and  $(x_1^1, y_1^1), \ldots, (x_n^1, y_n^1)$ , subject to the restriction  $\left|\left\{i \in [n] : (x_i^0, y_i^0) \neq (x_i^1, y_i^1)\right\}\right| \in \{0, 1\}$ .
2. If  $\exists i$  s.t.  $(x_i^0, y_i^0) \neq (x_i^1, y_i^1)$  then set  $\mathsf{Flag} = 1$ . Otherwise set  $\mathsf{Flag} = 0$ .
3. Algorithm  $\mathcal{A}$  gets  $(x_1^b, y_1^b), \ldots, (x_n^b, y_n^b)$  and selects a hypothesis  $h_0: X \to \{0, 1\}$ .  $\backslash^*$  the adversary  $\mathcal{B}$  does not get to see the hypothesis  $h_0^{*} \backslash$
Prediction phase:
4. For round  $r = 1,2,\dots ,t$
(a) If  $\operatorname{Flag} = 1$  then the adversary  $\mathcal{B}$  chooses two elements  $x_{n + r}^{0} = x_{n + r}^{1} \in X$ . Otherwise, the adversary  $\mathcal{B}$  chooses two elements  $x_{n + r}^{0}, x_{n + r}^{1} \in X$ .
(b) If  $x_{n + r}^0\neq x_{n + r}^1$  then Flag is set to 1.
(c) If  $x_{n + r}^{0} = x_{n + r}^{1}$  then the adversary  $\mathcal{B}$  gets  $h_{r - 1}(x_{n + r}^{b})$ .  
\* the adversary  $\mathcal{B}$  does not get to see the label if  $x_{n + r}^{0} \neq x_{n + r}^{1}$ .
(d) Algorithm  $\mathcal{A}$  gets  $x_{n + r}^{b}$  and selects a hypothesis  $h_r:X\to \{0,1\}$ .  $\backslash^*$  the adversary  $\mathcal{B}$  does not get to see the hypothesis  $h_r^* \backslash$
Let  $\text{View}_{\mathcal{B},t}^b$  be  $\mathcal{B}$ 's view of the execution, i.e., its own randomness and the sequence of predictions in Step 4c.
Figure 1: Definition of  $\mathrm{View}_{\mathcal{B},t}^{0}$  and  $\mathrm{View}_{\mathcal{B},t}^{1}$ .
6 A Generic Construction
Our generic construction Algorithm GenericBBL transforms a (non-private) learner for a concept class  $C$  into a private everlasting predictor for  $C$ . The theorem below follows from Theorem 6.2 and Claim 6.3 which are proved in Appendix E.
Theorem 6.1. Given  $\alpha, \beta, \delta < 1/16, \epsilon < 1$ , Algorithm GenericBBL is a  $(4\alpha, 4\beta, \epsilon, \delta, n)$ -private everlasting predictor, where  $n$  is set as in Algorithm GenericBBL.
Theorem 6.2 (accuracy of algorithm GenericBBL). Given  $\alpha, \beta, \delta < 1/16$ ,  $\varepsilon < 1$ , for any concept  $c$  and any round  $r$ , algorithm GenericBBL can predict the label of  $x_r$  as  $h_r(x_r)$ , such that  $\operatorname*{Pr}[error_{\mathcal{D}}(c(x_r) \neq h_r(x_r)) \leq 4\alpha] \geq 1 - 4\beta$ .
Claim 6.3 (privacy of algorithm GenericBBL). GenericBBL is  $(\varepsilon, \delta)$ -differentially private.
Remark 6.4. For simplicity, we analyzed GenericBBL in the realizable setting, i.e., under the assumption that the training set  $S$  is consistent with the target class  $C$ . Our construction carries over to the agnostic setting via standard arguments (ignoring computational efficiency). We refer the reader to [Beimel et al., 2021] and [Alon et al., 2020] for generic agnostic-to-realizable reductions in the context of private learning.
6.1 Improving the sample complexity dependency on accuracy
We briefly sketch how to improve the sample complexity of Algorithm GenericBBL from  $n = \tilde{\Theta}\left(\frac{\mathrm{VC}^2(C)}{\alpha^2\epsilon^2}\right)$  to  $n = \tilde{\Theta}\left(\frac{\mathrm{VC}^2(C)}{\alpha\epsilon^2}\right)$  by modifying steps 3(d)ii and 3(d)iii of Algorithm GenericBBL. To simplify the description, we consider a constant  $\epsilon$ , we ignore the privacy amplification by subsampling occurring in steps 2 and 3f and illustrate how the modified algorithm would execute by considering round  $i = 1$  of Algorithm GenericBBL.
Note that  $S_{1} = S$  where  $n = |S_{1}| = T_{1} \cdot \lambda_{1}$  and  $\lambda_{1} = \tilde{\Theta}(\mathrm{VC}(C) / \alpha_{1})$  is the sample complexity needed such that each of the hypotheses computed in Step 3b has error  $\alpha_{1}$  except for
Algorithm GenericBBL
Initial input: A labeled database  $S \in (X \times \{0,1\})^n$  where  $n = \frac{8\tau}{\alpha^2\varepsilon^2} \cdot \left( 8\mathrm{VC}(C)\log \left(\frac{26}{\alpha}\right) + 4\log \left(\frac{4}{\beta}\right) \right)^2 \cdot \log \left(\frac{1}{\delta}\right) \cdot \log^2\left(\frac{64\mathrm{VC}(C)\log\left(\frac{26}{\alpha}\right) + 32\log\left(\frac{4}{\beta}\right)}{\varepsilon\alpha^2\beta\delta}\right) \cdot (3 + \exp (\varepsilon + 4))$ .
1. Let  $\tau > 50$ . Set  $\alpha_{1} = \alpha / 2$ ,  $\beta_{1} = \beta / 2$ . Define  $\lambda_{i} = \frac{8\mathrm{VC}(C)\log\left(\frac{13}{\alpha_{i}}\right) + 4\log\left(\frac{2}{\beta_{i}}\right)}{\alpha_{i}}$ .
/\* by Theorem A.2  $\lambda_{i}$  samples suffice for PAC learning C with parameters  $\alpha_{i},\beta_{i}^{*}/$
2. Let  $S_{1}\subseteq S$  be a random subset of size  $n\cdot \frac{\varepsilon}{3 + \exp(\varepsilon + 4)} = \frac{\tau\cdot\lambda_{i}^{2}\cdot\log(\frac{1}{\delta})\cdot\log^{2}(\frac{\lambda_{i}}{\varepsilon\alpha_{i}\beta_{i}\delta})}{\varepsilon}$
3. Repeat for  $i = 1,2,3,\ldots$
(a) Divide  $S_{i}$  into  $T_{i} = \frac{\tau\cdot\lambda_{i}\cdot\log(\frac{1}{\delta})\cdot\log^{2}(\frac{\lambda_{i}}{\varepsilon\alpha_{i}\beta_{i}\delta})}{\varepsilon}$  disjoint databases  $S_{i,1},\ldots ,S_{i,T_i}$  of size  $\lambda_{i}$
(b) For  $t\in [T_i]$  let  $f_{t}\in C$  be a hypothesis minimizing error  $\mathcal{S}_{i,t}(\cdot)$ . Define  $F_{i} = (f_{1},\dots ,f_{T_{i}})$
(c) Set  $R_{i} = \frac{12800|S_{i}|}{\varepsilon}$ . Set the privacy parameters  $\varepsilon_{i}' = \frac{1}{3\sqrt{R_{i}\ln(\frac{2}{\delta})}}$  and  $\delta_{i}' = \frac{\delta}{2R_{i}}$ . Instantiate noises  $\eta_{1},\ldots,\eta_{R_{i}} \sim \mathrm{Lap}(1/T_{i}\varepsilon_{i}')$ .
(d) For  $\ell = 1$  to  $R_{i}$ :
i. Receive as input a prediction query  $x_{i,\ell}\in X$
ii. Compute  $q_{x_{i,\ell}}(F_i) = \frac{1}{T_i}\sum_{t\in [T_i]}f_t(x_{i,\ell})$ , and let  $y_{i,\ell} = q_{x_{i,\ell}}(F_i) + \eta_\ell$ .
iii. Respond with the label 0 if  $y_{i,\ell} < 1/2$  and 1 if  $y_{i,\ell} \geq 1/2$ .
(e) Denote  $D_{i} = (x_{i,1},\dots ,x_{i,R_{i}})$
(f) Let  $\hat{S}_i \subseteq S_i$  and  $\hat{D}_i \subseteq D_i$  be random subsets of size  $\frac{\varepsilon|S_i|}{3 + \exp(\varepsilon + 4)}$  and  $\frac{\varepsilon|D_i|}{3 + \exp(\varepsilon + 4)}$  respectively, and let  $\hat{S}_i' \circ \hat{D}_i' \gets \text{LabelBoost}(\hat{S}_i \circ \hat{D}_i)$ . Let  $S_{i+1} \subseteq \hat{D}_i'$  be a random subset of size  $\lambda_{i+1}T_{i+1}$ .
(g) Set  $\alpha_{i + 1}\gets \alpha_i / 2$  and  $\beta_{i + 1}\leftarrow \beta_{i} / 2$
probability  $\beta_{i}$ . Applying advanced composition, we get that  $\tilde{\Theta}(T_1^2)$  noisy majority queries implemented in steps steps 3(d)ii and 3(d)iii can be performed, i.e.,  $R_1 = \tilde{\Theta}(T_1^2)$ . Finally, to feed the next phase with  $|S_2| = \Theta(|S_1|)$  labeled samples, we need that  $R_1 = \Theta(T_1 \cdot \lambda_1) = \tilde{\Theta}(T_1 \cdot \mathrm{VC}(C) / \alpha)$  and hence  $T_1 = \tilde{\Theta}(\mathrm{VC}(C) / \alpha)$  resulting in  $n = \tilde{\Theta}(\mathrm{VC}^2(C) / \alpha^2)$ .
However, when queries are made from the same underlying distribution  $S$  was selected, we expect that most of them would exhibit a clear majority in steps 3(d)ii and 3(d)iii, except for a fraction of  $O(\alpha)$ . Hence a natural way to improve on the number of majority queries that can be performed is to replace steps 3(d)ii and 3(d)iii with a decision based on the sparse vector technique of Dwork et al. [2009]. In particular, we can use the BetweenThresholds mechanism of Bun et al. [2016] to get an improved  $R_{1} = \tilde{\Theta}(T_{1}^{2}/\alpha)$  and hence get  $T_{1} = \tilde{\Theta}(VC(C))$  and  $n = \tilde{\Theta}(VC^{2}/\alpha)$ .
A final important wrinkle is that the above calculation is based on the queries coming from the same underlying distribution  $S$  was selected, while our worst-case privacy requirement allows for an adversarial choice of queries that may in turn cause the execution of BetweenThresholds to halt too often and hence exhaust the privacy budget for the phase. It is hence important to estimate the number of times BetweenThresholds halts within a phase and to stop the execution of Algorithm GenericBBL when the estimate crosses a threshold. This estimate needs to be done in an online manner and should preserve differential privacy with respect to the queries.<sup>9</sup> This can be done, e.g., using the private counter algorithm of Dwork et al. [2010a], which preserves differential privacy under continual observation.
For example, stopping GenericBBL after the count of halts crosses a precise threshold would reveal that the last query caused BetweenThresholds to halt, and hence breach differential privacy.
Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. Boosting and differential privacy. In FOCS, pages 51-60, 2010b.
Vitaly Feldman and David Xiao. Sample complexity bounds on differentially private learning via communication complexity. SIAM J. Comput., 44(6):1740-1764, 2015. doi: 10.1137/140991844. URL http://dx.doi.org/10.1137/140991844.
Haim Kaplan, Katrina Ligett, Yishay Mansour, Moni Naor, and Uri Stemmer. Privately learning thresholds: Closing the exponential gap. In  $COLT$ , volume 125 of Proceedings of Machine Learning Research, pages 2263-2285. PMLR, 2020.
Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What can we learn privately? SIAM J. Comput., 40(3):793-826, 2011.
Michael J. Kearns, Mallesh M. Pai, Ryan M. Rogers, Aaron Roth, and Jonathan R. Ullman. Robust mediators in large games. CoRR, abs/1512.02698, 2015.
Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In FOCS, pages 94-103. IEEE, Oct 20-23 2007.
Anupama Nandi and Raef Bassily. Privately answering classification queries in the agnostic PAC model. In ALT, volume 117 of Proceedings of Machine Learning Research, pages 687-703. PMLR, 2020.
L. G. Valiant. A theory of the learnable. Commun. ACM, 27(11):1134-1142, November 1984. ISSN 0001-0782. doi: 10.1145/1968.1972. URL http://doi.acm.org/10.1145/1968.1972.
Vladimir N. Vapnik and Alexey Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16 (2):264-280, 1971.
A Additional Preliminaries from PAC Learning
It is well known that a sample of size  $\Theta (\mathrm{VC}(C))$  is necessary and sufficient for the PAC learning of a concept class  $C$ , where the Vapnik-Chervonenkis (VC) dimension of a class  $C$  is defined as follows:
Definition A.1 (VC-Dimension [Vapnik and Chervonenkis, 1971]). Let  $C$  be a concept class over a domain  $X$ . For a set  $B = \{b_1, \ldots, b_\ell\} \subseteq X$  of  $\ell$  points, let  $\Pi_C(B) = \{(c(b_1), \ldots, c(b_\ell)): c \in C\}$  be the set of all dichotomies that are realized by  $C$  on  $B$ . We say that the set  $B \subseteq X$  is shattered by  $C$  if  $C$  realizes all possible dichotomies over  $B$ , in which case we have  $|\Pi_C(B)| = 2^{|B|}$ .
The VC dimension of the class  $C$ , denoted  $\mathrm{VC}(C)$ , is the cardinality of the largest set  $B \subseteq X$  shattered by  $C$ .
Theorem A.2 (VC bound). Let  $C$  be a concept class over a domain  $X$ . For  $\alpha, \beta < 1/2$ , there exists an  $(\alpha, \beta, n)$ -PAC learner for  $C$ , where  $n = \frac{8\mathrm{VC}(C)\log(\frac{13}{\alpha}) + 4\log(\frac{2}{\beta})}{\alpha}$ .
The proof of Theorem 3.3 follows from algorithms HypothesisLearner, AccuracyBoost and claims B.1, B.2, all described below.
In Algorithm HypothesisLearner we assume that the everlasting differentially private prediction interface  $\mathcal{A}$  was fed with  $n$  i.i.d. samples taken from some (unknown) distribution  $\mathcal{D}$  and labeled by an unknown concept  $c\in C$ . Assuming the sequence of hypotheses  $\{h_r\}_{r\geq 0}$  produced by  $\mathcal{A}$  satisfies
$$
\forall r \operatorname {e r r o r} _ {\mathcal {D}} \left(c, h _ {r}\right) \leq \alpha \tag {1}
$$
we use it to construct - with constant probability - a hypothesis  $h$  with error bounded by  $O(\alpha)$ .
Algorithm HypothesisLearner
Parameters:  $0 < \beta \leq 1/8$ ,  $R = |X| \cdot \log(|X|) \cdot \log(1/\beta)$
Input: hypothesis sequence  $\{h_r\}_{r\geq 0}$
1. for all  $x \in X$  let  $L_x = \emptyset$
2. for  $r = 0,1,2,\ldots ,R$
(a) select  $x$  uniformly at random from  $X$  and let  $L_{x} = L_{x}\cup \{h_{r}(x)\}$
3. if  $L_{x} = \emptyset$  for some  $x \in X$  then fail, output an arbitrary hypothesis, and halt  $/\ast \operatorname{Pr}[\exists x \text{ such that } L_{x} = \emptyset ] \leq |X| (1 - \frac{1}{|X|})^{R} \approx |X| e^{-R / |X|} = \beta^{*} /$
4. for all  $x \in X$  let  $r_x$  be sampled uniformly at random from  $L_x$
5. construct the hypothesis  $h$ , where  $h(x) = r_x$ .
Claim B.1. If executed on a hypothesis sequence satisfying Equation 1 then with probability at least  $3/4$  Algorithm HypothesisLearner outputs a hypothesis  $h$  satisfying error  $\mathcal{D}(c, h) \leq 8\alpha$ .
Proof. Having  $\mathcal{D}, c \in C$  fixed, and given a hypothesis  $h$ , we define  $e_h(x)$  to be 1 if  $h(x) \neq c(x)$  and 0 otherwise. Thus, we can write error  $\mathcal{D}(c, h) = \mathbb{E}_{x \sim \mathcal{D}}[e_h(x)]$ .