SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haim Kaplan*
Yishay Mansour†
Shay Moran†
In this work we revisit an interactive variant of joint differential privacy, recently introduced by Naor et al. [2023], and generalize it towards handling online processes in which existing privacy definitions seem too restrictive. We study basic properties of this definition and demonstrate that it satisfies (suitable variants) of group privacy, composition, and post processing.
In order to demonstrate the advantages of this privacy definition compared to traditional forms of differential privacy, we consider the basic setting of online classification. We show that any (possibly non-private) learning rule can be effectively transformed to a private learning rule with only a polynomial overhead in the mistake bound. This demonstrates a stark difference with traditional forms of differential privacy, such as the one studied by Golowich and Livni [2021], where only a double exponential overhead in the mistake bound is known (via an information theoretic upper bound).
In this work we study privacy of interactive machine learning processes. As a motivating story, consider a chatbot that continuously improves itself by learning from the conversations it conducts with users. As these conversations might contain sensitive information, we would like to provide privacy guarantees to the users, in the sense that the content of their conversations with the chatbot would not leak. This setting fleshes out the following two requirements.
(1) Clearly, the answers given by the chatbot to user  $u_{i}$  must depend on the queries made by user  $u_{i}$ . For example, the chatbot should provide different answers when asked by user  $u_{i}$  for the weather forecast in Antarctica, and when asked by  $u_{i}$  for a pasta recipe.
This is in contrast to the plain formulation of differential privacy, where it is required that all of the mechanism outputs would be (almost) independent of any single user input. Therefore, the privacy requirement we are aiming for is that the conversation of user  $u_{i}$  will remain "hidden" from other users, and would not leak through the other users' interactions with the chatbot. Moreover, this should remain true even if a "privacy attacker" (aiming to extract information about the conversation user  $u_{i}$  had) conducts many different conversations with the chatbot.
(2) The interaction with the chatbot is, by design, interactive and adaptive, as it aims to conduct dialogues with the users. This allows the privacy attacker (mentioned above) to choose its queries to the chatbot adaptively. Privacy, hence, needs to be preserved even in the presence of adaptive attackers.
While each of these two requirements was studied in isolation, to the best of our knowledge, they have not been (explicitly) unified into a combined privacy framework. Requirement (1) was formalized
*Tel Aviv University and Google Research. haimk@tau.ac.il.
$^{\dagger}$ Tel Aviv University and Google Research. mansour.yishay@gmail.com.
$^{\ddagger}$ Technion and Google Research. smoran@technion.ac.il.
$^{\S}$ Georgetown University. kobbi.nissim@georgetown.edu.
Tel Aviv University and Google Research. u@uri.co.il.
by Kearns et al. [2015] as joint differential privacy (JDP). It provides privacy against non-adaptive attackers. Intuitively, in the chatbot example, JDP aims to hide the conversation of user  $u_{i}$  from any privacy attacker that chooses in advance all the queries it poses to the chatbot. This is unsatisfactory since the adaptive nature of this process invites adaptive attackers.
Requirement (2) was studied in many different settings, but to the best of our knowledge, only w.r.t. the plain formulation of DP, where the (adaptive) privacy attacker sees all of the outputs of the mechanism. Works in this vein include [Dwork et al., 2009, Chan et al., 2010, Hardt and Rothblum, 2010, Dwork et al., 2010b, Bun et al., 2017, Kaplan et al., 2021, Jain et al., 2021]. In the chatbot example, plain DP would require, in particular, that even the messages sent from the chatbot to user  $u_{i}$  reveals (almost) no information about  $u_{i}$ . In theory, this could be obtained by making sure that the entire chatbot model is computed in a privacy preserving manner, such that even its full description leaks almost no information about any single user. Then, when user  $u_{i}$  comes, we can "simply" share the model with her, and let her query it locally on her device. But this is likely unrealistic with large models involving hundreds of billions of parameters.
In this work we use challenge differential privacy, which was recently introduced by Naor et al. [2023] in the context of PAC learning. $^{6}$  As discussed below, challenge differential privacy is particularly suitable for addressing interactive and adaptive learning processes, such as the one illustrated above. Challenge DP can be viewed as an interactive variant of JDP, aimed at maintaining privacy against adaptive privacy attackers. Intuitively, in the chatbot example, this definition would guarantee that even an adaptive attacker that controls all of the users except for user  $u_{i}$ , learns (almost) no information about the conversation user  $u_{i}$  had with the chatbot.
1.1 Private Online Classification
We initiate the study of challenge differential privacy in the basic setting of online classification. Let  $\mathcal{X}$  be the domain,  $\mathcal{V}$  be the label space, and  $\mathcal{Z} = \mathcal{X} \times \mathcal{V}$  be a set of labeled examples. An online learner is a (possibly randomized) mapping  $\mathcal{A}: \mathcal{Z}^{\star} \times \mathcal{X} \to \mathcal{V}$ . That is, it is a mapping that maps a finite sequence  $S \in \mathcal{Z}^{\star}$  (the past examples), and an unlabeled example  $x$  (the current query point) to a label  $y$ , which is denoted by  $y = \mathcal{A}(x; S)$ .
Let  $\mathcal{H} \subseteq \mathcal{V}^{\mathcal{X}}$  be a hypothesis class. A sequence  $S \in \mathcal{Z}^{\star}$  is said to be realizable by  $\mathcal{H}$  if there exists  $h \in \mathcal{H}$  such that  $h(x_{i}) = y_{i}$  for every  $(x_{i},y_{i}) \in S$ . For a sequence  $S = \{(x_{t},y_{t})\}_{t=1}^{T} \in \mathcal{Z}^{\star}$  we write  $\mathcal{M}(\mathcal{A};S)$  for the random variable denoting the number of mistakes  $\mathcal{A}$  makes during the execution on  $S$ . That is  $\mathcal{M}(\mathcal{A};S) = \sum_{t=1}^{T} 1\{\hat{y}_{t} \neq y_{t}\}$ , where  $\hat{y}_{t} = \mathcal{A}(x_{t};S_{<t})$  is the (randomized) prediction of  $\mathcal{A}$  on  $x_{t}$ .
Definition 1.1 (Online Learnability: Realizable Case). We say that a hypothesis class  $\mathcal{H}$  is online learnable if there exists a learning rule  $\mathcal{A}$  such that  $\mathbb{E}\left[\mathcal{M}(\dot{\mathcal{A}};S)\right] = o(T)$  for every sequence  $S$  which is realizable by  $\mathcal{H}$ .
Remark 1.2. Notice that Definition 1.1 corresponds to an oblivious adversary, as it quantifies over the input sequence in advance. This should not be confused with the adversaries considered in the context of privacy which are always adaptive in this work. In the non-private setting, focusing on oblivious adversaries does not affect generality in terms of utility. This is less clear when privacy constraints are involved.<sup>7</sup> We emphasize that our results (our mistake bounds) continue to hold even when the realizable sequence is chosen by an adaptive (stateful) adversary, that at every point in time chooses the next input to the algorithm based on all of the previous outputs of the algorithm.
A classical result due to Littlestone [1988] characterizes online learnability (without privacy constraints) in terms of the Littlestone dimension. The latter is a combinatorial parameter of  $\mathcal{H}$  which was named after Littlestone by Ben-David et al. [2009].
In particular, Littlestone's characterization implies the following dichotomy: if  $\mathcal{H}$  has finite Littlestone dimension  $d$  then there exists a (deterministic) learning rule which makes at most  $d$  mistakes on
The privacy notion we study, Challenge DP, is a generalization of the privacy notion presented by Naor et al. [2023]. They focused on a special case in which there is no interaction with individual users. In the chatbot example, this corresponds to having each user submitting only a single query to the chatbot without conducting an adaptive dialog with it. In addition, we analyze useful properties of Challenge DP, such as composition, post-processing, and group-privacy. These properties were not studied by Naor et al. [2023].
In particular, Golowich and Livni [2021] studied both oblivious and adaptive adversaries, and obtained very different results in these two cases.
every realizable input sequence. In the complementing case, when the Littlestone dimension of  $\mathcal{H}$  is infinite, for every learning rule  $\mathcal{A}$  and every  $T\in \mathbb{N}$  there exists a realizable sequence  $S$  of length  $T$  such that  $\mathbb{E}\left[\mathcal{M}(\mathcal{A};S)\right]\geq T / 2$ . In other words, as a function of  $T$ , the optimal mistake bound is either uniformly bounded by the Littlestone dimension, or it is  $\geq T / 2$ . Because of this dichotomy, in some places online learnability is defined with respect to a uniform bound on the number of mistakes (and not just a sublinear one as in the above definition). In this work we follow the more general definition.
We investigate the following questions: Can every online learnable class be learned by an algorithm which satisfies challenge differential privacy? What is the optimal mistake bound attainable by private learners?
Our main result in this part provides an affirmative answer to the first question. We show that for any class  $\mathcal{H}$  with Littlestone dimension  $d$  there exists an  $(\varepsilon, \delta)$ -challenge-DP learning rule which makes at most  $\tilde{O}\left(\frac{d^2}{\varepsilon^2} \log^2\left(\frac{1}{\delta}\right) \log^2\left(\frac{T}{\beta}\right)\right)$  mistakes, with probability  $1 - \beta$ , on every realizable sequence of length  $T$ . Remarkably, our proof provides an efficient transformation taking a non-private learner to a private one: that is, given a black box access to a learning rule  $\mathcal{A}$  which makes at most  $M$  mistakes in the realizable case, we efficiently construct an  $(\varepsilon, \delta)$ -challenge-DP learning rule  $\mathcal{A}'$  which makes at most  $\tilde{O}\left(\frac{M^2}{\varepsilon^2} \log^2\left(\frac{1}{\delta}\right) \log^2\left(\frac{T}{\beta}\right)\right)$  mistakes.
1.1.1 Construction overview
We now give a simplified overview of our construction, called POP, which transforms a non-private online learning algorithm into a private one (while maintaining computational efficiency). Let  $\mathcal{A}$  be a non-private algorithm, guaranteed to make at most  $d$  mistakes in the realizable setting. We maintain  $k$  copies of  $\mathcal{A}$ . Informally, in every round  $i\in [T]$  we do the following:
1. Obtain an input point  $x_{i}$ .
2. Give  $x_{i}$  to each of the  $k$  copies of  $\mathcal{A}$  to obtain predicted labels  $\hat{y}_{i,1},\dots,\hat{y}_{i,k}$ .
3. Output a "privacy preserving" aggregation  $\hat{y}_i$  of  $\{\hat{y}_{i,1},\dots ,\hat{y}_{i,k}\}$ , which is some variant of noisy majority. This step will only satisfy challenge-DP, rather than (standard) DP.
4. Obtain the "true" label  $y_{i}$ .
5. Let  $\ell \in [k]$  be chosen at random.
6. Rewind all copies of algorithm  $\mathcal{A}$  except for the  $\ell$ th copy, so that they "forget" ever seeing  $x_{i}$ .
7. Give the true label  $y_{i}$  to the  $\ell$ th copy of  $\mathcal{A}$ .
As we aggregate the predictions given by the copies of  $\mathcal{A}$  using (noisy) majority, we know that if the algorithm errs then at least a constant fraction of the copies of  $\mathcal{A}$  err. As we feed the true label  $y_{i}$  to a random copy, with constant probability, the copy which we do not rewind incurs a mistake at this moment. That is, whenever we make a mistake then with constant probability one of the copies we maintain incurs a mistake. This can happen at most  $\approx k\cdot d$  times, since we have  $k$  copies and each of them makes at most  $d$  mistakes. This allows us to bound the number of mistakes made by our algorithm (w.h.p.). The privacy analysis is more involved. Intuitively, by rewinding all of the copies of  $\mathcal{A}$  (except one) in every round, we make sure that a single user can affect the inner state of at most one of the copies. This allows us to efficiently aggregate the predictions given by the copies in a privacy preserving manner. The subtle point is that the prediction we release in time  $i$  does require querying all the experts on the current example  $x_{i}$  (before rewinding them). Nevertheless, we show that this algorithm is private.
1.1.2 Comparison with Golowich and Livni [2021]
The closest prior work to this manuscript is by Golowich and Livni who also studied the problem of private online classification, but under a more restrictive notion of privacy than challenge-DP. In particular their definition requires that the sequence of predictors which the learner uses to predict in each round does not compromise privacy. In other words, it is as if at each round the learner publishes the entire truth-table of its predictor, rather than just its current prediction. This might be too prohibitive in certain applications such as the chatbot example illustrated above. Golowich and Livni show that even with respect to their more restrictive notion of privacy it is possible to online
learn every Littlestone class. However, their mistake bound is doubly exponential in the Littlestone dimension (whereas ours is quadratic), and their construction requires more elaborate access to the non-private learner. In particular, it is not clear whether their construction can be implemented efficiently (while our construction is efficient).
1.2 Additional Related Work
Several works studied the related problem of private learning from expert advice [Dwork et al., 2010a, Jain et al., 2012, Thakurta and Smith, 2013, Dwork and Roth, 2014, Jain and Thakurta, 2014, Agarwal and Singh, 2017, Asi et al., 2022]. These works study a variant of the experts problem in which the learning algorithm has access to  $k$  experts; on every time step the learning algorithm chooses one of the experts to follow, and then observes the loss of each expert. The goal of the learning algorithm is that its accumulated loss will be competitive with the loss of the best expert in hindsight. In this setting the private data is the sequence of losses observed throughout the execution, and the privacy requirement is that the sequence of experts chosen by the algorithm should not compromise the privacy of the sequence of losses. When applying these results to our context, the set of experts is the set of hypotheses in the class  $\mathcal{H}$ , which means that the outcome of the learner (on every time step) is a complete model (i.e., a hypothesis). That is, in our context, applying prior works on private prediction from expert advice would result in a privacy definition similar to that of Golowich and Livni [2021] that accounts (in the privacy analysis) for releasing complete models, rather than just the predictions, which is significantly more restrictive.
There were a few works that studied private learning in online settings under the constraint of JDP. For example, Shariff and Sheffet [2018] studied the stochastic contextual linear bandits problem under JDP. Here, in every round  $t$  the learner receives a context  $c_t$ , then it selects an action  $a_t$  (from a fixed set of actions), and finally it receives a reward  $y_t$  which depends on  $(c_t, a_t)$  in a linear way. The learner's objective is to maximize cumulative reward. The (non-adaptive) definition of JDP means that action  $a_t$  is revealed only to user  $u_t$ . Furthermore, it guarantees that the inputs of user  $u_t$  (specifically the context  $c_t$  and the reward  $y_t$ ) do not leak to the other users via the actions they are given, provided that all these other users fix their data in advance. This non-adaptive privacy notion fits the stochastic setting of Shariff and Sheffet [2018], but (we believe) is less suited for adversarial processes like the ones we consider in this work. We also note that the algorithm of Shariff and Sheffet [2018] in fact satisfies the more restrictive privacy definition which applies to the sequence of predictors (rather than the sequence of predictions), similarly to the that of Golowich and Livni [2021].
Notation. Two datasets  $S$  and  $S'$  are called neighboring if one is obtained from the other by adding or deleting one element, e.g.,  $S' = S \cup \{x'\}$ . For two random variables  $Y, Z$  we write  $X \approx_{(\varepsilon, \delta)} Y$  to mean that for every event  $F$  it holds that  $\operatorname*{Pr}[X \in F] \leq e^{\varepsilon} \cdot \operatorname*{Pr}[Y \in F] + \delta$ , and  $\operatorname*{Pr}[Y \in F] \leq e^{\varepsilon} \cdot \operatorname*{Pr}[X \in F] + \delta$ . Throughout the paper we assume that the privacy parameter  $\varepsilon$  satisfies  $\varepsilon = O(1)$ , but our analyses trivially extend to larger values of epsilon.
The standard definition of differential privacy is,
Definition 2.1 ([Dwork et al., 2006]). Let  $\mathcal{M}$  be a randomized algorithm that operates on datasets. Algorithm  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -differentially private (DP) if for any two neighboring datasets  $S, S'$  we have  $\mathcal{M}(S) \approx_{(\varepsilon, \delta)} \mathcal{M}(S')$ .
The Laplace mechanism. The most basic constructions of differentially private algorithms are via the Laplace mechanism as follows.
Definition 2.2. A random variable has probability distribution  $\mathrm{Lap}(\gamma)$  if its probability density function is  $f(x) = \frac{1}{2\gamma}\exp (-|x| / \gamma)$ , where  $x\in \mathbb{R}$ .
Definition 2.3 (Sensitivity). A function  $f$  that maps datasets to the reals has sensitivity  $\Delta$  if for every two neighboring datasets  $S$  and  $S'$  it holds that  $|f(S) - f(S')| \leq \Delta$ .
<sup>8</sup>Asi et al. [2022] study a more general framework of adaptive privacy in which the private data is an auxiliary sequence  $(z_{1},\ldots ,z_{T})$ . During the interaction with the learner, these  $z_{t}$ 's are used (possibly in an adaptive way) to choose the sequence of loss functions.
Theorem 2.4 (The Laplace Mechanism [Dwork et al., 2006]). Let  $f$  be a function that maps datasets to the reals with sensitivity  $\Delta$ . The mechanism  $\mathcal{A}$  that on input  $S$  adds noise with distribution  $\mathrm{Lap}\left(\frac{\Delta}{\varepsilon}\right)$  to the output of  $f(S)$  preserves  $(\varepsilon, 0)$ -differential privacy.
Joint differential privacy. The standard definition of differential privacy (Definition 2.1) captures a setting in which the entire output of the computation may be publicly released without compromising privacy. While this is a very desirable requirement, it is sometimes too restrictive. Indeed, Kearns et al. [2015] considered a relaxed setting in which we aim to analyze a dataset  $S = (x_{1},\ldots ,x_{n})$ , where every  $x_{i}$  represents the information of user  $i$ , and to obtain a vector of outcomes  $(y_{1},\dots ,y_{n})$ . This vector, however, is not made public. Instead, every user  $i$  only receives its "corresponding outcome"  $y_{i}$ . This setting potentially allows the outcome  $y_{i}$  to strongly depend on the the input  $x_{i}$ , without compromising the privacy of the  $i$ th user from the view point of the other users.
Definition 2.5 ([Kearns et al., 2015]). Let  $\mathcal{M}: X^n \to Y^n$  be a randomized algorithm that takes a dataset  $S \in X^n$  and outputs a vector  $\vec{y} \in Y^n$ . Algorithm  $\mathcal{M}$  satisfies  $(\varepsilon, \delta)$ -joint differential privacy (JDP) if for every  $i \in [n]$  and every two datasets  $S, S' \in X^n$  differing only on their  $i$ th point it holds that  $\mathcal{M}(S)_{-i} \approx_{(\varepsilon, \delta)} \mathcal{M}(S')_{-i}$ . Here  $\mathcal{M}(S)_{-i}$  denotes the (random) vector of length  $n - 1$  obtained by running  $(y_1, \ldots, y_n) \gets \mathcal{M}(S)$  and returning  $(y_1, \ldots, y_{i-1}, y_{i+1}, \ldots, y_n)$ .
In words, consider an algorithm  $\mathcal{M}$  that operates on the data of  $n$  individuals and outputs  $n$  outcomes  $y_{1},\ldots ,y_{n}$ . This algorithm is JDP if changing only the  $i$ th input point  $x_{i}$  has almost no affect on the outcome distribution of the other outputs (but the outcome distribution of  $y_{i}$  is allowed to strongly depend on  $x_{i}$ ). Kearns et al. [2015] showed that this fits a wide range of problems in economic environments.
Example 2.6 ([Nahmias et al., 2019]). Suppose that a city water corporation is interested in promoting water conservation. To do so, the corporation decided to send each household a customized report indicating whether their water consumption is above or below the median consumption in the neighborhood. Of course, this must be done in a way that protects the privacy of the neighbors. One way to tackle this would be to compute a privacy preserving estimation  $z$  for the median consumption (satisfying Definition 2.1). Then, in each report, we could safely indicate whether the household's water consumption is bigger or smaller than  $z$ . While this solution is natural and intuitive, it turns out to be sub-optimal: We can obtain better utility by designing a JDP algorithm that directly computes a different outcome for each user ("above" or "below"), which is what we really aimed for, without going through a private median computation.
Algorithm AboveThreshold. Consider a large number of low sensitivity functions  $f_{1}, f_{2}, \ldots, f_{T}$  which are given (one by one) to a data curator (holding a dataset  $S$ ). Algorithm AboveThreshold allows for privately identifying the queries  $f_{i}$  whose value  $f_{i}(S)$  is (roughly) greater than some threshold  $t$ .
Algorithm 1 AboveThreshold [Dwork et al., 2009, Hardt and Rothblum, 2010]
Input: Dataset  $S \in X^{*}$ , privacy parameters  $\varepsilon, \delta$ , threshold  $t$ , number of positive reports  $r$ , and an adaptively chosen stream of queries  $f_{i}: X^{*} \to \mathbb{R}$  with sensitivity  $\Delta$
1. Denote  $\gamma = O\left(\frac{\Delta}{\varepsilon}\sqrt{r}\ln \left(\frac{r}{\delta}\right)\right)$
2. In each round  $i$ , when receiving a query  $f_{i} \in Q$ , do the following:
(a) Let  $\dot{f}_i \gets f_i(S) + \mathrm{Lap}(\gamma)$
(b) If  $\hat{f}_i\geq t$  , then let  $\sigma_{i} = 1$  and otherwise let  $\sigma_{i} = 0$
(c) Output  $\sigma_{i}$
(d) If  $\sum_{j=1}^{i} \sigma_{i} \geq r$  then HALT
Even though the number of possible rounds is unbounded, algorithm AboveThreshold preserves differential privacy. Note, however, that AboveThreshold is an interactive mechanism, while the standard definition of differential privacy (Definition 2.1) is stated for non-interactive mechanisms, that process their input dataset, release an output, and halt. The adaptation of DP to such interactive settings is done via a game between the (interactive) mechanism and an adversary that specifies the inputs to the mechanism and observes its outputs. Intuitively, the privacy requirement is that the view
of the adversary at the end of the execution should be differentially private w.r.t. the inputs given to the mechanism. Formally,
Definition 2.7 (DP under adaptive queries [Dwork et al., 2006, Bun et al., 2017]). Let  $\mathcal{M}$  be a mechanism that takes an input dataset and answers a sequence of adaptively chosen queries (specified by an adversary  $\mathcal{B}$  and chosen from some family  $Q$  of possible queries). Mechanism  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -differentially private if for every adversary  $\mathcal{B}$  we have that  $\text{AdaptiveQuery}_{\mathcal{M}, \mathcal{B}, Q}$  (defined below) is  $(\varepsilon, \delta)$ -differentially private (w.r.t. its input bit  $b$ ).
Algorithm 2 AdaptiveQuery $_{\mathcal{M},B,Q}$  [Bun et al., 2017]
Input: A bit  $b \in \{0,1\}$ . (The bit  $b$  is unknown to  $\mathcal{M}$  and  $\mathcal{B}$ .)