Title: Black-Box Differential Privacy for Interactive ML
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy

online learning
Abstract: In this work we revisit an interactive variant of joint differential privacy, recently introduced by Naor et al. [2023], and generalize it towards handling online processes in which existing privacy definitions seem too restrictive. We study basic properties of this definition and demonstrate that it satisfies (suitable variants) of group privacy, composition, and post processing.

In order to demonstrate the advantages of this privacy definition compared to traditional forms of differential privacy,
we consider the basic setting of online classification. We show that any (possibly non-private) learning rule can be effectively transformed to a private learning rule with only a polynomial overhead in the mistake bound. This demonstrates a stark difference with traditional forms of differential privacy, such as the one studied  by Golowich and Livni [2021], where only a double exponential overhead in the mistake bound is known (via an information theoretic upper bound).

Black-Box Differential Privacy for Interactive ML

Abstract

In this work we revisit an interactive variant of joint differential privacy, recently introduced by Naor et al. [2023], and generalize it towards handling online processes in which existing privacy definitions seem too restrictive. We study basic properties of this definition and demonstrate that it satisfies (suitable variants) of group privacy, composition, and post processing.

In order to demonstrate the advantages of this privacy definition compared to traditional forms of differential privacy, we consider the basic setting of online classification. We show that any (possibly non-private) learning rule can be effectively transformed to a private learning rule with only a polynomial overhead in the mistake bound. This demonstrates a stark difference with traditional forms of differential privacy, such as the one studied by Golowich and Livni [2021], where only a double exponential overhead in the mistake bound is known (via an information theoretic upper bound).

1 Introduction

In this work we study privacy of interactive machine learning processes. As a motivating story, consider a chatbot that continuously improves itself by learning from the conversations it conducts with users. As these conversations might contain sensitive information, we would like to provide privacy guarantees to the users, in the sense that the content of their conversations with the chatbot would not leak. This setting fleshes out the following two requirements.

Clearly, the answers given by the chatbot to user u i must depend on the queries made by user u i . For example, the chatbot should provide different answers when asked by user u i for the weather forecast in Antarctica, and when asked by u i for a pasta recipe.

This is in contrast to the plain formulation of differential privacy, where it is required that all of the mechanism outputs would be (almost) independent of any single user input. Therefore, the privacy requirement we are aiming for is that the conversation of user u i will remain 'hidden' from other users, and would not leak through the other users' interactions with the chatbot. Moreover, this should remain true even if a 'privacy attacker' (aiming to extract information about the conversation user u i had) conducts many different conversations with the chatbot.

The interaction with the chatbot is, by design, interactive and adaptive , as it aims to conduct dialogues with the users. This allows the privacy attacker (mentioned above) to choose its queries to the chatbot adaptively . Privacy, hence, needs to be preserved even in the presence of adaptive attackers.

While each of these two requirements was studied in isolation, to the best of our knowledge, they have not been (explicitly) unified into a combined privacy framework. Requirement (1) was formalized

by Kearns et al. [2015] as joint differential privacy (JDP) . It provides privacy against non-adaptive attackers. Intuitively, in the chatbot example, JDP aims to hide the conversation of user u i from any privacy attacker that chooses in advance all the queries it poses to the chatbot. This is unsatisfactory since the adaptive nature of this process invites adaptive attackers.

Requirement (2) was studied in many different settings, but to the best of our knowledge, only w.r.t. the plain formulation of DP, where the (adaptive) privacy attacker sees all of the outputs of the mechanism. Works in this vein include [Dwork et al., 2009, Chan et al., 2010, Hardt and Rothblum, 2010, Dwork et al., 2010b, Bun et al., 2017, Kaplan et al., 2021, Jain et al., 2021]. In the chatbot example, plain DP would require, in particular, that even the messages sent from the chatbot to user u i reveals (almost) no information about u i . In theory, this could be obtained by making sure that the entire chatbot model is computed in a privacy preserving manner, such that even its full description leaks almost no information about any single user. Then, when user u i comes, we can 'simply' share the model with her, and let her query it locally on her device. But this is likely unrealistic with large models involving hundreds of billions of parameters.

In this work we use challenge differential privacy , which was recently introduced by Naor et al. [2023] in the context of PAC learning. 6 As discussed below, challenge differential privacy is particularly suitable for addressing interactive and adaptive learning processes, such as the one illustrated above. Challenge DP can be viewed as an interactive variant of JDP, aimed at maintaining privacy against adaptive privacy attackers. Intuitively, in the chatbot example, this definition would guarantee that even an adaptive attacker that controls all of the users except for user u i , learns (almost) no information about the conversation user u i had with the chatbot.

1.1 Private Online Classification

We initiate the study of challenge differential privacy in the basic setting of online classification. Let X be the domain, Y be the label space, and Z = X ×Y be set of labeled examples. An online learner is a (possibly randomized) mapping A : Z ⋆ ×X → Y . That is, it is a mapping that maps a finite sequence S ∈ Z ⋆ (the past examples), and an unlabeled example x (the current query point) to a label y , which is denoted by y = A ( x ; S ) .

̸

Let H ⊆ Y X be a hypothesis class. A sequence S ∈ Z ⋆ is said to be realizable by H if there exists h ∈ H such that h ( x i ) = y i for every ( x i , y i ) ∈ S . For a sequence S = { ( x t , y t ) } T t =1 ∈ Z ⋆ we write M ( A ; S ) for the random variable denoting the number of mistakes A makes during the execution on S . That is M ( A ; S ) = ∑ T t =1 1 { ˆ y t = y t } , where ˆ y t = A ( x t ; S <t ) is the (randomized) prediction of A on x t .

Definition 1.1 (Online Learnability: Realizable Case) . We say that a hypothesis class H is online learnable if there exists a learning rule A such that ❊ [ M ( A ; S )] = o ( T ) for every sequence S which is realizable by H .

Remark 1.2. Notice that Definition 1.1 corresponds to an oblivious adversary, as it quantifies over the input sequence in advance. This should not be confused with the adversaries considered in the context of privacy which are always adaptive in this work. In the non-private setting, focusing on oblivious adversaries does not affect generality in terms of utility. This is less clear when privacy constraints are involved. 7 We emphasize that our results (our mistake bounds) continue to hold even when the realizable sequence is chosen by an adaptive (stateful) adversary, that at every point in time chooses the next input to the algorithm based on all of the previous outputs of the algorithm.

A classical result due to Littlestone [1988] characterizes online learnability (without privacy constraints) in terms of the Littlestone dimension. The latter is a combinatorial parameter of H which was named after Littlestone by Ben-David et al. [2009].

In particular, Littlestone's characterization implies the following dichotomy: if H has finite Littlestone dimension d then there exists a (deterministic) learning rule which makes at most d mistakes on

every realizable input sequence. In the complementing case, when the Littlestone dimension of H is infinite, for every learning rule A and every T ∈ N there exists a realizable sequence S of length T such that ❊ [ M ( A ; S )] ≥ T/ 2 . In other words, as a function of T , the optimal mistake bound is either uniformly bounded by the Littlestone dimension, or it is ≥ T/ 2 . Because of this dichotomy, in some places online learnability is defined with respect to a uniform bound on the number of mistakes (and not just a sublinear one as in the above definition). In this work we follow the more general definition.

We investigate the following questions: Can every online learnable class be learned by an algorithm which satisfies challenge differential privacy? What is the optimal mistake bound attainable by private learners?

Our main result in this part provides an affirmative answer to the first question. We show that for any class H with Littlestone dimension d there exists an ( ε, δ ) -challenge-DP learning rule which makes at most ˜ O ( d 2 ε 2 log 2 ( 1 δ ) log 2 ( T β )) mistakes, with probability 1 -β , on every realizable sequence of length T . Remarkably, our proof provides an efficient transformation taking a non-private learner to a private one: that is, given a black box access to a learning rule A which makes at most M mistakes in the realizable case, we efficiently construct an ( ε, δ ) -challenge-DP learning rule A ′ which makes at most ˜ O ( M 2 ε 2 log 2 ( 1 δ ) log 2 ( T β )) mistakes.

1.1.1 Construction overview

We now give a simplified overview of our construction, called POP , which transforms a non-private online learning algorithm into a private one (while maintaining computational efficiency). Let A be a non-private algorithm, guaranteed to make at most d mistakes in the realizable setting. We maintain k copies of A . Informally, in every round i ∈ [ T ] we do the following:

Obtain an input point x i .

Give x i to each of the k copies of A to obtain predicted labels ˆ y i, 1 , . . . , ˆ y i,k .

Output a 'privacy preserving' aggregation ˆ y i of { ˆ y i, 1 , . . . , ˆ y i,k } , which is some variant of noisy majority. This step will only satisfy challenge-DP, rather than (standard) DP.

Obtain the 'true' label y i .

Let ℓ ∈ [ k ] be chosen at random.

Rewind all copies of algorithm A except for the ℓ th copy, so that they 'forget' ever seeing x i .

Give the true label y i to the ℓ th copy of A .

As we aggregate the predictions given by the copies of A using (noisy) majority, we know that if the algorithm errs then at least a constant fraction of the copies of A err. As we feed the true label y i to a random copy, with constant probability, the copy which we do not rewind incurs a mistake at this moment. That is, whenever we make a mistake then with constant probability one of the copies we maintain incurs a mistake. This can happen at most ≈ k · d times, since we have k copies and each of them makes at most d mistakes. This allows us to bound the number of mistakes made by our algorithm (w.h.p.). The privacy analysis is more involved. Intuitively, by rewinding all of the copies of A (except one) in every round, we make sure that a single user can affect the inner state of at most one of the copies. This allows us to efficiently aggregate the predictions given by the copies in a privacy preserving manner. The subtle point is that the prediction we release in time i does require querying all the experts on the current example x i (before rewinding them). Nevertheless, we show that this algorithm is private.

1.1.2 Comparison with Golowich and Livni [2021]

The closest prior work to this manuscript is by Golowich and Livni who also studied the problem of private online classification, but under a more restrictive notion of privacy than challenge-DP. In particular their definition requires that the sequence of predictors which the learner uses to predict in each round does not compromise privacy. In other words, it is as if at each round the learner publishes the entire truth-table of its predictor, rather than just its current prediction. This might be too prohibitive in certain applications such as the chatbot example illustrated above. Golowich and Livni show that even with respect to their more restrictive notion of privacy it is possible to online

learn every Littlestone class. However, their mistake bound is doubly exponential in the Littlestone dimension (whereas ours is quadratic), and their construction requires more elaborate access to the non-private learner. In particular, it is not clear whether their construction can be implemented efficiently (while our construction is efficient).

1.2 Additional Related Work

Several works studied the related problem of private learning from expert advice [Dwork et al., 2010a, Jain et al., 2012, Thakurta and Smith, 2013, Dwork and Roth, 2014, Jain and Thakurta, 2014, Agarwal and Singh, 2017, Asi et al., 2022]. These works study a variant of the experts problem in which the learning algorithm has access to k experts ; on every time step the learning algorithm chooses one of the experts to follow, and then observes the loss of each expert. The goal of the learning algorithm is that its accumulated loss will be competitive with the loss of the best expert in hindsight . In this setting the private data is the sequence of losses observed throughout the execution, and the privacy requirement is that the sequence of experts chosen by the algorithm should not compromise the privacy of the sequence of losses. 8 When applying these results to our context, the set of experts is the set of hypotheses in the class H , which means that the outcome of the learner (on every time step) is a complete model (i.e., a hypothesis). That is, in our context, applying prior works on private prediction from expert advice would result in a privacy definition similar to that of Golowich and Livni [2021] that accounts (in the privacy analysis) for releasing complete models, rather than just the predictions, which is significantly more restrictive.

There were a few works that studied private learning in online settings under the constraint of JDP. For example, Shariff and Sheffet [2018] studied the stochastic contextual linear bandits problem under JDP. Here, in every round t the learner receives a context c t , then it selects an action a t (from a fixed set of actions), and finaly it receives a reward y t which depends on ( c t , a t ) in a linear way. The learner's objective is to maximize cumulative reward. The (non-adaptive) definition of JDP means that action a t is revealed only to user u t . Furthermore, it guarantees that the inputs of user u t (specifically the context c t and the reward y t ) do not leak to the other users via the actions they are given, provided that all these other users fix their data in advance . This non-adaptive privacy notion fits the stochastic setting of Shariff and Sheffet [2018], but (we believe) is less suited for adversarial processes like the ones we consider in this work. We also note that the algorithm of Shariff and Sheffet [2018] in fact satisfies the more restrictive privacy definition which applies to the sequence of predictors (rather than the sequence of predictions), similarly to the that of Golowich and Livni [2021].

2 Preliminaries

Notation. Two datasets S and S ′ are called neighboring if one is obtained from the other by adding or deleting one element, e.g., S ′ = S ∪ { x ′ } . For two random variables Y, Z we write X ≈ ( ε,δ ) Y to mean that for every event F it holds that Pr[ X ∈ F ] ≤ e ε · Pr[ Y ∈ F ] + δ , and Pr[ Y ∈ F ] ≤ e ε · Pr[ X ∈ F ] + δ . Throughout the paper we assume that the privacy parameter ε satisfies ε = O (1) , but our analyses trivially extend to larger values of epsilon.

The standard definition of differential privacy is,

Definition 2.1 ([Dwork et al., 2006]) . Let M be a randomized algorithm that operates on datasets. Algorithm M is ( ε, δ ) -differentially private (DP) if for any two neighboring datasets S, S ′ we have M ( S ) ≈ ( ε,δ ) M ( S ′ ) .

The Laplace mechanism. The most basic constructions of differentially private algorithms are via the Laplace mechanism as follows.

Definition 2.2. A random variable has probability distribution Lap( γ ) if its probability density function is f ( x ) = 1 2 γ exp( -| x | /γ ) , where x ∈ R .

Definition 2.3 (Sensitivity) . A function f that maps datasets to the reals has sensitivity ∆ if for every two neighboring datasets S and S ′ it holds that | f ( S ) -f ( S ′ ) | ≤ ∆ .

Theorem 2.4 (The Laplace Mechanism [Dwork et al., 2006]) . Let f be a function that maps datasets to the reals with sensitivity ∆ . The mechanism A that on input S adds noise with distribution Lap( ∆ ε ) to the output of f ( S ) preserves ( ε, 0) -differential privacy.

Joint differential privacy. The standard definition of differential privacy (Definition 2.1) captures a setting in which the entire output of the computation may be publicly released without compromising privacy. While this is a very desirable requirement, it is sometimes too restrictive. Indeed, Kearns et al. [2015] considered a relaxed setting in which we aim to analyze a dataset S = ( x 1 , . . . , x n ) , where every x i represents the information of user i , and to obtain a vector of outcomes ( y 1 , . . . , y n ) . This vector, however, is not made public. Instead, every user i only receives its 'corresponding outcome' y i . This setting potentially allows the outcome y i to strongly depend on the the input x i , without compromising the privacy of the i th user from the view point of the other users.

Definition 2.5 ([Kearns et al., 2015]) . Let M : X n → Y n be a randomized algorithm that takes a dataset S ∈ X n and outputs a vector ⃗ y ∈ Y n . Algorithm M satisfies ( ε, δ ) -joint differential privacy (JDP) if for every i ∈ [ n ] and every two datasets S, S ′ ∈ X n differing only on their i th point it holds that M ( S ) -i ≈ ( ε,δ ) M ( S ′ ) -i . Here M ( S ) -i denotes the (random) vector of length n -1 obtained by running ( y 1 , . . . , y n ) ←M ( S ) and returning ( y 1 , . . . , y i -1 , y i +1 , . . . , y n ) .

In words, consider an algorithm M that operates on the data of n individuals and outputs n outcomes y 1 , . . . , y n . This algorithm is JDP if changing only the i th input point x i has almost no affect on the outcome distribution of the other outputs (but the outcome distribution of y i is allowed to strongly depend on x i ). Kearns et al. [2015] showed that this fits a wide range of problems in economic environments.

Example 2.6 ([Nahmias et al., 2019]) . Suppose that a city water corporation is interested in promoting water conservation. To do so, the corporation decided to send each household a customized report indicating whether their water consumption is above or below the median consumption in the neighborhood. Of course, this must be done in a way that protects the privacy of the neighbors. One way to tackle this would be to compute a privacy preserving estimation z for the median consumption (satisfying Definition 2.1). Then, in each report, we could safely indicate whether the household's water consumption is bigger or smaller than z . While this solution is natural and intuitive, it turns out to be sub-optimal: We can obtain better utility by designing a JDP algorithm that directly computes a different outcome for each user ('above' or 'below'), which is what we really aimed for, without going through a private median computation.

Algorithm AboveThreshold . Consider a large number of low sensitivity functions f 1 , f 2 , . . . , f T which are given (one by one) to a data curator (holding a dataset S ). Algorithm AboveThreshold allows for privately identifying the queries f i whose value f i ( S ) is (roughly) greater than some threshold t .

Algorithm 1 AboveThreshold [Dwork et al., 2009, Hardt and Rothblum, 2010]

Input: Dataset S ∈ X ∗ , privacy parameters ε, δ , threshold t , number of positive reports r , and an adaptively chosen stream of queries f i : X ∗ → R with sensitivity ∆ √

Denote γ = O ( ∆ ε r ln( r δ ) )

In each round i , when receiving a query f i ∈ Q , do the following:

(a) Let ˆ f i ← f i ( S ) + Lap( γ )

(b) If ˆ f i ≥ t , then let σ i = 1 and otherwise let σ i = 0

(c) Output σ i

(d) If ∑ i j =1 σ i ≥ r then HALT

Even though the number of possible rounds is unbounded, algorithm AboveThreshold preserves differential privacy. Note, however, that AboveThreshold is an interactive mechanism, while the standard definition of differential privacy (Definition 2.1) is stated for non-interactive mechanisms, that process their input dataset, release an output, and halt. The adaptation of DP to such interactive settings is done via a game between the (interactive) mechanism and an adversary that specifies the inputs to the mechanism and observes its outputs. Intuitively, the privacy requirement is that the view

of the adversary at the end of the execution should be differentially private w.r.t. the inputs given to the mechanism. Formally,

Definition 2.7 (DP under adaptive queries [Dwork et al., 2006, Bun et al., 2017]) . Let M be a mechanism that takes an input dataset and answers a sequence of adaptively chosen queries (specified by an adversary B and chosen from some family Q of possible queries). Mechanism M is ( ε, δ ) -differentially private if for every adversary B we have that AdaptiveQuery M , B ,Q (defined below) is ( ε, δ ) -differentially private (w.r.t. its input bit b ).

Algorithm 2 AdaptiveQuery M , B ,Q [Bun et al., 2017]

Input: A bit b ∈ { 0 , 1 } . (The bit b is unknown to M and B .)

The adversary B chooses two neighboring datasets S 0 and S 1 .

The dataset S b is given to the mechanism M .

For i = 1 , 2 , . . .

(a) The adversary B chooses a query q i ∈ Q .

(b) The mechanism M is given q i and returns a i .

(c) a i is given to B .

When M or B halts, output B 's view of the interaction.

Theorem 2.8 ([Dwork et al., 2009, Hardt and Rothblum, 2010, Kaplan et al., 2021]) . Algorithm AboveThreshold is ( ε, δ ) -differentially private.

A private counter. In the setting of algorithm AboveThreshold , the dataset is fixed in the beginning of the execution, and the queries arrive sequentially one by one. Dwork et al. [2010a] and Chan et al. [2010] considered a different setting, in which the data arrives sequentially. In particular, they considered the counter problem where in every time step i ∈ [ T ] we obtain an input bit x i ∈ { 0 , 1 } (representing the data of user i ) and must immediately respond with an approximation for the current sum of the bits. That is, at time i we wish to release an approximation for x 1 + x 2 + · · · + x i .

Similarly to our previous discussion, this is an interactive setting, and privacy is defined via a game between a mechanism M and an adversary B that adaptively determines the inputs for the mechanism.

Definition 2.9 (DP under adaptive inputs [Dwork et al., 2006, 2010a, Chan et al., 2010, Kaplan et al., 2021, Jain et al., 2021]) . Let M be a mechanism that in every round i obtains an input point x i (representing the information of user i ) and outputs a response a i . Mechanism M is ( ε, δ ) -differentially private if for every adversary B we have that AdaptiveInput M , B (defined below) is ( ε, δ ) -differentially private (w.r.t. its input bit b ).

Algorithm 3 AdaptiveInput M , B [Jain et al., 2021]

Input: A bit b ∈ { 0 , 1 } . (The bit b is unknown to M and B .)

For i = 1 , 2 , . . .

(a) The adversary B outputs a bit c i ∈ { 0 , 1 } , under the restriction that ∑ i j =1 c j ≤ 1 . %The round i in which c i = 1 is called the challenge round. Note that there could be at most one challenge round throughout the game.

(b) The adversary B chooses two input points x i, 0 and x i, 1 , under the restriction that if c i = 0 then x i, 0 = x i, 1 .

(c) Algorithm M obtains x i,b and outputs a i .

(d) a i is given to B .

When M or B halts, output B 's view of the interaction.

Theorem 2.10 (Private counter [Dwork et al., 2010a, Chan et al., 2010, Jain et al., 2021]) . There exists an ( ε, 0) -differentially private mechanism M (as in Definition 2.9) that in each round i ∈ [ T ] obtains an input bit x i ∈ { 0 , 1 } and outputs a response a i ∈ N with the following properties. Let s denote the random coins of M . Then there exists an event E such that: (1) Pr[ s ∈ E ] ≥ 1 -β , and (2) Conditioned on every s ∈ E , for every input sequence ( x 1 , . . . , x T ) , the answers ( a 1 , . . . , a T )

3 Challenge Differential Privacy

We now introduce the privacy definition we consider in this work is. Intuitively, the requirement is that even an adaptive adversary controlling all of the users except Alice, cannot learn much information about the interaction Alice had with the algorithm.

Definition 3.1 (Extension of Naor et al. [2023]) . Consider an algorithm M that, in each round i ∈ [ T ] obtains an input point x i , outputs a 'predicted' label ˆ y i , and obtains a 'true' label y i . We say that algorithm M is ( ε, δ ) -challenge differentially private if for any adversary B we have that OnlineGame M , B ,T , defined below, is ( ε, δ ) -differentially private (w.r.t. its input bit b ).

Algorithm 4 OnlineGame M , B ,T,g

Setting: T ∈ N denotes the number of rounds and g ∈ N is a 'group privacy' parameter. If not explicitly stated we assume that g = 1 . M is an online algorithm and B is an adversary that determines the inputs adaptively.

Input of the game: A bit b ∈ { 0 , 1 } . (The bit b is unknown to M and B .)

For i = 1 , 2 , . . . , T

(a) The adversary B outputs a bit c i ∈ { 0 , 1 } , under the restriction that ∑ i j =1 c j ≤ g . %Weinterpret rounds i in which c i = 1 as challenge rounds. Note that there could be at most g challenge rounds throughout the game.

(b) The adversary B chooses two labeled inputs ( x i, 0 , y i, 0 ) and ( x i, 1 , y i, 1 ) , under the restriction that if c i = 0 then ( x i, 0 , y i, 0 ) = ( x i, 1 , y i, 1 ) .

(c) Algorithm M obtains x i,b , then outputs ˆ y i , and then obtains y i,b .

(d) If c i = 0 then set ˜ y i = ˆ y i . Otherwise set ˜ y i = ⊥ .

(e) The adversary B obtains ˜ y i .

%Note that the adversary B does not get to see the outputs of M in challenge rounds.

Output B 's view of the game, that is ˜ y 1 , . . . , ˜ y T and the internal randomness of B .

%Note that from this we can reconstruct all the inputs specified by B throughout the game.

Remark 3.2. Naor et al. [2023] studied a special case of this definition, suitable to their stochastic setting. More specifically, they considered a setting where the algorithm initially gets a dataset containing labeled examples. Then, on every time step, a new user arrives and submits its unlabeled example to the algorithm, and the algorithm responds with a predicted label. We extend the definition to capture settings in which every user interacts with the algorithm (rather than just submitting its input). In the concrete application we consider (online learning) this corresponds to the user submitting its input, then obtaining the predicted label, and then submitting the 'true' label. Our generalized definition (Section A) captures this as a special case and allows for arbitrary interactions with each user.

Composition and post-processing. Composition and post-processing for challenge-DP follows immediately from their analogues for (standard) DP. Formally, composition is defined via the following game, called CompositionGame , in which a 'meta adversary' B ∗ is trying to guess an unknown bit b ∈ { 0 , 1 } . The meta adversary B ∗ is allowed to (adaptively) invoke k executions of the game specified in Algorithm 4, where all of these k executions are done with the same (unknown) bit b . See Algorithm 5.

Algorithm 5 CompositionGame B ∗ ,m,ε,δ

Input of the game: A bit b ∈ { 0 , 1 } . (The bit b is unknown to B ∗ .)

For ℓ = 1 , 2 , . . . , m

(a) The adversary B ∗ outputs an ( ε, δ ) -challenge-DP algorithm M ℓ , an adversary B ℓ , and an integer T ℓ .

(b) The adversary B ∗ obtains the outcome of OnlineGame M ℓ , B ℓ ,T ℓ ( b ) .

Output B ∗ 's view of the game (its internal randomness and all of the outcomes of OnlineGame it obtained throughout the execution).

The following theorem follows immediately from standard composition theorems for differential privacy [Dwork et al., 2010b].

Theorem 3.3 (special case of [Dwork et al., 2010b]) . For every B ∗ , every m ∈ N and every ε, δ, δ ′ ≥ 0 it holds that CompositionGame B ∗ ,m,ε,δ is ( ε ′ , mδ + δ ′ ) -differentially private (w.r.t. the input bit b ) for ε ′ = √ 2 m ln(1 /δ ′ ) ε + mε ( e ε -1) .

Group privacy. We show that challenge-DP is closed under group privacy. This is more subtle than the composition argument. In fact, we first need to define what do we mean by 'group privacy' in the context of challenge-DP, which we do using the parameter g in algorithm OnlineGame . Recall that throughout the execution of OnlineGame , the adversary is allowed g challenge rounds. We show that if an algorithm satisfies challenge-DP when the adversary is allowed only a single challenge round, then it also satisfies challenge-DP (with related privacy parameters) when the adversary is allowed g > 1 challenge rounds. This is captured by the following theorem; see Appendix B for the proof.

Theorem 3.4. Let M be an algorithm that in each round i ∈ [ T ] obtains an input point x i , outputs a 'predicted' label ˆ y i , and obtains a 'true' label y i . If M is ( ε, δ ) -challenge-DP then for every g ∈ N and every adversary B (posing at most g challenges) we have that OnlineGame M , B ,T,g is ( gε, g · e εg · δ ) -differentially private.

4 Algorithm ChallengeAT

Towards presenting our private online learner, we introduce a variant of algorithm AboveThreshold with additional guarantees, which we call ChallengeAT . Recall that AboveThreshold 'hides' arbitrary modifications to a single input point. Intuitively, the new variant we present aims to hide both an arbitrary modification to a single input point and an arbitrary modification to a single query throughout the execution. Consider algorithm ChallengeAT .

Algorithm 6 ChallengeAT

Input: Dataset S ∈ X ∗ , privacy parameters ε, δ , threshold t , number of positive reports r , and an adaptively chosen stream of queries f i : X ∗ → R each with sensitivity ∆

Tool used: An ( ε, 0) -DP algorithm, PrivateCounter , for counting bits under continual observation, guaranteeing error at most λ with probability at least 1 -δ

Instantiate PrivateCounter √

Denote γ = O ( ∆ ε r + λ ln( r + λ δ ) )

In each round i , when receiving a query f i , do the following:

(a) Let ˆ f i ← f i ( S ) + Lap( γ )

(b) If ˆ f i ≥ t , then let σ i = 1 and otherwise let σ i = 0

(c) Output σ i

(d) Feed σ i to PrivateCounter and let count i denote its current output

(e) If count i ≥ r then HALT

Remark 4.1. When we apply ChallengeAT , it sets λ = O ( 1 ε log( T ) log ( T β )) . Technically, for this it has to know T and β . To simplify the description this is not explicit in our algorithms.

The utility guarantees of ChallengeAT are straightforward. The following theorem follows by bounding (w.h.p.) all the noises sampled throughout the execution (when instantiating ChallengeAT with the private counter from Theorem 2.10).

Theorem 4.2. Let s denote the random coins of ChallengeAT . Then there exists an event E such that: (1) Pr[ s ∈ E ] ≥ 1 -β , and (2) Conditioned on every s ∈ E , for every input dataset S and every sequence of T queries ( f 1 , . . . , f T ) it holds that

Algorithm ChallengeAT does not halt before the r th time in which it outputs σ i = 1 ,

For every i such that σ i = 1 it holds that f i ( S ) ≥ t -O ( ∆ ε √ r + λ ln( r + λ δ ) log( T β ) ) ,

For every i such that σ i = 0 it holds that f i ( S ) ≤ t + O ( ∆ ε √ r + λ ln( r + λ δ ) log( T β ) ) ,

The event E in Theorem 4.2 occurs when all the Laplace noises of the counter and ChallengeAT are within a factor of log( T/β ) of their expectation. The privacy guarantees of ChallengeAT are more involved. They are defined via a game with an adversary B whose goal is to guess a secret bit b . At the beginning of the game, the adversary chooses two neighboring datasets S 0 , S 1 , and ChallengeAT is instantiated with S b . Then throughout the game the adversary specifies queries f i and observes the output of ChallengeAT on these queries. At some special round i ∗ , chosen by the adversary, the adversary specifies two queries f 0 i ∗ , f 1 i ∗ , where only f b i ∗ is fed into ChallengeAT . In round i ∗ the adversary does not get to see the answer of ChallengeAT on f b i ∗ (otherwise it could easily learn the bit b since f 0 i ∗ , f 1 i ∗ may be very different). We show that any such adversary B has only a small advantage in guessing the bit b . The formal details are given in Appendix C.

5 Online Classification under Challenge Differential Privacy

We are now ready to present our private online prediction algorithm. Consider algorithm POP .

Theorem 5.1. When executed with a learner A that makes at most d mistakes and with parameters k = ˜ O ( d ε 2 log 2 ( 1 δ ) log 2 ( T β ) ) and r = O ( dk +ln ( 1 β )) , then with probability at least (1 -β ) the number of mistakes made by algorithm POP is bounded by ˜ O ( d 2 ε 2 log 2 ( 1 δ ) log 2 ( T β ) ) .

Algorithm 7 POP (Private Online Procedure)

Setting: T ∈ N denotes the number of rounds in the game. A is a non-private online-algorithm. Parameters: k determines the number of copies of A we maintain. r determines the number of positive reports we aim to receive from ChallengeAT .

Instantiate k copies A 1 , . . . , A k of algorithm A

Instantiate algorithm ChallengeAT on an empty dataset with threshold t = -k/ 4 , privacy parameters ε, δ , number of positive reports r , and sensitivity parameter ∆ = 1 .

For i = 1 , 2 , . . . , T

(a) Obtain input x i

(b) Let A temp 1 , . . . , A temp k be duplicated copies of A 1 , . . . , A k

(c) Let ℓ i ∈ [ k ] be chosen uniformly at random

(d) Let ˆ y i,ℓ i ←A ℓ i ( x i ) . For j ∈ [ k ] \ { ℓ i } let ˆ y i,j ←A temp j ( x i )

(e) Feed ChallengeAT the query f i ≡ -∣ ∣ ∣ k 2 -∑ j ∈ [ k ] ˆ y i,j ∣ ∣ ∣ and obtain an outcome σ i . (If ChallengeAT halts then POP also halts.)

%Recall that σ i = 1 indicates that -∣ ∣ ∣ k 2 -∑ j ∈ [ k ] ˆ y i,j ∣ ∣ ∣ ≳ -k 4 , meaning that there is 'a lot' of disagreement among ˆ y i, 1 , . . . , ˆ y i,k .

(f) If σ i = 1 then sample ˆ y i ∈ { 0 , 1 } at random. Else let ˆ y i = majority { ˆ y i, 1 , . . . , ˆ y i,k }

(g) Output the bit ˆ y i as a prediction, and obtain a 'true' label y i

(h) Feed y i to A ℓ i

% Note that A ℓ is the only copy of A that changes its state during this iteration

Informally, the privacy guarantees of POP follow from those of ChallengeAT . We give the formal details in Appendix D, obtaining the following theorem.

Theorem 5.2. Algorithm POP is ( O ( ε ) , O ( δ )) -Challenge-DP. That is, For every adversary B it holds that OnlineGamePOP , B is ( O ( ε ) , O ( δ )) -DP w.r.t. the bit b (the input of the game).

We proceed with the utility guarantees of POP . See Appendix F for an extension to the agnostic setting.

Theorem 5.3. When executed with a learner A that makes at most d mistakes and with parameters k = ˜ O ( d ε 2 log 2 ( 1 δ ) log 2 ( T β ) ) and r = O ( dk +ln ( 1 β )) , then with probability at least (1 -β ) the number of mistakes made by algorithm POP is bounded by ˜ O ( d 2 ε 2 log 2 ( 1 δ ) log 2 ( T β ) ) .

Proof. By Theorem 4.2, with probability (1 -β ) over the internal coins of ChallengeAT , for every input sequence, its answers are accurate up to error of

where in our case, the sensitivity ∆ is 1 , and the error of the counter λ is at most O ( 1 ε log( T ) log ( T δ )) by Theorem 2.10. We continue with the proof assuming that this event occurs. Furthermore, we set k = Ω(error CAT ) , large enough, such that if less than 1 5 the experts disagree with the other experts, then algorithm POP returns the majority vote with probability 1.

Consider the execution of algorithm POP and define 1 / 5 -Err be a random variable that counts the number of time steps in which at least 1 / 5 th of the experts make an error. That is

̸

̸

We also define the random variable

That is expertAdvance counts the number of times steps in which the random expert we choose (the ℓ i th expert) errs. Note that the ℓ i th expert is the expert that gets the 'true' label y i as feedback. As we run k experts, and as each of them is guaranteed to make at most d mistakes, we get that

We now show that with high probability 1/5-Err is not much larger than expertAdvance . Let i be a time step in which at least 1 / 5 fraction of the experts err. As the choice of ℓ i (the expert we update) is random, then with probability at least 1 5 the chosen expert also errs. It is therefore unlikely that 1/5-Err is much larger than expertAdvance , which is bounded by kd . Specifically, by standard concentration arguments (see Appendix E for the precise version we use) it holds that

Note that when at least 1 / 5 of the experts disagree with other experts then at least 1 / 5 of the experts err. It follows that 1/5-Err upper bounds the number of times in which algorithm ChallengeAT returns an 'above threshold' answer. Hence, by setting r > 18 dk +18 + ln ( 1 β ) we ensure that w.h.p. algorithm ChallangeAT does not halt prematurely (and hence POP does not either).

Furthermore our algorithm errs either when there is a large disagreement between the experts or when all experts err. It follows that 1/5-Err also upper bounds the number of times which our algorithm errs.

Overall, by setting r = O ( dk +ln ( 1 β )) we ensure that POP does not halt prematurely, and by setting k = O ( ∆ ε √ r + λ ln( r + λ δ ) log( T β ) ) we ensure that POP does not err too many times throughout the execution. Combining the requirement on r and on k , it suffices to take

in which case algorithm POP makes at most ˜ O ( d 2 ε 2 log 2 ( 1 δ ) log 2 ( T β ) ) with high probability.

6 Conclusion

Our work presents a new privacy model for online classification, together with an efficiency preserving transformation from non-private online classification, that exhibits a doubly exponential improvement in the error compared to prior works on this topic. We leave open the possibility that such an improvement could also be achieved in the setting of Golowich and Livni [2021], i.e., under the more restrictive notion of privacy where the sequence of predictors does not compromise privacy.

Acknowledgments and Disclosure of Funding

Haim Kaplan was artially supported by Israel Science Foundation (grant 1595/19), and the Blavatnik Family Foundation.

Yishay Mansour was partially funded from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science Foundation (grant number 993/17), Tel Aviv University Center for AI and Data Science (TAD), and the Yandex Initiative for Machine Learning at Tel Aviv University.

Shay Moran is a Robert J. Shillman Fellow; he acknowledges support by ISF grant 1225/20, by BSF grant 2018385, by an Azrieli Faculty Fellowship, by Israel PBC-VATAT, by the Technion Center for Machine Learning and Intelligent Systems (MLIS), and by the European Union (ERC, GENERALIZATION, 101039692). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

Kobbi Nissim was partially funded by NSF grant No. CNS 2001041 and by a gift to Georgetown University.

Uri Stemmer was partially supported by the Israel Science Foundation (grant 1871/19) and by Len Blavatnik and the Blavatnik Family foundation.

References

Naman Agarwal and Karan Singh. The price of differential privacy for online learning. In ICML , volume 70 of Proceedings of Machine Learning Research , pages 32-40. PMLR, 06-11 Aug 2017.

Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private online prediction from experts: Separations and faster rates. CoRR , abs/2210.13537, 2022.

Shai Ben-David, Dávid Pál, and Shai Shalev-Shwartz. Agnostic online learning. In COLT , 2009.

Mark Bun, Thomas Steinke, and Jonathan Ullman. Make up your mind: The price of online queries in differential privacy. In Proceedings of the twenty-eighth annual ACM-SIAM symposium on discrete algorithms , pages 1306-1325. SIAM, 2017.

T.-H. Hubert Chan, Elaine Shi, and Dawn Song. Private and continual release of statistics. In ICALP (2) , volume 6199 of Lecture Notes in Computer Science , pages 405-417. Springer, 2010.

Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9(3-4):211-407, 2014.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC , pages 265-284, 2006.

Cynthia Dwork, Moni Naor, Omer Reingold, Guy N. Rothblum, and Salil P. Vadhan. On the complexity of differentially private data release: efficient algorithms and hardness results. In STOC , pages 381-390, 2009.

Cynthia Dwork, Moni Naor, Toniann Pitassi, and Guy N. Rothblum. Differential privacy under continual observation. In Symposium on Theory of Computing (STOC) , pages 715-724. ACM, 2010a.

Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. Boosting and differential privacy. In FOCS , pages 51-60, 2010b.

Noah Golowich and Roi Livni. Littlestone classes are privately online learnable. In NeurIPS , pages 11462-11473, 2021.

Anupam Gupta, Katrina Ligett, Frank McSherry, Aaron Roth, and Kunal Talwar. Differentially private combinatorial optimization. In SODA , pages 1106-1125, 2010.

Moritz Hardt and Guy N. Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In FOCS , pages 61-70, 2010.

Palak Jain, Sofya Raskhodnikova, Satchit Sivakumar, and Adam D. Smith. The price of differential privacy under continual observation. CoRR , abs/2112.00828, 2021.

Prateek Jain and Abhradeep Guha Thakurta. (near) dimension independent risk bounds for differentially private learning. In ICML , volume 32 of JMLR Workshop and Conference Proceedings , pages 476-484. JMLR.org, 2014.

Prateek Jain, Pravesh Kothari, and Abhradeep Thakurta. Differentially private online learning. In COLT , volume 23 of JMLR Proceedings , pages 24.1-24.34. JMLR.org, 2012.

Haim Kaplan, Yishay Mansour, and Uri Stemmer. The sparse vector technique, revisited. In COLT , volume 134 of Proceedings of Machine Learning Research , pages 2747-2776. PMLR, 2021.

Michael J. Kearns, Mallesh M. Pai, Ryan M. Rogers, Aaron Roth, and Jonathan R. Ullman. Robust mediators in large games. CoRR , abs/1512.02698, 2015.

Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. In Machine Learning , pages 285-318, 1988.

Yifat Nahmias, Oren Perez, Yotam Shlomo, and Uri Stemmer. Privacy preserving social norm nudges. Mich. Tech. L. Rev. , 26:43, 2019.

Moni Naor, Kobbi Nissim, Uri Stemmer, and Chao Yan. Private everlasting prediction. CoRR , abs/2305.09579, 2023.

Roshan Shariff and Or Sheffet. Differentially private contextual linear bandits. In NeurIPS , pages 4301-4311, 2018.

Abhradeep Guha Thakurta and Adam D. Smith. (nearly) optimal algorithms for private online learning in full-information and bandit settings. In NIPS , pages 2733-2741, 2013.

A General Variant of challenge-DP

Definition A.1. Consider an algorithm M that, in each phase i ∈ [ T ] , conducts an arbitrary interaction with the i th user. We say that algorithm M is ( ε, δ ) -challenge differentially private if for any adversary B we have that GeneralGame M , B ,T , defined below, is ( ε, δ ) -differentially private (w.r.t. its input bit b ).

Algorithm 8 GeneralGame M , B ,T

Setting: T ∈ N denotes the number of phases. M is an interactive algorithm and B is an adaptive and interactive adversary.

Input of the game: A bit b ∈ { 0 , 1 } . (The bit b is unknown to M and B .)

For i = 1 , 2 , . . . , T

(a) The adversary B outputs a bit c i ∈ { 0 , 1 } , under the restriction that ∑ i j =1 c j ≤ 1 .

(b) The adversary B chooses two interactive algorithms I i, 0 and I i, 1 , under the restriction that if c i = 0 then I i, 0 = I i, 1 .

(c) Algorithm M interacts with I i,b . Let ˆ y i denote the view of I i,b at the end of this interaction.

(d) If c i = 0 then set ˜ y i = ˆ y i . Otherwise set ˜ y i = ⊥ .

(e) The adversary B obtains ˜ y i .

Output B 's view of the game.

B Group privacy

In this section we prove Theorem 3.4.

Proof of Theorem 3.4. Fix g ∈ N and fix an adversary B (that poses at most g challenge rounds). We consider a sequence of games W 0 , W 1 , . . . , W g , where W ℓ is defined as follows.

Initialize algorithm M and the adversary B .

For round i = 1 , 2 , . . . , T :

(a) Obtain a challenge indicator c i and two labeled inputs ( x i, 0 , y i, 0 ) and ( x i, 1 , y i, 1 ) from B .

(b) If ∑ i j =1 c j > ℓ then set ( w i , z i ) = ( x i, 0 , y i, 0 ) . Otherwise set ( w i , z i ) = ( x i, 1 , y i, 1 ) .

(c) Feed w i to algorithm M , obtain an outcome ˆ y i , and feed it z i .

(d) If c i = 0 then set ˜ y i = ˆ y i . Otherwise set ˜ y i = ⊥ .

(e) Give ˜ y i to B .

Output ˜ y 1 , . . . , ˜ y T and the internal randomness of B .

That is, W ℓ simulates the online game between M and B , where during the first ℓ challenge rounds algorithm M is given ( x i, 1 , y i, 1 ) , and in the rest of the challenge rounds algorithm M is given ( x i, 0 , y i, 0 ) . Note that

We claim that for every 0 < ℓ ≤ g it holds that W ℓ -1 ≈ ( ε,δ ) W ℓ . To this end, fix 0 < ℓ ≤ g and consider an adversary ̂ B , that poses at most one challenge, defined as follows. Algorithm ̂ B runs B internally. In every round i , algorithm ̂ B obtains from B a challenge bit c i and two labeled inputs ( x i, 0 , y i, 0 ) and ( x i, 1 , y i, 1 ) . As long as B did not pose its ℓ th challenge, algorithm ̂ B outputs ( x i, 1 , y i, 1 ) , ( x i, 1 , y i, 1 ) . During the round i in which B poses its ℓ th challenge, algorithm B outputs ( x i, 0 , y i, 0 ) , ( x i, 1 , y i, 1 ) . This is the challenge round posed by algorithm ̂ B . In every round t afterwards, algorithm ̂ B outputs ( x i, 0 , y i, 0 ) , ( x i, 0 , y i, 0 ) . When algorithm ̂ B obtains an answer ˜ y i it

As ̂ B is an adversary that poses (at most) one challenge, by the privacy properties of M we know that OnlineGame M , ̂ B ,T is ( ε, δ ) -DP. Recall that the output of OnlineGame M , ̂ B ,T includes all of the randomness of ̂ B , as well as the answers ˜ y t generated throughout the game. This includes the randomness of B (which ̂ B runs internally), and hence, determines also all of the ˜ ˜ y i 's defined by ̂ B throughout the interaction. Let P be a post-processing procedure that takes the output of OnlineGame M , ̂ B ,T and returns the randomness of B as well as ( ˜ ˜ y 1 , . . . , ˜ ˜ y T ) . By closure of DP to post-processing, we have that P ( OnlineGame M , ̂ B ,T (0)) ≈ ( ε,δ ) P ( OnlineGame M , ̂ B ,T (1)) . Now note that P ( OnlineGame M , ̂ B ,T (0)) ≡ W ℓ -1 , and P ( OnlineGame M , ̂ B ,T (1)) ≡ W ℓ , and hence W ℓ -1 ≈ ( ε,δ ) W ℓ . Overall we have that

This shows that OnlineGame A , B ,T,g is ( gε, g · e εg · δ ) -differentially private, thereby completing the proof.

C Privacy Analysis of ChallengeAT

The privacy guarantees of ChallengeAT are captured using the game specified in algorithm ChallengeAT-Game B .

Theorem C.1. For every adversary B it holds that ChallengeAT-Game B is ( O ( ε ) , O ( δ )) -DP w.r.t. the bit b (the input of the game).

Algorithm 9 ChallengeAT-Game B

Setting: B is an adversary that adaptively determines the inputs to ChallengeAT .

Input of the game: A bit b ∈ { 0 , 1 } . (The bit b is unknown to ChallengeAT and B .)

The adversary B specifies two neighboring datasets S 0 , S 1 ∈ X ∗ .

Instantiate ChallengeAT with the dataset S b and parameters ε, δ , threshold t , and number of positive reports r .

For i = 1 , 2 , 3 , . . .

(a) Get bit c i ∈ { 0 , 1 } from B subject to the restriction that ∑ i j =1 c j ≤ 1 . %When c i = 1 this is the Challange round .

(b) Get two queries f 0 i : X ∗ → R and f 1 i : X ∗ → R from B , each with sensitivity ∆ , subject to the restriction that if c i = 0 then f 0 i ≡ f 1 i .

(c) Give the query f b i to ChallengeAT and get back the bit σ i .

(d) If c i = 0 then set ˆ y i = σ i . Otherwise set ˆ y t = ⊥ .

(e) Give ˆ y i to the adversary B .

Publish B 's view of the game, that is ˆ y 1 , . . . , ˆ y T and the internal randomness of B .

Proof. Fix an adversary B . Let CATG denote the algorithm ChallengeAT-Game B with this fixed B . Consider a variant of algorithm CATG , which we call CATG -noCount defined as follows. During the challenge round i , inside the call to ChallengeAT , instead of feeding σ i to the PrivateCounter we simply feed it 0 (in Step 3d of ChallengeAT ).

By the privacy properties of PrivateCounter (Theorem 2.10), for every b ∈ { 0 , 1 } we have that

so it suffices to show that CATG -noCount is DP (w.r.t. b ). Now observe that the execution of PrivateCounter during the execution of CATG -noCount can be simulated from the view of the adversary B (the only bit that ChallengeAT feeds the counter which is not in the view of the adversary is the one of the challange round which we replaced by zero in CATG -noCount ). Hence, we can generate the view of B in algorithm CATG by interacting with AboveThreshold instead of with ChallengeAT . This is captured by algorithm CAT-G -AboveThrehold .

Algorithm 10 CATGAboveThreshold

Setting: B is an adversary that adaptively determines the inputs to ChallengeAT . Input of the game: A bit b ∈ { 0 , 1 } . (The bit b is unknown to ChallengeAT and B

The adversary B specifies two neighboring datasets S 0 , S 1 ∈ X ∗ .

Instantiate PrivateCounter

Instantiate AboveThreshold on the dataset S b with parameters ε, δ, t, ( r + λ ) .

For i = 1 , 2 , 3 , . . .

(a) Get bit c i ∈ { 0 , 1 } from the adversary B subject to the restriction that ∑ i j =1 c j ≤ 1 .

(b) Get two queries f 0 i : X ∗ → R and f 1 i : X ∗ → R , each with sensitivity ∆ from B , subject to the restriction that if c i = 0 then f 0 i ≡ f 1 i .

(c) Give the query f b i to Algorithm AboveThreshold and get back a bit σ i .

(d) If c i = 0 then set ˆ y i = σ i . Otherwise set ˆ y t = ⊥ .

(e) Give ˆ y i to the adversary B .

(f) If c i = 0 then feed σ i to PrivateCounter , and otherwise feed it 0 .

(g) Let count i denote the current output of PrivateCounter , and HALT if count i ≥ r

Publish B 's view of the game, that is ˆ y 1 , . . . , ˆ y T and the internal randomness of B .

This algorithm is almost identical to CATG -noCount , except for the fact that AboveThreshold might halt the execution itself (even without the halting condition on the outcome of PrivateCounter ). However, by the utility guarantees of PrivateCounter , with probability at least 1 -δ it never errs

.)

by more than λ , in which case algorithm AboveThreshold never halts prematurely. Hence, for every bit b ∈ { 0 , 1 } we have that

So it suffices to show that CATG -AboveThrehold is DP (w.r.t. its input bit b ). This almost follows directly from the privacy guarantees of AboveThreshold , since CATG -AboveThrehold interacts only with this algorithm, except for the fact that during the challenge round i the adversary B specifies two queries (and only one of them is fed into AboveThreshold ). To bridge this gap, we consider one more (and final) modification to the algorithm, called ̂ CATG -AboveThrehold . This algorithm is identical to CATG -AboveThrehold , except that in Step 4c we do not feed f b i to AboveThreshold if c i = 1 . That is, during the challenge round we do not interact with AboveThreshold .

Now, by the privacy properties of AboveThreshold we have that ̂ CATG -AboveThrehold is DP (w.r.t. its input bit b ). Furthermore, when algorithm AboveThreshold does not halt prematurely, we have that ̂ CATG -AboveThrehold is identical to CATG -AboveThrehold . Therefore, for every bit b ∈ { 0 , 1 } we have

Overall we get that

D Privacy Analysis of POP

In this section we prove Theorem 5.2.

Proof of Theorem 5.2. Let B be an adversary that playes in OnlineGame against POP , posing at most 1 challenge. That is, at one time step i , the adversary specifies two inputs ( x 0 i , y 0 i ) , ( x 1 i , y 1 i ) , algorithm POP processes ( x b i , y b i ) , and the adversary does not see the prediction ˆ y i at this time step. We need to show that the view of the adversary is DP w.r.t. the bit b . To show this, we observe that the view of B can be generated (up to a small statistical distance of δ ) by interacting with ChallengeAT as in the game ChallengeAT-Game . Formally, consider the following adversary ˆ B that simulates B while interacting with ChallengeAT instead of POP .

As ˆ B only interacts with ChallengeAT , its view at the end of the execution (which includes the view of the simulated B ) is DP w.r.t. the bit b . Furthermore, the view of the simulated B generated in this process is almost identical to the view of B had it interacted directly with POP . Specifically, the only possible difference is that the computation of ˆ y i in Step 3(e)ii of ˆ B might not be well-defined. But this does not happen when ChallengeAT maintains correctness, which holds with probability at least 1 -δ .

Overall, letting ChallengeAT-Game ˆ B | B denote the view of the simulated B at the end of the interaction of ˆ B with ChallengeAT , we have that

Algorithm 11 ˆ B

Setting:

This is an adversary that plays against ChallengeAT in the game ChallengeAT-Game .

Specify two datasets S 0 = { 0 } and S 1 = { 1 } .

Instantiate algorithm B

For i = 1 , 2 , . . . , T

(a) Obtain a challenge indicator c i and inputs x 0 i , x 1 i from B (where x 0 i = x 1 i if c i = 0 ).

(b) Let ℓ i ∈ [ k ] be chosen uniformly at random

(c) Define the query q i : { 0 , 1 } → R , where q i ( b ) = f i and where f i is defined as in Step 3e of POP .

%Note that, given b , this can be computed from ( x 0 1 , x 1 1 ) , . . . , ( x 0 i , x 1 i ) and ℓ 1 , . . . , ℓ i and y 1 , . . . , y i -1 . Furthermore, whenever c i = 0 then this is a query of sensitivity at most 1 . When c i = 1 the sensitivity might be large, which we view it as two separate queries, corresponding to a challenge round when playing against ChallengeAT .

(d) Output the challenge bit c i and the query q i , which is given to ChallengeAT .

(e) If c i = 0 then

Obtain an outcome σ i from ChallengeAT

Define ˆ y i as in Step 3f of POP , as a function of σ i and ( x 0 1 , x 1 1 ) , . . . , ( x 0 i , x 1 i ) and ℓ 1 , . . . , ℓ i and y 1 , . . . , y i -1 .

Feed the bit ˆ y i to the adversary B

(f) Obtain a 'true' label y i from the adversary B .

E A Coin Flipping Game

Consider algorithm 12 which specifies an m -round 'coin flipping game' against an adversary B . In this game, the adaptively chooses the biases of the coins we flip. In every flip, the adversary might gain a reward or incur a 'budget loss'. The adversary aims to maximize the rewards it collects before its budget runs out.

Algorithm 12 CoinGame B ,k,m

Setting: B is an adversary that determins the coin biases adaptively. k denotes the 'budget' of the adversary. m denotes the number of iterations.

Set budget = k and reward = 0 .

In each round i = 1 , 2 , . . . , m :

(a) The adversary chooses 0 ≤ p i ≤ 5 6 and p i 5 ≤ q i ≤ 1 -p i , possibly based on the first ( i -1) rounds.

(b) A random variable X i ∈ { 0 , 1 , 2 } is sampled, where Pr[ X i = 1] = p i and Pr[ X i = 2] = q i and Pr[ X i = 0] = 1 -p i -q i .

(c) The adversary obtains X i

(d) If X i = 1 and budget > 0 then reward = reward + 1 .

(e) Else if X i = 2 then budget = budget -1 .

Output reward .

The next theorem states that no adversary can obtain reward much larger than k in this game. Intuitively, this holds because in every time step i , the probability of X i = 2 is not much smaller than the probability that X i , then (w.h.p.) it is very unlikely that the number of rewards would be much larger than k .

Theorem E.1 ([Gupta et al., 2010, Kaplan et al., 2021]) . For every adversary's strategy, every k ≥ 0 , every m ∈ N , and every λ ∈ R , we have

F Extension to the Agnostic Case

In this section we extend the analysis of POP to the agnostic setting. We use the tilde-notation to hide logarithmic factors in T, 1 δ , 1 β , 1 ε .

Theorem F.1 ([Ben-David et al., 2009]) . For any hypothesis class H and scalar M ∗ ≥ 0 there exists an online learning algorithm such that for any sequence (( x 1 , y 1 ) , . . . , ( x T , y T )) satisfying min h ∈ H ∑ T i =1 | h ( x i ) -y i | ≤ M ∗ the predictions ˆ y 1 , . . . , ˆ y T given by the algorithm satisfy

Definition F.2. For parameters u < w , let POP [ u,w ] denote a variant of POP in which we halt the execution after the v th time in which we err, for some arbitrary value u ≤ v ≤ w . (Note that the execution might halt even before that, by the halting condition of POP itself.) This could be done while preserving privacy (for appropriate values of u < w ) by using the counter of Theorem 2.10 for privately counting the number of mistakes.

Lemma F.3. Let H be a hypothesis class with d = Ldim( H ) , and let A denote the non-private algorithm from Theorem F.1 with M ∗ = d ln( T ) . Denote k = ˜ Θ ( d 2 ε ) , r = u = Θ( kd ln( T )) , and w = 2 u . Consider executing POP [ u,w ] with A and with parameters k, r on an adaptively chosen sequence of inputs ( x 1 , y 1 ) , . . . , ( x i ∗ , y i ∗ ) , where i ∗ ≤ T denotes the time at which POP [ u,w ] halts. Then, with probability at least (1 -β ) it holds that

Proof sketch. Similarly to the proof of Theorem 5.3, we set k = ˜ Ω ( d 2 ε ) , and assume that if less than 1 5 the experts disagree with the other experts, then algorithm POP [ u,w ] returns the majority vote with probability 1.

Let 1 / 5 -Err denote the random variable that counts the number of time steps in which at least 1 / 5 th of the experts make an error. As in the proof of Theorem 5.3, 1 / 5 -Err upper bounds both the number of mistakes made by POP [ u,w ] , which we denote by OurError , as well as the number of times in which algorithm ChallengeAT returns an 'above threshold' answer, which we denote by NumTop . By Theorem 4.2, we know that (w.h.p.) NumTop ≥ r . Also let WorstExpert denote the largest number of mistakes made by a single expert.

Consider the time i ∗ at which POP [ u,w ] halts. If it halts because u ≤ v ≤ w mistakes have been made, then

Alternatively, if POP [ u,w ] halts after r 'above threshold' answer, then

At any case, when POP [ u,w ] halts it holds that at least one expert made at least Ω( d ln( T )) mistakes. Therefore, by Theorem F.1, we have that OPT i ∗ ≥ d ln( T ) .

Theorem F.4. Let H be a hypothesis class with Ldim( H ) = d . There exists an ( ε, δ ) -ChallengeDP online learning algorithm providing the following guarantee. When executed on an adaptively

chosen sequence of inputs ( x 1 , y 1 ) , . . . , ( x T , y T ) , then the algorithm makes at most ˜ O ( d · OPT ε 2 + d 2 ε 2 ) mistakes (w.h.p.), where

Proof sketch. This is obtained by repeatedly re-running POP [ u,w ] , with the parameter setting specified in Lemma F.3. We refer to the time span of every single execution of POP [ u,w ] as a phase .

By construction, in every phase, POP [ u,w ] makes at most w = ˜ Θ( kd ) mistakes. By Lemma F.3 every hypothesis in H makes at least d · ln( T ) mistakes in this phase. Therefore, there could be at most ˜ O ( max { 1 , OPT d }) phases, during which we incur a total of at most ˜ O ( d · OPT ε 2 + d 2 ε 2 ) mistakes.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel method to apply DP to the context of interactive ML.\n\nSTRENGTHS:\nThe method itself is an interesting proposal, well-supported with proofs and theory. This work also extensively describes prior advances in the field in great detail.\n\nWEAKNESSES:\nThe paper has numerous issues with the way it is written. Firstly, there should not be any citations in the abstract, so, please reword it. Secondly, and much more importantly, this paper lacks a proper conclusion. There is no way for the reader to get a clear overview of the implications of this work as well as its relevance to the broader ML community. Instead, the work ends with a large algorithm definition, which really either belongs to the methods section or to the appendix. While I understand the overall idea of what the work proposes, there are no clear contributions either. I strongly encourage authors to rework these, as otherwise it is incredibly difficult to identify the merits of their method.\n\nCertain sections really don’t belong where they are right now: 1.1 in my eyes has nothing to do with the introduction; example 2.6, while helps contextualise the method is really taking the space away from the rest of the work; many algorithms can be placed in the appendix etc. Point being that the work is very difficult to process because some crucial sections are missing, yet some sections which were included really do not add the value to the submission.\n\nFinally, it looks to me as this is a purely theoretical paper, there are no experiments to support what the authors proposed. Given that the submission is presented as application-driven and useful for ML practitioners, this is rather odd.\n\nQUESTIONS:\nAre there any practical results to showcase the advantages of the proposed method?\n\nLIMITATIONS:\nOverall, I do not see many reasons to accept this work in its current state: While the method looks promising and may have broad implications for the use of DP in interactive setting, the manuscript does not do the method any justice and really struggles to make this accessible to a wide range of ML researchers.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of privacy preserving interactive learning. In this problem, a recommendation algorithm improves its model by answering private queries performed by a set of parties in sequential rounds. The responses to users queries should adapt to the query made by each party. Therefore, in order to provide an accurate service, the algorithm's responses must be strongly sensitive to the users. \nThis imposes an important restriction on the privacy guarantees that can be provided using the classical notion of differential privacy (DP): classical DP requires that all responses to queries are equally privacy preserving. Therefore, if the algorithm satisfies this notion of privacy, its accuracy is limited. \n\nTo tackle this problem, the contribution proposes challenge DP: a relaxed notion of DP in which the mechanism is allowed to provide a sensitive output to the party that generated the query, while still providing a less sensitive view to other parties. Using challenge DP, the contribution proposes a construction that, only by having black-box access to a interactive learning algorithm, can construct\na privacy preserving algorithm. The construction improves the accuracy of previous work, reducing the number of errors from exponential to quadratic in d, where d is the number of mistakes made by the black-box algorithm. Furthermore, these improvements are achieved while assuming an adversary that participates in the learning process controlling multiple parties and that can adapt its input in one round of the protocol.\n\nSTRENGTHS:\nThe paper is very well written and key aspects of the contribution are presented clearly. The contribution is novel: the relaxation of classical DP and the POP construction are clever. The results are significant, as improvements with respect previous work are important while \nassumptions on the adversary remain realistic. The overall quality is good.\n\nWEAKNESSES:\nI only have fairly minor remarks with respect to the presentation of the contribution. While the introduction to the problem is very nicely achieved, the space dedicated to the main contributions is a bit short. I would have appreciated if it had been bigger and had provided more details. Also, there are multiple DP definitions (such as Definitions 2.1, 2.7 and 2.9) that overload the statement \"X satisfies (epsilon, delta)-DP\" and then understanding which algorithm satisfies which definition of DP becomes a bit confusing. Different terminology for each case could help in clarity.\n\nQUESTIONS:\n1- In the definitions of DP under adaptive queries (Def 2.7) and adaptive inputs (Def 2.9), it seems that the neighboring databases are\nalways defined with respect to the bit b. As I understand, this means that the database only changes on the query in which the adversary\nplans to be adaptive. I have the impression that this might not reflect some realistic scenarios in interactive ML. For example, consider the case in which the adversary controls all the parties except of one party: Alice. It would be nice that the output (i.e. the view of the adversary) is differentially private with respect to the participation of Alice or not (which, in my understanding, is independent of bit b). Do the aforementioned definitions contemplate this case? \n\n2- In Theorems 4.1 and 4.2, you show how the POP construction (Algorithm 7) performs when it assumes an adversary that can adapt in a single round. However, you define challenge DP to be compatible with many adaptive/challenge rounds (group privacy). How an increase in the number of challenge rounds allowed to the adversary (i.e. an increase of g) would impact in the privacy/accuracy trade-offs?\n\nLIMITATIONS:\nWhile theoretical improvements are significant, more experimentation is required to understand if the practical deployment of the construction is feasible.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies privacy in the setting of interactive machine learning processes.  Challenge DP is presented as a new relaxation of DP that is satisfies many of the desirable properties of DP and any non-private online prediction algorithm can be constructed into a Challenge DP online prediction algorithm.\n\nSTRENGTHS:\nThe paper tackles an important problem and provides a nice motivating example in the setting of a continually improving chatbot.  It is clear from the example that the interaction is adaptive.  Rather than requiring that the interaction with the chatbot be differentially private, they study a relaxation called joint differential privacy, allowing the transcript of responses to depend arbitrarily on each individual’s prompts, but the chatbot should not leak too much information about others’ conversations in each response.  There has been work for this problem in the setting of (traditional) DP, but this is the first work to study mistake bounds in the setting of Joint DP, or more specifically Challenge DP (interactive variant of JDP), which they define and prove basic properties of.\n\nWEAKNESSES:\nThe motivating example is with a chatbot, but the problem setting is for private online classification, which might not be a relevant problem for a chatbot, as it is not clear what the labels or mistakes of a chatbot would be.  Is there a motivating example that is more related to private online classification?\n\nThe only proof in the paper is proving group privacy of challenge DP.  Although group privacy is an important property of any privacy definition, I do not think it adds to the overall narrative of the paper and not surprising as challenge DP is still related to DP.  The main results of this paper are in Theorems 4.1 and 4.2, so the paper should highlight the analysis there, or at the least provide a proof sketch. From the introduction, it is not clear why the reader should care about a group privacy property.   Furthermore, designing challenge DP algorithms is not clear and its relation with creating an online game that is DP is not clear (as in Theorem 4.1).  \n\nAlthough the problem is well motivated, the full story still seems incomplete. In particular, is there an actual gap between what is achievable under joint DP and (traditional) DP?  Currently, there is an existing DP approach that achieves very large mistake bound and this paper achieves a much better mistake bound, but is there a lower bound result?  \n\nIs there just a typo in Algorithm 1 and 6 where noise is not added to the threshold?  The privacy analysis of AboveThreshold depends on noise being added to the threshold that is reused at each round that is not above the noisy threshold.  I am willing to increase my score if this is merely a typo but would like the authors to verify.\n\n\nMinor: \n- \\mathcal{M} denoted mistake bounds and then in Definition 2.5 \\mathcal{M} is a mechanism.\n\n- Line 116 “if the algorithm errs than at least”\n\n- What is the optimal JDP approach for example 2.6?  Seems like you need to know the median to know which ones are above or below.\n\n- Footnote 1 on page 2, “as =composition”.\n\n- Line 309 “That is, For every”\n\n[Update] I have increased my score based on the rebuttal.\n\nQUESTIONS:\nis there an actual gap between what is achievable under joint (challenge) DP and (traditional) DP? \n\nIs there just a typo in Algorithm 1 and 6 where noise is not added to the threshold?\n\nLIMITATIONS:\nThe problem setting is for private online classification, yet the relation to a chatbot example is not quite clear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new differential privacy definition with desirable online learning properties. In this new variant named Challenge Differential Privacy, the adversary can observe the output of a sequence of online queries, except a \"challenge query\". In this query, two possible pairs of inputs are picked by an adversary. Based on the transcript of the queries without this challenge step, the adversary needs to decide which data point has been used for the hidden query. The authors introduce an algorithm that satisfies Challenge-DP, in which an ensemble of expert models is used to decide on a majority answer, but only one of the experts is allowed to use the ground truth label of the given sample.\n\nSTRENGTHS:\n* The setting proposed by the authors is interesting. Indeed, the standard definition of Differential Privacy is not well-tailored for online learning.\n* The authors discuss and prove the fundamental properties of Differential Privacy for their proposed variant: post-processing, composition, and group privacy.\n* The authors propose a technique to achieve ChallengeDP, POP, for which they introduce privacy and utility guarantees.\n* The work is compositional; it does not introduce new learning algorithms, but it introduces a technique to convert an existing learning algorithm to a private one that satisfies ChallengeDP.\n\nWEAKNESSES:\n* The current work is heavily reliant on the appendix. Consider making the main statements self-contained (ChallengeDP, POP). \n* The paper ends with a theorem statement. There is no conclusion/future work/discussion, giving the feel of possibly unfinished work.\n* $\\overset{\\sim}{O}$ notation inside $O$ notation (theorem 4.4) makes it very hard for the reader to compare it to other work/really understand at a granular level the privacy guarantees. As other results have previously shown, constants/linear terms matter a lot in differential privacy; please consider adding explicit guarantees (maybe in the appendix); otherwise, it is hard to understand the benefits of using this approach.\n* I would strongly suggest the authors state more clearly that in this online setting, the training algorithm that sees a sample uses it as a training point afterwards (possibly a more explicit protocol like the one in Naor et al. [0], figure 1 could be used).\n* A clearer discussion about the difference between private everlasting predictions from Naor et al. [0] and challenge differential privacy could be useful, and why it is a generalization of their work (as stated in the abstract). I acknowledge that the authors provided remark 3.2, but this could be more in-detail justified and discussed, as it is one of the paper's key contributions.\n\n[0]: https://arxiv.org/abs/2305.09579\n\nQUESTIONS:\n* The authors provided concise and clear proof for composition in the interactive setting, but I assume that this setting satisfies even the generalization of composition, namely concurrent composition[0], which is even more suitable for the described problem proposed by the authors. Did the authors consider proving/looking into concurrent composition, as it seems that is a definition well suited for this setting\n* In this setting, as far as I understand, the release of the learning algorithms results in a total loss of privacy, is that true, or could it be bounded/there might be a possibility of making a model release in this setting (considering that the initial learning algorithms, before the stream of queries, is not private).\n* Do the authors believe that this definition of privacy could replace $(\\epsilon, \\delta)$-DP for machine learning tasks, or should it be an alternative?\n* Is there an equivalence/relationship that the authors observed between $(\\epsilon, \\delta)$-DP and ChallengeDP?\n\n[0]: https://arxiv.org/abs/2207.08335\n\nLIMITATIONS:\n* Their proposed technique involves maintaining multiple copies of the same model, which is a possibly prohibitive approach when solving the proposed problem in the introduction, given the significant size of expert models.\n* The paper ends with a theorem statement. There is no conclusion, further work or discussion. I am willing to increase the score of the paper if the presentation at the end is improved.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}