SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Naman Agarwal and Karan Singh. The price of differential privacy for online learning. In ICML, volume 70 of Proceedings of Machine Learning Research, pages 32-40. PMLR, 06-11 Aug 2017.
Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private online prediction from experts: Separations and faster rates. CoRR, abs/2210.13537, 2022.
Shai Ben-David, David Pál, and Shai Shalev-Shwartz. Agnostic online learning. In *COLT*, 2009.
Mark Bun, Thomas Steinke, and Jonathan Ullman. Make up your mind: The price of online queries in differential privacy. In Proceedings of the twenty-eighth annual ACM-SIAM symposium on discrete algorithms, pages 1306-1325. SIAM, 2017.
T.-H. Hubert Chan, Elaine Shi, and Dawn Song. Private and continual release of statistics. In ICALP (2), volume 6199 of Lecture Notes in Computer Science, pages 405-417. Springer, 2010.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In TCC, pages 265-284, 2006.
Cynthia Dwork, Moni Naor, Omer Reingold, Guy N. Rothblum, and Salil P. Vadhan. On the complexity of differentially private data release: efficient algorithms and hardness results. In STOC, pages 381-390, 2009.
Cynthia Dwork, Moni Naor, Toniann Pitassi, and Guy N. Rothblum. Differential privacy under continual observation. In Symposium on Theory of Computing (STOC), pages 715-724. ACM, 2010a.
Cynthia Dwork, Guy N. Rothblum, and Salil P. Vadhan. Boosting and differential privacy. In FOCS, pages 51-60, 2010b.
Noah Golowich and Roi Livni. Littlestone classes are privately online learnable. In NeurIPS, pages 11462-11473, 2021.
Anupam Gupta, Katrina Ligett, Frank McSherry, Aaron Roth, and Kunal Talwar. Differentially private combinatorial optimization. In SODA, pages 1106-1125, 2010.
Moritz Hardt and Guy N. Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In FOCS, pages 61-70, 2010.
Palak Jain, Sofya Raskhodnikova, Satchit Sivakumar, and Adam D. Smith. The price of differential privacy under continual observation. CoRR, abs/2112.00828, 2021.
Prateek Jain and Abhradeep Guha Thakurta. (near) dimension independent risk bounds for differentially private learning. In ICML, volume 32 of JMLR Workshop and Conference Proceedings, pages 476-484. JMLR.org, 2014.
Prateek Jain, Pravesh Kothari, and Abhradeep Thakurta. Differentially private online learning. In  $COLT$ , volume 23 of JMLR Proceedings, pages 24.1-24.34. JMLR.org, 2012.
Haim Kaplan, Yishay Mansour, and Uri Stemmer. The sparse vector technique, revisited. In  $COLT$ , volume 134 of Proceedings of Machine Learning Research, pages 2747-2776. PMLR, 2021.
Michael J. Kearns, Mallesh M. Pai, Ryan M. Rogers, Aaron Roth, and Jonathan R. Ullman. Robust mediators in large games. CoRR, abs/1512.02698, 2015.
Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. In Machine Learning, pages 285-318, 1988.
Yifat Nahmias, Oren Perez, Yotam Shlomo, and Uri Stemmer. Privacy preserving social norm nudges. Mich. Tech. L. Rev., 26:43, 2019.
Moni Naor, Kobbi Nissim, Uri Stemmer, and Chao Yan. Private everlasting prediction. CoRR, abs/2305.09579, 2023.
Roshan Shariff and Or Sheffet. Differentially private contextual linear bandits. In NeurIPS, pages 4301-4311, 2018.
Abhradeep Guha Thakurta and Adam D. Smith. (nearly) optimal algorithms for private online learning in full-information and bandit settings. In NIPS, pages 2733-2741, 2013.
A General Variant of challenge-DP
Definition A.1. Consider an algorithm  $\mathcal{M}$  that, in each phase  $i\in [T]$ , conducts an arbitrary interaction with the  $i$ th user. We say that algorithm  $\mathcal{M}$  is  $(\varepsilon ,\delta)$ -challenge differentially private if for any adversary  $\mathcal{B}$  we have that GeneralGame $_{\mathcal{M},\mathcal{B},T}$ , defined below, is  $(\varepsilon ,\delta)$ -differentially private (w.r.t. its input bit  $b$ ).
Algorithm 8 GeneralGame $_{\mathcal{M}, \mathcal{B}, T}$
Setting:  $T \in \mathbb{N}$  denotes the number of phases.  $\mathcal{M}$  is an interactive algorithm and  $\mathcal{B}$  is an adaptive and interactive adversary.
(a) The adversary  $\mathcal{B}$  outputs a bit  $c_{i} \in \{0,1\}$ , under the restriction that  $\sum_{j=1}^{i} c_{j} \leq 1$ .
(b) The adversary  $\mathcal{B}$  chooses two interactive algorithms  $\mathcal{I}_{i,0}$  and  $\mathcal{I}_{i,1}$ , under the restriction that if  $c_{i} = 0$  then  $\mathcal{I}_{i,0} = \mathcal{I}_{i,1}$ .
(c) Algorithm  $\mathcal{M}$  interacts with  $\mathcal{I}_{i,b}$ . Let  $\hat{y}_i$  denote the view of  $\mathcal{I}_{i,b}$  at the end of this interaction.
(d) If  $c_{i} = 0$  then set  $\tilde{y}_i = \hat{y}_i$ . Otherwise set  $\tilde{y}_i = \bot$ .
(e) The adversary  $\mathcal{B}$  obtains  $\tilde{y}_i$ .
2. Output  $\mathcal{B}$ 's view of the game.
B Group privacy
In this section we prove Theorem 3.4.
Proof of Theorem 3.4. Fix  $g \in \mathbb{N}$  and fix an adversary  $\mathcal{B}$  (that poses at most  $g$  challenge rounds). We consider a sequence of games  $\mathcal{W}_0, \mathcal{W}_1, \ldots, \mathcal{W}_g$ , where  $\mathcal{W}_{\ell}$  is defined as follows.
1. Initialize algorithm  $\mathcal{M}$  and the adversary  $\mathcal{B}$ .
2. For round  $i = 1,2,\ldots ,T$  :
(a) Obtain a challenge indicator  $c_{i}$  and two labeled inputs  $(x_{i,0},y_{i,0})$  and  $(x_{i,1},y_{i,1})$  from  $\mathcal{B}$ .
(b) If  $\sum_{j=1}^{i} c_{j} > \ell$  then set  $(w_{i}, z_{i}) = (x_{i,0}, y_{i,0})$ . Otherwise set  $(w_{i}, z_{i}) = (x_{i,1}, y_{i,1})$ .
(c) Feed  $w_{i}$  to algorithm  $\mathcal{M}$ , obtain an outcome  $\hat{y}_i$ , and feed it  $z_{i}$ .
(d) If  $c_{i} = 0$  then set  $\tilde{y}_{i} = \hat{y}_{i}$ . Otherwise set  $\tilde{y}_{i} = \bot$ .
(e) Give  $\tilde{y}_i$  to  $\mathcal{B}$ .
3. Output  $\tilde{y}_1, \ldots, \tilde{y}_T$  and the internal randomness of  $\mathcal{B}$ .
That is,  $\mathcal{W}_{\ell}$  simulates the online game between  $\mathcal{M}$  and  $\mathcal{B}$ , where during the first  $\ell$  challenge rounds algorithm  $\mathcal{M}$  is given  $(x_{i,1},y_{i,1})$ , and in the rest of the challenge rounds algorithm  $\mathcal{M}$  is given  $(x_{i,0},y_{i,0})$ . Note that
$$
\text {O n l i n e G a m e} _ {\mathcal {M}, \mathcal {B}, T, g} (0) \equiv \mathcal {W} _ {0} \quad \text {a n d} \quad \text {O n l i n e G a m e} _ {\mathcal {M}, \mathcal {B}, T, g} (1) \equiv \mathcal {W} _ {g}.
$$
We claim that for every  $0 < \ell \leq g$  it holds that  $\mathcal{W}_{\ell -1} \approx_{(\varepsilon, \delta)} \mathcal{W}_{\ell}$ . To this end, fix  $0 < \ell \leq g$  and consider an adversary  $\widehat{\mathcal{B}}$ , that poses at most one challenge, defined as follows. Algorithm  $\widehat{\mathcal{B}}$  runs  $\mathcal{B}$  internally. In every round  $i$ , algorithm  $\widehat{\mathcal{B}}$  obtains from  $\mathcal{B}$  a challenge bit  $c_i$  and two labeled inputs  $(x_{i,0}, y_{i,0})$  and  $(x_{i,1}, y_{i,1})$ . As long as  $\mathcal{B}$  did not pose its  $\ell$ th challenge, algorithm  $\widehat{\mathcal{B}}$  outputs  $(x_{i,1}, y_{i,1})$ ,  $(x_{i,1}, y_{i,1})$ . During the round  $i$  in which  $\mathcal{B}$  poses its  $\ell$ th challenge, algorithm  $\mathcal{B}$  outputs  $(x_{i,0}, y_{i,0})$ ,  $(x_{i,1}, y_{i,1})$ . This is the challenge round posed by algorithm  $\widehat{\mathcal{B}}$ . In every round  $t$  afterwards, algorithm  $\widehat{\mathcal{B}}$  outputs  $(x_{i,0}, y_{i,0})$ ,  $(x_{i,0}, y_{i,0})$ . When algorithm  $\widehat{\mathcal{B}}$  obtains an answer  $\tilde{y}_i$  it sets  $\tilde{\tilde{y}}_i = \left\{ \begin{array}{ll} \tilde{y}_i, & \text{if } c_i = 0 \\ \perp, & \text{if } c_i = 1 \end{array} \right.$  and gives  $\tilde{y}_i$  to algorithm  $\mathcal{B}$ .
As  $\widehat{B}$  is an adversary that poses (at most) one challenge, by the privacy properties of  $\mathcal{M}$  we know that OnlineGame $_{\mathcal{M},\widehat{\mathcal{B}},T}$  is  $(\varepsilon, \delta)$ -DP. Recall that the output of OnlineGame $_{\mathcal{M},\widehat{\mathcal{B}},T}$  includes all of the randomness of  $\widehat{B}$ , as well as the answers  $\tilde{y}_t$  generated throughout the game. This includes the randomness of  $\mathcal{B}$  (which  $\widehat{B}$  runs internally), and hence, determines also all of the  $\tilde{y}_i$ 's defined by  $\widehat{B}$  throughout the interaction. Let  $P$  be a post-processing procedure that takes the output of OnlineGame $_{\mathcal{M},\widehat{\mathcal{B}},T}$  and returns the randomness of  $\mathcal{B}$  as well as  $(\tilde{y}_1, \ldots, \tilde{y}_T)$ . By closure of DP to post-processing, we have that  $P(\text{OnlineGame}_{\mathcal{M},\widehat{\mathcal{B}},T}(0)) \approx_{(\varepsilon, \delta)} P(\text{OnlineGame}_{\mathcal{M},\widehat{\mathcal{B}},T}(1))$ . Now note that  $P(\text{OnlineGame}_{\mathcal{M},\widehat{\mathcal{B}},T}(0)) \equiv \mathcal{W}_{\ell-1}$ , and  $P(\text{OnlineGame}_{\mathcal{M},\widehat{\mathcal{B}},T}(1)) \equiv \mathcal{W}_{\ell}$ , and hence  $\mathcal{W}_{\ell-1} \approx_{(\varepsilon, \delta)} \mathcal{W}_{\ell}$ . Overall we have that
$$
\mathsf {O n l i n e G a m e} _ {\mathcal {A}, \mathcal {B}, T, g} (0) \equiv \mathcal {W} _ {0} \approx_ {(\varepsilon , \delta)} \mathcal {W} _ {1} \approx_ {(\varepsilon , \delta)} \mathcal {W} _ {2} \approx_ {(\varepsilon , \delta)} \dots \approx_ {(\varepsilon , \delta)} \mathcal {W} _ {g} \equiv \mathsf {O n l i n e G a m e} _ {\mathcal {A}, \mathcal {B}, T, g} (1).
$$
This shows that OnlineGame $_{A,B,T,g}$  is  $(g\varepsilon, g \cdot e^{\varepsilon g} \cdot \delta)$ -differentially private, thereby completing the proof.
C Privacy Analysis of ChallengeAT
The privacy guarantees of ChallengeAT are captured using the game specified in algorithm ChallengeAT-Game $_B$ .
Theorem C.1. For every adversary  $\mathcal{B}$  it holds that ChalengeAT-Game is  $(O(\varepsilon), O(\delta))$ -DP w.r.t. the bit  $b$  (the input of the game).
Algorithm 9 ChallengeAT-Game $_{3}$
Setting:  $\mathcal{B}$  is an adversary that adaptively determines the inputs to ChallengeAT.
Input of the game: A bit  $b \in \{0,1\}$ . (The bit  $b$  is unknown to ChallengeAT and  $\mathcal{B}$ .)
1. The adversary  $\mathcal{B}$  specifies two neighboring datasets  $S_0, S_1 \in X^*$ .
2. Instantiate ChallengeAT with the dataset  $S_{b}$  and parameters  $\varepsilon, \delta$ , threshold  $t$ , and number of positive reports  $r$ .
3. For  $i = 1,2,3,\ldots$
(a) Get bit  $c_i \in \{0,1\}$  from  $\mathcal{B}$  subject to the restriction that  $\sum_{j=1}^{i} c_j \leq 1$ . % When  $c_i = 1$  this is the Challenge round.
(b) Get two queries  $f_{i}^{0}: X^{*} \to \mathbb{R}$  and  $f_{i}^{1}: X^{*} \to \mathbb{R}$  from  $\mathcal{B}$ , each with sensitivity  $\Delta$ , subject to the restriction that if  $c_{i} = 0$  then  $f_{i}^{0} \equiv f_{i}^{1}$ .
(c) Give the query  $f_{i}^{b}$  to ChallengeAT and get back the bit  $\sigma_{i}$ .
(d) If  $c_{i} = 0$  then set  $\hat{y}_i = \sigma_i$ . Otherwise set  $\hat{y}_t = \bot$ .
(e) Give  $\hat{y}_i$  to the adversary  $\mathcal{B}$ .
4. Publish  $\mathcal{B}$ 's view of the game, that is  $\hat{y}_1, \dots, \hat{y}_T$  and the internal randomness of  $\mathcal{B}$ .
Proof. Fix an adversary  $\mathcal{B}$ . Let CATG denote the algorithm ChallengeAT-Game $_{\mathcal{B}}$  with this fixed  $\mathcal{B}$ . Consider a variant of algorithm CATG, which we call CATG-noCount defined as follows. During the challenge round  $i$ , inside the call to ChallengeAT, instead of feeding  $\sigma_i$  to the PrivateCounter we simply feed it 0 (in Step 3d of ChallengeAT).
By the privacy properties of PrivateCounter (Theorem 2.10), for every  $b \in \{0,1\}$  we have that
$$
\operatorname {C A T G} (b) \approx_ {(\varepsilon , 0)} \operatorname {C A T G - n o C o u n t} (b),
$$
so it suffices to show that CATG-noCount is DP (w.r.t.  $b$ ). Now observe that the execution of PrivateCounter during the execution of CATG-noCount can be simulated from the view of the adversary  $\mathcal{B}$  (the only bit that ChallengeAT feeds the counter which is not in the view of the adversary is the one of the challenge round which we replaced by zero in CATG-noCount). Hence, we can generate the view of  $\mathcal{B}$  in algorithm CATG by interacting with AboveThreshold instead of with ChallengeAT. This is captured by algorithm CAT-G-AboveThreshold.
Algorithm 10 CATG-AboveThreshold
1. The adversary  $\mathcal{B}$  specifies two neighboring datasets  $S_0, S_1 \in X^*$ .
2. Instantiate PrivateCounter
3. Instantiate AboveThreshold on the dataset  $S_{b}$  with parameters  $\varepsilon, \delta, t, (r + \lambda)$ .
4. For  $i = 1,2,3,\ldots$
(a) Get bit  $c_{i}\in \{0,1\}$  from the adversary  $\mathcal{B}$  subject to the restriction that  $\sum_{j = 1}^{i}c_{j}\leq 1$
(b) Get two queries  $f_{i}^{0}: X^{*} \to \mathbb{R}$  and  $f_{i}^{1}: X^{*} \to \mathbb{R}$ , each with sensitivity  $\Delta$  from  $\mathcal{B}$ , subject to the restriction that if  $c_{i} = 0$  then  $f_{i}^{0} \equiv f_{i}^{1}$ .
(c) Give the query  $f_{i}^{b}$  to Algorithm AboveThreshold and get back a bit  $\sigma_{i}$ .
(d) If  $c_{i} = 0$  then set  $\hat{y}_i = \sigma_i$ . Otherwise set  $\hat{y}_t = \bot$ .
(e) Give  $\hat{y}_i$  to the adversary  $\mathcal{B}$ .
(f) If  $c_{i} = 0$  then feed  $\sigma_{i}$  to PrivateCounter, and otherwise feed it 0.
(g) Let  $\mathrm{count}_i$  denote the current output of PrivateCounter, and HALT if  $\mathrm{count}_i \geq r$
5. Publish  $\mathcal{B}$ 's view of the game, that is  $\hat{y}_1, \dots, \hat{y}_T$  and the internal randomness of  $\mathcal{B}$ .
This algorithm is almost identical to CATG-noCount, except for the fact that AboveThreshold might halt the execution itself (even without the halting condition on the outcome of PrivateCounter). However, by the utility guarantees of PrivateCounter, with probability at least  $1 - \delta$  it never errs
by more than  $\lambda$ , in which case algorithm AboveThreshold never halts prematurely. Hence, for every bit  $b \in \{0,1\}$  we have that
$$
\operatorname {C A T G - A b o v e T h r e h o l d} (b) \approx_ {(0, \delta)} \operatorname {C A T G - n o C o u n t} (b).
$$
So it suffices to show that CATG-AboveThreshold is DP (w.r.t. its input bit  $b$ ). This almost follows directly from the privacy guarantees of AboveThreshold, since CATG-AboveThreshold interacts only with this algorithm, except for the fact that during the challenge round  $i$  the adversary  $B$  specifies two queries (and only one of them is fed into AboveThreshold). To bridge this gap, we consider one more (and final) modification to the algorithm, called CATG-AboveThreshold. This algorithm is identical to CATG-AboveThreshold, except that in Step 4c we do not feed  $f_{i}^{b}$  to AboveThreshold if  $c_{i} = 1$ . That is, during the challenge round we do not interact with AboveThreshold.
Now, by the privacy properties of AboveThreshold we have that  $\widehat{\mathsf{CATG - AboveThreshold}}$  is DP (w.r.t. its input bit  $b$ ). Furthermore, when algorithm AboveThreshold does not halt prematurely, we have that  $\widehat{\mathsf{CATG - AboveThreshold}}$  is identical to CATG-AboveThreshold. Therefore, for every bit  $b \in \{0,1\}$  we have
$$
\operatorname {C A T G - A b o v e T h r e h o l d} (b) \approx_ {(0, \delta)} \widehat {\operatorname {C A T G - A b o v e T h r e h o l d}} (b).
$$
Overall we get that
$$
\begin{array}{l} \operatorname {C A T G} (0) \approx_ {(\varepsilon , 0)} \operatorname {C A T G - n o C u n t} (0) \\ \approx_ {(0, \delta)} \text {C A T G - A b o v e T h r e h o l d} (0) \\ \approx_ {(0, \delta)} \widehat {\mathbf {C A T G}} - \text {A b o v e T h r e h o l d} (0) \\ \approx_ {(\varepsilon , \delta)} \widehat {\mathbf {C A T G}} - \text {A b o v e T h r e h o l d} (1) \\ \approx_ {(0, \delta)} \text {C A T G - A b o v e T h r e h o l d} (1) \\ \approx_ {(0, \delta)} \operatorname {C A T G - n o C o u n t} (1) \\ \approx_ {(\varepsilon , 0)} \text {C A T G} (1) \\ \end{array}
$$
D Privacy Analysis of POP
In this section we prove Theorem 5.2.
Proof of Theorem 5.2. Let  $\mathcal{B}$  be an adversary that plays in OnlineGame against POP, posing at most 1 challenge. That is, at one time step  $i$ , the adversary specifies two inputs  $(x_i^0, y_i^0)$ ,  $(x_i^1, y_i^1)$ , algorithm POP processes  $(x_i^b, y_i^b)$ , and the adversary does not see the prediction  $\hat{y}_i$  at this time step. We need to show that the view of the adversary is DP w.r.t. the bit  $b$ . To show this, we observe that the view of  $\mathcal{B}$  can be generated (up to a small statistical distance of  $\delta$ ) by interacting with ChallengeAT as in the game ChallengeAT-Game. Formally, consider the following adversary  $\hat{\mathcal{B}}$  that simulates  $\mathcal{B}$  while interacting with ChallengeAT instead of POP.
As  $\hat{\mathcal{B}}$  only interacts with ChallengeAT, its view at the end of the execution (which includes the view of the simulated  $\mathcal{B}$ ) is DP w.r.t. the bit  $b$ . Furthermore, the view of the simulated  $\mathcal{B}$  generated in this process is almost identical to the view of  $\mathcal{B}$  had it interacted directly with POP. Specifically, the only possible difference is that the computation of  $\hat{y}_i$  in Step 3(e)ii of  $\hat{\mathcal{B}}$  might not be well-defined. But this does not happen when ChallengeAT maintains correctness, which holds with probability at least  $1 - \delta$ .
Overall, letting ChallengeAT-Game  $\mathcal{B}_{\hat{\mathcal{B}}_{|\mathcal{B}}}$  denote the view of the simulated  $\mathcal{B}$  at the end of the interaction of  $\hat{\mathcal{B}}$  with ChallengeAT, we have that
$$
\begin{array}{l} \text {O n l i n e G a m e} _ {\mathcal {P O P}, \mathcal {B}} (0) \approx_ {(0, \delta)} \text {C h a l l e N E A T - G a m e} _ {\left. \hat {\mathcal {B}} \right| _ {\mathcal {B}}} (0) \\ \approx_ {(\varepsilon , \delta)} \text {C h a l l e n g e A T - G a m e} _ {\hat {\mathcal {B}} | _ {\mathcal {B}}} (1) \\ \approx_ {(0, \delta)} \text {O n l i n e G a m e} _ {\mathrm {P O P}, B} (1). \\ \end{array}
$$
Algorithm 11  $\hat{\mathcal{B}}$
Setting: This is an adversary that plays against ChallengeAT in the game ChallengeAT-Game.
1. Specify two datasets  $S_0 = \{0\}$  and  $S_1 = \{1\}$ .
2. Instantiate algorithm  $\mathcal{B}$
3. For  $i = 1,2,\ldots ,T$
(a) Obtain a challenge indicator  $c_{i}$  and inputs  $x_{i}^{0}, x_{i}^{1}$  from  $\mathcal{B}$  (where  $x_{i}^{0} = x_{i}^{1}$  if  $c_{i} = 0$ ).
(b) Let  $\ell_i\in [k]$  be chosen uniformly at random
(c) Define the query  $q_{i}:\{0,1\} \to \mathbb{R}$ , where  $q_{i}(b) = f_{i}$  and where  $f_{i}$  is defined as in Step 3e of POP.
$\%$  Note that, given  $b$ , this can be computed from  $(x_1^0,x_1^1),\ldots ,(x_i^0,x_i^1)$  and  $\ell_1,\dots ,\ell_i$  and  $y_{1},\ldots ,y_{i - 1}$ . Furthermore, whenever  $c_{i} = 0$  then this is a query of sensitivity at most 1. When  $c_{i} = 1$  the sensitivity might be large, which we view it as two separate queries, corresponding to a challenge round when playing against ChallengeAT.
(d) Output the challenge bit  $c_{i}$  and the query  $q_{i}$ , which is given to ChallengeAT.
(e) If  $c_{i} = 0$  then
i. Obtain an outcome  $\sigma_{i}$  from ChallengeAT
ii. Define  $\hat{y}_i$  as in Step 3f of POP, as a function of  $\sigma_i$  and  $(x_1^0,x_1^1),\ldots ,(x_i^0,x_i^1)$  and  $\ell_1,\dots ,\ell_i$  and  $y_{1},\ldots ,y_{i - 1}$
iii. Feed the bit  $\hat{y}_i$  to the adversary  $\mathcal{B}$
(f) Obtain a "true" label  $y_{i}$  from the adversary  $\mathcal{B}$ .
E A Coin Flipping Game
Consider algorithm 12 which specifies an  $m$ -round "coin flipping game" against an adversary  $\mathcal{B}$ . In this game, the adaptively chooses the biases of the coins we flip. In every flip, the adversary might gain a reward or incur a "budget loss". The adversary aims to maximize the rewards it collects before its budget runs out.
Algorithm 12 CoinGame $_{B, k, m}$
Setting:  $\mathcal{B}$  is an adversary that determines the coin biases adaptively.  $k$  denotes the "budget" of the adversary.  $m$  denotes the number of iterations.
1. Set budget  $= k$  and reward  $= 0$ .
2. In each round  $i = 1, 2, \ldots, m$ :
(a) The adversary chooses  $0 \leq p_i \leq \frac{5}{6}$  and  $\frac{p_i}{5} \leq q_i \leq 1 - p_i$ , possibly based on the first  $(i - 1)$  rounds.
(b) A random variable  $X_{i} \in \{0,1,2\}$  is sampled, where  $\operatorname*{Pr}[X_i = 1] = p_i$  and  $\operatorname*{Pr}[X_i = 2] = q_i$  and  $\operatorname*{Pr}[X_i = 0] = 1 - p_i - q_i$ .
(c) The adversary obtains  $X_{i}$
(d) If  $X_{i} = 1$  and budget  $> 0$  then reward  $= \text{reward} + 1$ .
(e) Else if  $X_{i} = 2$  then budget  $=$  budget  $-1$ .
3. Output reward.
The next theorem states that no adversary can obtain reward much larger than  $k$  in this game. Intuitively, this holds because in every time step  $i$ , the probability of  $X_{i} = 2$  is not much smaller than the probability that  $X_{i}$ , then (w.h.p.) it is very unlikely that the number of rewards would be much larger than  $k$ .
Theorem E.1 ([Gupta et al., 2010, Kaplan et al., 2021]). For every adversary's strategy, every  $k \geq 0$ , every  $m \in \mathbb{N}$ , and every  $\lambda \in \mathbb{R}$ , we have
$$
\Pr \left[ \text {C o i n G a m e} _ {\mathcal {B}, k, m} > \lambda \right] \leq \exp \left(- \frac {\lambda}{6} + 3 (k + 1)\right).
$$
F Extension to the Agnostic Case
In this section we extend the analysis of P0P to the agnostic setting. We use the tilde-notation to hide logarithmic factors in  $T$ ,  $\frac{1}{\delta}$ ,  $\frac{1}{\beta}$ ,  $\frac{1}{\varepsilon}$ .
Theorem F.1 ([Ben-David et al., 2009]). For any hypothesis class  $H$  and scalar  $M^{*} \geq 0$  there exists an online learning algorithm such that for any sequence  $((x_{1},y_{1}),\ldots ,(x_{T},y_{T}))$  satisfying  $\min_{h\in H}\sum_{i = 1}^{T}|h(x_i) - y_i|\leq M^*$  the predictions  $\hat{y}_1,\dots ,\hat{y}_T$  given by the algorithm satisfy
$$
\sum_ {i = 1} ^ {T} \left| \hat {y} _ {i} - y _ {i} \right| \leq O \left(M ^ {*} + \operatorname {L d i m} (H) \ln (T)\right).
$$
Definition F.2. For parameters  $u < w$ , let  $POP_{[u,w]}$  denote a variant of  $POP$  in which we halt the execution after the  $v$ th time in which we err, for some arbitrary value  $u \leq v \leq w$ . (Note that the execution might halt even before that, by the halting condition of  $POP$  itself.) This could be done while preserving privacy (for appropriate values of  $u < w$ ) by using the counter of Theorem 2.10 for privately counting the number of mistakes.
Lemma F.3. Let  $H$  be a hypothesis class with  $d = \mathrm{Ldim}(H)$ , and let  $\mathcal{A}$  denote the non-private algorithm from Theorem F.1 with  $M^{*} = d\ln (T)$ . Denote  $k = \tilde{\Theta}\left(\frac{d^2}{\varepsilon}\right)$ ,  $r = u = \Theta$  ( $kd\ln (T)$ ), and  $w = 2u$ . Consider executing  $POP_{[u,w]}$  with  $\mathcal{A}$  and with parameters  $k, r$  on an adaptively chosen sequence of inputs  $(x_1,y_1),\ldots,(x_{i^*},y_{i^*})$ , where  $i^* \leq T$  denotes the time at which  $POP_{[u,w]}$  halts. Then, with probability at least  $(1 - \beta)$  it holds that
$$
\mathrm {O P T} _ {i ^ {*}} \triangleq \min  _ {h \in H} \sum_ {i = 1} ^ {i ^ {*}} | h (x _ {i}) - y _ {i} | > d \cdot \ln (T).
$$
Proof sketch. Similarly to the proof of Theorem 5.3, we set  $k = \tilde{\Omega}\left(\frac{d^2}{\varepsilon}\right)$ , and assume that if less than  $\frac{1}{5}$  the experts disagree with the other experts, then algorithm  $\mathsf{POP}_{[u,w]}$  returns the majority vote with probability 1.