Title: Federated Online Prediction from Experts with Differential Privacy: Separations and Regret Speed-ups
Venue: NeurIPS 2024
Accepted: yes
Keywords: online learning

federated learning

differential privacy
Abstract: We study the problems of differentially private federated online prediction from experts against both *stochastic adversaries* and *oblivious adversaries*. We aim to minimize the average regret on $m$ clients working in parallel over time horizon $T$ with explicit differential privacy (DP) guarantees. With stochastic adversaries, we propose a **Fed-DP-OPE-Stoch** algorithm that achieves $\sqrt{m}$-fold speed-up of the per-client regret compared to the single-player counterparts under both pure DP and approximate DP constraints, while maintaining logarithmic communication costs. With oblivious adversaries, we establish non-trivial lower bounds indicating that *collaboration among clients does not lead to regret speed-up with general oblivious adversaries*. We then consider a special case of the oblivious adversaries setting, where there exists a low-loss expert. We design a new algorithm **Fed-SVT** and show that it achieves an $m$-fold regret speed-up under both pure DP and approximate DP constraints over the single-player counterparts. Our lower bound indicates that Fed-SVT is nearly optimal up to logarithmic factors. Experiments demonstrate the effectiveness of our proposed algorithms. To the best of our knowledge, this is the first work examining the differentially private online prediction from experts in the federated setting.

Federated Online Prediction from Experts with Differential Privacy: Separations and Regret Speed-ups

Abstract

We study the problems of differentially private federated online prediction from experts against both stochastic adversaries and oblivious adversaries . We aim to minimize the average regret on m clients working in parallel over time horizon T with explicit differential privacy (DP) guarantees. With stochastic adversaries, we propose a Fed-DP-OPE-Stoch algorithm that achieves √ m -fold speed-up of the per-client regret compared to the single-player counterparts under both pure DP and approximate DP constraints, while maintaining logarithmic communication costs. With oblivious adversaries, we establish non-trivial lower bounds indicating that collaboration among clients does not lead to regret speed-up with general oblivious adversaries . We then consider a special case of the oblivious adversaries setting, where there exists a low-loss expert. We design a new algorithm Fed-SVT and show that it achieves an m -fold regret speed-up under both pure DP and approximate DP constraints over the single-player counterparts. Our lower bound indicates that Fed-SVT is nearly optimal up to logarithmic factors. Experiments demonstrate the effectiveness of our proposed algorithms. To the best of our knowledge, this is the first work examining the differentially private online prediction from experts in the federated setting.

1 Introduction

Federated Learning (FL) (McMahan et al., 2017) is a distributed machine learning framework, where numerous clients collaboratively train a model by exchanging model update through a server. Owing to its advantage in protecting the privacy of local data and reducing communication overheads, FL is gaining increased attention in the research community, particularly in the online learning framework (Mitra et al., 2021; Park et al., 2022; Kwon et al., 2023; Gauthier et al., 2023). Noticeable advancements include various algorithms in federated multi-armed bandits (Shi et al., 2021; Huang et al., 2021; Li and Wang, 2022; Yi and Vojnovic, 2022, 2023), federated online convex optimization (Patel et al., 2023; Kwon et al., 2023; Gauthier et al., 2023), etc.

Meanwhile, differential privacy (DP) has been integrated into online learning, pioneered by Dwork et al. (2010). Recently, Asi et al. (2022b) studied different types of adversaries, developing some of the best existing algorithms and establishing lower bounds. Within the federated framework, although differentially private algorithms have been proposed for stochastic bandits (Li et al., 2020b; Zhu et al., 2021; Dubey and Pentland, 2022) and linear contextual bandits (Dubey and Pentland, 2020; Li et al., 2022a; Zhou and Chowdhury, 2023; Huang et al., 2023), to the best of our knowledge, federated online learning in the adversarial setting with DP considerations remain largely unexplored.

In this work, we focus on federated online prediction from experts (OPE) with rigorous differential privacy (DP) guarantees 1 . OPE (Arora et al., 2012) is a classical online learning problem under which, a player chooses one out of a set of experts at each time slot and an adversary chooses a loss function. The player incurs a loss based on its choice and observes the loss function. With all previous observations, the player needs to decide which expert to select each time to minimize the cumulative expected loss. We consider two types of adversaries in the context of OPE. The first type, stochastic adversary , chooses a distribution over loss functions and samples a loss function independently and identically distributed (IID) from this distribution at each time step. The second type, oblivious adversary , chooses a sequence of loss functions in advance. 2

When extending the OPE problem to the federated setting, we assume that the system consists of a central server and m local clients, where each client chooses from d experts to face an adversary at each time step over time horizon T . The server coordinates the behavior of the clients by aggregating the clients' updates to form a global update (predicting a new global expert), while the clients use the global expert prediction to update its local expert selection and compute local updates. The local updates will be sent to the server periodically. In the Federated OPE framework, clients face either stochastic adversaries , receiving loss functions from the same distribution in an IID fashion, or oblivious adversaries , which arbitrarily select loss functions for each client at each time step beforehand (Yi and Vojnovic, 2023). Specifically, we aim to answer the following question:

Can we design differentially private federated OPE algorithms to achieve regret speed-up against both stochastic and oblivious adversaries?

In this paper, we give definitive answers to the question. Our contributions are summarized as follows.

Speed-up for stochastic adversaries. We develop a communication-efficient algorithm Fed-DPOPE-Stoch for stochastic adversaries with DP guarantees. The algorithm features the following elements in its design: 1) Local loss function gradient estimation for global expert determination. To reduce communication cost, we propose to estimate the gradient of each client's previous loss functions locally, and only communicate these estimates to the server instead of all previous loss functions. 2) Local privatization process. Motivated by the need for private communication in FL, we add noise to client communications (local gradient estimates) adhering to the DP principles, thereby building differentially private algorithms. √

We show that Fed-DP-OPE-Stoch achieves 1 / m -fold reduction of the per-client regret compared to the single-player counterparts (Asi et al., 2022b) under both pure DP and approximate DP constraints, while maintaining logarithmic communication costs.

Lower bounds for oblivious adversaries. We establish new lower bounds for federated OPE with oblivious adversaries. Our findings reveal a critical insight: collaboration among clients does not lead to regret speed-up in this context. Moreover, these lower bounds highlight a separation between oblivious adversaries and stochastic adversaries, as the latter is necessary to reap the benefits of collaboration in this framework.

Formulating an instance of oblivious adversaries for the corresponding lower bounds is a non-trivial challenge, because it requires envisioning a scenario where the collaborative nature of FL does not lead to the expected improvements in regret minimization. To deal with this challenge, we propose a policy reduction approach in FL . By defining an 'average policy' among all clients against a uniform loss function generated by an oblivious adversary, we reduce the federated problem to a single-player one, highlighting the equivalence of per-client and single-player regret. Our lower bounds represent the first of their kind for differentially private federated OPE problems.

Speed-up for oblivious adversaries under realizability assumption. We design a new algorithm Fed-SVT that obtains near-optimal regret when there is a low-loss expert. We show that Fed-SVT achieves an m -fold speed-up in per-client regret when compared to single-player models (Asi et al., 2023). This shows a clear separation from the general setting where collaboration among clients does not yield benefits in regret reduction when facing oblivious adversaries. Furthermore, we establish a new lower bound in the realizable case. This underlines that our upper bound is nearly optimal up to logarithmic factors.

m : number of clients; T : time horizon; d : number of experts; ε , δ : DP parameters; SING and FED stand for single-client and federated settings, respectively.

A summary of our main results and how they compare with the state of the art is shown in Table 1. A comprehensive literature review is provided in Appendix A.

2 Preliminaries

Federated online prediction from experts. Federated OPE consists of a central server, m clients and an interactive T -round game between an adversary and an algorithm. At time step t , each client i ∈ [ m ] first selects an expert x i,t ∈ [ d ] , and then, the adversary releases a loss function l i,t . Stochastic adversaries choose a distribution over loss functions and sample a sequence of loss functions l 1 , 1 , . . . , l m,T in an IID fashion from this distribution, while oblivious adversaries choose a sequence of loss functions l 1 , 1 , . . . , l m,T at the beginning of the game.

For federated OPE, the utility of primary interest is the expected cumulative regret among all clients defined as:

Differential privacy. We define differential privacy in the online setting following (Dwork et al., 2010). If an adversary chooses a loss sequence S = ( l 1 , 1 , . . . , l m,T ) , we denote A ( S ) = ( x 1 , 1 , . . . , x m,T ) the output of the interaction between the federated online algorithm A and the adversary. We say S = ( l 1 , 1 , . . . , l m,T ) and S ′ = ( l ′ 1 , 1 , . . . , l ′ m,T ) are neighboring datasets if S and S ′ differ in a one element. 3 .

Definition 1 ( ( ε, δ ) -DP) . A randomized federated algorithm A is ( ε, δ ) -differentially private against an adversary if, for all neighboring datasets S and S ′ and for all events O in the output space of A , we have

Communication model. Our setup involves a central server facilitating periodic communication with zero latency with all clients. Specifically, the clients can send 'local model updates' to the central server, which then aggregates and broadcasts the updated 'global model' to the clients. We assume full synchronization between clients and the server (McMahan et al., 2017). Following Wang et al. (2019), we define the communication cost of an algorithm as the number of scalars (integers or real numbers) communicated between the server and clients.

3 Federated OPE with Stochastic Adversaries

In this section, we aim to design an algorithm for Fed-DP-OPE with stochastic adversaries that achieves regret speed-up compared to the single-player setting under DP constraints with low

communication cost. We consider the loss functions l 1 , 1 ( · ) , . . . , l m,T ( · ) to be convex, α -Lipschitz and β -smooth w.r.t. ∥ · ∥ 1 in this section.

3.1 Intuition behind Algorithm Design

To gain a better understanding of our algorithm design, we first elaborate the difficulties encountered when extending prevalent OPE models to the FL setting under DP constraints. It is worth noting that all current OPE models with stochastic adversaries rely on gradient-based optimization methods. The central task in designing OPE models with stochastic adversaries lies in leveraging past loss functions to guide the generation of expert predictions. Specifically, we focus on the prominent challenges associated with the widely adopted Frank-Wolfe-based methods (Asi et al., 2022b). This algorithm iteratively moves the expert selection x t towards a point that minimizes the gradient estimate derived from the past loss functions l 1 , . . . , l t -1 over the decision space X , where X = ∆ d = { x ∈ R d : x i ≥ 0 , ∑ d i =1 x i = 1 } , and each x ∈ X represents a probability distribution over d experts. With DP constraints, a tree-based method is used for private aggregation of the gradients of loss functions (Asi et al., 2021b).

In the federated setting, it is infeasible for the central server to have full access to past loss functions due to the high communication cost. To overcome this, we use a design of local loss function gradient estimation for global expert determination . Our solution involves locally estimating the gradient of each client's loss functions, and then communicating these estimates to the server, which globally generates a new prediction. This strategy bypasses the need for full access to all loss functions, reducing the communication overhead while maintaining efficient expert selection.

To enhance privacy in the federated system, we implement a local privatization process . When communication is triggered, clients send 'local estimates' of the gradients of their loss functions to the server. These local estimates include strategically added noise, adhering to DP principles. The privatization method is crucial as it extends beyond privatizing the selection of experts; it ensures the privacy of all information exchanged between the central server and clients within the FL framework.

3.2 Algorithm Design

To address the aforementioned challenges, we propose the Fed-DP-OPE-Stoch algorithm. The Fed-DP-OPE-Stoch algorithm works in phases. In total, it has P phases, and each phase p ∈ [ P ] contains 2 p -1 time indices. Fed-DP-OPE-Stoch contains a client-side subroutine (Algorithm 1) and a server-side subroutine (Algorithm 2). The framework of our algorithm is outlined as follows.

At the initialization phase, the server selects an arbitrary point z ∈ X and broadcasts to all clients. Subsequently, each client initializes its expert selection x i, 1 = z , pays cost l i, 1 ( x i, 1 ) and observes the loss function.

Starting at phase p = 2 , each client uses loss functions from the last phase to update its local loss function gradient estimation, and then coordinates with the server to update its expert selection. After that, it sticks with its decision throughout the current phase and observes the loss functions. We elaborate this procedure as follows.

Private Local Loss Function Gradient Estimation. At the beginning of phase p , each client privately estimates the gradient using local loss functions from the last phase B i,p = { l i, 2 p -2 , . . . , l i, 2 p -1 -1 } . We employ the tree mechanism for the private aggregation at each client, as in the DP-FW algorithm from Asi et al. (2021b). Roughly speaking, DP-FW involves constructing binary trees and allocating sets of loss functions to each vertex. The gradient at each vertex is then estimated using the loss functions of that vertex and the gradients along the path to the root. Specifically, we run a DP-FW subroutine at each client i , with sample set B i,p , parameter T 1 and batch size b . In DP-FW, each vertex s in the binary tree j corresponds to a gradient estimate v i,j,s . DP-FW iteratively updates gradient estimates by visiting the vertices of binary trees. The details of DP-FW can be found in Algorithm 3 in Appendix C.1.

Intuitively, in the DP-FW subroutine, reaching a leaf vertex marks the completion of gradient refinement for a specific tree path. At these critical points, we initiate communication between the central server and individual clients. Specifically, as the DP-FW subroutine reaches a leaf vertex s of tree j , each client sends the server a set of noisy inner products for each decision set vertex

c n with their gradient estimate v i,j,s . In other words, each client communicates with the server by sending {⟨ c n , v i,j,s ⟩ + ξ i,n } n ∈ [ d ] , where c 1 , . . . , c d represents d vertices of decision set X = ∆ d , ξ i,n ∼ Lap ( λ i,j,s ) , and λ i,j,s = 4 α 2 j bε .

Private Global Expert Prediction. After receiving {⟨ c n , v i,j,s ⟩ + ξ i,n } n ∈ [ d ] from all clients, the central server privately predicts a new expert:

Subsequently, the server broadcasts the 'global prediction' ¯ w j,s to all clients.

Local Expert Selection Updating. Denote the index of the leaf s of tree j as k . Then, upon receiving the global expert selection ¯ w j,s , each client i updates its expert prediction for leaf k +1 , denoted as x i,p,k +1 ∈ X , as follows:

where η i,p,k = 2 k +1 .

After updating all leaf vertices of the trees, the client obtains x i,p,K , the final state of the expert prediction in phase p . Then, each client i sticks with expert selection x i,p,K throughout phase p and collects loss functions l i, 2 p -1 , . . . , l i, 2 p -1 .

Remark 1 (Comparison with Asi et al. (2022b)) . The key difference between Fed-DP-OPE-Stoch and non-federated algorithms (Asi et al., 2022b) lies in our innovative approach to centralized coordination and communication efficiency. Unlike the direct application of DP-FW in each phase in Asi et al. (2022b), our algorithm employs local loss function gradient estimation , global expert prediction and local expert selection updating . Additionally, our strategic communication protocol, where clients communicate with the server only when DP-FW subroutine reaches leaf vertices, significantly reduces communication costs. Moreover, the integration of DP at the local level in our algorithm distinguishes it from non-federated approaches.

Algorithm 1 Fed-DP-OPE-Stoch: Client i

3.3 Theoretical Guarantees

Now we are ready to present theoretical guarantees for Fed-DP-OPE-Stoch.

Theorem 1. Assume that loss function l i,t ( · ) is convex, α -Lipschitz, β -smooth w.r.t. ∥ · ∥ 1 . Setting λ i,j,s = 4 α 2 j bε , b = 2 p -1 ( p -1) 2 and T 1 = 1 2 log ( bεβ √ m α log d ) , Fed-DP-OPE-Stoch (i) satisfies ε -DP and (ii) achieves the per-client regret of

Algorithm 2 Fed-DP-OPE-Stoch: Central server

1: Input: Phases P , number of clients m , decision set X = ∆ d with vertices { c 1 , . . . , c d } .

2: Initialize: Pick any z ∈ X and broadcast to clients.

3: for p = 2 to P do

4: Receive from clients: {⟨ c n , v i,j,s ⟩ + ξ i,n } n ∈ [ d ]

5: Update ¯ w j,s according to Equation (1)

6: Communicate to clients: ¯ w j,s

7: end for

A similar result characterizing the performance of Fed-DP-OPE-Stoch under ( ε, δ ) -DP constraint is shown in Theorem 6.

Proof sketch of DP guarantees and communication costs. Given neighboring datasets S and S ′ differing in l i 1 ,t 1 or l ′ i 1 ,t 1 , we first note that l i 1 ,t 1 or l ′ i 1 ,t 1 is used in only one phase, denoted as p 1 . Furthermore, note that l i 1 ,t 1 or l ′ i 1 ,t 1 is used in the gradient computation for at most 2 j -| s | iterations, corresponding to descendant leaf nodes. This insight allows us to set noise levels λ i,j,s sufficiently large to ensure each of these iterations is ε/ 2 j -| s | -DP regarding output {⟨ c n , v i 1 ,j,s ⟩} n ∈ [ d ] ,j ∈ [ T 1 ] , | s | = j . By basic composition and post-processing, we can make sure the final output is ε -DP.

The communication cost is obtained by observing that there are p phases, and within each phase, there are O (2 T 1 ) leaf vertices. Thus, the communication frequency scales in O ( ∑ p 2 T 1 ) and the communication cost scales in O ( md ∑ p 2 T 1 ) .

Proof sketch of regret upper bound. We first give bounds for the total regret in phase p ∈ [ P ] . We can show that for every t in phase p , we have

To upper bound the regret, we bound the following two quantities separately.

Step 1: bound ∥∇ L t ( x i,p,k ) -¯ v p,k ∥ ∞ in Lemma 3. We show that every index of the d -dimensional vector ∇ L t ( x i,p,k ) -¯ v p,k is O ( α 2 + β 2 bm ) -sub-Gaussian by induction on the depth of vertex in Lemma 2.

Step 2: bound ⟨ ¯ v p,k , ¯ w p,k ⟩ -min w ∈X ⟨ ¯ v p,k , w ⟩ in Lemma 5. We establish that ⟨ ¯ v p,k , ¯ w p,k ⟩ -min w ∈X ⟨ ¯ v p,k , w ⟩ is upper bounded by 2 m max n :1 ≤ n ≤ d ∣ ∣ ∣ ∑ m i =1 ξ i,n ∣ ∣ ∣ . This bound incorporates the maximum absolute sum of m IID Laplace random variables ξ i,n . To quantify the bound on the sum of m IID Laplace random variables, we refer to Lemma 4. Consequently, we have E [ ⟨ ¯ v p,k , ¯ w p,k ⟩ -min w ∈X ⟨ ¯ v p,k , w ⟩ ] upper bounded by O ( λ i,j,s ln d √ m ) .

⋆

With the two steps above, we can show that for each time step t in phase p , E [ L t ( x i,p,K ) -L t ( x )] is upper bounded by O ( ( α + β ) √ log d 2 p m + √ αβ log d m 1 4 √ 2 p ε ) . Summing up all the phases gives us the final upper bound. The full proof of Theorem 1 can be found in Appendix C.2.

When β is small, Theorem 1 reduces to the following corollary.

Corollary 1. If β = O ( log d √ mTε ) , then, Fed-DP-OPE-Stoch (i) satisfies ε -DP and (ii) achieves the per-client regret of

with (iii) a communication cost of O ( md log T ) .

The full proof of Corollary 1 can be found in Appendix C.2.

Remark 2 . We note that the regret for the single-player counterpart Asi et al. (2022b) scales in O ( √ T log d + log d log T ε ) when β = O ( log d Tε ) . Compared with this upper bound, Fed-DP-OPE-√

Stoch achieves m -fold speed-up, with a communication cost of O ( md log T ) . This observation underscores the learning performance acceleration due to collaboration. Furthermore, our approach extends beyond the mere privatization of selected experts; we ensure the privacy of all information exchanged between the central server and clients.

Remark 3 . We remark that Fed-DP-OPE-Stoch can be slightly modified into a centrally differentially private algorithm, assuming client-server communication is secure. Specifically, we change the local privatization process to a global privatization process on the server side. This mechanism results in less noise added and thus better utility performance. The detailed design is deferred to Appendix C.4 and the theoretical guarantees are provided in Theorem 7.

4 Federated OPE with Oblivious Adversaries: Lower bounds

In this section, we shift our attention to the more challenging oblivious adversaries. We establish new lower bounds for general oblivious adversaries.

To provide a foundational understanding, we start with some intuition behind FL. FL can potentially speed up the learning process by collecting more data at the same time to gain better insights to future predictions. In the stochastic setting, the advantage of collaboration lies in the ability to collect more observations from the same distribution, which leads to variance reduction. However, when facing oblivious adversaries, the problem changes fundamentally. Oblivious adversaries can select loss functions arbitrarily, meaning that having more data does not necessarily help with predicting their future selections.

The following theorem formally establishes the aforementioned intuition.

Theorem 2. For any federated OPE algorithm against oblivious adversaries, the per-client regret is lower bounded by Ω( √ T log d ) . Let ε ∈ (0 , 1] and δ = o (1 /T ) , for any ( ε, δ ) -DP federated OPE algorithm, the per-client regret is lower bounded by Ω ( min ( log d ε , T )) .

Remark 4 . Theorem 2 states that with oblivious adversaries, the per-client regret under any federated OPE is fundamentally independent of the number of clients m , indicating that, the collaborative effort among multiple clients does not yield a reduction in the regret lower bound. This is contrary to typical scenarios where collaboration can lead to shared insights and improve overall performance. The varied and unpredictable nature of oblivious adversaries nullifies the typical advantages of collaboration. Theorem 2 also emphasizes the influence of the DP guarantees. Our lower bounds represent the first non-trivial impossibility results for the federated OPE to the best of our knowledge.

Proof sketch. We examine the case when all clients receive the same loss function from the oblivious adversary at each time step, i.e. l i,t = l ′ t . Within this framework, we define the 'average policy" among all clients, i.e., p ′ t ( k ) = 1 m ∑ m i =1 p i,t ( k ) , ∀ k ∈ [ d ] . This leads us to express the per-client regret as: Reg ( T, m ) = ∑ T t =1 ∑ d k =1 p ′ t ( k ) · l ′ t ( k ) -∑ T t =1 l ′ t ( x ⋆ ) . Note that p ′ t ( k ) is defined by p 1 ,t ( k ) , . . . , p m,t ( k ) , which in turn are determined by l 1 , 1 , . . . , l m,t -1 . According to our choice of l i,t = l ′ t , p ′ t ( k ) is determined by l ′ 1 , l ′ 2 , . . . , l ′ t -1 . Therefore, p ′ 1 , p ′ 2 , . . . , p ′ t are generated by a legitimate algorithm for online learning with expert advice problems. Through our policy reduction approach in FL , we can reduce the federated problem to a single-player setting, showing the equivalence of per-client and single-player regret against the oblivious adversary we construct, thus obtaining the regret lower bound. We believe that this technique will be useful in future analysis of other FL algorithm. Incorporating DP constraints, we refer to Lemma 9, which transforms the DP

online learning problem to a well-examined DP batch model. The full proof of Theorem 2 can be found in Appendix D.

5 Federated OPE with Oblivious Adversaries: Realizable Setting

Given the impossibility results in the general oblivious adversaries setting, one natural question we aim to answer is, is there any special scenarios where we can still harness the power of federation even in presence of oblivious adversaries? Towards this end, in this section, we focus on the (near) realizable setting, formally defined below.

Definition 2 (Realizability) . A federated OPE problem is realizable if there exists a feasible solution x ⋆ ∈ [ d ] such that ∑ T t =1 l i,t ( x ⋆ ) = 0 , ∀ i ∈ [ m ] . If the best expert achieves small loss L ⋆ ≪ T , i.e., there exists x ⋆ ∈ [ d ] such that ∑ T t =1 l i,t ( x ⋆ ) ≤ L ⋆ , ∀ i ∈ [ m ] , the problem is near-realizable.

Intuitively, collaboration is notably advantageous in this context, as all clients share the same goal of reaching the zero-loss solution x ⋆ . As more clients participate, the shared knowledge pool expands, making the identification of the optimal solution more efficient. In the following, we first provide regret lower bounds to quantify this benefit formally, and then show that a sparse vector based federated algorithm can almost achieve such lower bounds thus is nearly optimal.

5.1 Lower Bound

Theorem 3. Let ε ≤ 1 and δ ≤ ε/d . For any ( ε, δ ) -DP federated OPE algorithm against oblivious adversaries in the realizable setting, the per-client regret is lower bounded by Ω ( log( d ) mε ) .

Remark 5 . In the single-player setting, the regret bound is Ω ( log( d ) ε ) (Asi et al., 2023). In the federated setting, our results imply that the per-client regret is reduced to 1 m times the single-player regret. This indicates a possible m -fold speed-up in the federated setting.

̸

Proof sketch. To begin, we consider a specific oblivious adversary. We introduce two prototype loss functions: l 0 ( x ) = 0 for all x ∈ [ d ] and for j ∈ [ d ] l j ( x ) = 0 if x = j and otherwise l j ( x ) = 1 . An oblivious adversary picks one of the d sequences S 1 , . . . , S d uniformly at random, where S j = ( l j 1 , 1 , . . . , l j m,T ) such that l j i,t = l 0 if t = 1 , . . . , T -k , otherwise, l j i,t = l j , and k = log d 2 mε . Assume there exists an algorithm such that the per-client regret is upper bounded by log( d ) 32 mε . This would imply that for at least d/ 2 of S 1 , . . . , S d , the per-client regret is upper bounded by log( d ) 16 mε . Assume without loss of generality these sequences are S 1 , . . . , S d/ 2 . We let B j be the set of expert selections that has low regret on S j . We can show that B j and B j ′ are disjoint for j = j ′ , implying choosing any expert j leads to low regret for loss sequence S j but high regret for S j ′ . By group privacy, we have P ( A ( S j ) ∈ B j ′ ) ≥ e -mkε P ( A ( S j ′ ) ∈ B j ′ ) -mkδ, leading to a contradiction when d ≥ 32 . The full proof of Theorem 3 can be found in Appendix E.1.

5.2 Algorithm Design

We develop a new algorithm Fed-SVT. Our algorithm operates as follows:

Periodic communication. We adopt a fixed communication schedule in our federated setting, splitting the time horizon T into T/N phases, each with length N . In Fed-SVT, every client selects the same expert, i.e., x i,t = x t at each time step t . Initially, each client starts with a randomly chosen expert x 1 . At the beginning of each phase n , each client sends the accumulated loss of the last phase ∑ nN -1 t ′ =( n -1) N l i,t ′ ( x ) to the central server.

Global expert selection. The server, upon receiving ∑ nN -1 t ′ =( n -1) N l i,t ′ ( x ) from all clients, decides whether to continue with the current expert or switch to a new one. This decision is grounded in the Sparse-Vector algorithm (Asi et al., 2023), where the accumulated loss from all clients over a phase is treated as a single loss instance in the Sparse-Vector algorithm. Based on the server's expert decision, clients update their experts accordingly. The full algorithm is provided in Appendix E.2.

5.3 Theoretical Guarantees

Theorem 4. Let l i,t ∈ [0 , 1] d be chosen by an oblivious adversary under nearrealizability assumption. Fed-SVT is ε -DP, the communication cost scales in O ( mdT/N ) , and with probability at least 1 -O ( ρ ) , the pre-client regret is upper bounded by O ( log 2 ( d )+log ( T 2 N 2 ρ ) log ( d ρ ) mε +( N + L ⋆ ) log ( d ρ ) ) . Moreover, Fed-SVT is ( ε, δ ) -DP, the communication cost scales in O ( mdT/N ) , and with probability at least 1 -O ( ρ ) , the pre-client regret is upper bounded by O ( log 3 2 ( d ) √ log( 1 δ )+log ( T 2 N 2 ρ ) log ( d ρ ) mε +( N + L ⋆ ) log ( d ρ ) ) .

Remark 6 . Note that when N + L ⋆ = O ( log T mε ) , then, under Fed-SVT, the per-client regret upper

the single-player scenario (Asi et al., 2023), our results indicate an m -fold regret speed-up. Note that our lower bound (Theorem 3) scales in Ω ( log d mε ) . Our upper bound matches with the lower bound in terms of m and ϵ , and is nearly optimal up to logarithmic factors.

The proof of Theorem 4 is presented in Appendix E.3.

6 Numerical Experiments

Fed-DP-OPE-Stoch. We conduct experiments in a synthetic environment to validate the theoretical performances of Fed-DP-OPE-Stoch, and compare them with its single-player counterpart Limited Updates (Asi et al., 2022b).

We first generate true class distributions for each client i ∈ [ m ] and timestep t ∈ [ T ] , which are sampled IID from Gaussian distributions with means and variances sampled uniformly. Following Gaussian sampling, we apply a softmax transformation and normalization to ensure the outputs

are valid probability distributions. Then we generate a sequence of IID loss functions l 1 , 1 , . . . , l m,T for m clients over T timesteps using cross-entropy between true class distributions and predictions. We set T 1 = 1 in Fed-DP-OPE-Stoch, therefore the communication cost scales in O ( md log T ) . We set m = 10 , T = 2 14 , ε = 10 , δ = 0 and d = 100 . The per-client cumulative regret as a function of T is plotted in Figure 1. We see that Fed-DP-OPE-Stoch outperforms Limited Updates significantly, indicating the regret speed-up due to collaboration. Results with different seeds are provided in Appendix F.

Fed-SVT. Weconduct experiments in a synthetic environment, comparing Fed-SVT with the single-player model Sparse-Vector (Asi et al., 2023).

We generate random losses for each expert at every timestep and for each client, ensuring that one expert always has zero loss to simulate an optimal choice. We set m = 10 , T = 2 9 , ε = 10 , δ = 0 and d = 100 . In FedSVT, we experiment with communication intervals N = 1 , 30 , 50 , where communication cost scales in O ( mdT/N ) . The per-client cumulative regret as a function of T is plotted in Figure 2. Our results show that

Fed-SVT significantly outperforms the Sparse-Vector, highlighting the benefits of collaborative expert selection in regret speed-up, even at lower communication costs (notably in the N = 50 case). Results with different seeds are provided in Appendix F. Additionally, we evaluate the performances of Fed-SVT on the MovieLens-1M dataset (Harper and Konstan, 2015) in Appendix G.

7 Conclusions

In this paper, we have advanced the state-of-the-art of differentially private federated online prediction from experts, addressing both stochastic and oblivious adversaries. Our Fed-DP-OPE-Stoch algorithm

showcases a significant √ m -fold regret speed-up compared to single-player models with stochastic adversaries, while effectively maintaining logarithmic communication costs. For oblivious adversaries, we established non-trivial lower bounds, highlighting the limited benefits of client collaboration. Additionally, our Fed-SVT algorithm demonstrates an m -fold speed-up, indicating near-optimal performance in settings with low-loss experts. One limitation of this work is the lack of experiments on real-world federated learning scenarios such as recommender systems and healthcare. We leave this exploration to future work.

Acknowledgments

The work of F. Gao, R. Huang and J. Yang was supported in part by the U.S. National Science Foundation under the grants CNS-1956276, CNS-2114542 and ECCS-2133170.

References

Agarwal, A., Foster, D. P., Hsu, D. J., Kakade, S. M., and Rakhlin, A. (2011). Stochastic convex optimization with bandit feedback. Advances in Neural Information Processing Systems , 24.

Agarwal, N., Kale, S., Singh, K., and Thakurta, A. (2023a). Differentially private and lazy online convex optimization. In The Thirty Sixth Annual Conference on Learning Theory , pages 4599-4632. PMLR.

Agarwal, N., Kale, S., Singh, K., and Thakurta, A. G. (2023b). Improved differentially private and lazy online convex optimization. arXiv preprint arXiv:2312.11534 .

Agarwal, N. and Singh, K. (2017). The price of differential privacy for online learning. In International Conference on Machine Learning , pages 32-40. PMLR.

Altschuler, J. and Talwar, K. (2018). Online learning over a finite action set with limited switching. In Conference On Learning Theory , pages 1569-1573. PMLR.

Arora, S., Hazan, E., and Kale, S. (2012). The multiplicative weights update method: a meta-algorithm and applications. Theory of computing , 8(1):121-164.

Asi, H., Chadha, K., Cheng, G., and Duchi, J. (2022a). Private optimization in the interpolation regime: faster rates and hardness results. In International Conference on Machine Learning , pages 1025-1045. PMLR.

Asi, H., Duchi, J., Fallah, A., Javidbakht, O., and Talwar, K. (2021a). Private adaptive gradient methods for convex optimization. In International Conference on Machine Learning , pages 383-392. PMLR.

Asi, H., Feldman, V., Koren, T., and Talwar, K. (2021b). Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning , pages 393-403. PMLR.

Asi, H., Feldman, V., Koren, T., and Talwar, K. (2022b). Private online prediction from experts: Separations and faster rates. arXiv preprint arXiv:2210.13537 .

Asi, H., Feldman, V., Koren, T., and Talwar, K. (2023). Near-optimal algorithms for private online optimization in the realizable regime. arXiv preprint arXiv:2302.14154 .

Azize, A. and Basu, D. (2022). When privacy meets partial information: A refined analysis of differentially private bandits. Advances in Neural Information Processing Systems , 35:3219932210.

Bar-On, Y. and Mansour, Y. (2019). Individual regret in cooperative nonstochastic multi-armed bandits. Advances in Neural Information Processing Systems , 32.

Bassily, R., Feldman, V., Talwar, K., and Guha Thakurta, A. (2019). Private stochastic convex optimization with optimal rates. Advances in neural information processing systems , 32.

Bassily, R., Smith, A., and Thakurta, A. (2014). Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science , pages 464-473. IEEE.

Bubeck, S., Li, Y., Luo, H., and Wei, C.-Y. (2019). Improved path-length regret bounds for bandits. In Conference On Learning Theory , pages 508-528. PMLR.

Buccapatnam, S., Eryilmaz, A., and Shroff, N. B. (2014). Stochastic bandits with side observations on networks. In The 2014 ACM international conference on Measurement and modeling of computer systems , pages 289-300.

Caron, S., Kveton, B., Lelarge, M., and Bhagat, S. (2012). Leveraging side observations in stochastic bandits. arXiv preprint arXiv:1210.4839 .

Cesa-Bianchi, N., Gentile, C., Mansour, Y., and Minora, A. (2016). Delay and cooperation in nonstochastic bandits. In Conference on Learning Theory , pages 605-622. PMLR.

Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, learning, and games . Cambridge university press.

Cesa-Bianchi, N., Mansour, Y., and Stoltz, G. (2007). Improved second-order bounds for prediction with expert advice. Machine Learning , 66:321-352.

Charisopoulos, V., Esfandiari, H., and Mirrokni, V. (2023). Robust and private stochastic linear bandits. In International Conference on Machine Learning , pages 4096-4115. PMLR.

Chen, L., Yu, Q., Lawrence, H., and Karbasi, A. (2020). Minimax regret of switching-constrained online convex optimization: No phase transition. Advances in Neural Information Processing Systems , 33:3477-3486.

Cheng, D., Zhou, X., and Ji, B. (2023). Understanding the role of feedback in online learning with switching costs. arXiv preprint arXiv:2306.09588 .

Chowdhury, S. R. and Zhou, X. (2022a). Distributed differential privacy in multi-armed bandits. arXiv preprint arXiv:2206.05772 .

Chowdhury, S. R. and Zhou, X. (2022b). Shuffle private linear contextual bandits. arXiv preprint arXiv:2202.05567 .

Cotter, A., Shamir, O., Srebro, N., and Sridharan, K. (2011). Better mini-batch algorithms via accelerated gradient methods. Advances in neural information processing systems , 24.

Dubey, A. and Pentland, A. (2020). Differentially-private federated linear bandits. Advances in Neural Information Processing Systems , 33:6003-6014.

Dubey, A. and Pentland, A. (2022). Private and byzantine-proof cooperative decision-making. arXiv preprint arXiv:2205.14174 .

Duchi, J., Hazan, E., and Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7).

Duchi, J. C., Chaturapruek, S., and Ré, C. (2015). Asynchronous stochastic convex optimization. arXiv preprint arXiv:1508.00882 .

Duchi, J. C., Lafferty, J., Zhu, Y., et al. (2016). Local minimax complexity of stochastic convex optimization. Advances in Neural Information Processing Systems , 29.

Dwork, C., Naor, M., Pitassi, T., and Rothblum, G. N. (2010). Differential privacy under continual observation. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 715-724.

Dwork, C., Roth, A., et al. (2014). The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407.

Feldman, V., Koren, T., and Talwar, K. (2020). Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449.

Fichtenberger, H., Henzinger, M., and Upadhyay, J. (2023). Constant matters: Fine-grained error bound on differentially private continual observation. In International Conference on Machine Learning , pages 10072-10092. PMLR.

Freund, Y. and Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119-139.

Garcelon, E., Chaudhuri, K., Perchet, V., and Pirotta, M. (2022). Privacy amplification via shuffling for linear contextual bandits. In International Conference on Algorithmic Learning Theory , pages 381-407. PMLR.

Gauthier, F., Gogineni, V. C., Werner, S., Huang, Y.-F., and Kuh, A. (2023). Asynchronous online federated learning with reduced communication requirements. arXiv preprint arXiv:2303.15226 .

Geulen, S., Vöcking, B., and Winkler, M. (2010). Regret minimization for online buffering problems using the weighted majority algorithm. In COLT , pages 132-143. Citeseer.

Golowich, N. and Livni, R. (2021). Littlestone classes are privately online learnable. Advances in Neural Information Processing Systems , 34:11462-11473.

Gonen, A., Hazan, E., and Moran, S. (2019). Private learning implies online learning: An efficient reduction. Advances in Neural Information Processing Systems , 32.

Guha Thakurta, A. and Smith, A. (2013). (nearly) optimal algorithms for private online learning in full-information and bandit settings. Advances in Neural Information Processing Systems , 26.

Hanna, O. A., Girgis, A. M., Fragouli, C., and Diggavi, S. (2022). Differentially private stochastic linear bandits:(almost) for free. arXiv preprint arXiv:2207.03445 .

Harper, F. M. and Konstan, J. A. (2015). The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) , 5(4):1-19.

He, J., Wang, T., Min, Y., and Gu, Q. (2022). A simple and provably efficient algorithm for asynchronous federated contextual linear bandits. Advances in neural information processing systems , 35:4762-4775.

Hong, S. and Chae, J. (2021). Communication-efficient randomized algorithm for multi-kernel online federated learning. IEEE transactions on pattern analysis and machine intelligence , 44(12):98729886.

Hu, B. and Hegde, N. (2022). Near-optimal thompson sampling-based algorithms for differentially private stochastic bandits. In Uncertainty in Artificial Intelligence , pages 844-852. PMLR.

Hu, C., Pan, W., and Kwok, J. (2009). Accelerated gradient methods for stochastic optimization and online learning. Advances in Neural Information Processing Systems , 22.

Huang, R., Wu, W., Yang, J., and Shen, C. (2021). Federated linear contextual bandits. Advances in neural information processing systems , 34:27057-27068.

Huang, R., Zhang, H., Melis, L., Shen, M., Hejazinia, M., and Yang, J. (2023). Federated linear contextual bandits with user-level differential privacy. In International Conference on Machine Learning , pages 14060-14095. PMLR.

Jain, P., Kothari, P., and Thakurta, A. (2012). Differentially private online learning. In Conference on Learning Theory , pages 24-1. JMLR Workshop and Conference Proceedings.

Jain, P., Raskhodnikova, S., Sivakumar, S., and Smith, A. (2023). The price of differential privacy under continual observation. In International Conference on Machine Learning , pages 1465414678. PMLR.

Jain, P. and Thakurta, A. G. (2014). (near) dimension independent risk bounds for differentially private learning. In International Conference on Machine Learning , pages 476-484. PMLR.

Kalai, A. and Vempala, S. (2005). Efficient algorithms for online decision problems. Journal of Computer and System Sciences , 71(3):291-307.

Kaplan, H., Mansour, Y., Moran, S., Nissim, K., and Stemmer, U. (2023a). Black-box differential privacy for interactive ml. In Thirty-seventh Conference on Neural Information Processing Systems .

Kaplan, H., Mansour, Y., Moran, S., Nissim, K., and Stemmer, U. (2023b). On differentially private online predictions. arXiv preprint arXiv:2302.14099 .

Kwon, D., Park, J., and Hong, S. (2023). Tighter regret analysis and optimization of online federated learning. IEEE Transactions on Pattern Analysis and Machine Intelligence .

Li, C. and Wang, H. (2022). Asynchronous upper confidence bound algorithms for federated linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 6529-6553. PMLR.

Li, C., Zhou, P., Xiong, L., Wang, Q., and Wang, T. (2018). Differentially private distributed online learning. IEEE Transactions on Knowledge and Data Engineering , 30(8):1440-1453.

Li, F., Zhou, X., and Ji, B. (2022a). Differentially private linear bandits with partial distributed feedback. In 2022 20th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt) , pages 41-48. IEEE.

Li, F., Zhou, X., and Ji, B. (2023). (private) kernelized bandits with distributed biased feedback. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 7(1):1-47.

Li, S., Chen, W., Wen, Z., and Leung, K.-S. (2020a). Stochastic online learning with probabilistic graph feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 46754682.

Li, T., Song, L., and Fragouli, C. (2020b). Federated recommendation system via differential privacy. In 2020 IEEE international symposium on information theory (ISIT) , pages 2592-2597. IEEE.

Li, W., Song, Q., Honorio, J., and Lin, G. (2022b). Federated x-armed bandit. arXiv preprint arXiv:2205.15268 .

Littlestone, N. and Warmuth, M. K. (1994). The weighted majority algorithm. Information and computation , 108(2):212-261.

Liu, C. and Belkin, M. (2018). Accelerating sgd with momentum for over-parameterized learning. arXiv preprint arXiv:1810.13395 .

M Ghari, P. and Shen, Y. (2022). Personalized online federated learning with multiple kernels. Advances in Neural Information Processing Systems , 35:33316-33329.

Ma, S., Bassily, R., and Belkin, M. (2018). The power of interpolation: Understanding the effectiveness of sgd in modern over-parametrized learning. In International Conference on Machine Learning , pages 3325-3334. PMLR.

Mahdavi, M., Yang, T., and Jin, R. (2013). Stochastic convex optimization with multiple objectives. Advances in neural information processing systems , 26.

McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communicationefficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Mishra, N. and Thakurta, A. (2015). (nearly) optimal differentially private stochastic multi-arm bandits. In Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence , pages 592-601.

Mitra, A., Hassani, H., and Pappas, G. J. (2021). Online federated learning. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 4083-4090. IEEE.

Park, J., Kwon, D., et al. (2022). Ofedqit: Communication-efficient online federated learning via quantization and intermittent transmission. arXiv preprint arXiv:2205.06491 .

Patel, K. K., Wang, L., Saha, A., and Srebro, N. (2023). Federated online and bandit convex optimization. In International Conference on Machine Learning , pages 27439-27460. PMLR.

Rakhlin, A., Sridharan, K., and Tewari, A. (2011). Online learning: Stochastic, constrained, and smoothed adversaries. Advances in neural information processing systems , 24.

Ren, W., Zhou, X., Liu, J., and Shroff, N. B. (2020). Multi-armed bandits with local differential privacy. arXiv preprint arXiv:2007.03121 .

Sajed, T. and Sheffet, O. (2019). An optimal private stochastic-mab algorithm based on optimal private stopping rule. In International Conference on Machine Learning , pages 5579-5588. PMLR.

Shalev-Shwartz, S. et al. (2012). Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194.

Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. (2009). Stochastic convex optimization. In COLT , page 5.

Shariff, R. and Sheffet, O. (2018). Differentially private contextual linear bandits. Advances in Neural Information Processing Systems , 31.

Sherman, U. and Koren, T. (2021). Lazy oco: Online convex optimization on a switching budget. In Conference on Learning Theory , pages 3972-3988. PMLR.

Shi, C. and Shen, C. (2021). Federated multi-armed bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 9603-9611.

Shi, C., Shen, C., and Yang, J. (2021). Federated multi-armed bandits with personalization. In International conference on artificial intelligence and statistics , pages 2917-2925. PMLR.

Srebro, N., Sridharan, K., and Tewari, A. (2010). Smoothness, low noise and fast rates. Advances in neural information processing systems , 23.

Steinhardt, J. and Liang, P. (2014). Adaptivity and optimism: An improved exponentiated gradient algorithm. In International conference on machine learning , pages 1593-1601. PMLR.

Tao, Y., Wu, Y., Zhao, P., and Wang, D. (2022). Optimal rates of (locally) differentially private heavytailed multi-armed bandits. In International Conference on Artificial Intelligence and Statistics , pages 1546-1574. PMLR.

Tenenbaum, J., Kaplan, H., Mansour, Y., and Stemmer, U. (2021). Differentially private multi-armed bandits in the shuffle model. Advances in Neural Information Processing Systems , 34:2495624967.

Tossou, A. and Dimitrakakis, C. (2016). Algorithms for differentially private multi-armed bandits. In Proceedings of the AAAI Conference on Artificial Intelligence .

Tossou, A., Dimitrakakis, C., and Dubhashi, D. (2017). Thompson sampling for stochastic bandits with graph feedback. In Proceedings of the AAAI Conference on Artificial Intelligence .

Vaswani, S., Bach, F., and Schmidt, M. (2019). Fast and faster convergence of sgd for overparameterized models and an accelerated perceptron. In The 22nd international conference on artificial intelligence and statistics , pages 1195-1204. PMLR.

Wang, C.-H., Li, W., Cheng, G., and Lin, G. (2022a). Federated online sparse decision making. arXiv preprint arXiv:2202.13448 .

Wang, H., Zhao, D., and Wang, H. (2022b). Dynamic global sensitivity for differentially private contextual bandits. In Proceedings of the 16th ACM Conference on Recommender Systems , pages 179-187.

Wang, H., Zhao, Q., Wu, Q., Chopra, S., Khaitan, A., and Wang, H. (2020). Global and local differential privacy for collaborative bandits. In Proceedings of the 14th ACM Conference on Recommender Systems , pages 150-159.

Wang, Y., Hu, J., Chen, X., and Wang, L. (2019). Distributed bandit learning: Near-optimal regret with efficient communication. arXiv preprint arXiv:1904.06309 .

Wei, C.-Y. and Luo, H. (2018). More adaptive algorithms for adversarial bandits. In Conference On Learning Theory , pages 1263-1291. PMLR.

Woodworth, B. E. and Srebro, N. (2021). An even more optimal stochastic optimization algorithm: minibatching and interpolation learning. Advances in Neural Information Processing Systems , 34:7333-7345.

Wu, Y., György, A., and Szepesvári, C. (2015). Online learning with gaussian payoffs and side observations. Advances in Neural Information Processing Systems , 28.

Wu, Y., Zhou, X., Tao, Y., and Wang, D. (2023). On private and robust bandits. arXiv preprint arXiv:2302.02526 .

Yi, J. and Vojnovic, M. (2022). On regret-optimal cooperative nonstochastic multi-armed bandits. arXiv preprint arXiv:2211.17154 .

Yi, J. and V ojnovic, M. (2023). Doubly adversarial federated bandits. In International Conference on Machine Learning , pages 39951-39967. PMLR.

Zheng, K., Cai, T., Huang, W., Li, Z., and Wang, L. (2020). Locally differentially private (contextual) bandits learning. Advances in Neural Information Processing Systems , 33:12300-12310.

Zhou, X. and Chowdhury, S. R. (2023). On differentially private federated linear contextual bandits. arXiv preprint arXiv:2302.13945 .

Zhu, Z., Zhu, J., Liu, J., and Liu, Y . (2021). Federated bandit: A gossiping approach. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 5(1):1-29.

Zinkevich, M. (2003). Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936.

A Related Work

Online learning with stochastic adversaries. Online learning with stochastic adversaries has been extensively studied (Rakhlin et al., 2011; Duchi et al., 2011; Hu et al., 2009; Li et al., 2020a; Caron et al., 2012; Buccapatnam et al., 2014; Tossou et al., 2017; Wu et al., 2015). This problem is also closely related to stochastic convex optimization (Shalev-Shwartz et al., 2009; Mahdavi et al., 2013; Duchi et al., 2015; Agarwal et al., 2011; Duchi et al., 2016) since online learning with stochastic adversaries can be transformed into the stochastic convex optimization problem using online-tobatch conversion. In the federated setting, Mitra et al. (2021) proposed a federated online mirror descent method. Patel et al. (2023) presented a federated projected online stochastic gradient descent algorithm. Research on online learning with stochastic adversaries incorporating DP has also seen significant advancements (Bassily et al., 2014, 2019; Feldman et al., 2020; Asi et al., 2021a,b, 2022b).

Online learning with non-stochastic adversaries. Online learning with non-stochastic adversaries has been extensively studied (Cesa-Bianchi and Lugosi, 2006; Littlestone and Warmuth, 1994; Freund and Schapire, 1997; Zinkevich, 2003). Federated online learning with non-stochastic adversaries, a more recent development, was introduced by Hong and Chae (2021). Mitra et al. (2021) developed a federated online mirror descent method. Park et al. (2022) presented a federated online gradient descent algorithm. Kwon et al. (2023) studied data heterogeneity. Furthermore, Li et al. (2018) and Patel et al. (2023) explored distributed online and bandit convex optimization. Gauthier et al. (2023) focused on the asynchronous settings. Research into online learning with limited switching has also been explored by Kalai and Vempala (2005); Geulen et al. (2010); Altschuler and Talwar (2018); Chen et al. (2020); Sherman and Koren (2021); He et al. (2022); Li et al. (2022b); Cheng et al. (2023).

Differentially private online learning with non-stochastic adversaries was pioneered by Dwork et al. (2010). Several studies have explored OPE problem with DP constraints (Jain et al., 2012; Guha Thakurta and Smith, 2013; Jain and Thakurta, 2014; Agarwal and Singh, 2017; Asi et al., 2022b). Specifically, Jain and Thakurta (2014) and Agarwal and Singh (2017) focused on methods based on follow the regularized leader. Asi et al. (2022b) proposed an algorithm using the shrinking dartboard method.

Studies addressing differentially private online learning with non-stochastic adversaries in other contexts have also made significant strides. Kaplan et al. (2023b) focused on online classification problem with joint DP constraints. Fichtenberger et al. (2023) introduced a constant improvement. Gonen et al. (2019) and Kaplan et al. (2023a) studied the relationship between private learning and online learning. Agarwal et al. (2023a) and Agarwal et al. (2023b) studied the DP online convex optimization problem. Research on differentially private federated online learning with non-stochastic adversaries is very limited to the best of our knowledge.

Online learning under realizability assumption. Several works have studied online learning in the realizable setting. Srebro et al. (2010) studied mirror descent for online optimization. Shalev-Shwartz et al. (2012) studied the weighted majority algorithm. There is also a line of work studying stochastic convex optimization in the realizable setting (Cotter et al., 2011; Ma et al., 2018; Vaswani et al., 2019; Liu and Belkin, 2018; Woodworth and Srebro, 2021; Asi et al., 2022a). Additionally, some researchers have recently focused on the variance or the path length of the best expert (Cesa-Bianchi et al., 2007; Steinhardt and Liang, 2014; Wei and Luo, 2018; Bubeck et al., 2019). When DP constraint is considered, Asi et al. (2023) introduced algorithms using the sparse vector technique. Golowich and Livni (2021) studied DP online classification.

Multi-armed bandits with DP. Research on DP multi-armed bandits has developed along various lines (Mishra and Thakurta, 2015; Tossou and Dimitrakakis, 2016; Sajed and Sheffet, 2019; Azize and Basu, 2022). Further, Hu and Hegde (2022) proposed a Thompson-sampling-based approach, Tao et al. (2022) explored the heavy-tailed rewards scenario, and Chowdhury and Zhou (2022a) achieved optimal regret in a distributed setting. Additionally, Ren et al. (2020) and Zheng et al. (2020) investigated local DP constraints. For linear contextual bandits with DP, significant studies were conducted by Shariff and Sheffet (2018); Wang et al. (2020, 2022b) and Hanna et al. (2022). Furthermore, Hanna et al. (2022); Chowdhury and Zhou (2022b); Garcelon et al. (2022) and Tenenbaum et al. (2021) focused on the shuffle model. Li et al. (2023) studied kernelized bandits with distributed biased feedback. Wu et al. (2023) and Charisopoulos et al. (2023) tackled privacy and robustness simultaneously.

Federated multi-armed bandits. Federated bandits have been studied extensively recently. Shi and Shen (2021) and Shi et al. (2021) investigated federated stochastic multi-armed bandits without and with personalization, respectively. Wang et al. (2019) considered the distributed setting. Huang et al. (2021), Wang et al. (2022a) and Li and Wang (2022) studied federated linear contextual bandits. Li et al. (2022b) focused on federated X -armed bandits problem. Additionally, Cesa-Bianchi et al. (2016), Bar-On and Mansour (2019), Yi and Vojnovic (2022) and Yi and Vojnovic (2023) have studied cooperative multi-armed bandits problem with data exchange among neighbors. M Ghari and Shen (2022) have explored online model selection where each client learns a kernel-based model, utilizing the specific characteristics of kernel functions.

When data privacy is explicitly considered, Li et al. (2020b) and Zhu et al. (2021) studied federated bandits with DP guarantee. Dubey and Pentland (2022) investigated private and byzantine-proof cooperative decision-making in the bandits setting. Dubey and Pentland (2020) and Zhou and Chowdhury (2023) considered the linear contextual bandit model with joint DP guarantee. Li et al. (2022a) studied private distributed linear bandits with partial feedback. Huang et al. (2023) studied federated linear contextual bandits with user-level DP guarantee.

B Applications of Differentially Private Federated OPE

Differentially private federated online prediction has many important real-world applications. We provide three examples below.

Personalized Healthcare: Consider a federated online prediction setting where patients' wearable devices collect and process health data locally, and the central server aggregates privacy-preserving updates from devices to provide health recommendations or alerts. DP federated online prediction can speed up the learning process and improve prediction accuracy without exposing individual users' health data, thus ensuring patient privacy.

Financial Fraud Detection: DP federated online prediction can also enhance fraud detection systems across banking and financial services. Each client device (e.g. PC) locally analyzes transaction patterns and flags potential fraud without revealing sensitive transaction details to the central server. The server's role is to collect privacy-preserving updates from these clients to improve the global fraud detection model. This method ensures that the financial company can dynamically adapt to new fraudulent tactics, improving detection rates while safeguarding customers' financial privacy.

Personalized Recommender Systems: Each client (e.g. smartphone) can personalize content recommendations by analyzing user interactions and preferences locally. The central server (e.g. company) aggregates privacy-preserving updates from all clients to refine the recommendation model. Thus, DP federated online prediction improves the whole recommender system performance while maintaining each client's privacy.

C Algorithms and Proofs for Fed-DP-OPE-Stoch

C.1 DP-FW

In Fed-DP-OPE-Stoch, we run a DP-FW subroutine (Asi et al., 2021b) at each client i in each phase p . DP-FW maintains T 1 binary trees indexed by 1 ≤ j ≤ T 1 , each with depth j , where T 1 is a predetermined parameter. An example of the tree structure is shown below. We introduce the notation s ∈ { 0 , 1 } ≤ j to denote vertices within binary tree j . ∅ signifies the tree's root. For any s, s ′ ∈ { 0 , 1 } ≤ j , if s = s ′ 0 , then s denotes the left child of s ′ . Conversely, when s = s ′ 1 , s is attributed as the right child of s ′ . For each client i , each vertex s in the binary tree j corresponds to a parameter x i,j,s and a gradient estimate v i,j,s . Each client iteratively updates parameters and gradient estimates by visiting the vertices of a binary tree according to the Depth-First Search (DFS) order: as it visits a left child vertex s , the algorithm maintains the parameter x i,j,s and the gradient estimate v i,j,s identical to those of its parent vertex s ′ , i.e. v i,j,s = v i,j,s ′ and x i,j,s = x i,j,s ′ . As it proceeds to a right child vertex, we uniformly select 2 -| s | b loss functions from set B i,p = { l i, 2 p -2 , . . . , l i, 2 p -1 -1 } to subset B i,j,s without replacement, where b denoting the predetermined batch size and | s | representing the depth of the vertex s . Then the algorithm improves the gradient estimate v i,j,s at the current vertex s of tree j using the estimate v i,j,s ′ at the parent vertex s ′ , i.e.

Algorithm 3 DP-FW at client i (Asi et al., 2021b)

where ∇ l ( · ; B i,j,s ) = 1 |B i,j,s | ∑ l ∈B i,j,s ∇ l ( · ) , and B i,j,s is the subset of loss functions at vertex s in the binary tree j for client i . The full algorithm is shown in Algorithm 3.

∅

0

00

01

1

10

11

C.2 Proof of Theorem 1 and Corollary 1 (for pure DP)

Theorem 5 (Restatement of Theorem 1) . Assume that loss function l i,t ( · ) is convex, α -Lipschitz, β -smooth w.r.t. ∥·∥ 1 . Setting λ i,j,s = 4 α 2 j bε , b = 2 p -1 ( p -1) 2 and T 1 = 1 2 log ( bεβ √ m α log d ) , Fed-DP-OPE-Stoch (i) satisfies ε -DP and (ii) achieves the per-client regret of

with (iii) a communication cost of O ( m 5 4 d √ Tεβ α log d ) .

We define population loss as L t ( x ) = E [ 1 m ∑ m i =1 l i,t ( x i,t )] . The per-client regret can be expressed as E [ ∑ T t =1 L t ( x i,t ) -min x ⋆ ∈X ∑ T t =1 L t ( x ⋆ ) ] . We use v i,p,k , w i,p,k , x i,p,k and η i,p,k to denote quantities corresponding to phase p , iteration k , and client i . Also we introduce some average quantities ¯ v p,k = 1 m ∑ m i =1 v i,p,k and ¯ w p,k = 1 m ∑ m i =1 w i,p,k .

To prove the theorem, we start with Lemma 1 that gives pure privacy guarantees.

Lemma 1. Assume that 2 T 1 ≤ b . Setting λ i,j,s = 4 α 2 j bε , Fed-DP-OPE-Stoch is ε -DP.

Proof. Let S and S ′ be two neighboring datasets and assume that they differ in sample l i 1 ,t 1 or l ′ i 1 ,t 1 , where 2 p 1 -1 ≤ t 1 < 2 p 1 . Let B i 1 ,j,s be the set that contains l i 1 ,t 1 or l ′ i 1 ,t 1 . Recall that |B i 1 ,j,s | = 2 -| s | b . The key point is that this set is used in the calculation of v i 1 ,p 1 ,k for at most 2 j -| s |

iterates, i.e. the leaves that are descendants of the vertex. Let k 0 and k 1 be the first and last iterate such that B i 1 ,j,s is used for the calculation of v i 1 ,p 1 ,k , hence k 1 -k 0 +1 ≤ 2 j -| s | .

First, we show that ( ⟨ c n , v 1 , 1 , 1 ⟩ , . . . , ⟨ c n , v m,P,K ⟩ ) ≈ ( ε, 0) ( ⟨ c n , v ′ 1 , 1 , 1 ⟩ , . . . , ⟨ c n , v ′ m,P,K ⟩ ) holds for 1 ≤ n ≤ d . For our purpose, it suffices to establish that ( ⟨ c n , v i 1 ,p 1 ,k 0 ⟩ , . . . , ⟨ c n , v i 1 ,p 1 ,k 1 ⟩ ) ≈ ( ε, 0) ( ⟨ c n , v ′ i 1 ,p 1 ,k 0 ⟩ , . . . , ⟨ c n , v ′ i 1 ,p 1 ,k 1 ⟩ ) holds for 1 ≤ n ≤ d , since l i 1 ,t 1 or l ′ i 1 ,t 1 is only used in client i 1 , at phase p 1 and iteration k 0 , . . . , k 1 . Therefore it is enough to show that ⟨ c n , v i 1 ,p 1 ,k ⟩ ≈ ( ε 2 j -| s | , 0) ⟨ c n , v ′ i 1 ,p 1 ,k ⟩ holds for k 0 ≤ k ≤ k 1 and 1 ≤ n ≤ d , because k 1 -k 0 + 1 ≤ 2 j -| s | . Note that |⟨ c n , v i 1 ,p 1 ,k -v ′ i 1 ,p 1 ,k ⟩| ≤ 4 α 2 -| s | b . Setting λ i,j,s = 4 α 2 j bε and applying standard results of Laplace mechanism (Dwork et al. (2014)) lead to our intended results. Finally, we can show that ( x 1 , 1 ,K , . . . , x m,P,K ) ≈ ( ε, 0) ( x ′ 1 , 1 ,K , . . . , x ′ m,P,K ) by post-processing.

To help prove the upper bound of the regret, we introduce some lemmas first.

Lemma 2. Let t be the time-step, p be the index of phases, i be the index of the clients, and ( j, s ) be a vertex. For every index 1 ≤ k ≤ d of the vectors, we have

where ¯ v p,j,s = 1 m ∑ m i =1 v i,p,j,s .

Proof. Let us fix p , t , k and i for simplicity and let A j,s = ¯ v p,j,s,k -∇ L t,k ( x i,p,j,s ) . We prove the lemma by induction on the depth of the vertex, i.e., | s | . If | s | = 0 , then v i,p,j, ∅ = ∇ l ( x i,p,j, ∅ ; B i,p,j, ∅ ) where B i,p,j, ∅ is a sample set of size b . Therefore we have

where the last inequality holds because for a random variable X ∈ [ -α, α ] , we have E [exp c ( X -E [ X ])] ≤ exp ( c 2 α 2 2 ) .

Assume the depth of the vertex | s | ≥ 0 and let s = s ′ a where a ∈ { 0 , 1 } . If a = 0 , clearly the lemma holds. If a = 1 , recall that v i,p,j,s = v i,p,j,s ′ + ∇ l ( x i,p,j,s ; B i,p,j,s ) -∇ l ( x i,p,j,s ′ ; B i,p,j,s ) , then

Let B i,p,< ( j,s ) = ∪ ( j 1 ,s 1 ) < ( j,s ) B i,p,j 1 ,s 1 be the set containing all the samples used up to vertex ( j, s ) in phase p at client i . We have

Since l i,t ( · ; s ) is β -smooth w.r.t. ∥ · ∥ 1 , we have

Since vertex ( j, s ) is the right son of vertex ( j, s ′ ) , the number of updates between x i,p,j,s and x i,p,j,s ′ is at most the number of leafs visited between these two vertices i.e. 2 j -| s | . Therefore we have

By using similar arguments to the case | s | = 0 , we can get

Then we get

Applying this inductively, we have for every index 1 ≤ k ≤ d

Lemma 3 upper bounds the variance of the average gradient.

Lemma 3. At phase p , for each vertex ( j, s ) and 2 p -1 ≤ t < 2 p -1 , we have

where ¯ v p,j,s = 1 m ∑ m i =1 v i,p,j,s .

Proof. Lemma 2 implies that ¯ v p,j,s,k -∇ L t,k ( x i,p,j,s ) is O ( α 2 + β 2 bm ) -sub-Gaussian for every index 1 ≤ k ≤ d of the vectors. Applying standard results of the maximum of d sub-Gaussian random variables, we get

Lemma 4 gives the tail bound of the sum of i.i.d. random variables following Laplace distribution. Lemma 4. Let ξ i,n be IID random variables following the distribution Lap ( λ i,j,s ) . Then we have

Proof. ξ i,n 's are md IID random variables following the distribution Lap ( λ i,j,s ) . We note that

Since 1 1 -λ i,j,s 2 u 2 ≤ 1 + 2 λ i,j,s 2 u 2 ≤ exp(2 λ i,j,s 2 u 2 ) , when | u | ≤ 1 2 λ i,j,s , ξ i,n is sub-exponential with parameter (4 λ i,j,s 2 , 2 λ i,j,s ) . Applying standard results of linear combination of sub-exponential random variables, we can conclude that ∑ m i =1 ξ i,n , denoted as Y n , is sub-exponential with parameter (4 mλ i,j,s 2 , 2 λ i,j,s ) . From standard results of tail bounds of sub-exponential random variables, we have

Then we have

Lemma 5. Setting λ i,j,s = 4 α 2 j bε , we have

Proof. Since ¯ w p,k = arg min c n :1 ≤ n ≤ d [ 1 m ∑ m i =1 ( ⟨ c n , v i,p,k ⟩ + ξ i,n )] , where ξ i,n ∼ Lap ( λ i,j,s ) , we denote ¯ w p,k as c n ⋆ and we have

Applying Lemma 4, we get

With the lemmas above, we are ready to prove Theorem 1.

Proof. Lemma 1 implies the claim about privacy. We proceed to prove the regret upper bound.

where ( a ) is due to β -smoothness, ( b ) is because of the updating rule of x i,p,k , and ( c ) follows from the convexity of the loss function and Hölder's inequality.

Subtracting L t ( x ⋆ ) from each side and taking expectations, we have

Applying Lemma 5 and Lemma 3, we have

Let α k = η i,p,k ( α + β ) O (√ log d bm ) + η i,p,k α 2 j bε O ( ln d √ m ) + 1 2 βη i,p,k 2 . We simplify the notion of η i,p,k to η k . Then we have

where η k = 2 k +1 .

Since K = 2 T 1 , simply algebra implies that

At iteration 2 p -1 ≤ t < 2 p , setting b = 2 p -1 ( p -1) 2 and T 1 = 1 2 log ( bεβ √ m α log d ) in Equation (4), we have

Therefore, the total regret from time step 2 p to 2 p +1 -1 is at most

Summing over p , we can get

Now we turn our focus to communication cost. Since there are log T phases, and within each phase, there are O (2 T 1 ) leaf vertices where communication is initiated, the communication frequency scales in O ( ∑ p 2 T 1 ) . Therefore, the communication cost scales in O ( m 5 / 4 d √ Tεβ/ ( α log d )) .

Corollary 2 (Restatement of Corollary 1) . If β = O ( log d √ mTε ) , then, Fed-DP-OPE-Stoch (i) satisfies ε -DP and (ii) achieves the per-client regret of

with (iii) a communication cost of O ( md log T ) .

Proof. Similar to the proof of Theorem 1, we can show that

At iteration 2 p -1 ≤ t < 2 p , setting b = 2 p -1 and T 1 = 1 in Equation (5), we have

Since β = O ( log d √ mTε ) , we have

Therefore, the total regret from time step 2 p to 2 p +1 is at most

Summing over p , we can get

The proof of the DP guarantee and communication costs follows from the proof of Theorem 1.

C.3 Theorem 6 (for approximate DP)

Theorem 6. Let δ ≤ 1 /T . Assume that loss function l i,t ( · ) is convex, α -Lipschitz, β -smooth w.r.t. ∥ · ∥ 1 . Assume ε ≤ ( α log(2 p -1 /δ ) log d ) 1 / 4 √ p -1 ( β 2 p -1 ) 1 / 4 . Setting λ i,j,s = α 2 T 1 / 2 log(2 p -1 /δ ) bε , b = 2 p -1 ( p -1) 2 , and T 1 = 2 3 log ( bε √ mβ α log(2 p -1 /δ ) log d ) , Fed-DP-OPE-Stoch is ( ε, δ ) -DP, the communication cost scales in O ( md 2 T 1 log T ) and the per-client regret is upper bounded by

Proof. We apply the following lemma to prove privacy in this setting.

Lemma 6 (Asi et al. (2021b), Lemma 4.4) . Let b ≥ 2 T 1 , δ ≤ 1 /T and ε ≤ √ 2 -T 1 log(1 /δ ) . Setting λ i,j,s = α 2 T 1 / 2 log(2 p -1 /δ ) bε , we have Fed-DP-OPE is ( O ( ε ) , O ( δ )) -DP.

Theorem 6 then follows using similar arguments to the proof of Theorem 1.

C.4 Extension to Central DP setting

In this section, we extend our Fed-DP-OPE-Stoch algorithm to the setting where client-server communication is secure with some modifications, achieving better utility performance.

We present the algorithm design in Algorithm 4 and Algorithm 5. We change the local privatization process to a global privatization process on the server side. Specifically, in Line 8, when communication is triggered, each client sends {⟨ c n , v i,j,s ⟩} n ∈ [ d ] to the server, where c 1 , . . . , c d represents d vertices of decision set X = ∆ d . In Line 5, after receiving {⟨ c n , v i,j,s ⟩} n ∈ [ d ] from all clients, the central server privately predicts a new expert:

where ζ n ∼ Lap ( µ j,s ) and µ j,s = 4 α 2 j bmε . Other components remain the same.

Algorithm 4 Fed-DP-OPE-Stoch (CDP): Client i

Algorithm 5 Fed-DP-OPE-Stoch (CDP): Central server

Now we present the theoretical guarantees in this setting.

Theorem 7. Assume that loss function l i,t ( · ) is convex, α -Lipschitz, β -smooth w.r.t. ∥ · ∥ 1 . Fed-DPOPE-Stoch (i) satisfies ε -DP and (ii) achieves the per-client regret of

with (iii) a communication cost of O ( m 3 2 d √ Tεβ α log d ) .

To prove the theorem, we start with Lemma 7 that gives pure privacy guarantees.

Proof. Let S and S ′ be two neighboring datasets and assume that they differ in sample l i 1 ,t 1 or l ′ i 1 ,t 1 , where 2 p 1 -1 ≤ t 1 < 2 p 1 . The algorithm is ε -DP if we have ( x 1 , 1 ,K , . . . , x m,P,K ) ≈ ( ε, 0) ( x ′ 1 , 1 ,K , . . . , x ′ m,P,K ) .

Let B i 1 ,j,s be the set that contains l i 1 ,t 1 or l ′ i 1 ,t 1 . Recall that |B i 1 ,j,s | = 2 -| s | b . The key point is that this set is used in the calculation of v i 1 ,p 1 ,k for at most 2 j -| s | iterates, i.e. the leaves that are descendants of the vertex. Let k 0 and k 1 be the first and last iterate such that B i 1 ,j,s is used for the calculation of v i 1 ,p 1 ,k , hence k 1 -k 0 +1 ≤ 2 j -| s | . For a sequence a i , . . . , a j , we use the shorthand a i : j = { a i , . . . , a j } .

Step 1: x 1 : m , p 1 , k 0 : k 1 ≈ ( ε, 0 ) x ′ 1 : m , p 1 , k 0 : k 1 and ¯ w p 1 , k 0 : k 1 ≈ ( ε, 0 ) ¯ w ′ p 1 , k 0 : k 1 by basic composition, post-processing and report noisy max

Wewill show that ( x 1 ,p 1 ,k 0 , . . . , x m,p 1 ,k 1 ) and ( x ′ 1 ,p 1 ,k 0 , . . . , x ′ m,p 1 ,k 1 ) are ε -indistinguishable. Since B i 1 ,j,s is used to calculate v i 1 ,p 1 ,k for at most 2 j -| s | iterates, i.e. k 1 -k 0 +1 ≤ 2 j -| s | , it is enough to show that ¯ w p 1 ,k ≈ ( ε 2 j -| s | , 0) ¯ w ′ p 1 ,k for k 0 ≤ k ≤ k 1 and then apply basic composition and postprocessing. Note that for every k 0 ≤ k ≤ k 1 , the sensitivity |⟨ c n , v i 1 ,p 1 ,k -v ′ i 1 ,p 1 ,k ⟩| ≤ 4 α 2 -| s | b , therefore, | 1 m ∑ m i =1 ⟨ c n , v i,p 1 ,k ⟩ -1 m ∑ m i =1 ⟨ c n , v ′ i,p 1 ,k ⟩| ≤ 4 α 2 -| s | mb . Using privacy guarantees of report noisy max (Lemma 15), we have ¯ w p 1 ,k ≈ ( ε 2 j -| s | , 0) ¯ w ′ p 1 ,k for k 0 ≤ k ≤ k 1 with µ j,s = 4 α 2 j bmε .

Step 2: x 1 : m , 1 : P , K ≈ ( ε, 0 ) x ′ 1 : m , 1 : P , K by post-processing

In order to show ( x 1 , 1 ,K , . . . , x m,P,K ) ≈ ( ε, 0) ( x ′ 1 , 1 ,K , . . . , x ′ m,P,K ) , we only need to prove that ( x 1 ,p 1 ,K , . . . , x m,p 1 ,K ) ≈ ( ε, 0) ( x ′ 1 ,p 1 ,K , . . . , x ′ m,p 1 ,K ) and apply post-processing. It is enough to show that iterates ( x 1 ,p 1 , 1 , . . . , x m,p 1 ,K ) and ( x ′ 1 ,p 1 , 1 , . . . , x ′ m,p 1 ,K ) is ε -indistinguishable.

The iterates x 1: m,p 1 , 1: k 0 -1 and x ′ 1: m,p 1 , 1: k 0 -1 do not depend on l i 1 ,t 1 or l ′ i 1 ,t 1 , hence 0indistinguishable. Moreover, given that ( x 1 ,p 1 ,k 0 , . . . , x m,p 1 ,k 1 ) and ( x ′ 1 ,p 1 ,k 0 , . . . , x ′ m,p 1 ,k 1 ) are ε -indistinguishable, it is clear that ( x 1 ,p 1 ,k 1 +1 , . . . , x m,p 1 ,K ) and ( x ′ 1 ,p 1 ,k 1 +1 , . . . , x ′ m,p 1 ,K ) are ε -indistinguishable by post-processing.

To help prove the upper bound of the regret, we introduce some lemmas first.

Lemma 8. Setting µ j,s = 4 α 2 j mbε , we have

Proof. Since ¯ w p,k = arg min c n :1 ≤ n ≤ d [ 1 m ∑ m i =1 ⟨ c n , v i,p,k ⟩ + ζ n ] , where ζ n ∼ Lap ( µ j,s ) , we denote ¯ w p,k as c n ⋆ and we have

Standard results for the expectation of maximum of d i.i.d. Laplace random variables imply that E [ max n :1 ≤ n ≤ d | ζ n | ] ≤ O ( µ j,s ln d ) . Therefore,

Then we are ready to prove Theorem 7.

Proof. Lemma 7 implies the claim about privacy. Following the same arguments in the proof of Theorem 1, we can get

Applying Lemma 8 and Lemma 3, we have

Let α k = η i,p,k ( α + β ) O (√ log d bm ) + η i,p,k α 2 j bε O ( ln d m ) + 1 2 βη i,p,k 2 . We simplify the notion of η i,p,k to η k . Then we have

where η k = 2 k +1 .

Since K = 2 T 1 , simply algebra implies that

At iteration 2 p -1 ≤ t < 2 p , setting b = 2 p -1 ( p -1) 2 and T 1 = 1 2 log ( bεβm α log d ) , we have

Summing over all the timesteps, we have

The proof of communication cost is similar to that in proof of Theorem 1.

D Proof of Lower Bounds

Theorem 8 (Restatement of Theorem 2) . For any federated OPE algorithm against oblivious adversaries, the per-client regret is lower bounded by Ω( √ T log d ) . Let ε ∈ (0 , 1] and δ = o (1 /T ) , for any ( ε, δ ) -DP federated OPE algorithm, the per-client regret is lower bounded by Ω ( min ( log d ε , T )) .

Proof. Consider the case when all clients receive the same loss function from the oblivious adversary at each time step, i.e. l i,t = l ′ t . Then we define the average policy among all clients p ′ t ( k ) = 1 m ∑ m i =1 p i,t ( k ) , ∀ k ∈ [ d ] . Now the regret is

Therefore, we have

From Lemma 9, if ε ∈ (0 , 1] and δ = o (1 /T ) , then there exists a sequence of losses l ′ 1 , l ′ 2 , . . . , l ′ t such that for any ( ε, δ ) -DP algorithm for online learning with expert advice problem against oblivious adversaries, the expected regret satisfies

Therefore, we have for any ( ε, δ ) -DP algorithm,

Lemma 9. Let ε ∈ (0 , 1] and δ = o (1 /T ) . There exists a sequence of losses l 1 , . . . , l T such that for any ( ε, δ ) -DP algorithm A for DP-OPE against oblivious adversaries satisfies

Proof. Let n, d ∈ N . Define y ∈ Y n containing n records, where Y = { 0 , 1 } d . The function 1 -Select d : Y n → [ d ] corresponds to selecting a coordinate b ∈ [ d ] in the batch model.

Then we define the regret of 1 -Select d . For a batched algorithm M with input dataset y and output b ∈ [ d ] , define

Let A be an ( ε, δ ) -DP algorithm ( { 0 , 1 } d ) T → ([ d ]) T for DP-OPE against oblivious adversaries with regret ∑ T t =1 l t ( x t ) -min x ⋆ ∈ [ d ] ∑ T t =1 l t ( x ⋆ ) ≤ α . Setting T = n , we can use A to construct

Note that p ′ t ( k ) is defined by p 1 ,t ( k ) , . . . , p m,t ( k ) , which in turn are determined by l 1 , 1 , . . . , l m,t -1 . According to our choice of l i,t = l ′ t , p ′ t ( k ) is determined by l ′ 1 , l ′ 2 , . . . , l ′ t -1 . Therefore p ′ 1 , p ′ 2 , . . . , p ′ t are generated by a legitimate algorithm for online learning with expert advice problems.

There exists a sequence of losses l ′ 1 , l ′ 2 , . . . , l ′ t such that for any algorithm for online learning with expert advice problem, the expected regret satisfies (Cesa-Bianchi and Lugosi (2006), Theorem 3.7)

an ( ε, δ ) -DP algorithm M for 1 -Select d in the batch model. The details of the algorithm appear in Algorithm 6.

Algorithm 6 Batch algorithm M for 1 -Select (Jain et al. (2023), Algorithm 2 with k = 1 )

1: Input: y = ( y 1 , . . . , y n ) ∈ Y n , where Y = { 0 , 1 } d , and black-box access to a DP-OPE algorithm A for oblivious adversaries.

2: Construct a stream z ← y with n records.

3: for t = 1 to n do

4: Send the record z t to A and get the corresponding output x t .

5: end for

6: Output b = x n .

Let ε > 0 , α ∈ R + , and T, d, n ∈ N , where T = n . If a DP-OPE algorithm A : ( { 0 , 1 } d ) T → ([ d ]) T for oblivious adversaries is ( ε, δ )-DP and the regret is upper bounded by α , i.e. ∑ T t =1 l t ( x t ) -min x ⋆ ∈ [ d ] ∑ T t =1 l t ( x ⋆ ) ≤ α , then by Lemma 10, we have the batch algorithm M for 1 -Select d is ( ε, δ ) -DP and Reg 1 -Select d ( M ) ≤ α n .

If δ = o (1 /T ) , then n = Ω ( n log d εα ) (Lemma 11). We have α = min ( Ω ( log d ε ) , n ) = min ( Ω ( log d ε ) , T ) . So α ≥ Ω ( min ( log d ε , T )) . Therefore, if an algorithm for DP-OPE against oblivious adversaries is ( ε, δ ) -differentially private and ∑ T t =1 l t ( x t ) -min x ⋆ ∈ [ d ] ∑ T t =1 l t ( x ⋆ ) ≤ α holds, then α ≥ Ω ( min ( log d ε , T )) . This means that there exists a sequence of loss functions l 1 , . . . , l T such that ∑ T t =1 l t ( x t ) -min x ⋆ ∈ [ d ] ∑ T t =1 l t ( x ⋆ ) ≥ Ω ( min ( log d ε , T )) .

Lemma 10. Let M be the batch algorithm for 1 -Select d . For all ε > 0 , δ ≥ 0 , α ∈ R + , and T, d, n ∈ N , where T = n , if a DP-OPE algorithm A : ( { 0 , 1 } d ) T → ([ d ]) T for oblivious adversaries is ( ε, δ ) -differentially private and the regret is upper bounded by α , i.e. ∑ T t =1 l t ( x t ) -min x ⋆ ∈ [ d ] ∑ T t =1 l t ( x ⋆ ) ≤ α , then batch algorithm M for 1 -Select d is ( ε, δ ) -differentially private and Reg 1 -Select d ( M ) ≤ α n .

Proof. DP guarantee: Fix neighboring datasets y and y ′ that are inputs to algorithm M . According to the algorithm design for 1 -Select d , we stream y and y ′ to a DP-OPE algorithm A . Since A is ( ε, δ ) -DP, and M only post-processes the outputs received from A , therefore M is ( ε, δ ) -DP.

Regret upper bound: Fix a dataset y . Note that if α ≥ n , the accuracy guarantee for M is vacuous. Now assume α < n . Let γ be the regret of M , that is,

where b is the output of M .

The main observation in the regret analysis is that if α is small, so is γ . Specifically, the regret of M is at most α n . Therefore, γ ≤ α n .

Lemma 11 (Jain et al. (2023), Lemma 4.2) . For all d, n ∈ N , ε ∈ (0 , 1] , δ = o (1 /n ) , γ ∈ [ 0 , 1 20 ] , and ( ε, δ )-DP 1 -Select d algorithms M : ( { 0 , 1 } d ) n → [ d ] with Reg 1 -Select d ( M ) ≤ γ , we have n = Ω ( log d εγ ) .

E Algorithms and Proofs for Fed-SVT

E.1 Proof of Theorem 3

Theorem 9 (Restatement of Theorem 3) . Let ε ≤ 1 and δ ≤ ε/d . For any ( ε, δ ) -DP federated OPE algorithm against oblivious adversaries in the realizable setting, the per-client regret is lower bounded by Ω ( log( d ) mε ) .

Proof. I introduce two prototype loss functions first: let l 0 ( x ) = 0 for all x ∈ [ d ] and for j ∈ [ d ] let l j ( x ) be the function that has l j ( x ) = 0 for x = j and otherwise l j ( x ) = 1 . Then we define d loss sequences S j = ( l j 1 , 1 , . . . , l j m,T ) , such that

where k = log d 2 mε and j ∈ [ d ] .

The oblivious adversary picks one of the d sequences S 1 , . . . , S d uniformly at random. Assume towards a contradiction that E [ ∑ m i =1 ∑ T t =1 l i,t ( x i,t ) -∑ m i =1 ∑ T t =1 l i,t ( x ⋆ ) ] ≤ log( d ) 32 ε . This implies that there exists d/ 2 sequences such that the expected regret satisfies E [ ∑ m i =1 ∑ T t =1 l j i,t ( x i,t ) -∑ m i =1 ∑ T t =1 l j i,t ( x ⋆ ) ] ≤ log( d ) 16 ε . Assume without loss of generality these sequences are S 1 , . . . , S d/ 2 . Let B j be the set of outputs that has low regret on S j , that is,

Note that B j ∩ B j ′ = ∅ since if x 1: m, 1: T ∈ B j then among the mk outputs x 1: m,T -k +1: T at least 3 mk 4 = 3 log( d ) 8 ε of them must be equal to j . Now Markov inequality implies that

Moreover, group privacy gives

where the last inequality is due to δ ≤ ε/d . Overall we get that

which is a contradiction for d ≥ 32 .

E.2 Algorithm Design of Fed-SVT

Fed-SVT contains a client-side subroutine (Algorithm 7) and a server-side subroutine (Algorithm 8).

E.3 Proof of Theorem 4

Theorem 10 (Restatement of Theorem 4) . Let l i,t ∈ [0 , 1] d be chosen by an oblivious adversary under near-realizability assumption. Set 0 < ρ < 1 / 2 , κ = O (log( d/ρ )) , L = mL ⋆ + 8 log(2 T 2 / ( N 2 ρ )) ε +4 /η , and η = ε/ 2 κ . Then the algorithm is ε -DP, the communication cost

Algorithm 7 Fed-SVT: Client i

Algorithm 8 Fed-SVT: Central server

scales in O ( mdT/N ) , and with probability at least 1 -O ( ρ ) , the pre-client regret is upper bounded by O ( log 2 ( d )+log ( T 2 N 2 ρ ) log ( d ρ ) mε +( N + L ⋆ ) log ( d ρ ) ) .

Moreover, setting η = ε/ √ κ log(1 /δ ) , we have the algorithm is ( ε, δ ) -DP, the communication cost scales in O ( mdT/N ) , and with probability at least 1 -O ( ρ ) , the pre-client regret is upper bounded

Proof. DP guarantee: There are κ applications of exponential mechanism with privacy parameter η . Moreover, sparse vector technique is applied over each sample once, hence the κ applications of sparse-vector are ε/ 2 -DP. Overall, the algorithm is ( ε/ 2 + κη ) -DP and ( ε/ 2 + √ 2 κ log(1 /δ ) η + κη ( e η -1) , δ ) -DP (Lemma 14). Setting η = ε/ 2 κ results in ε -DP and η = ε/ √ κ log(1 /δ ) results in ( ε, δ ) -DP.

Communication cost: The number of communication between the central server and clients scales in O ( mT/N ) . Moreover, within each communication, the number of scalars exchanged scales in O ( d ) . Therefore the communication cost is O ( mdT/N ) .

Regret upper bound: We define a potential at phase n ∈ [ T/N ] :

where L n ( x ) = max ( ∑ m i =1 ∑ nN -1 t ′ =1 l i,t ′ ( x ) , mL ⋆ ) . Note that ϕ 0 = de -ηmL ⋆ / 2 and ϕ n ≥ e -ηmL ⋆ / 2 for all n ∈ [ T/N ] since there is x ∈ [ d ] such that ∑ m i =1 ∑ T t =1 l i,t ( x ) ≤ mL ⋆ . We split the iterates to s = ⌈ log d ⌉ rounds n 0 N,n 1 N,... , n s N where n p is the largest n ∈ [ T/N ] such that ϕ n p ≥ ϕ 0 / 2 p . Let Z p be the number of switches in [ n p N, ( n p +1 -1) N ] (number of times the exponential mechanism is used to pick x t ). Let Z = ∑ s -1 p =0 Z p be the total number of switches. Note that Z ≤ 3 s + ∑ s -1 p =0 max( Z p -3 , 0) and Lemma 12 implies max( Z p -3 , 0) is upper bounded by a geometric random variable with success probability 1 / 3 . Therefore, using concentration of geometric random variables (Lemma 13), we get that

Since K ≥ 3 s +24log(1 /ρ ) , the algorithm does not reach the switching budget with probability 1 -O ( ρ ) . So the total number of switching scales as O (log( d/ρ )) . Now we analyze the regret. Define T 1 N,... T C N as the switching time steps with T C N = T , where C = O (log( d/ρ )) . Lemma 16 implies that with probability at least 1 -ρ ,

Case 1: Setting η = ε/ 2 κ in Equation (8), we have

Case 2: Setting η = ε/ √ κ log(1 /δ ) in Equation (8), we have

Lemma 12. Fix 0 ≤ p ≤ s -1 . Then for any 1 ≤ k ≤ T/N , it holds that

Proof. Let n p N ≤ nN ≤ n p +1 N be a time-step when a switch happens (exponential mechanism is used to pick x t ). Note that ϕ n p +1 ≥ ϕ n / 2 . We prove that the probability that x t is switched between nN and n p +1 N is at most 2 / 3 . To this end, note that if x t is switched before n p +1 N then ∑ m i =1 ∑ n p +1 N -1 t ′ = nN l i,t ′ ( x t ′ ) ≥ L -8 log(2 T 2 / ( N 2 ρ )) ε , therefore L n p +1 ( x ) -L n ( x ) ≥ L -8 log(2 T 2 / ( N 2 ρ )) ε ≥ 4 /η . Thus we have that

where the second inequality follows the fact that 1 { a ≥ b } ≤ 1 -e -ηa 1 -e -ηb for a, b, η ≥ 0 , and the last inequality since ϕ n p +1 /ϕ n ≥ 1 / 2 . This argument shows that after the first switch inside the range [ n p N,n p +1 N -1] , each additional switch happens with probability at most 2 / 3 . So we have

Lemma 13 (Asi et al. (2023), Lemma A.2) . Let W 1 , . . . , W n be i.i.d. geometric random variables with success probability p . Let W = ∑ n i =1 W i . Then for any k ≥ n

F Experimental Supplementary

The simulations were conducted on a system with a 2.3 GHz Dual-Core Intel Core i5, Intel Iris Plus Graphics 640 with 1536 MB, and 16 GB of 2133 MHz LPDDR3 RAM. Approximately 10 minutes are required to reproduce the experiments.

We present our numerical results with different seeds in Figure 4 and Figure 5.

G Experiments on MovieLens-1M

We use the MovieLens-1M dataset (Harper and Konstan, 2015) to evaluate the performances of FedSVT, comparing it with the single-player model Sparse-Vector (Asi et al., 2023). We first compute the rating matrix of 6040 users to 18 movie genres (experts) R = [ r u,g ] ∈ R 6040 × 18 , and then calculate L = [max(0 , r u,g ⋆ -r u,g )] ∈ R 6040 × 18 where g ⋆ = arg max g ( 1 6040 ∑ 6040 u =1 r u,g ) . We generate the sequence of loss functions { l u } u ∈ [6040] where l u = L u, : . In our experiments, we set m = 10 , T = 604 , ε = 10 , δ = 0 and run 10 trials. In Fed-SVT, we experiment with communication intervals N = 1 , 30 , 50 , where communication cost scales in O ( mdT/N ) . The per-client cumulative regret as a function of T is plotted in Figure 6. Our results show that Fed-SVT significantly outperforms Sparse-Vector with low communication costs (notably in the N = 50 case). These results demonstrate the effectiveness of our algorithm in real-world applications.

Stochastic(m=10,ε=10,d=100)

Stochastic(m=10,ε=10,d=100)

Stochastic(m=10,ε=10,d=100)

30

100

80

25

Per-client Regret

Per-client Regret

80

60

60

15

40

40

10

20

Limited Updates

20

Limited Updates

5

Limited Updates

0

Fed-DP-OPE-Stoch

0

Fed-DP-OPE-Stoch

0

Fed-DP-OPE-Stoch

0

2500

5000

7500

10000

12500

15000

2500

5000

7500

10000

12500 15000

0

2500

5000

7500

10000

12500 15000

T

T

T

Stochastic(m=10,ε=10,d=100)

Stochastic(m=10,ε=10,d=100)

Stochastic(m=10,ε=10,d=100)

80

40

40

Per-client Regret

30

40

20

20

10

20

10

Limited Updates

Limited Updates

Limited Updates

0

Fed-DP-OPE-Stoch

0

Fed-DP-OPE-Stoch

0

Fed-DP-OPE-Stoch

0

2500

5000

7500

10000

12500 15000

0

2500

5000

7500

10000

12500 15000

0

2500

5000

7500

10000

1250015000

T

T

T

Oblivious(Realizable) (m=10,ε=10,d=100)

Oblivious (Realizable)(m=10,ε=10,d=100)

Oblivious(Realizable)(m=10,ε=10,d=100)

100

100

80

Per-client Regret

80

Per-client Regret

80

60

60

40

40

40

Sparse-Vector

Sparse-Vector

Sparse-Vector

20

Fed-SVT (N = 50)

20

Fed-SVT (N= 50)

20

Fed-SVT (N =50)

Fed-SVT (N = 30)

Fed-SVT (N=30)

Fed-SVT (N = 30)

0

Fed-SVT (N = 1)

0

Fed-SVT (N = 1)

0

Fed-SVT (N = 1)

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

T

T

T

Oblivious (Realizable)(m=10,ε=10,d=100)

Oblivious (Realizable)(m =10,ε=10, d=100)

Oblivious (Realizable) (m=10,ε=10, d=100)

70

100

100

60

Per-client Regret

80

Per-client Regret

80

Per-client Regret

50

60

60

40

40

Sparse-Vector

Sparse-Vector

20

Sparse-Vector

20

Fed-SVT (N=30)

Fed-SVT (N = 50)

20

Fed-SVT (N= 50)

Fed-SVT (N=30)

10

Fed-SVT (N = 30)

Fed-SVT (N =50)

Fed-SVT (N = 1)

Fed-SVT (N =1)

Fed-SVT (N = 1)

0

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

T

T

T

H Background on Differential Privacy

H.1 Advanced Composition

Lemma 14 (Advanced composition, Dwork et al. (2014)) . If A 1 , . . . , A k are randomized algorithms that each is ( ε, δ ) -DP, then their composition ( A 1 ( S ) , . . . , A k ( S )) is ( √ 2 k log(1 /δ ′ ) ε + kε ( e ε -1) , δ ′ + kδ ) -DP.

H.2 Report Noisy Max

The "Report Noisy Max" mechanism can be used to privately identify the counting query among m queries with the highest value. This mechanism achieves this by adding Laplace noise independently generated from Lap (∆ /ε ) to each count and subsequently determining the index corresponding to the largest noisy count (we ignore the possibility of a tie), where ∆ is the sensitivity of the queries. Report noisy max gives us the following guarantee.

MovieLens-1M(m=10,ε=10)

50

Regret

40

Sparse-Vector

30

Per-client F

Fed-SVT(N=50)

Fed-SVT (N=30)

20

Fed-SVT (N=1)

10

0

0

100

200

300

400

500

600

T

Lemma 15 (Dwork et al. (2014), Claim 3.9) . The Report Noisy Max algorithm is ε -differentially private.

H.3 Sparse Vector Technique

We recall the sparse-vector technique here. Given an input S = ( z 1 , . . . , z n ) ∈ Z n , the algorithm takes a stream of queries q 1 , q 2 , . . . , q T in an online manner. We assume that each q i is 1-sensitive, i.e., | q i ( S ) -q i ( S ′ ) | ≤ 1 for neighboring datasets S , S ′ ∈ Z n that differ in a single element. We have the following guarantee.

Lemma 16 (Dwork et al. (2014), Theorem 3.24) . Let S = ( z 1 , . . . , z n ) ∈ Z n . For a threshold L and ρ > 0 , there is an ε -DP algorithm (AboveThreshold) that halts at time k ∈ [ T +1] such that for α = 8(log T +log(2 /ρ )) ε with probability at least 1 -ρ , we have q i ( S ) ≤ L + α for all t < k , and q k ( S ) ≥ L -α or k = T +1 .

I Broader Impacts

This work improves online learning and decision-making through collaboration among multiple users without exposing personal information, which helps balance the benefits of big data with the need to protect individual privacy, promoting ethical data usage and fostering societal trust in an increasingly data-driven world.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Sections 3, 4 and 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We make sure to follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: We do not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of online prediction from experts (OPE) under differential privacy (DP) constraints in a federated setting. OPE operates in a set of rounds, and consists on choosing at each round the expert that minimizes the regret over observations of data. The selection of the expert at each round is based on previous adversarially chosen observations of data. Differentially private OPE (DP-OPE) ensures that the selection of experts is not significantly different (up to $\\epsilon$ and $\\delta$ parameters) if one observation of the data changes. \n\nThe current protocol studies private OPE in the federated setting (Fed-DP-OPE), where the expert selection is done by a set of clients that collaborate via a server. Under local DP (i.e. all messages between server and clients are part of the output of the protocol) and $m$ clients, the current work provides fundamental bounds of over the impact of the federated collaboration on the regret. First, it proposes a protocol that achieves an order-wise $\\sqrt{m}$ multiplicative regret reduction with respect to DP-OPE when the adversarial observations are chosen randomly. Second, it shows that the communication cost is logarithmic in the number of expert selection rounds. Third, it shows that under an oblivious adversary that chooses the the observations arbitrarily in advance, it is not possible to obtain an order-wise improvement regret in Fed-DP-OPE over DP-OPE. Finally, it shows that improvement is possible in the federated setting for a relaxation of the oblivious adversary.\n\nSTRENGTHS:\nThe paper is in general clearly written.\n\nThe contributions provide a significant impact, showing that federated collaboration can improve accuracy under privacy constraints without incurring in large communication cost (i.e. the cost is logarithmic in $T$ for the stochastic adversary setting). The $\\sqrt{m}$ factor speedup is significant. The relaxation of the oblivious adversary seems realistic, therefore improvements in this setting seem also important. \n\nThe paper also proposes novel techniques. Although the natural adaptation to federated setting and the use tree based private aggregation have been widely studied in the past, the paper also proposes novel ways to reduce (i) communication in the stochastic adversary setting, (ii)  regret and communication in the (relaxed) oblivious adversary setting and (iii) a novel proof technique for the lower bounds on the classical oblivious case. \n\nThe work is of good quality, appropriately backing it's claims with proofs and empirically illustrating the impact of the results with reproducible experiments.\n\nWEAKNESSES:\nThere are certain non-major weaknesses in motivating the problem and clarity at preliminaries:  \n\n1- The implications of accurate OPE could be better motivated in the introduction. \n\n2- The notion of DP in the online setting could be better clarified. First, a link between the chosen sequence of loss functions and possibly sensitive data is missing. Second, the definition specifies \"DP against an adversary\". The same terminology for DP adversary and OPE adversary in charge of selecting observations over the sensitive dataset is confusing.\n\nQUESTIONS:\nCould you please clarify what is the link between sensitive data and the notion of neighboring input used in the paper? \n\nFor instance, it would seem reasonable that the sequence $\\mathcal{S}$ of loss functions chosen by the adversary depend on a set of sensitive data points. Then, it could be possible that more than one loss function changes if a single entry of this dataset changes. However, your setting is a bit different and I am wondering where could it be applicable.\n\nLIMITATIONS:\nThe limitations of the work have been appropriately addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of online federated expert selection. In order to make the proposed algorithm robust against adversaries, the paper proposes algorithms with differential privacy guarantees. Both stochastic and oblivious adversaries are investigated by the paper. The paper provides theoretical guarantees for the proposed algorithms. The paper provides experimental results to verify theoretical findings.\n\nSTRENGTHS:\n1. The paper proposes differentially private online federated learning algorithm which is an important and unexplored research area.\n2. The paper provides regret analysis with differential privacy guarantees for the proposed algorithms. Although I am not expert at differential privacy, theoretical results looks convincing to me.\n3. The paper is well-written and clear.\n\nWEAKNESSES:\n1. The paper can benefit from extending its experimental study. It would be great if authors can add more datasets and also do more ablation studies to strengthen the contribution of the paper.\n2. The paper can add more explanations about applicability of the study. Usually federated learning is used to train a model. However, reading the paper, the practical applications of the paper is not obvious.\n\nQUESTIONS:\n1. I do not understand the statement in the paper that \"*In the federated setting, it is infeasible for the central server to have full access to past loss functions due to the high communication cost.*\" I think in these cases the server can collect losses over time and store them in such a way that clients can afford communication cost. I believe in these cases probably storage can be more important bottleneck than communication cost. Furthermore, I do not think sending a loss would be bottleneck for the system in both communication and storage aspects even if the number of clients is large. Can you explain more about this?\n2. I suggest extending the literature review of the paper by including the work \"Personalized online federated learning with multiple kernels. *Advances in Neural Information Processing Systems (NeurIPS)*, 35:33316–33329, 2022.\". This work studies online model selection for communication and prediction efficiency in online federated learning.\n\nLIMITATIONS:\nThe paper discusses the limitations adequately. For example, the paper clearly presents the assumption in section 3 that the loss functions are assumed to be convex and smooth with respect to $||\\cdot||_1$.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies differentially private federated online prediction from experts against stochastic and oblivious adversaries. The goal is to minimize average regret across clients over time with privacy guarantees. For stochastic adversaries, the proposed Fed-DP-OPE-Stoch algorithm achieves regret improvement over single-player counterparts with logarithmic communication costs. For oblivious adversaries with a low-loss expert, the new Fed-SVT algorithm demonstrates an m-fold regret speed-up under pure and approximate differential privacy, nearly optimal up to logarithmic factors according to established lower bounds. Simulation experiments have been done.\n\nSTRENGTHS:\n1, The paper is well written, presented: The background, related previous work, problem formulation and key concepts (e.g., federated online prediction, differential privacy, stochastic/oblivious adversaries) are clearly explained. The algorithms are described in detail. \n\n2,  The theoretical analysis appears rigorous, with the authors establishing regret bounds for their proposed algorithms and deriving  lower bounds for the oblivious adversary case. The Fed-SVT algorithm achieves near-optimal regret performance (up to logarithmic factors) in the special case of oblivious adversaries with a low-loss expert, demonstrating the quality of the proposed solution.\n\nWEAKNESSES:\n1, Lack of Real-World Evaluation: The lack of experiments on real-world dataset scenarios is a significant limitation. The authors should address how their proposed algorithms would perform and be applicable to practical problems in domains like recommender systems or healthcare. Without evaluations on real-world datasets, it is challenging to judge the practical utility and potential challenges of their methods.\n\n2, Novelty Concerns:  After given a tree-based method is used for private aggregation of the gradients of loss functions (Asi et al., 2021b), what's the challenge for the theoretical analysis of proposed algorithms given Asi et al., 2022b and Asi et al., 2023?\n\nQUESTIONS:\n1, Please add at least one experiment with real-world dataset. So reader can have better sense regarding the real-world application and how practical of the proposed algorithms.\n2, Please clarify the challenges in theoretical analysis given existing methods.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problems of differentially private federated online prediction from experts against both stochastic adversaries and oblivious adversaries. The main contributions are three-fold. First, for stochastic adversaries, the paper proposes a differentially private mixture of experts algorithms and provide theoretical guarantees along with. Besides, for oblivious adversaries, the paper first shows a pessimistic result by providing a lower bound showing that federated learning cannot improve over the single machine learning in general. Furthermore, the paper shows that under special realizable case, however, federated learning can benefit from more machines. The paper proposes an algorithm that achieves that benefit with theoretical guarantees.\n\nSTRENGTHS:\nBoth the theoretical analysis and algorithm design seems original and non-trivial. The technical contribution seems solid. The paper is also clear in its problem setup, algorithm description and theoretical justification. The problem solved by the paper is a concrete theoretical problem, and the paper does a good job in solving the problem overall.\n\nWEAKNESSES:\nMy only concern of the paper is on the application side. While the problem studied by the paper makes perfect sense from a theoretical perspective, it does not seem very clear to me where it can have applications. With that being said, I think it is fine for authors to focus on a theoretical problem and leave its potential applications for future work.\n\nQUESTIONS:\nWould you please list some potential applications of the proposed algorithms? It will be better to have some specific applications for readers to keep in mind.\n\nLIMITATIONS:\nLike admitted by the authors and pointed out in the weakness section, the main limitation of the paper is the lack of real-world data experiments and applications. But given the theoretical nature of the paper, I do not think this limitation is unacceptable.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}