SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhepei Wei\* Chuanhao Li\* Haifeng Xu2 Hongning Wang1
University of Virginia<sup>1</sup> University of Chicago<sup>2</sup>
{tqf5qb, c15ev, hw5x}@virginia.edu
haifengxu@chicago.edu
Most existing works on federated bandits take it for granted that all clients are altruistic about sharing their data with the server for the collective good whenever needed. Despite their compelling theoretical guarantee on performance and communication efficiency, this assumption is overly idealistic and oftentimes violated in practice, especially when the algorithm is operated over self-interested clients, who are reluctant to share data without explicit benefits. Negligence of such self-interested behaviors can significantly affect the learning efficiency and even the practical operability of federated bandit learning. In light of this, we aim to spark new insights into this under-explored research area by formally introducing an incentivized communication problem for federated bandits, where the server shall motivate clients to share data by providing incentives. Without loss of generality, we instantiate this bandit problem with the contextual linear setting and propose the first incentivized communication protocol, namely, INC-FEDUCB, that achieves near-optimal regret with provable communication and incentive cost guarantees. Extensive empirical experiments on both synthetic and real-world datasets further validate the effectiveness of the proposed method across various environments.
Federated bandit learning has recently emerged as a promising new direction to promote the application of bandit models while preserving privacy by enabling collaboration among multiple distributed clients [10, 40, 22, 13, 23, 24, 25, 15, 37, 9]. The main focus in this line of research is on devising communication-efficient protocols to achieve near-optimal regret in various settings. Most notably, the direction on federated contextual bandits has been actively gaining momentum, since the debut of several benchmark communication protocols for contextual linear bandits in the P2P [18] and star-shaped [40] networks. Many subsequent studies have explored diverse configurations of the clients' and environmental modeling factors and addressed new challenges arising in these contexts. Notable recent advancements include extensions to asynchronous linear bandits [22, 13], generalized liner bandits [23], and kernelized contextual bandits [24, 25].
Despite the extensive exploration of various settings, almost all existing federated bandit algorithms rely on the assumption that every client in the system is willing to share their local data/model with the server, regardless of the communication protocol design. For instance, synchronous protocols [40, 23] require all clients to simultaneously engage in data exchange with the server in every communication round. Similarly, asynchronous protocols [22, 25, 13] also assume clients must participate in communication as long as the individualized upload or download event is triggered, albeit allowing interruptions by external factors (e.g., network failure).
In contrast, our work is motivated by the practical observation that many clients in a federated system are inherently self-interested and thus reluctant to share data without receiving explicit benefits from the server [16]. For instance, consider the following scenario: a recommendation platform
(server) wants its mobile app users (clients) to opt in its new recommendation service, which switches previous on-device local bandit algorithm to a federated bandit algorithm. Although the new service is expected to improve the overall recommendation quality for all clients, particular clients may not be willing to participate in this collaborative learning, as the expected gain for them might not compensate their locally increased cost (e.g., communication bandwidth, added computation, lost control of their data, and etc). In this case, additional actions have to be taken by the server to encourage participation, as it has no power to force clients. This exemplifies the most critical concern in the real-world application of federated learning [16]. And a typical solution is known as incentive mechanism, which motivates individuals to contribute to the social welfare goal by offering incentives such as monetary compensation.
While recent studies have explored incentivized data sharing in federated learning [30, 38], most of which only focused on the supervised offline learning setting [16]. To our best knowledge, ours is the first work that studies incentive design for federated bandit learning, which inherently imposes new challenges. First, there is a lack of well-defined metric to measure the utility of data sharing, which rationalizes a client's participation. Under the context of bandit learning, we measure data utility by the expected regret reduction from the exchanged data for each client. As a result, each client values data (e.g., sufficient statistics) from the server differently, depending on how such data aligns with their local data (e.g., the more similar the less valuable). Second, the server is set to minimize regret across all clients through data exchange. But as the server does not generate data, it can be easily trapped by the situation where its collected data cannot pass the critical mass to ensure every participating client's regret is close to optimal (e.g., the data under server's possession cannot motivate the clients who have more valuable data to participate). To break the deadlock, we equip the server to provide monetary incentives. Subsequently, the server needs to minimize its cumulative monetary payments, in addition to the regret and communication minimization objectives as required by federated bandit learning. We propose a provably effective incentivized communication protocol, based on a heuristic search strategy to balance these distinct learning objectives. Our solution obtains near-optimal regret  $O(d\sqrt{T}\log T)$  with provable communication and incentive cost guarantees. Extensive empirical simulations on both synthetic and real-world datasets further demonstrate the effectiveness of the proposed protocol in various federated bandit learning environments.
Federated Bandit Learning One important branch in this area is federated multi-armed bandits (MABs), which has been well-studied in the literature [27, 36, 19, 4, 20, 28, 32, 39, 34, 33, 43]. The other line of work focuses on the federated contextual bandit setting [18, 40], which has recently attracted increasing attention. Wang et al. [40] and Korda et al. [18] are among the first to investigate this problem, where multiple communication protocols for linear bandits [1, 26] in star-shaped and P2P networks are proposed. Many follow-up works on federated linear bandits [10, 15, 22, 13] have emerged with different client and environment settings, such as investigating fixed arm set [15], incorporating differential privacy [10], and introducing asynchronous communication [13, 22]. Li et al. [23] extend the federated linear bandits to generalized linear bandits [11]. And they further investigated federated learning for kernelized contextual bandits in both synchronous and asynchronous settings [24, 25].
In this work, we situate the incentivized federated bandit learning problem under linear bandits with time-varying arm sets, which is a popular setting in many recent works [40, 10, 22, 13]. But we do not assume the clients will always participate in data sharing: they will choose not to share their data with the server if the resulting benefit of data sharing is not deemed to outweigh the cost. Here we need to differentiate our setting from those with asynchronous communication, e.g., Asyn-LinUCB [22]. Such algorithms still assume all clients are willing to share, though sometimes the communication can be interrupted by some external factors (e.g., network failure). We do not assume communication failures and leave it as our future work. Instead, we assume the clients need to be motivated to participate in federated learning, and our focus is to devise the minimum incentives to obtain the desired regret and communication cost for all participating clients.
Incentivized Federated Learning Data sharing is essential to the success of federated learning [30], where client participation plays a crucial role. However, participation involves costs, such as the need for additional computing and communication resources, and the risk of potential privacy breaches, which can lead to opt-outs [5, 14]. In light of this, recent research has focused on investigating
incentive mechanisms that motivate clients to contribute, rather than assuming their willingness to participate. Most of the existing research involves multiple decentralized clients solving the same task, typically with different copies of IID datasets, where the focus is on designing data valuation methods that ensure fairness or achieve a specific accuracy objective [35, 41, 8]. On the other hand, Donahue et al. [7] study voluntary participation in model-sharing games, where clients may opt out due to biased global models caused by the aggregated non-IID datasets. More recently, Karimireddy et al. [16] investigated incentive mechanism design for data maximization while avoiding free riders. For a detailed discussion of this topic, we refer readers to recent surveys on incentive mechanism design in federated learning [42, 38].
However, most works on incentivized federated learning only focus on better model estimation among fixed offline datasets, which does not apply to the bandit learning problem, where the exploration of growing data is also part of the objective. More importantly, in our incentivized federated bandit problem, the server is obligated to improve the overall performance of the learning system, i.e., minimizing regret among all clients, which is essentially different from previous studies where the server only selectively incentivizes clients to achieve a certain accuracy [35] or to investigate how much accuracy the system can achieve without payment [16].
In this section, we formally introduce the incentivized communication problem for federated bandits under the contextual linear bandit setting.
3.1 Federated Bandit Learning
We consider a learning system consisting of (1)  $N$  clients that directly interact with the environment by taking actions and receiving the corresponding rewards, and (2) a central server that coordinates the communication among the clients to facilitate their learning collectively. The clients can only communicate with the central server, but not with each other, resulting in a star-shaped communication network. At each time step  $t \in [T]$ , an arbitrary client  $i_t \in [N]$  becomes active and chooses an arm  $\mathbf{x}_t$  from a candidate set  $\mathcal{A}_t \subseteq \mathbb{R}^d$ , and then receives the corresponding reward feedback  $y_t = f(\mathbf{x}_t) + \eta_t \in \mathbb{R}$ . Note that  $\mathcal{A}_t$  is time-varying,  $f$  denotes the unknown reward function shared by all clients, and  $\eta_t$  denotes zero mean sub-Gaussian noise with known variance  $\sigma^2$ .
The performance of the learning system is measured by the cumulative (pseudo) regret over all  $N$  clients in the finite time horizon  $T$ , i.e.,  $R_{T} = \sum_{t=1}^{T} r_{t}$ , where  $r_{t} = \max_{\mathbf{x} \in \mathcal{A}_{t}} \mathbf{E}[y|\mathbf{x}] - \mathbf{E}[y_{t}|\mathbf{x}_{t}]$  is the regret incurred by client  $i_{t}$  at time step  $t$ . Moreover, under the federated learning setting, the system also needs to keep the communication cost  $C_{T}$  low, which is measured by the total number of scalars [40] being transferred across the system up to time  $T$ . With the linear reward assumption, i.e.,  $f(\mathbf{x}) = \mathbf{x}^{\top} \theta_{\star}$ , where  $\theta_{\star}$  denotes the unknown parameter, a ridge regression estimator  $\hat{\theta}_{t} = V_{t}^{-1} b_{t}$  can be constructed based on sufficient statistics from all  $N$  clients at each time step  $t$ , where  $V_{t} = \sum_{s=1}^{t} \mathbf{x}_{s} \mathbf{x}_{s}^{\top}$  and  $b_{t} = \sum_{s=1}^{t} \mathbf{x}_{s} y_{s}$  [21]. Using  $\hat{\theta}_{t}$  under the Optimism in the Face of Uncertainty (OFUL) principle [1], one can obtain the optimal regret  $R_{T} = O(d\sqrt{T})$ . To achieve this regret bound in the federated setting, a naive method is to immediately share statistics of each newly collected data sample to all other clients in the system, which essentially recovers its centralized counterpart. However, this solution incurs a disastrous communication cost  $C_{T} = O(d^{2} N T)$ . On the other extreme, if no communication occurs throughout the entire time horizon (i.e.,  $C_{T} = 0$ ), the regret upper bound can be up to  $R_{T} = O(d\sqrt{NT})$  when each client interacts with the environment at the same frequency, indicating the importance of timely data/model aggregation in reducing  $R_{T}$ .
To balance this trade-off between regret and communication cost, prior research efforts centered around designing communication-efficient protocols for federated bandits that feature the "delayed update" of sufficient statistics [40, 22, 13]. Specifically, each client  $i$  only has a delayed copy of  $V_{t}$  and  $b_{t}$ , denoted as  $V_{i,t} = V_{t_{\mathrm{last}}} + \Delta V_{i,t}, b_{i,t} = b_{t_{\mathrm{last}}} + \Delta b_{i,t}$ , where  $V_{t_{\mathrm{last}}}, b_{t_{\mathrm{last}}}$  is the aggregated sufficient statistics shared by the server in the last communication, and  $\Delta V_{i,t}, \Delta b_{i,t}$  is the accumulated local updates that client  $i$  obtain from its interactions with the environment since  $t_{\mathrm{last}}$ . In essence, the success of these algorithms lies in the fact that  $V_{t}, b_{t}$  typically changes slowly and thus has little instant impact on the regret for most time steps. Therefore, existing protocols that only require occasional communications can still achieve nearly optimal regret, despite the limitation on assuming clients' willingness on participation as we discussed before.
3.2 Incentivized Federated Bandits
Different from the prior works in this line of research, where all clients altruistically share their data with the server whenever a communication round is triggered, we are intrigued in a more realistic setting where clients are self-interested and thus reluctant to share data with the server if not well motivated. Formally, each client in the federated system inherently experiences a cost $^2$  of data sharing, denoted by  $\widetilde{D}_i^p \in \mathbb{R}$ , due to their individual consumption of computing resources in local updates or concerns about potential privacy breaches caused by communication with the server. Moreover, as the client has nothing to lose when there is no local update to share in a communication round at time step  $t$ , in this case we assume the cost is 0, i.e.,  $D_i^p = \widetilde{D}_i^p \cdot \mathbb{I}(\Delta V_{i,t} \neq \mathbf{0})$ . As a result, the server needs to motivate clients to participate in data sharing via the incentive mechanism  $\mathcal{M}: \mathbb{R}^N \times \mathbb{R}^{d \times d} \to \mathbb{R}^N$ , which takes as inputs a collection of client local updates  $\Delta V_{i,t} \in \mathbb{R}^{d \times d}$  and a vector of cost values  $D^p = \{D_1^p, \dots, D_N^p\} \in \mathbb{R}^N$ , and outputs the incentive  $\mathcal{I} = \{\mathcal{I}_{1,t}, \dots, \mathcal{I}_{N,t}\} \in \mathbb{R}^N$  to be distributed among the clients. Specifically, to make it possible to measure gains and losses of utility in terms of real-valued incentives (e.g., monetary payment), we adopt the standard quasi-linear utility function assumption, as is standard in economic analysis [2, 31].
At each communication round, a client decides whether to share its local update with the server based on the potential utility gained from participation, i.e., the difference between the incentive and the cost of data sharing. This requires the incentive mechanism to be individually rational:
Definition 1 (Individual Rationality [29]) An incentive mechanism  $\mathcal{M}:\mathbb{R}^N\times \mathbb{R}^{d\times d}\to \mathbb{R}^N$  is individually rational if for any  $i$  in the participant set  $S_{t}$  at time step  $t$ , we have
$$
\mathcal {I} _ {i, t} \geq D _ {i} ^ {p} \tag {1}
$$
In other words, each participant must be guaranteed non-negative utility by participating in data sharing under  $\mathcal{M}$ .
The server coordinates with all clients and incentivizes them to participate in the communication to realize its own objective (e.g., collective regret minimization). This requires  $\mathcal{M}$  to be sufficient:
Definition 2 (Sufficiency) An incentive mechanism  $\mathcal{M}:\mathbb{R}^N\times \mathbb{R}^{d\times d}\to \mathbb{R}^N$  is sufficient if the resulting outcome satisfies the server's objective.
Typically, under different application scenarios, the server may have different objectives, such as regret minimization or best arm identification. In this work, we set the objective of the server to minimize the regret across all clients; and ideally the server aims to attain the optimal  $\widetilde{O}(d\sqrt{T})$  regret in the centralized setting via the incentivized communication. Therefore, we consider an incentive mechanism is sufficient if it ensures that the resulting accumulated regret is bounded by  $\widetilde{O}(d\sqrt{T})$ .
The communication backbone of our solution derives from DisLinUCB [40], which is a widely adopted paradigm for federated linear bandits. We adopt their strategy for arm selection and communication trigger, so as to focus on the incentive mechanism design. We name the resulting algorithm INC-FEDUCB, and present it in Algorithm 1. Note that the two incentive mechanisms to be presented in Section 4.2 and 4.3 are not specific to any federated bandit learning algorithms, and each of them can be easily extended to alternative workarounds as a plug-in to accommodate the incentivized federated learning setting. For clarity, a summary of technical notations can be found in Table 7.
4.1 A General Framework: INC-FEDUCB Algorithm
Our framework comprises three main steps: 1) client's local update; 2) communication trigger; and 3) incentivized data exchange among the server and clients. Specifically, after initialization, an active client performs a local update in each time step and checks the communication trigger. If a communication round is triggered, the system performs incentivized data exchange between clients and the server. Otherwise, no communication is needed.
Note that if the costs are trivially set to zero, then clients have no reason to opt-out of data sharing and our problem essentially reduces to the standard federated bandits problem [40].
Algorithm 1 INC-FEDUCB Algorithm
Require:  $D_{c}\geq 0,D^{p} = \{D_{1}^{p},\dots ,D_{N}^{p}\}$ $\sigma ,\lambda >0,\delta \in (0,1)$    
1: Initialize: [Server]  $V_{g,0} = \mathbf{0}_{d\times d}\in \mathbb{R}^{d\times d},b_{g,0} = \mathbf{0}_d\in \mathbb{R}^d$    
2:  $\Delta V_{-j,0} = \mathbf{0}_{d\times d},\Delta b_{-j,0} = \mathbf{0}_d,\forall j\in [N]$    
3: [All clients]  $V_{i,0} = \mathbf{0}_{d\times d},b_{i,0} = \mathbf{0}_d,\Delta V_{i,0} = \mathbf{0}_{d\times d},\Delta b_{i,0} = \mathbf{0}_d,\Delta t_{i,0} = 0,\forall i\in [N]$    
4: for  $t = 1,2,\ldots ,T$  do   
5: [Client  $i_t]$  Observe arm set  $\mathcal{A}_t$    
6: [Client  $i_t]$  Select arm  $\mathbf{x}_t\in \mathcal{A}_t$  by Eq. (2) and observe reward  $y_{t}$    
7: [Client  $i_t]$  Update:  $V_{i,t} + = \mathbf{x}_t\mathbf{x}_t^\top ,b_{i,t}, + = \mathbf{x}_t y_t$    
8:  $\Delta V_{i,t} + = \mathbf{x}_t\mathbf{x}_t^\top ,\Delta b_{i,t} + = \mathbf{x}_t y_t,\Delta t_{i,t} + = 1$    
9: if  $\Delta t_{i_t,t}\log \frac{\operatorname*{det}(V_{i_t,t} + \lambda I)}{\operatorname*{det}(V_{i_t,t} - \Delta V_{i_t,t} + \lambda I)} >D_c$  then   
10: [All clients  $\rightarrow$  Server] Upload  $\Delta V_{i,t}$  such that  $\widetilde{S}_t = \{\Delta V_{i,t}|\forall i\in [N]\}$    
11: [Server] Select incentivized participants  $S_{t} = \mathcal{M}(\tilde{S}_{t})$  Incentive Mechanism   
12: for  $i:\Delta V_{i,t}\in S_t$  do   
13: [Participant  $i\to$  Server] Upload  $\Delta b_{i,t}$    
14: [Server] Update:  $V_{g,t} + = \Delta V_{i,t},b_{g,t} + = \Delta b_{i,t}$    
15:  $\Delta V_{-j,t} + = \Delta V_{i,t},\Delta b_{-j,t} + = \Delta b_{i,t},\forall j\neq i$    
16: [Participant  $i]$  Update:  $\Delta V_{i,t} = 0,\Delta b_{i,t} = 0,\Delta t_{i,t} = 0$    
17: for  $\forall i\in [N]$  do   
18: [Server  $\rightarrow$  All Clients] Download  $\Delta V_{-i,t},\Delta b_{-i,t}$    
19: [Client  $i]$  Update:  $V_{i,t} + = \Delta V_{-i,t},b_{i,t} + = \Delta b_{-i,t}$    
20: [Server] Update:  $\Delta V_{-i,t} = 0,\Delta b_{-i,t} = 0$
Formally, at each time step  $t = 1,\dots ,T$ , an arbitrary client  $i_{t}$  becomes active and interacts with its environment using observed arm set  $\mathcal{A}_t$  (Line 5). Specifically, it selects an arm  $\mathbf{x}_t\in \mathcal{A}_t$  that maximizes the UCB score as follows (Line 6):
$$
\mathbf {x} _ {t} = \underset {\mathbf {x} \in \mathcal {A} _ {t}} {\arg \max } \mathbf {x} ^ {\top} \hat {\theta} _ {i _ {t}, t - 1} (\lambda) + \alpha_ {i _ {t}, t - 1} \| \mathbf {x} \| _ {V _ {i _ {t}, t - 1} ^ {- 1} (\lambda)} \tag {2}
$$
where  $\hat{\theta}_{i_t,t - 1}(\lambda) = V_{i_t,t - 1}^{-1}(\lambda)b_{i_t,t - 1}$  is the ridge regression estimator of  $\theta_{\star}$  with regularization parameter  $\lambda >0$ ,  $V_{i_t,t - 1}(\lambda) = V_{i_t,t - 1} + \lambda I$ , and  $\alpha_{i_t,t - 1} = \sigma \sqrt{\log\frac{\operatorname*{det}(V_{i_t,t - 1}(\lambda))}{\operatorname*{det}(\lambda I)} + 2\log 1 / \delta} +\sqrt{\lambda}$ .  $V_{i_t,t}(\lambda)$  denotes the covariance matrix constructed using data available to client  $i_t$  up to time  $t$ . After obtaining a new data point  $(\mathbf{x}_t,y_t)$  from the environment, client  $i_t$  checks the communication event trigger  $\Delta t_{i_t,t}\cdot \log \frac{\operatorname*{det}(V_{i_t,t}(\lambda))}{\operatorname*{det}(V_{i_t,t_{\mathrm{last}}}(\lambda))} >D_c$  (Line 9), where  $\Delta t_{i_t,t}$  denotes the time elapsed since the last time  $t_\mathrm{last}$  it communicated with the server and  $D_{c}\geq 0$  denotes the specified threshold.
Incentivized Data Exchange With the above event trigger, communication rounds only occur if (1) a substantial amount of new data has been accumulated locally at client  $i_{t}$ , and/or (2) significant time has elapsed since the last communication. However, in our incentivized setting, triggering a communication round does not necessarily lead to data exchange at time step  $t$ , as the participant set  $S_{t}$  may be empty (Line 11). This characterizes the fundamental difference between INC-FEDUCB and DisLinUCB [40]: we no longer assume all  $N$  clients will share their data with the server in an altruistic manner; instead, a rational client only shares its local update with the server if the condition in Eq. (1) is met. In light of this, to evaluate the potential benefit of data sharing, all clients must first reveal the value of their data to the server before the server determines the incentive. Hence, after a communication round is triggered, all clients upload their latest sufficient statistics update  $\Delta V_{i,t}$  to the server (Line 10) to facilitate data valuation and participant selection in the incentive mechanism (Line 11). Note that this disclosure does not compromise clients' privacy, as the clients' secret lies in  $\Delta b_{i,t}$  that is constructed by the rewards. Only participating clients will upload their  $\Delta b_{i,t}$  to the server (Line 13). After collecting data from all participants, the server downloads the aggregated updates  $\Delta V_{-i,t}$  and  $\Delta b_{-i,t}$  to every client  $i$  (Line 17-20). Following the convention in federated bandit learning [40], the communication cost is defined as the total number of scalars transferred during this data exchange process.