SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aligned with the findings presented in Section 5.1, the results on real-world dataset also confirm the advantage of the payment-efficient mechanism over the payment-free incentive mechanism in terms of both accumulative (normalized) reward and communication cost. As illustrated in Figure 3, this performance advantage is particularly notable in a more conservative environment, where clients have higher  $D_{\star}^{p}$ . And when the cost of data sharing for clients is relatively low, the performance gap between the two mechanisms becomes less significant. We attribute this to the fact that clients with low  $D_{i}^{p}$  values are more readily motivated by the data alone, thus alleviating the need for additional monetary incentive. On the other hand, higher values of  $D_{i}^{p}$  indicate that clients are more reluctant to share their data. As a result, the payment-free incentive mechanism fails to motivate a sufficient number of clients to participate in data sharing, leading to a noticeable performance gap, as evidenced in Figure 3(c). Note that the communication cost exhibits a sharp increase towards the end of the interactions. This is because the presence of highly skewed user distribution in the real-world dataset. For instance, in the last 2,032 rounds, only one client remains actively engaged with the environment, rapidly accumulating sufficient amounts of local updates, thus resulting in an increase in both communication frequency and cost.
E.2 Ablation Study on Heuristic Search (Supplement to Section 5.2)
We further study the effectiveness of each component in the heuristic search of INC-FEDUCB on the real-world dataset and compare the performance among different variants across different data sharing costs. As presented in Figure 4(a) and 4(b), the variants without payment-free (PF) component, which only rely on monetary incentive to motivate clients to participate, generally exhibit lower rewards and higher communication costs. The reason is a bit subtle: as the payment efficient mechanism is subject to both  $\beta$  gap constraint and minimum payment cost requirement, it tends to satisfy the  $\beta$  gap constraint with minimum amount of data collected. But the payment free mechanism will always collect the maximum amount data possible. As a result, without the PF component, the server
(a) Normalized Reward
(c) Payment Cost
Figure 4: Ablation study on heuristic search (w.r.t  $D_{\star}^{p}\in [1,10,100]$ ).
tends to collect less (but enough) data, which in turn leads to more communication rounds and worse regret. The side-effect of the increased communication frequency is the higher payment costs, with respect to the  $\beta$  gap requirement in each communication round. This is particularly notable in a more collaborative environment, where clients have lower data sharing costs. As exemplified in Figure 4(c), when the clients are more willing to share data (e.g.,  $D_{\star}^{p} = 1$ ), the variants without PF incur significantly higher payment costs compared to the those with PF, as the server misses the opportunity to get those easy to motivate clients. Therefore, providing data incentives becomes even more crucial in such scenarios to ensure effective client participation and minimize payment costs. On the other hand, the variants without iterative search (IS) tend to maintain competitive performance compared to the fully-fledged model, despite incurring a higher payment cost, highlighting the advantage of IS in minimizing payment.
E.3 Environment & Hyper-Parameter Study (Supplement to Section 5.3)
d=25,K=25, Dc=T/T2d log T - √T2/N2dR log T · log β	DisLinUCB	INC-FEDUCB (β=1)	INC-FEDUCB (β=0.7)	INC-FEDUCB (β=0.3)
MovieLens (Dp* = 0)	Reward (Acc.)	38,353	38,353	37,731 (Δ-1.6%)	36,829 (Δ-2.4%)
Commu. Cost	33,415,200	33,415,200	5,967,000 (Δ-82%)	2,457,000 (Δ-92.6%)
Pay. Cost	\	0	0 (Δ=0%)	0 (Δ=0%)
MovieLens (Dp* = 1)	Reward (Acc.)	\	38,353	37,717 (Δ-1.7%)	36,833 (Δ-4%)
Commu. Cost	\	33,415,200	13,372,250 (Δ-60%)	5,038,675 (Δ-84.9%)
Pay. Cost	\	7859.67	124.41 (Δ-98.4%)	0 (Δ-100%)
MovieLens (Dp* = 10)	Reward (Acc.)	\	38,353	37,648 (Δ-1.8%)	36,675 (Δ-4.4%)
Commu. Cost	\	33,415,200	10,041,250 (Δ-70%)	4,240,625 (Δ-87.3%)
Pay. Cost	\	110,737.62	8,590.43 (Δ-92.2%)	2,076.98 (Δ-98.1%)
MovieLens (Dp* = 100)	Regret (Acc.)	\	38,353	37,641 (Δ-1.9%)	36,562 (Δ-4.7%)
Commu. Cost	\	33,415,200	8,496,600 (Δ-74.6%)	5,136,700 (Δ-84.6%)
Pay. Cost	\	1,155,616.99	105,847.84 (Δ-90.8%)	32,618.34 (Δ-97.2%)
Table 2: Study on hyper-parameter of INC-FEDUCB and environment (w/ theoretical  $D_{c}$ ).
In contrast to the hyper-parameter study on synthetic dataset with fixed communication threshold reported in Section 5.3, in this section, we comprehensively investigate the impact of  $\beta$  and  $D_{\star}^{p}$  on the real-world dataset by varying the communication thresholds  $D_{c}$ . First, we empirically validate the effectiveness of the theoretical value of  $D_{c} = \frac{T}{N^{2}d\log T} - \sqrt{\frac{T^{2}}{N^{2}dR\log T}}$  as introduced in Theorem 4. The results presented in Table 2 are generally consistent with the findings in Section 5.3: decreasing  $\beta$  can substantially lower the payment cost while still maintaining competitive rewards. We can also find that using the theoretical value of  $D_{c}$  can also save the communication cost. This results from the fact that setting  $D_{c}$  as a function of  $\beta$  leads to a higher communication threshold for lower  $\beta$ , and therefore reducing communication frequency. This observation is essentially aligned with the intuition behind lower  $\beta$ : when the system has a higher tolerance for outdated sufficient statistics, it should not only pay less in each communication round but also trigger communication less frequently.
On the other hand, we investigate INC-FEDUCB's performance under two fixed communication thresholds  $D_{c} = T / (N^{2}d\log T)$  and  $D_{c} = T / (Nd\log T)$ , which are presented in Table 3 and 4,
d=25,K=25, Dc=T/N2d log T	DisLinUCB	INC-FEDUCB (β=1)	INC-FEDUCB (β=0.7)	INC-FEDUCB (β=0.3)
MovieLens (Dp=0)	Reward (Acc.)	38,353	38,353	38,353 (Δ=0%)
Commu. Cost	33,415,200	33,415,200	33,415,200 (Δ=0%)
Pay. Cost	\	0	0 (Δ=0%)
MovieLens (Dp=1)	Reward (Acc.)	\	38,353	38,207 (Δ-0.4%)
Commu. Cost	\	33,415,200	171,046,600 (Δ+412%)
Pay. Cost	\	7859.67	2095.73 (Δ-73.3%)
MovieLens (Dp=10)	Reward (Acc.)	\	38,353	38,251 (Δ-0.3%)
Commu. Cost	\	33,415,200	135,521,025 (Δ+306%)
Pay. Cost	\	110,737.62	33,271.39 (Δ-70%)
MovieLens (Dp=100)	Reward (Acc.)	\	38,353	38,251 (Δ-0.3%)
Commu. Cost	\	33,415,200	135,521,025 (Δ+306%)
Pay. Cost	\	1,155,616.99	352,231.39 (Δ-69.5%)
Table 3: Study on hyper-parameter of INC-FEDUCB and environment (w/ lower fixed  $D_{c}$ ).
d=25,K=25, Dc=TNd log T	DisLinUCB	INC-FEDUCB (β=1)	INC-FEDUCB (β=0.7)	INC-FEDUCB (β=0.3)
MovieLens (Dp=0)	Reward (Acc.)	37,308	37,308	37,308 (Δ=0%)	37,308 (Δ=0%)
Commu. Cost	2,737,800	2,737,800	2,737,800 (Δ=0%)	2,737,800 (Δ=0%)
Pay. Cost	\	0	0 (Δ=0%)	0 (Δ=0%)
MovieLens (Dp=1)	Reward (Acc.)	\	37,308	37,296 (Δ-0.1%)	37,306 (Δ-0.1%)
Commu. Cost	\	2,737,800	4,197,525 (Δ+53.3%)	5,948,950 (Δ+117.3%)
Pay. Cost	\	55.31	44.76 (Δ-19.1%)	0 (Δ-100%)
MovieLens (Dp=10)	Reward (Acc.)	\	37,308	37,297 (Δ-0.1%)	37,167 (Δ-0.1%)
Commu. Cost	\	2,737,800	3,696,350 (Δ+35%)	5,765,075 (Δ+110.6%)
Pay. Cost	\	4048.69	3779.77 (Δ-6.6%)	2242.22 (Δ-44.6%)
MovieLens (Dp=100)	Reward (Acc.)	\	37,308	37,273 (Δ-0.1%)	36,946 (Δ-0.1%)
Commu. Cost	\	2,737,800	3,484,850 (Δ+27.3%)	5,690,250 (Δ+107.8%)
Pay. Cost	\	77,041.04	65,286.90 (Δ-15.3%)	40,010.59 (Δ-48.1%)
Table 4: Study on hyper-parameter of INC-FEDUCB and environment (w/ higher fixed  ${D}_{c}$  ).
respectively. These two values are created by increasing the theoretical value of  $D_{c}$ . Overall, the main findings align with those reported in Section 5.3, confirming our previous statements. While reducing  $\beta$  can achieve competitive rewards with lower payment costs, it comes at the expense of increased communication costs, suggesting the trade-off between payment costs and communication costs. Interestingly, the setting under a higher  $D_{\star}^{p}$  and  $D_{c}$  can help mitigate the impact of  $\beta$ . Specifically, while increasing the client's cost of data sharing inherently brings additional incentive costs, raising the communication threshold results in fewer communication rounds, leading to reduced overall communication costs. This finding highlights the importance of thoughtful design in choosing  $D_{c}$  and  $\beta$  to balance the trade-off between payment costs and communication costs in real-world scenarios with diverse data sharing costs.
E.4 Extreme Case Study
To further investigate the utility of INC-FEDUCB in extreme cases, we conduct a set of case studies on both synthetic and real-world datasets with fixed data sharing costs. As shown in Table 5 and 6, when  $\beta$  is extremely small, we can achieve almost  $100\%$  savings in incentive cost compared to the case where every client has to be incentivized to participate in data sharing (i.e.,  $\beta = 1$ ). However, this extreme setting inevitably results in a considerable drop in regret/reward performance and potentially tremendous extra communication cost due to the extremely outdated local statistics in clients. Nevertheless, by strategically choosing the communication threshold, we can mitigate the additional communication costs associated with the low  $\beta$  values. For instance, in the synthetic dataset, the difference in performance drop between the theoretical  $D_{c}$  setting and heuristic  $D_{c}$  value
d=25,K=25,Dp=100	INC-FEDUCB (β=1)	INC-FEDUCB (β=0.7)	INC-FEDUCB (β=0.3)	INC-FEDUCB (β=0.01)
T=5000,N=50(Dc=T/N2d log T)	Regret (Acc.)	45.37	46.33 (Δ+2.1%)	48.49 (Δ+6.9%)	51.22 (Δ+12.9%)
Commu. Cost	174,720,000	264,193,275 (Δ+51.2%)	299,134,900 (Δ+71.2%)	314,667,500 (Δ+80.1%)
Pay. Cost	479,397.18	229,999.66 (Δ-52%)	115,600 (Δ-75.9%)	42,800 (Δ-91.1%)
T=5000,N=50(Dc=T/T2d log T-√T2/N2dRlog T·log β)	Regret (Acc.)	45.37	46.72 (Δ+3%)	49.13 (Δ+8.3%)	53.72 (Δ+18.4%)
Commu. Cost	174,720,000	17,808,725 (Δ-89.8%)	7,237,600 (Δ-95.9%)	2,981,175 (Δ-98.3%)
Pay. Cost	479,397.18	178,895.78 (Δ-62.7%)	84,989.39 (Δ-82.3%)	1,200 (Δ-99.7%)
Table 5: Case study on synthetic dataset.
d=25,K=25, Dc=100	INC-FEDUCB (β=1)	INC-FEDUCB (β=0.7)	INC-FEDUCB (β=0.3)	INC-FEDUCB (β=0.01)
MovieLens(Dc=T/N2d log T)	Reward (Acc.)	38,353	38,251 (Δ-0.3%)	37,970 (Δ-1%)	37,039 (Δ-3.4%)
Commu. Cost	33,415,200	135,521,025 (Δ+306%)	522,196,225 (Δ+1463%)	1,226,741,425 (Δ+3571.2%)
Pay. Cost	1,155,616.99	352,231.39 (Δ-69.5%)	346,619.77 (Δ-70%)	75,799.39 (Δ-93.4%)
MovieLens(Dc=T/N2d log T-√T2/N2dR log T·log β)	Reward (Acc.)	38,353	37,641 (Δ-1.9%)	36,562 (Δ-4.7%)	31,873 (Δ-16.9%)
Commu. Cost	33,415,200	8,496,600 (Δ-74.6%)	5,136,700 (Δ-84.6%)	1,880,450 (Δ-94.4%)
Pay. Cost	1,155,616.99	105,847.84 (Δ-90.8%)	32,618.34 (Δ-97.2%)	200 (Δ-99.9%)
is relatively small ( $\Delta + 18.4\%$  vs.  $\Delta + 12.9\%$ ). However, these two different choices of  $D_{c}$  exhibit opposite effects on communication costs, with the theoretical one achieving a significant reduction ( $\Delta - 98.3\%$ ) while the heuristic one incurred a significant increase ( $\Delta + 80.1\%$ ). On the other hand, in the real-world dataset, the heuristic choice of  $D_{c}$  may lead to a smaller performance drop compared to the theoretical setting of  $D_{c}$  (e.g.,  $\Delta - 3.4\%$  vs.  $\Delta - 16.9\%$ ), reflecting the specific characteristics of the environment (e.g., a high demand of up-to-date sufficient statistics). Similar to the findings in Section E.3, this case study also emphasizes the significance of properly setting the system hyper-parameter  $\beta$  and  $D_{c}$ . By doing so, we can effectively accommodate the trade-off between performance, incentive costs, and communication costs, even in extreme cases.
F Notation Table
Table 6: Case study on real-world dataset.
Notation	Meaning
d	context dimension
N	total number of clients
T	total number of time steps
β	hyperparameter that controls the regret level
Dc	communication threshold
Di	data sharing cost of client i
St	participant set at time step t
Di,t(St)	data offered by the sever to client i at time step t
Ii,t/Imi,t	data/monetary incentive for client i at time step t
Vt	covariance matrix constructed by all available data in the system
Vi,t,bi,t	local data of client i at time step t
Vg,t,gb,t	global data stored at the server at time step t
ΔVi,t,Δbi,t	data stored at client i that has not been shared with the server
ΔV-i,t,Δb-i,t	data stored at the server that has not been shared with the client i
Table 7: Main technical notations used in this paper.