SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rachael Hwee Ling Sim $^{1}$ , Yehong Zhang $^{2}$ , Trong Nghia Hoang $^{3}$
Xinyi Xu<sup>1</sup>, Bryan Kian Hsiang Low<sup>1</sup>, and Patrick Jaillet<sup>4</sup>
$^{1}$  Department of Computer Science, National University of Singapore, Republic of Singapore
2 Peng Cheng Laboratory, People's Republic of China
$^{3}$  School of Electrical Engineering and Computer Science, Washington State University, USA
$^{4}$  Dept. of Electrical Engineering and Computer Science, MIT, USA
$^{1}$  {rachaels, xuxinyi, lowkh} @ comp.nus.edu.sg,  $^{2}$  zhangyh02@pcl.ac.cn
trongnghia.hoang@wsu.edu, jaillet@mit.edu
Collaborative machine learning involves training models on data from multiple parties but must incentivize their participation. Existing data valuation methods fairly value and reward each party based on shared data or model parameters but neglect the privacy risks involved. To address this, we introduce differential privacy (DP) as an incentive. Each party can select its required DP guarantee and perturb its sufficient statistic (SS) accordingly. The mediator values the perturbed SS by the Bayesian surprise it elicits about the model parameters. As our valuation function enforces a privacy-valuation trade-off, parties are deterred from selecting excessive DP guarantees that reduce the utility of the grand coalition's model. Finally, the mediator rewards each party with different posterior samples of the model parameters. Such rewards still satisfy existing incentives like fairness but additionally preserve DP and a high similarity to the grand coalition's posterior. We empirically demonstrate the effectiveness and practicality of our approach on synthetic and real-world datasets.
Collaborative machine learning (ML) seeks to build ML models of higher quality by training on more data owned by multiple parties [47, 62]. For example, a hospital can improve its prediction of disease progression by training on data collected from more and diversified patients from other hospitals [6]. Likewise, a real-estate firm can improve its prediction of demand and price by training on data from others [9]. However, parties have two main concerns that discourage data sharing and participation in collaborative ML: (a) whether they benefit from the collaboration and (b) privacy.
Concern (a) arises as each party would expect the significant cost that it incurs to collect and share data (e.g., the risk of losing its competitive edge) to be covered. Some existing works [47, 51], among other data valuation methods, have recognized that parties require incentives to collaborate, such as a guaranteed fair higher reward from contributing more valuable data than the others, an individually rational higher reward from collaboration than in solitude, and a higher total reward (i.e., group welfare) whenever possible. Often, parties share and are rewarded with information (e.g., gradients [58] or parameters [47] of parametric ML models) computed from the shared data. However, these incentive-aware reward schemes expose parties to privacy risks.
<sup>1</sup>Data valuation methods study how much data is worth. As explained in [46], a party's data is first valued independently using a performance metric (e.g., see Def. 3.1 later) and then relative to the data contributed by others (e.g., Shapley value (Sec. 4)). The latter value is helpful to (i) model interpretability and (ii) deciding how much to compensate the data owners fairly.
On the other hand, some federated learning (FL) works [34] have addressed the privacy concern (b) and satisfied strict data protection laws (e.g., European Union's General Data Protection Regulation) by enforcing differential privacy (DP) [1, 36] during the collaboration. Each party injects noise before sharing information to ensure that its shared information would not significantly alter a knowledgeable collaborating party's or mediator's belief about whether a datum was input to the algorithm. Injecting more noise leads to a stronger DP guarantee. As raised in [64], adding DP can invalidate game-theoretic properties and hence affect participation. For example, in the next paragraph, we will see that adding DP may lead to the collaboration being perceived as unfair and a lower group welfare. However, to the best of our knowledge (and as discussed in Sec. 7 and Fig. 5), there are no works that address both concerns, i.e., ensure the fairness, individual rationality, and group welfare incentives (see Sec. 4), alongside privacy. Thus, we aim to fill in this gap and design an incentive-aware yet privacy-preserving reward scheme by addressing the following questions:
If a party (e.g., hospital) requires a stronger DP guarantee, what should the impact be on its valuation and reward? Our answer is that, on average, its valuation and reward should decrease. Intuitively, it is unfair when this party gets a higher valuation due to randomness in the DP noise. More importantly, parties require guaranteed higher rewards to consider a weaker privacy guarantee [22, 64] which will help maximize the utility of the collaboratively trained model(s). As observed in [14, 65], the weaker the DP guarantee, the smaller the loss in model accuracy from enforcing DP. Thus, we will (i) assign a value to each party to enforce a privacy-valuation trade-off and incentivize parties against unfetteredly selecting an excessively strong DP guarantee,[2] and (ii) flexibly allow each party to enforce a different DP guarantee without imposing a party's need for strong DP on others. This new perspective and its realization is our main contribution.
To enforce a privacy-valuation trade-off, how should DP be ensured and a party's data be valued (Sec. 3)? Initially, valuation using validation accuracy seems promising as the works of [18, 25] have empirically shown that adding noise will decrease the valuation. However, parties may be reluctant to contribute validation data due to privacy concerns and disagree on the validation set as they prioritize accurate predictions on different inputs (e.g., patient demographics). So, we revert to valuing parties based on the quality of inference of the model parameters under DP. Bayesian inference is a natural choice as it quantifies the impact of (additional DP) noise. In Sec. 2, we will explain how each party ensures DP by only sharing perturbed sufficient statistic (SS) with the mediator. The mediator values the perturbed SS by the surprise it elicits relative to the prior belief of model parameters. Intuitively, noisier perturbed SS is less valuable as the posterior belief of the model parameters will be more diffuse and similar to the prior. As parties prioritize obtaining a model for future predictions and may face legal/decision difficulties in implementing monetary payments, we reward each party with posterior samples of the model parameters (in short, model reward) instead.
How should the reward scheme be designed to satisfy the aforementioned privacy, individual rationality, and fairness incentives (Sec. 4)? Our scheme will naturally satisfy the privacy incentive as any post-processing of the perturbed SS will preserve DP. To satisfy fairness and individual rationality, we set the target reward value for every party using  $\rho$ -Shapley value [47]. Lastly, to realize these target reward values, how should the model reward be generated for each party (Sec. 5)? Instead of rewarding all parties with samples from the same (grand coalition's) posterior of the model parameters given all their perturbed SS (which would be unfair if their valuations differ), our reward control mechanism generates a different posterior for each party that still preserves a high similarity to the grand coalition's posterior. Concretely, the mediator scales the SS by a factor between 0 and 1 before sampling to control the impact of data on the posterior (by tempering the data likelihood). Scaling the SS by a factor of 0, 1, and between 0 and 1 yield the prior, posterior, and their interpolation, respectively. We then solve for the factor to achieve the target reward value.
By answering the above questions, our work here provides the following novel contributions<sup>3</sup>:
- A new privacy-valuation trade-off criterion for valuation functions that is provably satisfied by the combination of our Bayesian surprise valuation function with DP noise-aware inference (Sec. 3);
- New incentives including  $DP$  (while deterring excessive  $DP$ ) and similarity to grand coalition's model (Sec. 4);
2The work of [30] describes problems posed by excessive data privacy and "the need to balance privacy with fuller and representative data collection" (instead of privileging privacy). But, parties are still free to seek DP.
See App. B for the key differences of our work here vs. data valuation and DP/FL works.
- Reward control mechanisms (Sec. 5) to generate posterior samples of the model parameters for each party that achieve a target reward value and the aforementioned incentives; one such mechanism tempers the likelihood of the data by scaling the SS and data quantity.
2 Collaborative ML Problem with Privacy Incentive
Our private collaborative ML problem setup comprises a mediator coordinating information sharing, valuation, and reward, and  $n$  parties performing a common ML task (e.g., predicting disease progression). Let the set  $N \triangleq \{1, \dots, n\}$  denote the grand coalition of  $n$  parties. Each party  $i$  owns a private dataset  $\mathcal{D}_i$  which cannot be directly shared with others, including the mediator. What alternative information should each party provide to the mediator for collaborative training of an ML model?
To ease aggregation, this work focuses only on Bayesian models with sufficient statistic (SS), such as exponential family models [5], Bayesian linear regression [39], and generalized linear models, including Bayesian logistic regression [21] (with approximate SS).
Definition 2.1 (Sufficient Statistic (SS) [48, 52]). The statistic  $s_i$  is a SS for the dataset  $\mathcal{D}_i$  if the model parameters  $\theta$  and dataset  $\mathcal{D}_i$  are conditionally independent given  $s_i$ , i.e.,  $p(\theta | s_i, \mathcal{D}_i) = p(\theta | s_i)$ .
We propose that each party  $i$  shares its SS  $\boldsymbol{s}_i$  for and in place of its dataset  $\mathcal{D}_i$  to protect the privacy of  $\mathcal{D}_i$ . We assume that the parties have agreed to adopt a common Bayesian model with the same prior  $p(\theta)$  of model parameters  $\theta$ , and each party  $i$ 's dataset  $\mathcal{D}_i$  is independently drawn from the likelihood  $p(\mathcal{D}_i|\theta)$  that is conjugate to the prior  $p(\theta)$  (i.e., belonging to an exponential family). The mediator can compute the posterior belief  $p(\theta|\{\mathcal{D}_i\}_{i\in N})$  of model parameters  $\theta$  given the grand coalition  $N$ 's datasets using a function  $f_{\theta}$  of the sum over shared SS:  $p(\theta|\{\mathcal{D}_i\}_{i\in N}) \propto p(\theta)f_{\theta}(\sum_{i\in N}s_i)$ . We give a concrete example and the mathematical details of SS in Apps. A.1 and E, respectively.
Privacy Incentive. However, sharing the exact SS  $s_N \triangleq \{s_i\}_{i \in N}$  will not ensure privacy as the mediator can draw inferences about individual datum in the private datasets  $\mathcal{D}_N \triangleq \{\mathcal{D}_i\}_{i \in N}$ . To mitigate the privacy risk, each party  $i$  should choose its required privacy level  $\epsilon_i$  and enforce  $(\lambda, \epsilon_i)$ -Rényi differential privacy. In Def. 2.2, a smaller  $\epsilon_i$  corresponds to a stronger DP guarantee.
Definition 2.2 (Rényi Differential Privacy (DP) [38]). A randomized algorithm  $\mathcal{R}:\mathcal{D}\to \pmb{o}$  is  $(\lambda ,\epsilon)$ -Rényi differentially private if for all neighboring datasets  $\mathcal{D}$  and  $\mathcal{D}'$ , the Rényi divergence of order  $^4$ $\lambda >1$  is  $D_{\lambda}(\mathcal{R}(\mathcal{D})||\mathcal{R}(\mathcal{D}')|\leq \epsilon$ .
Party  $i$  can enforce (example-level) $^4$ $(\lambda, \epsilon_i)$ -Rényi DP by applying the Gaussian mechanism: It generates perturbed SS  $o_i \triangleq s_i + z_i$  by sampling a Gaussian noise vector  $z_i$  from the distribution  $p(Z_i) = \mathcal{N}(\mathbf{0}, 0.5(\lambda/\epsilon_i)\Delta_2^2(g)\mathbf{I})$  where  $\Delta_2^2(g)$  is the squared  $\ell_2$ -sensitivity $^4$  of the function  $g$  that maps the dataset  $\mathcal{D}_i$  to the SS  $s_i$ . We choose Rényi DP over the commonly used  $(\epsilon, \delta)$ -DP as it gives a stronger privacy definition and allows a more convenient composition of the Gaussian mechanisms [38], as explained in App. A.2.
Each party  $i$  will share (i) the number  $c_{i} \triangleq |\mathcal{D}_{i}|$  of data points in its dataset  $\mathcal{D}_{i}$ , (ii) its perturbed SS  $o_{i}$ , and (iii) its Gaussian distribution  $p(Z_{i})$  with the mediator. As DP algorithms are robust to post-processing, the mediator's subsequent operations of  $o_{i}$  (with no further access to the dataset) will preserve the same DP guarantees. The mediator uses such information to quantify the impact of the DP noise and compute the DP noise-aware posterior  $p(\theta | \{o_{i}\}_{i \in N})$  via Markov Chain Monte Carlo (MCMC) sampling steps outlined by [3, 4, 27].
In this section, we have satisfied the privacy incentive. In Sec. 3, we assign a value  $v_{C}$  to each coalition  $C \subseteq N$ 's perturbed SS  $o_{C} \triangleq \{o_{i}\}_{i \in C}$  that would decrease, on average, as the DP guarantee strengthens. In Secs. 4 and 5, we outline our reward scheme: Each party  $i$  will be rewarded with model parameters sampled from  $q_{i}(\theta)$  (in short, model reward) for future predictions with an appropriate reward value  $r_{i}$  (decided based on  $(v_{C})_{C \subseteq N}$ ) to satisfy collaborative ML incentives (e.g., individual
<sup>4</sup> Parties agree on  $\lambda$ . In App. A.2, we will define/explain the DP-related concepts and other DP notions.
${}^{5}$  A benefit of sharing perturbed SS is that each party only incurs the privacy cost once regardless of the number of samples drawn or coalitions considered.
Here,  $\pmb{o}_i$  should be interpreted as a random vector taking the value of its sample. Non-noise-aware inference incorrectly treats  $\pmb{o}_i$  as  $\pmb{s}_i$  and computes  $p(\theta | \{\pmb{s}_i = \pmb{o}_i\}_{i \in N})$  instead. See App. A.3.
rationality, fairness). Our work's main contributions, notations, and setup are detailed in Fig. 1. The main steps involved are detailed in Algo. 2.
Figure 1: An overview of our private collaborative ML problem setup from party  $i$ 's perspective and our novel contributions (ideas in blue, novel combination of solutions in blue). We (i) enforce a privacy-reward trade-off (using each party  $i$ 's desire for a higher-quality model reward in collaborative ML) to deter party  $i$  from unfetteredly/overcautiously selecting an excessive DP guarantee (small  $\epsilon_i$ ), (ii) ensure DP in valuation and rewards, and (iii) preserve similarity of its model reward  $q_i(\theta)$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  to achieve a high utility.
3 Valuation of Perturbed Sufficient Statistics
The perturbed SS  $o_{C}$  of coalition  $C$  is more valuable and assigned a higher value  $v_{C}$  if it yields a model (in our work here, the DP noise-aware posterior  $p(\theta | o_{C})$ ) of higher quality. Most data valuation methods [24, 18, 63] measure the quality of an ML model by its performance on a validation set. However, it may be challenging for collaborating parties (e.g., competing healthcare firms) to create and agree on a large, representative validation set as they may prioritize accurate predictions on different inputs (e.g., patient demographics) [47]. The challenge increases when each firm requires privacy and avoids data sharing. Other valuation methods [47, 59] have directly used the private inputs of the data (e.g., design matrix). Here, we propose to value the perturbed SS  $o_{C}$  of coalition  $C$  based on the surprise [23] that it elicits from the prior belief of model parameters, as defined below:
Definition 3.1 (Valuation via Bayesian Surprise). The value of coalition  $C$  or its surprise  $v_{C}$  is the KL divergence  $D_{\mathrm{KL}}(p(\theta | o_{C}); p(\theta))$  between posterior  $p(\theta | o_{C})$  vs. prior  $p(\theta)$ .
From Def. 3.1, a greater surprise would mean that more bits will be needed to encode the information in  $p(\theta | o_C)$  given that others already know  $p(\theta)$ . Otherwise, a smaller surprise means our prior belief has not been updated significantly. Moreover, as the valuation depends on the observed  $o_C$ , the surprise elicited by the exact SS and data points will indirectly influence the valuation. Next, by exploiting the equality of the expected Bayesian surprise and the information gain on model parameters  $\theta$  given perturbed SS  $o_C$  (i.e.,  $\mathbb{E}_{o_C}[v_C] = \mathbb{I}(\theta; o_C)$ ), we can establish the following essential properties of our valuation function:
V1 Non-negativity.  $\forall C\subseteq N\forall o_{C}v_{C}\geq 0$  . This is due to the non-negativity of KL divergence.
V2 Party monotonicity. In expectation w.r.t.  $o_C$ , adding a party will not decrease the valuation:  $\forall C \subseteq C' \subseteq N \, \mathbb{E}_{o_{C'}}[v_{C'}] \geq \mathbb{E}_{o_C}[v_C]$ . The proof (App. C.1) uses the "information never hurts" property.
V3 Privacy-valuation trade-off. When the DP guarantee is strengthened from  $\epsilon_{i}$  to a smaller  $\epsilon_{i}^{s}$  and independent Gaussian noise is added to  $o_{i}$  to generate  $o_{i}^{s}$ , in expectation, the value of any coalition  $C$  containing  $i$  will strictly decrease: Let  $v_{C}^{s}$  denote the value of coalition  $C$  with the random variable and realization of  $o_{i}$  replaced by  $o_{i}^{s}$ . Then,  $(i\in C)\wedge (\epsilon_i^s < \epsilon_i)\Rightarrow \mathbb{E}_{o_C}[v_C] > \mathbb{E}_{o_C^s}[v_C^s ]$ .
The proof of V3 (App. C.1) uses the data processing inequality of information gain and the conditional independence between  $\theta$  and  $o_i^s$  given  $o_i$ . Together, these properties address an important question of how to ensure DP and value a party's data to enforce a privacy-valuation trade-off (Sec. 1). Additionally, in App. C.2, we prove that in expectation, our Bayesian surprise valuation is equivalent to the alternative valuation that measures the similarity of  $p(\theta | o_C)$  to the grand coalition  $N$ 's DP noise-aware posterior  $p(\theta | o_N)$ .
Implementation. Computing the Bayesian surprise valuation is intractable since the DP noise-aware posterior  $p(\theta | o_C)$  and its KL divergence from  $p(\theta)$  do not have a closed-form expression. Nonetheless, there exist approximate inference methods like the Markov chain Monte Carlo (MCMC) sampling to estimate  $p(\theta | o_C)$  efficiently, as discussed in App. A.3. As our valuation function requires estimating the value of multiple coalitions and the posterior sampling step is costly, we prefer estimators with a low time complexity and a reasonable accuracy for a moderate number  $m$  of samples. We recommend KL estimation to be performed using the nearest-neighbors method [45], and repeated and averaged to reduce the variance of the estimate (see App. C.3 for a discussion). The nearest-neighbor KL estimator is also asymptotically unbiased; drawing more samples would reduce the bias and variance of our estimates and is more likely to ensure fairness — for example, party  $i$ 's sampled valuation is only larger than  $j$ 's if  $i$ 's true valuation is higher.
Remark. Our valuation is based on the submitted information  $\{c_i, o_i, p(Z_i)\}_{i \in N}$  without verifying or incentivizing their truthfulness. We discuss how this limitation is shared by existing works and can be overcome by legal contracts and trusted data-sharing platforms in App. I.
4 Reward Scheme for Ensuring Incentives
After valuation, the mediator should reward each party  $i$  with a model reward (i.e., consisting of samples from  $q_{i}(\theta)$ ) for future predictions. Concretely,  $q_{i}(\theta)$  is a belief of model parameters  $\theta$  after learning from the perturbed SS  $o_{N}$ . As in Sec. 3, we value party  $i$ 's model reward as the KL divergence from the prior:  $r_{i} \triangleq D_{\mathrm{KL}}(q_{i}(\theta); p(\theta))$ . The mediator will first decide the target reward value  $r_{i}^{*}$  for every party  $i \in N$  using  $\{v_{C}\}_{C \subseteq N}$  to satisfy incentives such as fairness. The mediator will then control and generate a different  $q_{i}(\theta)$  for every party  $i \in N$  such that  $r_{i} = r_{i}^{*}$  using reward control mechanisms from Sec. 5. We will now outline the incentives and desiderata for model reward  $q_{i}(\theta)$  and reward values  $r_{i}$  and  $r_{i}^{*}$  for every party  $i \in N$  when the grand coalition forms<sup>7</sup>.
P1 DP-Feasibility. In party  $i$ 's reward, any other party  $k$  is still guaranteed at least its original  $(\lambda, \epsilon_k)$ -DP guarantee or stronger. The implication is that the generation of party  $i$ 's reward should not require more private information (e.g., SS) from party  $k$ .
P2 Efficiency. There is a party  $i \in N$  whose model reward is the grand coalition  $N$ 's posterior, i.e.,  $q_{i}(\theta) = p(\theta | o_{N})$ . It follows that  $r_{i} = v_{N}$ .
P3 Fairness. The target reward values  $(r_i^*)_{i\in N}$  must consider the coalition values  $\{v_{C}\}_{C\subseteq N}$  and satisfy properties F1 to F4 given in [47] and reproduced in App. D.2. The monotonicity axiom F4 ensures using a valuation function which enforces that a privacy-valuation trade-off will translate to a privacy-reward trade-off and deter parties from selecting excessive DP guarantees.
P4 Individual Rationality. Each party should receive a model reward that is more valuable than the model trained on its perturbed SS alone:  $\forall i \in N r_i^* \geq v_i$ .
P5 Similarity to Grand Coalition's Model. Among multiple model rewards  $q_{i}(\theta)$  whose value  $r_{i}$  equates the target reward  $r_{i}^{*}$ , we secondarily prefer one with a higher similarity  $r_{i}' = -D_{\mathrm{KL}}(p(\theta | o_{N}); q_{i}(\theta))$  to  $p(\theta | o_{N})$ .<sup>8</sup>
P6 Group Welfare. The reward scheme should maximize the total reward value  $\sum_{i=1}^{n} r_i$  to increase the utility of model reward for each party and achieve the aims of collaborative ML.
Choice of desiderata. We adopt the desiderata from [47] but make P1 and P2 more specific (by considering each party's actual reward  $q_{i}(\theta)$  over just its values  $r_{i}$  and  $v_{N}$ ) and introduce P5. Firstly, for our Bayesian surprise valuation function, the feasibility constraint of [47] is inappropriate as removing a party or adding some noise realization may result in  $r_{i} > v_{N}$ , so we propose P1 instead. Next, we recognize that party  $i$  is not indifferent to all model rewards  $q_{i}(\theta)$  with the same target reward value as they may have different utility (predictive performance). Thus, we propose our more specific P2 and a secondary desideratum P5. As P5 is considered after other desiderata, it does not conflict with existing results, e.g., design for  $(r_{i}^{*})_{i\in N}$  to satisfy other incentives.
Remark on Rationality. In P4, a party's model reward is compared to the model trained its perturbed SS instead of its exact SS alone. This is because the mediator cannot access (and value the model
See App. D.1 for the modifications needed when the grand coalition does not form.
Expectation propagation, a common approximate inference technique, also maximizes  $r_i'$ . [5].
The valuation fn. is only monotonic in expectation (V2-V3). See further discussion in App. I, Question 9.