SlowGuess/NeurIPS_2023_OCR huggingface dataset:
trained on) the private exact SS. Moreover, with no restrictions on the maximum DP noise, the value of some party's exact SS may exceed the grand coalition's perturbed SS when parties require strong DP guarantees. P4 is sufficient when parties require DP when alone to protect data from curious users of their ML model [1, 3, 27]. For example, a hospital may not want doctors to infer specific patients' data. When parties do not require DP when alone, our reward scheme cannot theoretically ensure that the model reward from collaboration is better than using the exact SS. We further discuss this limitation in App. D.3.
Design of  $(r_i^*)_{i\in N}$ . To satisfy the desiderata from [47] (including our fairness P3 and rationality P4 incentives), we adopt their  $\rho$ -Shapley fair reward scheme with  $\rho \in (0,1]$  that sets  $r_i^* = v_N(\phi_i / \max_{k\in N}\phi_k)^\rho$  with Shapley value $^{10}$ $\phi_i \triangleq (1/n)\sum_{C \subseteq N \setminus i} \left[\binom{n-1}{|C|}^{-1}(v_{C \cup \{i\}} - v_C)\right]$ . Shapley value's consideration of marginal contribution (MC) to all coalitions is key to ensuring strict desirability F3 such that party  $i$  obtains a higher reward than party  $k$  (despite  $v_i = v_k$ ) if  $i$ 's perturbed SS adds more value to every other non-empty coalition. Applying Theorem 1 of [47], the mediator should set  $\rho$  between 0 and  $\min_{i\in N} \log(v_i / v_N) / \log(\phi_i / \max_k \phi_k)$  to guarantee rationality. Selecting a larger  $\rho$  incentivizes a party with a high-quality perturbed SS to share by fairly limiting the benefits to parties with lower-quality ones. Selecting a smaller  $\rho$  reward parties more equally and increase group welfare P6. Refer to Sec. 4.2 of [47] for a deeper analysis of the impact of varying  $\rho$ . These results hold for any choice of  $(\lambda, \epsilon_i)$ .
After explaining the desiderata for model reward  $q_{i}(\theta)$  and reward values  $r_{i}$  and  $r_{i}^{*}$  for every party  $i \in N$ , we are now ready to solve for  $q_{i}(\theta)$  such that  $r_{i} = r_{i}^{*}$ .
5 Reward Control Mechanisms
This section discusses two mechanisms to generate model reward  $q_{i}(\theta)$  with different attained reward value  $r_{i}$  for every party  $i \in N$  by controlling a single continuous parameter and solving for its value such that the attained reward value equates the target reward value:  $r_{i} = r_{i}^{*}$ . We will discuss the more obvious reward mechanism in Sec. 5.1 to contrast its cons with the pros of that in Sec. 5.2. Both reward mechanisms do not request new information from the parties; thus, the DP post-processing property applies, and every party  $k$  is still guaranteed at least its original DP guarantee or stronger in all model rewards (i.e., P1 holds).
5.1 Reward Control via Noise Addition
The work of [47] controls the reward values by adding Gaussian noise to the data outputs. We adapt it such that the mediator controls the reward value for party  $i \in N$  by adding Gaussian noise to the perturbed SS of each party  $k \in N$  instead. To generate the model reward for party  $i$  (superscripted), the mediator will reparameterize the sampled Gaussian noise vectors  $\{e_k^i \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\}_{k \in N}$  to generate the further perturbed SS<sup>11</sup>
$$
\boldsymbol {t} _ {N} ^ {i} \triangleq \left\{\boldsymbol {t} _ {k} ^ {i} \triangleq \boldsymbol {o} _ {k} + \left(0. 5 \lambda \Delta_ {2} ^ {2} (g _ {k}) \tau_ {i}\right) ^ {1 / 2} \boldsymbol {e} _ {k} ^ {i} \right\} _ {k \in N}
$$
where  $\Delta_2^2 (g_k)$  is the squared  $\ell_{2}$ -sensitivity of function  $g_{k}$  that computes the exact SS  $s_k$  from dataset  $\mathcal{D}_k$  (Sec. 2). Then, the mediator rewards party  $i$  with samples of model parameters  $\theta$  from the new DP noise-aware posterior  $q_{i}(\theta) = p(\theta |t_{N}^{i})$ .
Here, the scalar  $\tau_{i} \geq 0$  controls the additional noise variance and can be optimized via root-finding to achieve  $r_{i} = r_{i}^{*}$ . The main advantage of this reward control mechanism is its interpretation of strengthening party  $k$ 's DP guarantee in all parties' model rewards (see P1). For example, it can be derived that if  $\epsilon_{k} = \infty$ , then party  $k$  will now enjoy  $(\lambda, 1 / \tau_{i})$ -DP guarantee in party  $i$ 's reward instead. If  $\epsilon_{k} < \infty$ , then party  $k$  will now enjoy a stronger  $(\lambda, \epsilon_{k} / (1 + \tau_{i}\epsilon_{k}))$ -DP guarantee since  $\epsilon_{k} / (1 + \tau_{i}\epsilon_{k}) < \epsilon_{k}$ .
However, this mechanism has some disadvantages. Firstly, for the same scaled additional noise variance  $\tau_{i}$ , using different noise realizations  $\{e_k^i\}_{k\in N}$  will lead to model reward  $q_{i}(\theta)$  with varying
10 Party  $i$ 's MC to some coalitions and Shapley value  $\phi_i$  may be negative, which results in an unusable negative/undefined  $r_i^*$ . This issue can be averted while preserving P3 by upweighting non-negative MCs such as to the empty set.
To ease notation, we slightly abuse  $\pmb{t}_k^i$  to represent both a random vector and its sample.
similarity  $r_i'$  to the grand coalition  $N$ 's posterior. The mechanism cannot efficiently select the best model reward with higher  $r_i'$  (P5). Secondly, the value of  $r_i$  computed using such  $t_k^i$  may be non-monotonic $^{12}$  in  $\tau_i$  (see Fig. 2d), which makes it hard to bracket the smallest root  $\tau_i$  that solves for  $r_i = r_i^*$ . To address these disadvantages, we will propose the next mechanism.
5.2 Reward Control via Likelihood Tempering
Intuitively, a party  $i$  who is assigned a lower target reward value  $r_i^* < v_N$  should be rewarded with posterior samples of model parameters  $\theta$  that use less information from the datasets and SS of all parties. Sparked by the diffuse posterior algorithm [16], we propose that the mediator can generate such "less informative" samples for party  $i$  using the normalized posterior<sup>13</sup>
$$
q _ {i} (\theta) \propto p (\theta) [ p (\mathcal {D} _ {N} | \theta) ] ^ {\kappa_ {i}} \tag {1}
$$
involving the product of the prior  $p(\theta)$  and the data likelihood  $p(\mathcal{D}_N|\theta)$  to the power of (or, said in another way, tempered by a factor of)  $\kappa_{i}$ . Notice that setting  $\kappa_{i} = 0$  and  $\kappa_{i} = 1$  recover the prior  $p(\theta)$  and the posterior  $p(\theta |\mathcal{D}_N)$ , respectively. Thus, setting  $\kappa_{i}\in (0,1)$  should smoothly interpolate between both. We can optimize  $\kappa_{i}$  to control  $q_{i}(\theta)$  so that  $r_i = r_i^* < v_N$ .
But, how do we temper the likelihood? We start by examining the easier, non-private setting. In Sec. 2, we stated that under our assumptions, the posterior  $p(\theta | \mathcal{D}_N)$  can be computed by using the sum of data quantities  $\{c_k\}_{k \in N}$  and sum of exact SS  $s_N$ . In App. E, we further show that using the tempered likelihood  $[p(\mathcal{D}_N | \theta)]^{\kappa_i}$  is equivalent to scaling the data quantities and the exact SS  $s_N$  by the factor  $\kappa_i$  beforehand. In the private setting, the mediator can similarly scale the data quantities, the perturbed SS in  $o_N$  (instead of the inaccessible exact SS), and the  $\ell_2$ -sensitivity by the factor  $\kappa_i$  beforehand; see App. E.3 for details. This likelihood tempering mechanism addresses both disadvantages of Sec. 5.1:
- There is no need to sample additional DP noise. We empirically show that tempering the likelihood produces a model reward that interpolates between the prior vs. posterior (in App. G) and preserves a higher similarity  $r_i'$  to the grand coalition  $N$ 's posterior (P5 and hence, more group welfare P6) and better predictive performance than noise addition (see Sec. 6).
- Using a smaller tempering factor  $\kappa_{i} \in [0,1]$  provably decreases the attained reward value  $r_i$  (see App. E). Thus, as the relationship between  $r_i$  and  $\kappa_{i}$  is monotonic, we can find the only root by searching the interval  $[0,1]$ .
Remark. Our discussion on improving the estimate of  $v_{C}$  in the paragraph on implementation in Sec. 3 also applies to the estimate of  $r_i$  in Secs. 5.1 and 5.2. Thus, solving for  $\tau_i$  or  $\kappa_i$  to achieve  $r_i = r_i^*$  using any root-finding algorithm can only be accurate up to the variance in our estimate.
6 Experiments and Discussion
This section empirically evaluates the privacy-valuation and privacy-reward trade-offs (Sec. 6.1), reward control mechanisms (Sec. 6.2), and their relationship with the utility of the model rewards (Sec. 6.3). The time complexity of our scheme is analyzed in App. F and baseline methods are discussed in App. H.3. We consider Bayesian linear regression (BLR) with unknown variance on the Syn and CaliH datasets, and Bayesian logistic regression on the Diab dataset with 3 collaborating parties (see App. H.1 for details) and enforce  $(2,\epsilon_{i})$ -Renyi DP. For Synthetic BLR (Syn), we select and use a normal inverse-gamma distribution (i) to generate the true regression model weights, variance, and a 2D-dataset and (ii) as our model prior  $p(\theta)$ . We consider 3 parties with  $c_{1} = 100$ ,  $c_{2} = 200$ ,  $c_{3} = 400$  data points, respectively. For Californian Housing dataset (CaliH) [44], as in [47],  $60\%$  of the CaliH data is deemed "public/historic" and used to pre-train a neural network without DP. Real estate firms may only care about the privacy of their newest transactions. As the parties' features-house values relationship may differ from the "public" dataset, we do transfer learning and selectively retrain only the last layer with BLR using the parties' data. Parties 1 to 3 have,
$^{12}$ Increasing  $\tau_{i}$  (i.e., using  $\pmb{t}_k^i$  that diverges more from  $s_k$ ) can instead increase the surprise: The privacy-valuation trade-off (Sec. 3) only holds in expectation across all noise and SS realizations.
<sup>13</sup>The normalized posterior is also known as the power posterior. [37] discuss useful interpretation and benefits such as synthetically reducing the sample size, increasing the ease of computation/MCMC mixing and robustness to model misspecifications.
(a) Syn
(b) CaliH
Figure 2: (a-c) Graphs of party 2's valuation  $v_{2}$ , Shapley value  $\phi_{2}$ , attained reward value  $r_{2}$  vs. privacy guarantee  $\epsilon_{2}$  for various datasets. (d-e) Graphs of attained reward value  $r_{i}$  vs.  $\kappa_{i}$  (Sec. 5.2) and  $\tau_{i}$  (Sec. 5.1) for 2 different noise realizations. (f) Graph of similarity  $r_{i}'$  to grand coalition  $N$ 's posterior  $p(\theta | o_{N})$  vs.  $r_{i}$  for Syn dataset corresponding to (e).
(d)  $r_i$  vs.  $\kappa_{i},\tau_{i}$
(Non-monotonic in  $\tau_{i}$ )
(e)  $r_i$  vs.  $\kappa_{i}$ $\tau_{i}$
(Monotonic in  $\tau_{i}$ )
(f)  $r_i'$  vs.  $r_i$
respectively,  $20\%$ ,  $30\%$ , and  $50\%$  of the dataset with 6581 data points and 6 features. For PIMA Indian Diabetes classification dataset (Diab) [50], we use a Bayesian logistic regression model to predict whether a patient has diabetes based on sensitive inputs (e.g., patient's age, BMI, number of pregnancies). To reduce the training time, we only use the 4 PCA main components as features (to generate the approximate SS) [27]. Parties 1, 2, and 3 have, respectively,  $20\%$ ,  $30\%$ , and  $50\%$  of the dataset with 614 data points. As we are mainly interested in the impact of one party controlling its privacy guarantee  $\epsilon_{i}$ , for all experiments, we only vary party 2's from the default 0.1. We fix the privacy guarantees of others  $(\epsilon_{1} = \epsilon_{3} = 0.2)$  and  $\rho = 0.2$  in the  $\rho$ -Shapley fair reward scheme, and analyze party 2's reward and utility. Note that as  $\epsilon_{2}$  increases (decreases), party 2 becomes the most (least) valuable of all parties.
6.1 Privacy-valuation and Privacy-reward Trade-offs
For each dataset, we only vary the privacy guarantee of party  $i = 2$  with  $\epsilon_{2} \in [0.004, 0.02, 0.1, 0.5, 2.5, 12.5]$  and use the Gaussian mechanism and a fixed random seed to generate the perturbed SS  $o_{2}$  from the exact SS  $s_{2}$ . Fig. 2a-c plot the mean and shades the standard error of  $v_{i}, v_{N}, \phi_{i}$ , and  $r_{i}$  over 5 runs. The privacy-valuation and privacy-reward trade-offs can be observed: As the privacy guarantee weakens (i.e.,  $\epsilon_{2}$  increases), party 2's valuation  $v_{2}$ , Shapley value  $\phi_{2}$ , and attained reward value  $r_{2}$  increase. When  $\epsilon_{2}$  is large, party 2 will be the most valuable contributor and rewarded with  $p(\theta | o_{N})$ , hence attaining  $r_{i} = v_{N}$ . App. H.5 shows that the trade-offs do not hold for non-noise-aware inference.
6.2 Reward Control Mechanisms
We use the Syn experiment to compare the reward mechanisms that vary the noise addition using  $\tau_{i}$  (Sec. 5.1) vs. temper the likelihood using  $\kappa_{i}$  (Sec. 5.2). The mechanisms control  $q_{i}(\theta)$  (i.e., used to generate party  $i$ 's model reward) to attain the target reward values. For each value of  $\tau_{i}$  and  $\kappa_{i}$  considered, we repeat the posterior sampling and KL estimation method 5 times. Figs. 2d and 2e-f use different sets of sampled noise  $\{e_k^i\}_{k\in N}$  to demonstrate the stochastic relationship between  $r_i$  and  $\tau_{i}$ . In Fig. 2d, the non-monotonic disadvantage of noise addition can be observed: As  $\tau_{i}$  increases,  $r_i$  does not consistently decrease, hence making it hard to solve for the smaller  $\tau_{i}$  that attains  $r_i^* = 3$ . In contrast, as  $\kappa_{i}$  decreases from 1,  $r_i$  consistently decreases. Furthermore, in Fig. 2f, we demonstrate the other advantage of likelihood tempering: For the same  $r_i$ , tempering the likelihood leads to a higher similarity  $r_i'$  to the posterior  $p(\theta | o_N)$  than noise addition. In App. H.6, we report the relationship between  $r_i$  vs.  $\kappa_{i}$  and  $\tau_{i}$  for the other real-world datasets.
6.3 Utility of Model Reward
The utility (or the predictive performance) of both Bayesian models can be assessed by the mean negative log probability (MNLP) of a non-private test set. In short, MNLP reflects how unlikely the test set is given the perturbed SS and additionally cares about the uncertainty/confidence in the model predictions (e.g., due to the impact of DP noise). MNLP will be higher (i.e., worse) when the model
<sup>14</sup>We defer the analysis of valuation function (e.g., impact of varying coverage of input space) to App. H.4.
15 Such a test set is hard to obtain in practice and we are only using it for evaluation purposes.
(a) Syn (vs.  $\epsilon_{2}$ )
(b) CaliH (vs.  $\epsilon_{2}$ )
(c) Diab (vs.  $\epsilon_{2}$ )
Figure 3: (a-c) Graphs of utility of party 2's model reward measured by  $\mathrm{MNLP}_r$  vs. privacy guarantee  $\epsilon_{2}$  for various datasets. (d-f) Graphs of utility of model reward measured by  $\mathrm{MNLP}_r$  vs. attained reward value  $r_i$  under the two reward control mechanisms for various datasets.
(d) Syn (vs.  $r_i$ )
(e) CaliH (vs.  $r_i$ )
(f) Diab (vs.  $r_i$ )
is more uncertain of its accurate predictions or overconfident in inaccurate predictions on the test set; see App. H.2 for an in-depth definition.
Privacy trade-offs. Figs. 3a-c illustrate the privacy-utility trade-off described in Sec. 1: As  $\epsilon_{2}$  decreases (i.e., privacy guarantee strengthens), the  $\mathrm{MNLP}_N$  of grand coalition  $N$ 's collaboratively trained model and the  $\mathrm{MNLP}_i$  of party  $i = 2$ 's individually trained model generally increase, so their utilities drop  $(\dagger)$ . This motivates the need to incentivize party 2 against selecting an excessively small  $\epsilon_{2}$  by enforcing privacy-valuation and privacy-reward trade-offs. From Figs. 3a-c, the impact of our scheme enforcing the trade-offs can be observed: As  $\epsilon_{2}$  decreases, the  $\mathrm{MNLP}_r$  of party  $i = 2$ 's model reward increases.
Remark. In Fig. 3c, an exception to  $(\dagger)$  is observed. The exception illustrates that the privacy-valuation trade-off may not hold for a valuation function based on the performance on a validation set.
Individual rationality. It can be observed from Figs. 3a-c that as  $\epsilon_{2}$  decreases, the  $\mathrm{MNLP}_r$  of party  $i = 2$ 's model reward increases much less rapidly than the  $\mathrm{MNLP}_i$  of its individually trained model. So, it is rational for party  $i = 2$  to join the collaboration to get a higher utility.
Remark. Party  $i = 2$ 's utility gain appears small when  $\epsilon_{2}$  is large due to parties 1 and 3's selection of strong privacy guarantee  $\epsilon = 0.2$ . Party  $i$  can gain more when other parties require weaker privacy guarantees such as  $\epsilon = 2$  instead (see App. H.5).
Likelihood tempering is a better reward control mechanism. Extending Sec. 6.2, we compare the utility of party  $i$ 's model reward generated by noise addition vs. likelihood tempering in Figs. 3d-f. Across all experiments, likelihood tempering (with  $\kappa_{i}$ ) gives (i) a lower  $\mathrm{MNLP}_r$  and hence a higher utility, and (ii) a lower variance in  $\mathrm{MNLP}_r$  than varying the noise addition (with  $\tau_{i}$ ).
Fig. 5 in App. B gives a diagrammatic overview showing how our work fills the gap in existing works.
Data Valuation. Most data valuation methods are not differentially private and directly access the data. For example, computing the information gain [47] or volume [59] requires the design matrix. While it is possible to make these valuation methods differentially private (see App. H.3) or value DP trained models using validation accuracy (on an agreed, public validation set), the essential properties of our valuation function (V2-V3) may not hold.
Privacy Incentive. Though the works of [20, 60] reward parties directly proportional to their privacy budget, their methods do not incentivize data sharing as a party does not fairly receive a higher reward value for contributing a larger, more informative dataset. While the work of [28] artificially creates a privacy-reward trade-off by paying each party  $i$  the product of its raw data's Shapley value  $\phi_i$  and a monotonic transformation of  $\epsilon_i$ , it neither ensures DP w.r.t. the mediator nor fairly considers how a stronger DP guarantee may reduce party  $i$ 's marginal contribution to others (hence  $\phi_i$ ). The work of [13] considers data acquisition from parties with varying privacy requirements but focuses on the mean estimation problem and designing payment and privacy loss functions to get parties to report their true unit cost of privacy loss. Our work here distinctively considers Bayesian models and fairness and allows parties to choose their privacy guarantees directly while explicitly enforcing a privacy-reward trade-off.
Difficulties ensuring incentives with existing DP/FL works. The one posterior sampling (OPS) method [56, 16] proposes that each party  $i$  can achieve DP by directly releasing samples from the posterior  $p(\theta | \mathcal{D}_i)$  (if the log-likelihood is bounded). However, OPS is data inefficient and may not guarantee privacy for approximate inference [15]. It is unclear how we can privately value a coalition  $C$  and sample from the joint posterior  $p(\theta | \{\mathcal{D}_i\}_{i \in C})$ . DP-FedAvg/DP-FedSGD [36] or DP-SGD [1] enable collaborative but private training of neural networks by requiring each party  $i$  to clip and add Gaussian noise to its submitted gradient updates. However (in addition to the valuation function issue above), it is tricky to ensure that the parties' rewards satisfy data sharing incentives. In each round of FL, parties selected will receive the (same) latest model parameters to compute gradient updates. This setup goes against the fairness (P3) incentive as parties who share less informative gradients should be rewarded with lower quality model parameters instead. Although the unfairness may potentially be corrected via gradient-based [58] or monetary rewards, there is no DP reward scheme to guarantee a party better model reward from collaboration than in solitude or a higher monetary reward than its cost of participation, hence violating individual rationality.
Unlike existing works in collaborative ML that solely focus on the fairness incentive, our proposed scheme further (i) ensures privacy for the parties during valuation and in model rewards and (ii) enforces a privacy-valuation trade-off to deter parties from unfetteredly selecting excessive DP guarantees to maximize the utility of collaboratively trained models.[16] This involves novelly combining our proposed Bayesian surprise valuation function and reward control mechanism with DP noise-aware inference. We empirically evaluate our scheme on several datasets. Our likelihood tempering reward control mechanism consistently preserves better predictive performance.
Our work has two limitations which future work should overcome. Firstly, we only consider ML models with SS (see App. A.1 for applications) and a single round of sharing information with the mediator as a case study to show the incentives and trade-offs can be achieved. Future work should generalize our scheme to ML models without an explicit SS.
Next, like the works of [18, 17, 25, 40, 47, 51] and others, we do not consider the truthfulness of submitted information and value data as-is. This limitation is acceptable for two reasons. 1) Parties such as hospitals and firms will truthfully share information as they are primarily interested in building and receiving a model reward of high quality and may additionally be bound by the collaboration's legal contracts and trusted data-sharing platforms. For example, with the use of X-road ecosystem,[17] parties can upload a private database which the mediator can query for the perturbed SS. This ensures the authenticity of the data (also used by the owner) and truthful computation given the uploaded private database. 2) Each party would be more inclined to submit true information as any party  $k$  who submits fake SS will reduce its utility from the collaboration. This is because party  $k$ 's submitted SS is used to generate  $k$ 's model reward and cannot be replaced locally as party  $k$  will only receive posterior samples. Future work should seek to verify and incentivize truthfulness.
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-RP-2020-018). This work is also supported by the National Natural Science Foundation of China (62206139) and the Major Key Project of PCL (PCL2023AS7-1).
16App. I discusses the ethical implications and limitations of our work, and other questions a reader may have.
$^{17}$ https://joinup.ec.europa.eu/collection/ict-security/solution/x-road-data-exchange-layer/about, https://x-road.global/