SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- The predictive mean  $\widehat{\mu}(\boldsymbol{x}_*)$  is the averaged prediction of  $\boldsymbol{y}_*$  (i.e.,  $\boldsymbol{w}^\top \boldsymbol{x}_*$ , where  $\boldsymbol{w}$  is part of the model parameters  $\theta$ ) over all samples of the model parameters  $\theta$ .
- The predictive variance  $\widehat{\sigma^2}(\boldsymbol{x}_*)$  is computed using the variance decomposition formula, i.e., the sum of the averaged  $\sigma^2$  (the unknown variance parameter within  $\theta$ ) and the computed variance in predictions over samples, i.e.,  $= m^{-1}\sum_{j=1}^{m}\sigma_j^2 + \widehat{\mu}^2(\boldsymbol{x}_*) - \widehat{\mu}(\boldsymbol{x}_*)^2$ .
Classification. We can estimate  $p(y_{*}|\pmb{x}_{*},\pmb{o}_{i})$ , for test input  $\pmb{x}_{*}$  using the Monte Carlo approximation [39] and reusing the samples  $\theta$  from  $p(\theta |\pmb{o}_i)$ . Concretely,  $p(y = 1|\pmb{x}_{*},\pmb{o}_{i})\cong m^{-1}\sum_{j = 1}^{m}\sigma (\theta^{\top}\pmb{x}_{*})$ . The MNLP is equivalent to the cross-entropy loss.
H.3Baselines
To plot the figures in Sec. 6, the baseline DP and collaborative ML works must
1. work for similar models, i.e., Bayesian linear and logistic regression;
2. not use additional information to value coalitions and generate model rewards (to preserve the DP post-processing property); and
3. decide feasible model reward values and suggest how model rewards can be generated.
Work of [59]. Valuation by volume is model-agnostic (satisfying criteria 1). Each party  $i \in N$  can submit the noisy version of  $\mathbf{X}_i^\top \mathbf{X}_i$  with DP guarantees to the mediator who can sum them to value coalitions (satisfying criteria 2). The work does not propose a model reward scheme to satisfy criteria 3.
Work of [47]. [47] only considered Bayesian linear regression (with known variance) and it is not straightforward to compute information gain on model parameters for Bayesian linear regression (with unknown variance) and Bayesian logistic regression. Thus, the work does not satisfy criteria 1. For Bayesian linear regression (with known variance), each party  $i \in N$  can submit the noisy version of  $X_{i}^{\top}X_{i}$  with DP guarantees to the mediator who can sum them to value coalitions (satisfying criteria 2). The work proposed a model reward scheme which involves adding noise to the outputs  $y$  (satisfying criteria 3 but has to be adapted to ensure DP).
DP-FL works. A promising approach is to use DP-FedAvg/DP-FedSGD [36] to learn any model parameters (satisfying criteria 1) in conjunction with FedSV [55] to value coalitions without additional information (satisfying criteria 2). However, to our knowledge, these works will not satisfy criteria 3 as they do not suggest how to generate model rewards of a target reward value.
As no existing work satisfies all criteria, we compare against (1) using non-noise-aware inference instead of noise-aware inference, all else equal (in App. H.5); and (2) an adapted variant of the reward control via noise addition (in Sec. 5.1, Sec. 6, and App. G).
H.4 Valuation Function
In Sec. 6, we only vary the privacy guarantee  $\epsilon_{i}$  of one party  $i$ . In this subsection, we will analyze how other factors such as the coverage of the input space and the number of posterior samples on the valuation  $v_{i}$ .
Coverage of input space. We vary the coverage of the input space by only keeping those data points whose first feature value is not greater than the 25, 50, 75, 100-percentile. Across all experiments in Table 1, it can be observed that as the percentile increases (hence, data quantity and coverage improve), the surprise elicited by the perturbed SS  $o_N$  increases in tandem with the surprise elicited by the exact SS  $s_N$ .
Feature 0's Percentile Range	[0, 25]	[0, 50]	[0, 75]	[0, 100]
Syn
Surprise of sN	12.030	12.698	13.322	14.183
Surprise of oN	6.007	6.775	7.410	8.438
CaliH
Surprise of sN	21.261	22.401	26.578	28.422
Surprise of oN	9.282	10.212	12.121	17.959
Diab
Surprise of sN	5.450	6.279	7.019	7.258
Surprise of oN	1.854	2.712	3.909	5.394
Table 1: We report the surprise elicited by  $s_N$  and  $o_N$  (with  $\epsilon = 1$ ) when using the subset of data with first feature value not exceeding the 25, 50, 75, 100-percentile for all datasets.
Number of posterior samples. For a consistent KL estimator, the bias/variance of the KL estimator should decrease with a larger number of posterior samples.
Gibbs sampling. We compare the estimated surprise using various degrees of thinning (i.e., keeping only every  $t$ -th sample) to generate 30000 samples for the CaliH dataset. In Table 2, it can be observed that although the total number of samples is constant, the surprise differs significantly. Moreover,
as  $t$  increases, the surprise decreases at a decreasing rate and eventually converges. This may be because consecutive Gibbs samples are highly correlated and close, thus causing us to underestimate the distance to the  $k$ -th nearest-neighbors within  $\theta \mid o_N$  (see discussion in App. C.3). Increasing  $t$  reduces the correlation and closeness and better meets the i.i.d. samples assumption of the nearest-neighbor-based KL estimation method [45].
Thin every t-th sample	Surprise vN
1	14.849 ± 0.036
2	12.839 ± 0.033
4	11.626 ± 0.018
8	11.038 ± 0.022
16	10.834 ± 0.033
20	10.790 ± 0.032
30	10.793 ± 0.011
NUTS logistic regression. After drawing 10000 samples for the Diab dataset using the default setting, we analyze how using a subset of the samples will affect the estimated surprise. In particular, we consider using (i) the first  $m$  samples or (ii) thinned  $m$  samples where we only keep every  $10000 / m$ -th sample.
In Table 3, it can be observed that as the number  $m$  of samples increases, the standard deviation of the estimated surprise decreases. Moreover, there is no significant difference between using the first  $m$  samples or the thinned  $m$  samples. This suggests that the samples are sufficiently independent and thinning is not needed.
Table 2: Thinning factor  $t$  vs. surprise  ${v}_{N}$  for CaliH dataset.
No. m of Samples	Surprise vN
First 1000	2.227 ± 0.051
Thinned 1000	2.211 ± 0.034
First 2000	2.117 ± 0.049
Thinned 2000	2.117 ± 0.045
First 5000	2.145 ± 0.037
Thinned 5000	2.119 ± 0.038
All 10000	2.128 ± 0.030
Table 3: Number  $m$  of samples vs. surprise  ${v}_{N}$  for Diab dataset.
H.5 Additional Experiments on Valuation, Privacy-valuation Trade-off, and Privacy-reward Trade-off
(c) Diab
Figure 8: Graphs of Shapley value  $\phi_{i}$  of parties  $i = 1,2,3$  vs. party 2's privacy guarantee  $\epsilon_{2}$  for various datasets.
Shapley value. In Fig. 8, it can be observed that as party 2 weakens its privacy guarantee (i.e.,  $\epsilon_{2}$  increases), its Shapley value  $\phi_{2}$  increases while other parties' Shapley values (e.g.,  $\phi_{3}$ ) decrease.
When party 2 adds less noise to generate its perturbed SS  $o_2$ , others add less value (i.e., make lower marginal contributions (MC)) to coalitions containing party 2. Party 2 changes from being least valuable to being most valuable, even though it has more data than party 1 and less data than party 3.
(b) CaliH
Figure 9: Graphs of party 2's valuation  $v_{2}$ , Shapley value  $\phi_{2}$ , and attained reward value  $r_{2}$  vs. privacy guarantee  $\epsilon_{2}$  for various datasets when performing non-noise-aware (i.e., noise-naive) inference, i.e.,  $p(\theta | S_{N} = o_{N})$  and treating  $o_{N}$  as though it is  $s_{N}$ .
(c)Diab
Without DP noise-aware inference. In Fig. 9a, it can be observed that as  $\epsilon_{2}$  increases,  $v_{i}$  and  $\phi_{i}$  for party  $i = 2$  do not strictly increase. In Figs. 9b-c, it can be observed that as  $\epsilon_{2}$  increases,  $v_{i}$  and  $\phi_{i}$  for party  $i = 2$  decrease instead. The consequence of non-noise-aware inference is undesirable for incentivization — party 2 unfairly gets a lower valuation and reward for using a weaker privacy guarantee, i.e., a greater privacy sacrifice. Moreover, when  $\epsilon_{2}$  is small (i.e., under a strong privacy guarantee), party 2 is supposed to be least valuable. However, the significantly different  $o_{2}$  causes party 2 to have the highest valuation and be rewarded with the grand coalition  $N$ 's model (i.e.,  $r_{i}$  close to  $v_{N}$ ) instead.
Lastly, we also observe that without DP noise-aware inference, the utility of the model reward is small. For example, the naive posterior for the Syn dataset results in an MNLP larger than 100.
Conditions for larger improvement in MNLP. In Fig. 3, it seems that the utility of party  $i = 2$ 's model reward measured by  $\mathrm{MNLP}_r$  cannot improve significantly over that of its individually trained model when  $\epsilon_2$  is large. However, party  $i$ 's  $\mathrm{MNLP}_r$  can be improved by a larger extent when (i) any other party  $j \neq i$  selects a weaker privacy guarantee (i.e., a larger  $\epsilon_j$ ), thus improving the utility of the collaboratively trained model or (ii) party  $i$  and others have lower data quantity (i.e., smaller  $c_k$  for all  $k \in N$ ) and are unable to individually train a model of high utility. Figs. 10a, 10b, and 10c are examples of (i), (ii), and (i+ii), respectively. In Fig. 10a, the  $\mathrm{MNLP}_N$  of grand coalition  $N$ 's collaboratively trained model is lower than that in Fig. 3a. In Fig. 10b, the  $\mathrm{MNLP}_i$  of party  $i$ 's model is higher due to less data. In these examples, we observe that a party can still gain a significant improvement  $\mathrm{MNLP}_i - \mathrm{MNLP}_r$  when  $\epsilon_i > 1$ .
Condition (i) for a larger improvement in  $\mathrm{MNLP}_r$  is satisfied when the trade-off deters parties from selecting excessive DP guarantees, i.e., it incentivizes parties to select weaker DP guarantees that still meet their legal and customers' requirements. Condition (ii) should be satisfied in most real-life scenarios where a party wants to participate in collaborative ML and federated learning. The party (e.g., bank) is unable to achieve its desired utility with its individually trained model due to limited data and collaborates with others to unlock any improvement in the utility of a collaboratively trained model.
(a) Syn, larger  $\epsilon_{j}$
(b) Syn, halved  $\{c_k\}_{k\in N}$
(c) Diab, larger  $\epsilon_{k}$  & halved  $\{c_k\}_{k\in N}$
Figure 10: Graphs of utility of party  $i = 2$ 's model reward  $q_{i}(\theta)$  measured by  $\mathrm{MNLP}_r$  vs. privacy guarantee  $\epsilon_{2}$  for Syn dataset (a) when  $\epsilon_{1} = \epsilon_{3} = 2$  instead of 0.2, and (b) when only a subset of  $c_{k} / 2$  data points is available for every party  $k = 1, 2, 3$ . (c) Graph of utility of party  $i = 1$ 's model reward  $q_{i}(\theta)$  measured by  $\mathrm{MNLP}_r$  vs. privacy guarantee  $\epsilon_{1}$  for Diab dataset when  $\epsilon_{2} = \epsilon_{3} = 2$  instead of 0.2 and only a subset of  $c_{k} / 2$  data points is available for every party  $k = 1, 2, 3$ .
Higher  $\lambda = 10$ . In Fig. 11, the privacy-valuation, privacy-reward, and privacy-utility trade-offs are still observed when parties select a higher  $\lambda = 10$  when enforcing the Rényi DP guarantee. Moreover, the utility of party 2's model reward is higher (i.e., lower MNLP) than that of its individually trained model.
(a) Syn
(c) Diab
(d) Syn (MNLP vs.  $\epsilon_{2}$ )
Figure 11: Graphs of party 2's (a-c) valuation  $v_{2}$ , Shapley value  $\phi_{2}$ , and attained reward value  $r_{2}$ , and (d-f) utility of its model reward  $q_{i}(\theta)$  measured by  $\mathrm{MNLP}_r$  vs. privacy guarantee  $\epsilon_{2}$  for various datasets when enforcing  $(\lambda = 10, \epsilon_{i})$ -Rényi DP guarantee.
) CaliH (MNLP vs.  $\epsilon_{2}$ )
(f) Diab (MNLP vs.  $\epsilon_{2}$ )
H.6 Additional Experiments on Reward Control Mechanisms
For the CaliH dataset, there is a monotonic relationship between  $r_i$  vs. both  $\kappa_i$  and  $\tau_i$ , as shown in Fig. 12a. However, it can be observed from Figs. 12b-c that for the same attained reward value  $r_i$ , adding scaled noise variance  $\tau_i$  will lead to a lower similarity  $r_i'$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  and utility of model reward (higher  $\mathrm{MNLP}_r$ ) than tempering the likelihood by  $\kappa_i$ .
(a)  $r_i$  vs.  $\kappa_{i},\tau_{i}$
(b)  $r_i^\prime$  vs.  $r_i$
Figure 12: (a) Graph of attained reward value  $r_i$  vs.  $\kappa_i$  (Sec. 5.2) and  $\tau_i$  (Sec. 5.1), (b) graph of similarity  $r_i'$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  vs.  $r_i$ , and (c) graph of utility of party  $i = 2$ 's model reward  $q_i(\theta)$  measured by  $\mathrm{MNLP}_r$  vs.  $r_i$  for CaliH dataset.
(c)  $\mathrm{MNLP}_r$  vs.  $r_i$
For Diab dataset, there is a monotonic relationship between  $r_i$  vs. both  $\kappa_i$  and  $\tau_i$ , as shown in Fig. 13a. However, it can be observed from Fig. 13b-c that for the same attained reward value  $r_i$ , tempering the likelihood by  $\kappa_i$  leads to a higher similarity  $r_i'$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  and utility of model reward (lower  $\mathrm{MNLP}_r$ ) than adding scaled noise variance  $\tau_i$ .
(a)  $r_i$  VS.  $\kappa_{i2}$ $\tau_{i}$
Figure 13: (a) Graph of attained reward value  $r_i$  vs.  $\kappa_i$  (Sec. 5.2) and  $\tau_i$  (Sec. 5.1), (b) graph of similarity  $r_i'$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  vs.  $r_i$ , and (c) graph of utility of party  $i = 2$ 's model reward  $q_i(\theta)$  measured by  $\mathrm{MNLP}_r$  vs.  $r_i$  for Diab dataset.
(b)  $r_i^\prime$  vs.  $r_i$
Problematic noise realization. We will show here and in Fig. 14a that some (large) noise realization can result in a non-monotonic relationship between the attained reward value  $r_i$  vs. the scaled additional noise variance  $\tau_i$ . As a result, it is hard to bracket the smallest root  $\tau_i$  that solves for  $r_i = r_i^*$  (e.g.,  $= 2$  or  $= 3$ ). Moreover, it can be observed from Figs. 14b-c that the model reward's posterior  $q_i(\theta)$  has a low similarity  $r_i'$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  and a much higher  $MNLP_r$  than the prior. This suggests that injecting noise does not interpolate well between the prior and the posterior. In these cases, it is not suitable to add scaled noise variance  $\tau_i$  and our reward control mechanism via likelihood tempering with  $\kappa_i$ , is preferred instead.
(a)  $r_i$  vs.  $\kappa_{i}$ $\tau_{i}$
(Non-monotonic in  $\tau_{i}$ )
Figure 14: (a) Graph of attained reward value  $r_i$  vs.  $\kappa_i$  (Sec. 5.2) and  $\tau_i$  (Sec. 5.1), (b) graph of similarity  $r_i'$  to the grand coalition  $N$ 's posterior  $p(\theta | o_N)$  vs.  $r_i$ , and (c) graph of utility of party  $i = 2$ 's model reward  $q_i(\theta)$  measured by  $\mathrm{MNLP}_r$  vs.  $r_i$  for Syn dataset corresponding to (a).
I Other Questions
Question 1: Are there any ethical concerns we foresee with our proposed scheme?
Answer: Our privacy-valuation trade-off (V3) should deter parties from unfetteredly selecting excessively strong DP guarantees. Parties inherently recognize the benefits of stronger DP guarantees and may prefer such benefits in collaboration out of overcaution, mistrust of others, and convenience. The trade-off counteracts (see Fig. 1) the above perceived benefits by explicitly introducing costs (i.e., lower valuation and quality of model reward). Consequently, parties will carefully select a weaker yet satisfactory privacy guarantee they truly need.
However, a potential concern is that parties may opt to sacrifice their data's privacy to obtain a higher-quality model reward. The mediator can alleviate this concern by enforcing a minimum privacy guarantee (i.e., maximum  $\epsilon$ ) each party must select. The model rewards will preserve this minimum privacy guarantee due to P1. The mediator can also decrease the incentive by modifying  $v_{C}$ .
Another potential concern is that if parties have data with significantly different quantity/quality/privacy guarantees, the weaker party  $k$  with fewer data or requiring a stronger privacy guarantee will be denied the best model reward (i.e., trained on the grand coalition's SS) and instead rewarded with one that is of lower quality for fairness. The mediator can alleviate the concern and at least ensure individual rationality (P4) by using a smaller  $\rho$  so that a weaker party  $k$  can obtain a higher-quality model reward with a higher target reward value  $r_k^*$ .
Question 2: Is it sufficient and reasonable to value parties based on the submitted information  $\{c_i,o_i,p(Z_i)\}_{i\in N}$  instead of ensuring and incentivizing truthfulness? Would parties strategically declare other values to gain a higher valuation and reward?
Answer: An ideal collaborative ML scheme should additionally incentivize parties to be truthful and verify the authenticity of the information provided. However, achieving the "truthfulness" incentive is hard and has only been tackled by existing works to a limited extent. Existing work cannot discern if the data and information declared are collected or artificially created (e.g., duplicated) and thus, this non-trivial challenge is left to future work. The work of [32] assigns and considers each client's reputation from earlier rounds, while the works of [29, 31] measure the correlation in parties' predictions and model updates. The work of [7] proposes a payment rule based on the log pointwise mutual information between a party's dataset and the pooled dataset of others. This payment rule guarantees that when all other parties are truthful (i.e., a strong assumption), misreporting a dataset with an inaccurate posterior is worse (in expectation) than reporting a dataset with accurate posterior.[22]
Thus, like the works of [18, 17, 25, 40, 47, 51] and others, we value data as-is and leave achieving the "truthfulness" incentive to future work. In practice, parties such as hospitals and firms will truthfully share information as they are primarily interested in building and receiving a model reward of high quality and may additionally be bound by the collaboration's legal contracts and trusted data-sharing platforms like Ocean Protocol [43]. For example, with the use of X-road ecosystem,[23] parties can maintain a private database which the mediator can query for the perturbed SS. This ensures the authenticity of the data (also used by the owner) and truthful computation given the uploaded private database.
Lastly, a party  $k$  who submits fake SS will also reduce its utility from the collaboration. Party  $k$ 's fake SS will affect the grand coalition's posterior of the model parameters given all perturbed SS and is also used to generate  $k$ 's model reward. As party  $k$  only receives posterior samples,  $k$  cannot replace the fake SS with its exact SS locally. As party  $k$  have to bear the consequences of the fake SS, it would be more likely to submit true information.
Question 3: Why do we only consider Bayesian models with SS?
22The payment rule may be unfair as when two parties are present, they will always be paid equally.
$^{23}$ https://joinup.ec.europa.eu/collection/ict-security/solution/x-road-data-exchange-layer/about, https://x-road.global/
Answer: See App A.1 for a background on SS. Our approach would also work for Bayesian models with approximate SS, such as Bayesian logistic regression, and latent features extracted by a neural network.
1. The exact SS  $s_i$  captures all the information (i.e., required by the mediator) within party  $i$ 's dataset  $\mathcal{D}_i$ . Thus, the mediator can do valuation and generate model rewards from the perturbed SS  $\{o_i\}_{i \in N}$  without requesting more information from the parties. This limits the privacy cost and allows us to rely on the DP post-processing property.
2. In Sec. 3, the proof that Def. 3.1 satisfies a privacy-valuation trade-off (V3) uses the properties of SS.
Our work introduces privacy as an incentive and simultaneously offers a new perspective that excessive DP can and should be deterred by introducing privacy-valuation and privacy-reward trade-offs and accounting for the DP noise. We use Bayesian models with SS as a case study to show how the incentives and trade-offs can be achieved. It is up to the future work to address the non-trivial challenge of ensuring privacy-valuation and privacy-reward trade-offs for other models.
Question 4: Can alternative fair reward schemes be used in place of  $\rho$ -Shapley fair reward scheme [47]?
Answer: Yes, if they satisfy P3 and P4. For example, if the exchange rate between the perturbed SS quality and monetary payment is known, then the scheme of [40] can be used to decide the reward instead. Our work will still ensure the privacy-valuation trade-off and provide the mechanism to generate the model reward  $q_{i}(\theta)$  to attain any target reward value  $r_{i}^{*}$  while preserving similarity to the grand coalition  $N$ 's model (P5).
Question 5: What is the difference between our work here and that of [47]?
Answer: We clearly outlined our contributions in bullet points at the end of the introduction section (Sec. 1) and in Fig. 1.
At first glance, our work seems to only add a new privacy incentive. However, as discussed in the introduction section (Sec. 1), privacy is barely considered by existing collaborative ML works and raises significant challenges. The open questions/challenges in [64]'s survey on adopting DP in game-theoretic mechanism design (see Sec. 7.1 therein) inspire us to ask the following questions:
- How can DP and the aims of cooperative game theory-inspired collaborative ML be compatible? Will DP invalidate existing properties like fairness?
- How should parties requiring a strong DP guarantee be prevented from unfairly and randomly obtaining a high-quality model reward?
We propose to enforce a provable privacy-valuation trade-off to answer the latter. The enforcement involves novelly selecting and combining the right valuation function and tools, such as DP noise-aware inference.
Additionally, we propose a new reward control mechanism that involves tempering the likelihood (practically, scaling the SS) to preserve similarity to the grand coalition's model (P5) and hence increase the utility of the model reward.
Question 6: Will a party with high-quality data (e.g., a large data quantity, less need for DP guarantee) be incentivized to participate in the collaboration?
Answer: From Fig. 3, it may seem that a rich party  $i$  with ample data and a weak privacy guarantee (i.e., large  $\epsilon_{i}$ ) has a lower utility of model reward to gain from the collaboration. However, it may still be keen on a further marginal improvement in the utility of its model reward (e.g., increasing the classification accuracy from  $97\%$  to  $99\%$  and predicting better for some sub-groups) and can reasonably expect a better improvement as other parties are incentivized by our scheme (through enforcing a privacy-valuation trade-off and fairness F4) to contribute more data at a weaker yet satisfactory DP guarantee (see App. H.5). Moreover, a rich party does not need to be concerned about others unfairly benefiting from its contribution as our scheme guarantees fairness through Shapley value. In Fig. 8, as a party selects a weaker DP guarantee (and all
else being held constant), the Shapley values of others, which determine their model rewards, decrease.
Question 7: What is the impact of varying other hyperparameters?
Answer: The work of [47] proposes  $\rho$ -Shapley fairness and theoretically and empirically show that any  $\rho > 0$  guarantees fairness across parties and a smaller  $\rho$  will lead to a higher attained reward value  $r_i$  for all other parties which do not have the largest Shapley value. These properties apply to our problem setup, and using a larger  $\rho$  will worsen/reduce  $r_i$  and the utility of party  $i$ 's model reward  $q_i(\theta)$  measured by  $\mathrm{MNLP}_r$ . The work of [47] has empirically shown that the number of parties does not impact the scheme's effectiveness. However, it affects the time complexity to compute the exact and approximate SV.