SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} q _ {i} (\theta) \propto p ^ {\kappa_ {i}} \left(d _ {1} | \theta\right) \dots p ^ {\kappa_ {i}} \left(d _ {c _ {k}} | \theta\right) p (\theta | \eta) \\ \propto \left[ \left(\prod_ {l = 1} ^ {c _ {k}} h (d _ {l})\right) ^ {\kappa_ {i}} \exp \left[ \kappa_ {i} \left[ g \left(\mathcal {D} _ {k}\right) \cdot \theta - c _ {k} A (\theta) \right] \right] \right. [ h (\theta) \exp [ T (\theta) \cdot \eta - B (\eta) ] ] \\ \propto \exp \left[ \kappa_ {i} g \left(\mathcal {D} _ {k}\right) \cdot \theta - \kappa_ {i} c _ {k} A (\theta) + T (\theta) \cdot \eta - B (\eta) \right] \\ \propto \exp \left[ \left(\left[ \kappa_ {i} g (\mathcal {D} _ {k}) ^ {\top}, \kappa_ {i} c _ {k} \right] ^ {\top} + \eta\right) \cdot T (\theta) - C ^ {\prime} (\eta) \right] \\ \end{array}
$$
where  $C^\prime (\eta)$  is chosen such that the distribution is normalized.
Thus, tempering the likelihood by  $\kappa_{i}$  is equivalent to scaling the SS  $g(\mathcal{D}_k)$  and data quantity  $c_{k}$ . We additionally proved that the normalized posterior can be obtained from the scaled SS and data quantity via the conjugate update.
Bayesian Linear Regression (BLR). The Bayesian linear regression model was introduced in App. A.1 To recap, BLR model parameters  $\theta$  consists of the weight parameters  $\boldsymbol{w} \in \mathbb{R}^w$  and the noise variance  $\sigma^2$ . BLR models the relationship between the concatenated output vector  $\boldsymbol{y}$  and the design matrix  $\boldsymbol{X}$  as  $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{w} + \mathcal{N}(0,\sigma^2\boldsymbol{I})$ . The tempered likelihood
$$
\begin{array}{l} p ^ {\kappa_ {i}} (\boldsymbol {y} | \boldsymbol {X}, \boldsymbol {w}, \sigma^ {2}) \propto \left[ (2 \pi \sigma^ {2}) ^ {- \frac {c}{2}} \exp \left(- \frac {(\boldsymbol {y} - \boldsymbol {X} \boldsymbol {w}) ^ {\top} (\boldsymbol {y} - \boldsymbol {X} \boldsymbol {w})}{2 \sigma^ {2}}\right) \right] ^ {\kappa_ {i}} \\ = \left(2 \pi \sigma^ {2}\right) ^ {- \frac {c \kappa_ {i}}{2}} \exp \left[ \frac {- 1}{2 \sigma^ {2}} \kappa_ {i} \boldsymbol {y} ^ {\top} \boldsymbol {y} + \frac {1}{\sigma^ {2}} \boldsymbol {w} ^ {\top} \kappa_ {i} \boldsymbol {X} ^ {\top} \boldsymbol {y} - \frac {1}{2 \sigma^ {2}} \boldsymbol {w} ^ {\top} \kappa_ {i} \boldsymbol {X} ^ {\top} \boldsymbol {X} \boldsymbol {w} \right]. \\ \end{array}
$$
only depends on the scaled sufficient statistics  $\kappa_{i}(\pmb{y}^{\top}\pmb{y},\pmb{X}^{\top}\pmb{y},\pmb{X}^{\top}\pmb{X})$ . When the prior  $p(\pmb{\theta})$  follows the conjugate normal inverse-gamma distribution, the power posterior can be obtained from the scaled SS and data quantity via the conjugate update.
$^{20}p(\theta)$  and  $p(\theta |\mathcal{D}_i)$  belong to the same exponential family.
Generalized Linear Model (GLM). App. A.1 introduces GLMs and states that the polynomial approximation to the GLM mapping function is an exponential family model. Tempering the GLM likelihood function by  $\kappa_{i}$  is equivalent to scaling the GLM mapping function by  $\kappa_{i}$  and can achieved by scaling the polynomial approximate SS by the same factor.
E.2 Smaller Tempering Factor Decreases Reward Value Proof
The KL divergence between two members of the same exponential family with natural parameters  $\eta$  and  $\eta'$ , and log partition function  $B(\cdot)$  is given by  $(\eta - \eta')^\top \nabla B(\eta) - B(\eta) + B(\eta')$  [41]. To ease notational overload, we abuse some existing ones, which only apply in this subsection, by letting  $s_N \triangleq \sum_{k \in N} s_k$  and  $c_N \triangleq \sum_{k \in N} c_k$ . Let  $\eta'$  and  $\eta$  be the natural parameters of the prior and the normalized tempered posterior distribution (used to generate a model reward with value  $r_i$ ), respectively. Then,  $\eta = \eta' + \kappa_i \left[ s_N^\top, c_N \right]^\top$ . For  $\kappa_i \in [0,1]$ , the derivative of  $r_i$  w.r.t.  $\kappa_i$  is non-negative:
$$
\begin{array}{l} \frac {\mathrm {d} r _ {i}}{\mathrm {d} \kappa_ {i}} = \frac {\partial r _ {i}}{\partial \eta} \frac {\partial \eta}{\partial \kappa_ {i}} \\ = \left(\left(\eta - \eta^ {\prime}\right) ^ {\top} \nabla^ {2} B (\eta) + \nabla B (\eta) - \nabla B (\eta)\right) \left[ \boldsymbol {s} _ {N} ^ {\top}, c _ {N} \right] ^ {\top} \\ = \left[ \kappa_ {i} \boldsymbol {s} _ {N} ^ {\top}, \kappa_ {i} c _ {N} \right] \nabla^ {2} B (\eta) \left[ \boldsymbol {s} _ {N} ^ {\top}, c _ {N} \right] ^ {\top} \\ = \kappa_ {i} \left[ \boldsymbol {s} _ {N} ^ {\top}, c _ {N} \right] \nabla^ {2} B (\eta) \left[ \boldsymbol {s} _ {N} ^ {\top}, c _ {N} \right] ^ {\top} \geq 0. \\ \end{array}
$$
As  $B(\eta)$  is convex w.r.t.  $\eta$ , the second derivative<sup>21</sup>  $\nabla^2 B(\eta)$  is positive semi-definite, so  $\left[\boldsymbol{s}_N^\top, c_N\right] \nabla^2 B(\eta) \left[\boldsymbol{s}_N^\top, c_N\right]^\top \geq 0$ .
Hence, for  $\kappa_{i}\in [0,1]$ , the KL divergence is non-decreasing as  $\kappa_{i}$  increases to 1. In other words, as  $\kappa_{i}$  shrinks towards 0, the KL divergence is decreasing; equality only holds when the variance of the SS is 0.
E.3 Implementation of Reward Control Mechanisms
This subsection introduces how to obtain the model reward  $q_{i}(\theta)$  for each party  $i$  in Sec. 5.
Update for noise addition (varying  $\tau_{i}$ ). We update the inputs to Algorithm 1 for BLR or the No-U-Turn sampler for GLM. To generate party  $i$ 's posterior samples, for every party  $k \in N$ , the algorithm uses the further perturbed SS  $t_{k}^{i}$  instead of the perturbed SS  $o_{k}$ . Moreover, the algorithm considers the total DP noise  $Z_{k} + \mathcal{N}(0, 0.5 \lambda \Delta_{2}^{2}(g_{k}) \tau_{i} I)$  instead of only the noise  $Z_{k}$  added by party  $k$ .
Update for likelihood tempering (varying  $\kappa_{i}$ ). To generate party  $i$ 's posterior samples, for every party  $k \in N$ , Use  $\kappa_{i}c_{k}$ ,  $\kappa_{i}\pmb{o}_{k}$ , and  $p(\kappa_iZ_k)$  as the inputs to Algorithm 1 for BLR or the No-U-Turn [19] sampler for GLM instead. Scaling the perturbed SS would affect the sensitivity of party  $k$ 's submitted information and the DP noise needed.
F Time Complexity
Algorithm 2 An overview of our collaborative ML problem setup.
The computational complexity is given in App. F.
Require: Rényi DP  $\lambda$  parameter, Noise-aware inference algorithm, Shared prior  $p(\theta)$  of model parameters and prior  $p(\omega)$  of data parameters,  $\rho$ -Shapley fairness scheme parameter.
// Party's actions (ensure DP)
1: for each party  $i \in N$  do
2: Compute exact SS  $s_i$  from dataset  $\mathcal{D}_i$ .
3: Choose DP guarantee  $(\lambda, \epsilon_i)$ -Rényi DP.
21This second derivative is the variance of the sufficient statistic of  $\theta$ . It is non-negative and often positive.
4: Sample  $\mathbf{z}_i$  from the Gaussian distribution  $p(Z_i) = \mathcal{N}(\mathbf{0}, 0.5(\lambda/\epsilon_i)\Delta_2^2(g)\mathbf{I})$ .
5: Compute perturbed SS  $\pmb{o}_i \triangleq \pmb{s}_i + \pmb{z}_i$ .
6: Submit (i) number  $c_{i} \triangleq |\mathcal{D}_{i}|$  of data points in its dataset  $\mathcal{D}_i$ , (ii) perturbed SS  $o_i$  and (iii) Gaussian distribution  $p(Z_i)$  to the mediator.
7: end for
// Mediator's actions
// 1. Compute valuation of perturbed SS needed for Shapley value. The choice of  $v$  ensures a privacy-valuation trade-off.
8: Draw  $m$  samples from  $p(\theta)$ .
9: for each coalition  $C \subseteq N$  do
10: Draw  $m$  samples from the posterior  $p(\theta | o_C)$  by applying the noise-aware inference algorithm. The algorithm requires the perturbed SS  $o_C \triangleq \{o_i\}_{i \in C}$ , data quantities  $\{c_i\}_{i \in C}$  and noise distributions  $\{Z_i\}_{i \in C}$ .
11: Compute  $v_{C}$  by using the nearest-neighbors method [45] to estimate the KL divergence  $D_{\mathrm{KL}}(p(\theta | o_{C}); p(\theta))$  from the samples.
12: end for
// 2. Decide the target reward values using  $\rho$ -Shapley value [47] which ensure efficiency (P2), fairness (P3), rationality (P4) and control group welfare (P6).
13: for each party  $i \in N$  do
14: Compute Shapley value  $\phi_{i} = (1 / n)\sum_{C\subseteq N\setminus i}\left[\binom{n - 1}{|C|}^{-1}\left(v_{C\cup \{i\}} - v_{C}\right)\right]$
15: end for
16: Identify the maximum Shapley value  $\phi_{*} = \max_{k\in N}\phi_{k}$
17: for each party  $i \in N$  do
18: Compute  $\rho$ -Shapley fair target reward  $r_i^*$  for party  $i$  using the formula  $r_i^* = v_N \times (\phi_i / \phi_*)^\rho$ .
19: end for
// 3. Generate model reward  $q_{i}(\theta)$  with value  $r_{i} = r_{i}^{*}$  that preserves similarity (P5) with the grand coalition's model and privacy for others (P1).
20: for each party  $i\in N$  do
21: Initialize  $\mathrm{Kr} = ()$
22: while True do
23: Select  $\kappa_{i}\in [0,1]$  using a root finding algorithm and Kr.
24: Draw  $m$  samples from the normalized posterior  $q_{i}(\theta)$  (Eq. 1 by applying the noise-aware inference algorithm. Use the scaled perturbed SS  $\{\kappa_{i}\pmb{o}_{i}\}_{i\in N}$ , data quantities  $\{\kappa_{i}c_{i}\}_{i\in N}$  and noise distributions  $\{\kappa_{i}Z_{i}\}_{i\in N}$ .
25: Compute the reward value  $r_i$  by using the nearest-neighbors method [45] to estimate the KL divergence  $D_{\mathrm{KL}}(q_i(\theta); p(\theta))$  from the samples.
26: if attained reward value  $r_i = r_i^*$  then
27: Reward party  $i$  with the  $m$  posterior samples from  $q_{i}(\theta)$ .
28: break
29: end if
30: Update  $\mathrm{Kr}\gets \mathrm{Kr} + ((\kappa_i,r_i),)$
31: end while
32: end for
The main steps of our scheme are detailed in Algo. 2 and the time complexity of the steps are as follows:
1. Local SS  $s_i$  computation (Line 2 in Algo 2). Party  $i$  will need to sum the SS for its  $c_i$  data points. Subsequent steps will not depend on the number  $c_i$  of data points. The (approximate) SS is usually an  $\mathcal{O}(d^2)$  vector where  $d$  is the number of regression model features. Therefore, this step incurs  $\mathcal{O}(c_id^2)$  time.
2. Perturbed SS  $o_i$  computation (Lines 4-5 in Algo 2). Each party will need to use the Gaussian mechanism to perturb  $s_i$ . Therefore, this step incurs  $\mathcal{O}(d^2)$  time.
Valuation of Perturbed SS (Sec. 3). The valuation of  $o_N$  requires us to draw  $m$  posterior samples of model parameters using DP noise-aware inference (refer to App. A.3 and the cited references for the exact steps). As the methods of [27] and [4] incur  $\mathcal{O}(md^4)$  time for a single party, inference based on Fig. 4 will take  $n$  times longer to consider  $n$  parties. KL estimation using  $k$ -nearest neighbor will incur  $\mathcal{O}(m\log(m)\dim(\theta))$  time to value multiple (scaled) perturbed SS. Therefore, valuation incurs  $\mathcal{O}(nmd^4 + m\log(m)\dim(\theta))$  time.
3. Deciding target reward value  $r_i^*$  for every  $i \in N$  (Sec. 4, Lines 9-19 in Algo 2)). Computing the Shapley values exactly involves valuing  $o_C$  for each subset  $C \subseteq N$ , hence, repeating Step A  $\mathcal{O}(2^n)$  time. When the number of parties is small (e.g.,  $< 6$ ), we can compute the Shapley values exactly. For larger  $n$ , we can approximate the Shapley values  $(\phi_i)_{i \in N}$  with bounded  $\ell_2$ -norm error using  $\mathcal{O}(n(\log n)^2)$  samples [25, 53]. Moreover, the value of different coalitions can be computed in parallel. Therefore, this step incurs  $\mathcal{O}(2^n)$  or  $\mathcal{O}(n(\log n)^2)$  times the time in Step A.
4. Solving for  $\kappa_{i}$  to generate model reward (Sec. 5.2, Lines 21-31 in Algo 2). During root-finding, the mediator values different model rewards  $q_{i}(\theta)$  generated by scaling the perturbed SS  $o_{k}$ , data quantity  $c_{k}$  and DP noise distribution  $Z_{k}$  of each party  $k\in N$  by different  $\kappa_{i}$ , hence, repeats Step A. As we are searching for the root in a fixed interval [0, 1] and to a fixed precision, Step A is repeated a constant (usually  $< 10$ ) number of times. Therefore, this step incurs  $\mathcal{O}(nmd^4 +m\log (m)\dim (\theta))$  time per party.
The mediator can further reduce the number of valuation of model rewards (repetitions of Step A) by using the tuples of  $(\kappa_i, r_i)$  obtained when solving for  $\kappa_i$  to narrow the root-finding range for other parties after  $i$ .
Therefore, the total incurred time depends on the number of valuations performed in Step A. The time complexity may vary for other inference and KL estimation methods.
G Comparison of Reward Control Mechanisms via Noise Addition (Sec. 5.1) vs. Likelihood Tempering (Sec. 5.2)
See Fig. 7.
Figure 7: We contour plot the distribution of the regression model weights  $w_{1}$  and  $w_{2}$  for the prior, the grand coalition  $N$ 's posterior, and the model reward's posterior to attain the target reward value  $r_{2}^{*}$  utilizing noise addition (Sec. 5.1) vs. likelihood tempering (Sec. 5.2) as the reward control mechanism for the Syn dataset where  $\rho = .5$ . The tempered posterior interpolates the prior and grand coalition  $N$ 's posterior better as its mean/mode lies along the line connecting the prior's and grand coalition  $N$ 's posterior mean and the variance is scaled by the same extent for both weights.
The experiments are performed on a machine with Ubuntu 20.04 LTS,  $2 \times$  Intel Xeon Gold 6230 (2.1GHz) without GPU. The software environments used are Miniconda and Python. A full list of packages used is given in the file environment.yml attached.
H.1 Experimental Details
Synthetic BLR (Syn). The BLR parameters  $\theta$  consist of the weights for each dimension of the 2D dataset, the bias, and the variance  $\sigma^2$ . The normal inverse-gamma distribution used (i) to generate the true regression model weights, variance, and a 2D dataset and (ii) as our model prior is as follows:  $\sigma^2 \sim \mathrm{InvGamma}(\alpha_0 = 5, \beta_0 = 0.1)$  where  $\alpha_0$  and  $\beta_0$  are, respectively, the inverse-gamma shape and scale parameters, and  $\boldsymbol{w} | \sigma^2 \sim \mathcal{N}(\mathbf{0}, \sigma^2 \boldsymbol{\Lambda}_0^{-1})$  where  $\boldsymbol{\Lambda}_0 = 0.025 \boldsymbol{I}$ .
We consider three parties 1, 2, and 3 with  $c_{0} = 100$ ,  $c_{1} = 200$ , and  $c_{2} = 400$  data points, respectively. We fix  $\epsilon_{1} = \epsilon_{3} = 0.2$  and vary  $\epsilon_{2}$  from the default 0.1. As  $\epsilon_{2}$  increases (decreases), party 2 may become the most (least) valuable. We allow each party to have a different Gaussian distribution  $p(\pmb{x}_i)$  by maintaining a separate conjugate normal inverse-Wishart distribution  $p(\omega_i = (\mu_{\omega,i}, \Sigma_{\omega,i}))$  for each party. We set the prior  $\Sigma_{\omega,i} \sim \mathcal{W}^{-1}(\psi_0 = I, \nu_0 = 50)$  where  $\psi_0$  and  $\nu_0$  are the scale matrix and degrees of freedom (i.e., how strongly we believe the prior), respectively. Then,  $\mu_{\omega,i} \sim \mathcal{N}(0, \Sigma_{\omega,i})$ . The  $\ell_{2}$ -sensitivity is estimated using [26]'s analysis based on the norms/bounds of the dataset.
One posterior sampling run generates 16 Gibbs sampling chains in parallel. For each chain, we discard the first 10000 burn-in samples and draw  $m = 30000$  samples. To reduce the closeness/correlation between samples which will affect the nearest-neighbor-based KL estimation, we thin and only keep
every 16-th sample and concatenate the thinned samples across all 16 chains. For the experiment on reward control mechanisms, we use 5 independent runs of posterior sampling and KL estimation.
Californian Housing dataset (CaliH). As the CaliH dataset may contain outliers, we preprocess the "public" dataset ( $60\%$  of the CaliH data) by subtracting the median and scaling by the interquartile range for each feature. We train a neural network (NN) of 3 layers with [48, 24, 6] hidden units and the rectified linear unit (ReLU) as the activation function to minimize the mean squared error, which we will then use as a "pre-trained NN". The outputs of the last hidden layer have 6 features used as the inputs for BLR. We intentionally reduce the number of features in the BLR model by adding more layers to the pre-trained NN and reduce the magnitude of the BLR inputs by adding an activation regularizer on the pre-trained NN hidden layers (i.e.,  $\ell_2$  penalty weight of 0.005). These reduce the computational cost of Gibbs sampling/KL estimation and the  $\ell_2$ -sensitivity of the inputs to BLR (hence the added DP noise), respectively. We also add a weights/bias regularizer with an  $\ell_2$  penalty weight of 0.005 for the last layer connected to the outputs. Lastly, we standardize the outputs of the last hidden layer to have zero mean and unit variance.
We preprocess the private dataset for valuation and the held-out validation set (an 80-20 split) using the same pre-trained NN/translation process. To reduce the sensitivity and added DP noise, we filter and exclude any data point with a  $z$ -score  $> 4$  for any feature. There are 6581 training data points left. We divide the dataset randomly among 3 parties such that parties 1, 2, and 3 have, respectively,  $20\%$ ,  $30\%$  and  $50\%$  of the dataset and  $\epsilon_1 = \epsilon_3 = 0.2$  while  $\epsilon_2$  is varied from the default 0.1.
The BLR parameters  $\theta$  consist of the weights for each of the 6 features, the bias, and the variance  $\sigma^2$ . We assume  $\theta$  has a normal inverse-gamma distribution and set the prior as follows. The prior depends on the MLE estimate based on the public dataset, and we assume it has the same significance as  $n_0 = 10$  data points. Hence, we set  $\sigma^2 \sim \mathrm{InvGamma}(\alpha_0 = n_0 / 2, \beta_0 = n_0 / 2 \times \mathrm{MLE}$  estimate of  $\sigma^2)$  and  $\pmb{w}|\sigma^2 \sim \mathcal{N}(\mathbf{0}, \sigma^2 (n_0 \pmb{x}^\top \pmb{x})^{-1})$ .
We assume that each party has the same underlying Gaussian distribution for  $p(\pmb{x})$  and maintain only one conjugate normal inverse-Wishart distribution  $p(\omega = (\mu_{\omega}, \Sigma_{\omega}))$  shared across parties. We initialize the prior  $p(\omega)$  to be weakly dependent on the prior dataset [39]. The  $\ell_2$ -sensitivity is estimated using [26]'s analysis based on the norms/bounds of the private transformed dataset.
One posterior sampling run generates 16 Gibbs sampling chains in parallel. For each chain, we discard the first 10000 burn-in samples and draw  $m = 30000$  samples. To reduce the closeness/correlation between samples which will affect the nearest-neighbor-based KL estimation, we thin and only keep every 16-th sample and concatenate the thinned samples across all 16 chains. For the experiment on reward control mechanisms, we use 5 independent runs of posterior sampling and KL estimation.
PIMA Indian Diabetes classification dataset (Diab). This dataset has 8 raw features such as age, BMI, number of pregnancies, and a binary output variable. Patients with and without diabetes are labeled  $y = 1$  and  $y = -1$ , respectively. We split the training and the validation set using an 80-20 split. There are 614 training data points. There are  $35.6\%$  and  $31.8\%$  of patients with diabetes in the training and validation sets, respectively. Hence, random guessing would lead to a cross-entropy loss of 0.629.
We preprocess both sets by (i) subtracting the training set's median and scaling by the interquartile range for each feature, (ii) using principal component analysis (PCA) to select the 4 most important components of the feature space to be used as new features, and lastly, (iii) centering and scaling the new features to zero mean and unit variance. To reduce the effect of outliers and the  $\ell_2$ -sensitivity, we clip each training data point's feature values at  $\pm 2.2$ .
We divide the 614 training data points such that parties 1, 2, and 3 have, respectively,  $20\%$ ,  $30\%$ , and  $50\%$  of the dataset and  $\epsilon_{1} = \epsilon_{3} = 0.2$  while  $\epsilon_{2}$  is varied from the default 0.1. We compute the approximate SS [21] and perturb them for each party to achieve the selected  $\epsilon_{i}$  [27]. The  $\ell_{2}$ -sensitivity is also estimated based on the dataset.
We consider a Bayesian logistic regression model, and its parameters  $\theta$  consist of the bias and the weights for each of the 4 features. Like that of [27], we set an independent standard Gaussian prior for  $\theta$  but rescale it such that the squared norm  $\| \theta \| _2^2$  has a truncated Chi-square prior with  $d = 4$  degrees
of freedom. Truncation prevents sampling  $\theta$  with a norm larger than 2.5 times the non-private/true setting's  $\theta^{*}$  squared norm during inference.
We assume each party has the same distribution for  $p(\pmb{x})$ . Our data prior  $p(\pmb{x})$  has mean 0 and covariance  $\Sigma = \mathrm{diag}(\iota)\Omega \mathrm{diag}(\iota)$  where  $\iota \sim \mathcal{N}(\mathbf{1},\mathbf{I}),\iota \in [0,2]$ , and  $\Omega$  follows a Lewandowski-Kurowicka-Joe prior LKJ(2).
We use the No-U-Turn [19] sampler. We run 25 Markov chains with 400 burn-in samples and draw  $m = 2000$  samples with a target Metropolis acceptance rate of 0.86. We discard chains with a low Bayesian fraction of missing information (i.e.,  $< .3$ ) and split the concatenated samples across chains into 5 groups to estimate KL divergence. As sampling is slower and the generated samples tend to be less correlated, we can use fewer samples.
Remark. For the CaliH dataset, the preprocessing is based on the "public" dataset, but for the Diab dataset, the preprocessing (i.e., standardization, PCA) is based on the private, valued dataset. We have assumed that the data is preprocessed. However, in practice, before using our mechanism, the parties may have to reserve/separately expend some privacy budget for these processing steps. The privacy cost is ignored in our analysis of the privacy-valuation trade-off.
KL estimation. We estimate KL divergence using the  $k$ -nearest-neighbor-based KL estimator [45]. To reduce the bias due to the skew of the distribution, we apply a whitening transformation [54] where each parameter sample is centered and multiplied by the inverse of the sample covariance matrix based on all samples from  $\theta$  and  $\theta \mid o$ . As a default, we set  $k = 4$  and increase  $k$  until the distance to the  $k$ -th neighbor is non-zero.
H.2 Utility of Model Reward
The mean negative log probability (MNLP) on a test dataset  $\mathcal{D}_{*}$  given the perturbed SS  $o_{i}$  is defined as follows:
$$
\mathrm {M N L P} \triangleq \frac {1}{| \mathcal {D} _ {*} |} \sum_ {\left(\boldsymbol {x} _ {*}, y _ {*}\right) \in \mathcal {D} _ {*}} - \log p \left(y _ {*} \mid \boldsymbol {x} _ {*}, \boldsymbol {o} _ {i}\right).
$$
We prefer MNLP over the model accuracy or mean squared error metric. MNLP additionally measures if a model is uncertain of its accurate predictions or overconfident in inaccurate predictions. In contrast, the latter metrics penalize inaccurate predictions equally and ignore the model's confidence (which is affected by the DP noise).
Regression. Approximating the predictive distribution,  $p(y_{*}|\boldsymbol{x}_{*},\boldsymbol{o}_{i})$ , for test input  $\boldsymbol{x}_{*}$  as Gaussian, the MNLP formula becomes
$$
\mathrm {M N L P} \triangleq \frac {1}{| \mathcal {D} _ {*} |} \sum_ {\left(\boldsymbol {x} _ {*}, y _ {*}\right) \in \mathcal {D} _ {*}} \frac {1}{2} \left(\log \left(2 \pi \widehat {\sigma^ {2}} \left(\boldsymbol {x} _ {*}\right)\right) + \frac {\left(\widehat {\mu} \left(\boldsymbol {x} _ {*}\right) - y _ {*}\right) ^ {2}}{\widehat {\sigma^ {2}} \left(\boldsymbol {x} _ {*}\right)}\right)
$$
where  $\mu_{*}$  and  $\widehat{\sigma^2} (\pmb{x}_*)$  denote the predictive mean and variance, respectively. The first term penalizes large predictive variance while the second term penalizes inaccurate predictions more strongly when the predictive variance is small (i.e., overconfidence).