SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Definition 22 (Randomized Response on Bins,  $\left[\mathrm{GKK}^{+}23\right]$ ). Let  $\varepsilon >0$ , for map  $\Phi :\mathcal{V}\to \hat{\mathcal{V}}$  RR-on-Bins is defined as the mechanism that on input  $y$  samples  $\hat{y}\sim \hat{Y}$  where the random variable  $\hat{Y}$  is distributed as
$$
\operatorname * {P r} [ \hat {Y} = \hat {y} ]   =   \left\{ \begin{array}{l l} \frac {e ^ {\varepsilon}}{e ^ {\varepsilon} + | \hat {\mathcal {Y}} | - 1} & \text {i f} \hat {y} = \Phi (y) \\ \frac {1}{e ^ {\varepsilon} + | \hat {\mathcal {Y}} | - 1} & \text {o t h e r w i s e .} \end{array} \right.
$$
Definition 23 (Debiased Randomized Response). For  $\varepsilon >0$ , the  $\varepsilon$ -debiased randomized response ( $\varepsilon$ -dbRR) operates by performing randomized response on  $(\Phi(y))_{y \in \mathcal{Y}}$ , where
$$
\Phi (y) = \left(\left(e ^ {\varepsilon} + | \mathcal {Y} | - 1\right) y - \sum_ {y ^ {\prime} \in \mathcal {Y}} y ^ {\prime}\right) / \left(e ^ {\varepsilon} - 1\right),
$$
namely, the mechanism on input  $y$  samples  $\hat{y} \sim \hat{Y}$ , where the random variable  $\hat{Y}$  is distributed as
$$
\operatorname * {P r} [ \hat {Y} = \hat {y} ]   =   \left\{ \begin{array}{l l} \frac {e ^ {\varepsilon}}{e ^ {\varepsilon} + | \mathcal {Y} | - 1} & \text {i f} \hat {y} = \Phi (y) \\ \frac {1}{e ^ {\varepsilon} + | \mathcal {Y} | - 1} & \text {o t h e r w i s e .} \end{array} \right.
$$
Finally, we prove Proposition 7, restated below for convenience.
Proof. Let  $y_0 = \min(\mathcal{V})$  and  $y_1 = \max(\mathcal{V})$ . The smallest and largest values among the outputs of  $\varepsilon$ -dbRR are given as
$$
\hat {y} _ {0} = \left((e ^ {\varepsilon} + | \mathcal {Y} | - 1) \cdot y _ {0} - \sum_ {y \in \mathcal {Y}} y\right) / (e ^ {\varepsilon} - 1),
$$
$$
\hat {y} _ {1} = \left(\left(e ^ {\varepsilon} + | \mathcal {Y} | - 1\right) \cdot y _ {1} - \sum_ {y \in \mathcal {Y}} y\right) / \left(e ^ {\varepsilon} - 1\right).
$$
Consider the mechanism  $\mathcal{M}$  such that for any input  $y\in \mathcal{V}$  it holds that
$$
\mathcal {M} (y) = \left\{ \begin{array}{l l} \hat {y} _ {0} & \quad \text {w . p .} \frac {\hat {y} _ {1} - y}{\hat {y} _ {1} - \hat {y} _ {0}}, \\ \hat {y} _ {1} & \quad \text {w . p .} \frac {y - \hat {y} _ {0}}{\hat {y} _ {1} - \hat {y} _ {0}}. \end{array} \right.
$$
Note  $\mathcal{M}$  is unbiased, namely  $\mathbb{E}[\mathcal{M}(y)] = y$  for all  $y\in \mathcal{V}$ . We show
$$
\begin{array}{l} \frac {\max _ {y} \operatorname * {P r} [ \mathcal {M} (y) = \hat {y} _ {0} ]}{\min _ {y} \operatorname * {P r} [ \mathcal {M} (y) = \hat {y} _ {0} ]} = \frac {\operatorname * {P r} [ \mathcal {M} (y _ {0}) = \hat {y} _ {0} ]}{\operatorname * {P r} [ \mathcal {M} (y _ {1}) = \hat {y} _ {0} ]} \\ = \frac {(e ^ {\varepsilon} + | \mathcal {Y} | - 1) \cdot y _ {1} - \sum_ {y \in \mathcal {Y}} y - (e ^ {\varepsilon} - 1) y _ {0}}{(e ^ {\varepsilon} + | \mathcal {Y} | - 1) \cdot y _ {1} - \sum_ {y \in \mathcal {Y}} y - (e ^ {\varepsilon} - 1) y _ {1}} \\ = \frac {\sum_ {y \neq y _ {0}} \left(y _ {1} - y\right) + e ^ {\varepsilon} \left(y _ {1} - y _ {0}\right)}{\sum_ {y \neq y _ {0}} \left(y _ {1} - y\right) + \left(y _ {1} - y _ {0}\right)} \leq e ^ {\varepsilon}, \\ \end{array}
$$
where the last step follows because  $\sum_{y\neq y_0}(y_1 - y)\geq 0$  and for  $a,b,c > 0$  it holds that  $(a + b) / (a+$ $c)\leq b / c$  . Similarly, it holds that  $\max_y\operatorname *{Pr}[\mathcal{M}(y) = \hat{y}_1] / \min_y\operatorname *{Pr}[\mathcal{M}(y) = \hat{y}_1]\leq e^\varepsilon$
G Additional Details of Experiments
All our experiments were performed using NVidia P100 GPUs.
G.1 Hyperparameter Details
Criteo Sponsored Search Conversion. We use a neural network that takes as input concatenation of floating point features, as well as categorical features using an embedding table of dimension four for each. The neural network had two hidden layers of dimensions 64 and 32 respectively. The training was performed on a random  $80\%$  of the dataset using RMSProp algorithm using the squared loss objective, with learning rate of  $10^{-4}$ ,  $\ell_2$ -regularization of  $10^{-4}$ , batch size of 1,024 for 50 epochs. The remaining  $20\%$  of the dataset was used to report the test loss.
This architecture choice was chosen by performing a hyperparameter search over various choices, and this choice seemed to give the best results for all the randomizers considered.
US Census. We use a neural network that takes as input concatenation of floating point features, as well as categorical features using an embedding table of dimension eight for each. The neural network had two hidden layers of dimensions 128 and 64 respectively. The training was performed on a random  $80\%$  of the dataset using RMSProp algorithm using the squared loss objective, with  $\ell_2$ -regularization of  $10^{-4}$ , batch size of 8, 192. A varying learning rate in  $\{10^{-2}, 10^{-3}, 10^{-4}\}$  and a varying number of epochs in  $\{50, 200\}$  were used, and the best test loss was reported for training with each randomizer. The remaining  $20\%$  of the dataset was used to report the test loss.
G.2 Early Stopping
In training on the AppAds dataset, we found that the training would often be unstable, with the training loss blowing up after several steps, especially for smaller values of  $\varepsilon$ , which corresponded to faster blow up. We used early stopping, keeping track of intermediate states, to select the best model that minimizes the training loss, and used this to report the test loss.
G.3 Complexity of Computing the Mechanisms
Even though the wall clock time for computing the optimal unbiased randomizer using an LP solver is significantly larger than that of computing the optimal RR-on-Bins randomizer (for which there is a highly efficient dynamic programming algorithm  $\mathrm{[GKK^{+}23]}$ ), this is still orders of magnitude smaller than that of ML model training. We note that while the noisy label loss decreases as  $\Delta$  in Algorithm 2 is made smaller, this causes the computational cost of solving the LP to increase. This trade-off is demonstrated in the following table, which shows the running time of the LP for the unbiased randomizer, the noisy label loss and the final test loss, for different mesh sizes (parameter  $n$  in Algorithm 2) for  $\varepsilon = 1$  on the US Census dataset we study (prediction of number of weeks worked in  $\{1,\ldots ,52\}$ ).
Mechanism	Mesh size	Computing mechanism wall-clock time (secs)	Noisy label loss	Test loss
RR-on-Bins	n/a	0.154	79.71	172.44
Opt. Unbiased	52	2.38	1288.21	134.44
Opt. Unbiased	416	17.1	1275.22	134.43
Opt. Unbiased	1664	161	1274.71	134.43
The test loss is quite similar for various mesh sizes, even though the noisy label loss does improve slightly with finer discretization. This suggests that the unbiasedness was key to the improvements over RR-on-Bins and the discretization of the output set does not hurt performance as much.
The computation time increases considerably when the number of input labels is large, e.g., in the Criteo Search Sponsored Conversion Logs dataset, where there were 401 input labels  $\{0,\dots ,400\}$