SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At a high level, Lemma 3.6 from Amin et al. [2] is a statement about the volumes of regions of different depths. The next step is to formally define these volumes.
Definition 9.2 ([2]). Given database  $D$ , define  $S_{i,D} = \{y \in \mathbb{R}^d \mid \tilde{T}_D(y) \geq i\}$  to be the set of points with approximate Tukey depth at least  $i$  in  $D$  and  $V_{i,D} = \mathrm{vol}(S_{i,D})$  to be the volume of that set. When  $D$  is clear from context, we write  $S_i$  and  $V_{i}$  for brevity. We also use  $w_{D}(V_{d,D}) := \int_{S_{d,D}} \exp(\varepsilon \cdot \tilde{T}_{D}(y)) dy$  to denote the weight assigned to  $V_{d,D}$  by an exponential mechanism whose score function is  $\tilde{T}_{D}$ .
Amin et al. [2] define a family of mechanisms  $A_{1}, A_{2}, \ldots$ , where  $A_{t}$  runs the exponential mechanism to choose a point of approximately maximal Tukey depth, but restricted to the domain of points with Tukey depth at least  $t$ . Since this domain is a data-dependent quantity, they use the PTR framework to select a safe depth  $t$ . We briefly recall the definitions of "safe" and "unsafe" databases given by Brown et al. [7], together with the key PTR result from Amin et al. [2].
Definition 9.3 (Definitions 2.1 and 3.1 [7]). Two distributions  $\mathcal{P},\mathcal{Q}$  over domain  $\mathcal{W}$  are  $(\varepsilon ,\delta)$ -indistinguishable, denoted  $\mathcal{P}\approx_{\varepsilon ,\delta}\mathcal{Q}$ , if for any measurable subset  $W\subset \mathcal{W}$
$$
\mathbb {P} _ {w \sim \mathcal {P}} [ w \in W ] \leq e ^ {\varepsilon} \mathbb {P} _ {w \sim \mathcal {Q}} [ w \in W ] + \delta \text {a n d} \mathbb {P} _ {w \sim \mathcal {Q}} [ w \in W ] \leq e ^ {\varepsilon} \mathbb {P} _ {w \sim \mathcal {P}} [ w \in W ] + \delta .
$$
Database  $D$  is  $(\varepsilon, \delta, t)$ -safe if for all neighboring  $D' \sim D$ , we have  $A_t(D) \approx_{\varepsilon, \delta} A_t(D')$ . Let  $\mathsf{Safe}_{(\varepsilon, \delta, t)}$  be the set of safe databases, and let  $\mathsf{Unsafe}_{(\varepsilon, \delta, t)}$  be its complement.
We can now restate Lemma 3.6 from Amin et al. [2]. Informally, it states that if the volume of an "outer" region of Tukey depth is not much larger than the volume of an "inner" region, the difference
in depth between the two is a lower bound on the distance to an unsafe database. For the purpose of this paper, Tukey applies this result by finding such a  $k$ , adding noise for privacy, and checking that the resulting distance to unsafety is large enough that subsequent steps will be privacy-safe.
Lemma 9.4. Define  $M(D)$  to be a mechanism that receives as input database  $D$  and computes the largest  $k \in \{0, \dots, t - 1\}$  such that there exists  $g > 0$  where
$$
\frac {V _ {t - k - 1 , D}}{V _ {t + k + g + 1 , D}} \cdot e ^ {- \varepsilon g / 2} \leq \delta
$$
or outputs  $-1$  if the inequality does not hold for any such  $k$ . Then for arbitrary  $D$
1.  $M$  is 1-sensitive, and
2. for all  $z \in \operatorname{Unsafe}_{(\varepsilon, 4e^{\circ} \delta, t)}$ ,  $d_H(D, z) > M(D)$ .
We provide a drop-in replacement for Lemma 9.4 that slightly weakens the requirement placed on  $k$ .
Lemma 9.5. Define  $M(D)$  to be a mechanism that receives as input database  $D$  and computes the largest  $k \in \{0, \dots, t - 1\}$  such that
$$
\frac {V _ {t - k - 1 , D}}{w _ {D} \left(V _ {t + k - 1 , D}\right)} \cdot e ^ {\varepsilon (t + k + 1)} \leq \delta
$$
1.  $M$  is 1-sensitive, and
2. for all  $z\in \mathrm{Unsafe}_{(\varepsilon ,4e^{\circ}\delta ,t)}$ $d_{H}(D,z) > M(D)$
The new result therefore replaces the denominator  $V_{t + k + g + 1,D} \cdot e^{\varepsilon g / 2}$  with denominator  $\frac{w_D(V_{t + k - 1,D})}{e^{\varepsilon(t + k + 1)}}$ . Every point in  $V_{t + k - 1,D}$  of depth at least  $t + k + g + 1$  has score at least  $t + k + g + 1$ , so  $\frac{w_D(V_{t + k - 1,D})}{e^{\varepsilon(t + k + 1)}} \geq V_{t + k + g + 1,D} \cdot e^{\varepsilon g}$ , so  $V_{t + k + g + 1,D} \cdot e^{\varepsilon g / 2} \leq \frac{w_D(V_{t + k + g + 1,D})}{e^{\varepsilon(t + k + 1)}}$  and the check for the new result is no harder to pass. To see that it may be easier, note that only the new result takes advantage of the higher scores of deeper points in  $V_{t + k - 1,D}$ .
Proof of Lemma 9.5. First we prove item 1. Let  $D$  and  $D'$  be any neighboring databases and suppose WLOG that  $D' = D \cup \{x\}$ . We want to show that  $|M(D) - M(D')| \leq 1$ .
First we prove relationships between the points with approximate Tukey depth at least  $p$  and  $p - 1$  in datasets  $D$  and  $D'$ . From the definition of approximate Tukey depth, together with the fact that  $D'$  contains one additional point, for any point  $y$ , we are guaranteed that  $\tilde{T}_D(y) \leq \tilde{T}_{D'}(y) \leq \tilde{T}_D(y) + 1$ . Recall that  $S_{p,D}$  is the set of points with approximate Tukey depth at least  $p$  in  $D$ . This implies that  $S_{p+1,D} \subset S_{p,D}$ . Next, since for every point  $y$  we have  $\tilde{T}_{D'}(y) \geq \tilde{T}_D(y)$ , we have that  $S_{p,D} \subset S_{p,D'}$ . Finally, since  $\tilde{T}_D(y) \geq \tilde{T}_{D'}(y) - 1$ , we have that  $S_{p,D'} \subset S_{p-1,D}$ . Taken together, we have
$$
S _ {p + 1, D} \subset S _ {p, D} \subset S _ {p, D ^ {\prime}} \subset S _ {p - 1, D}.
$$
$$
V _ {p + 1, D} \leq V _ {p, D} \leq V _ {p, D ^ {\prime}} \leq V _ {p - 1, D}. \tag {5}
$$
Next, since the unnormalized exponential mechanism density  $y \mapsto \exp(y\hat{T}_D(y))$  is non-negative, we have that  $w_D(V_{p+1,D}) = \int_{S_{p+1,D}} \exp(\epsilon \tilde{T}_D(y)) dy \leq \int_{S_{p,D}} \exp(\epsilon \tilde{T}_D(y)) dy = w_D(V_{p,D})$ . Using the fact that  $\tilde{T}_D(y) \leq \tilde{T}_{D'}(y)$ , we have that  $w_D(V_{p,D}) = \int_{S_{p,D}} \exp(\epsilon \tilde{T}_D(y)) dy \leq \int_{S_{p,D'}} \exp(\epsilon \tilde{T}_{D'}(y)) dy = w_{D'}(V_{p,D'})$ . Finally, using the fact that  $\tilde{T}_{D'}(y) \leq \tilde{T}_D(y) + 1$ , we have  $w_{D'}(V_{p,D'}) = \int_{S_{p,D'}} \exp(\varepsilon \tilde{T}_{D'}(y)) dy \leq \int_{S_{p-1,D}} \exp(\epsilon \tilde{T}_D(y) + \epsilon) dy = e^\epsilon w_D(V_{p-1,D})$ . Together, this gives
$$
w _ {D} \left(V _ {p + 1, D}\right) \leq w _ {D} \left(V _ {p, D}\right) \leq w _ {D ^ {\prime}} \left(V _ {p, D ^ {\prime}}\right) \leq w _ {D} \left(V _ {p - 1, D}\right) \cdot e ^ {\varepsilon}. \tag {6}
$$
Now suppose there exists  $k_{D'}^* \geq 0$  such that  $\frac{V_{t-k_{D'}'-1,D'}}{w_{D'}(V_{t+k_{D'}'+2,D'})} \cdot e^{\varepsilon(t+k_{D'}'+1)} \leq \delta$ . Then by Equation 5,  $V_{t-k_{D'}'-1,D'} \geq V_{t-k_{D'}',D}$ , and by Equation 6  $w_{D'}(V_{t+k_{D'}'+2,D'}) \leq w_{D}(V_{t+k_{D'}'+1,D}) \cdot e^{\varepsilon}$ , so
$\frac{V_{t-k_{D'}', D}}{w_D(V_{t+k_{D'}' + 1, D})} \cdot e^{\varepsilon(t+k_{D'}')} \leq \delta$  and then  $k_D^* \geq k_{D'}^* - 1$ . Similarly, if there exists  $k_D^* \geq 0$  such that  $\frac{V_{t-k_{D'} - 1, D}}{w_D(V_{t+k_{D'} + 2, D})} \cdot e^{\varepsilon(t+k_{D'}+1)} \leq \delta$ , then by Equation 5  $V_{t-k_{D'} - 1, D} \geq V_{t-k_{D'}, D'}$ , and by Equation 6  $W_D(V_{t+k_{D'} + 2, D}) \leq W_{D'}(V_{t+k_{D'} + 2, D'})$ , so  $\frac{V_{t-k_{D'}', D'}}{w_{D'}(V_{t+k_{x}' + 2, D'})} \cdot e^{\varepsilon(t+k_{D'}^*)} \leq \delta$ , and  $k_{D'}^* \geq k_D^* - 1$ . Thus if  $k_D^* \geq 0$  or  $k_{D'}^* \geq 0$ ,  $|k_D^* - k_{D'}^*| \leq 1$ . The result then follows since  $k^* \geq -1$ .
As in Lemma 9.4, item 2 is a consequence of Lemma 3.8 from Brown et al. [7] and the fact that  $k^{*} = -1$  is a trivial lower bound on distance. The only change made to the proof of Lemma 3.8 of Brown et al. [7] is, in its notation, to replace its denominator lower bound with
$$
w _ {z} \left(V _ {t - 1, z}\right) \geq w _ {x} \left(V _ {t + k - 1, x}\right) \cdot e ^ {- k \varepsilon}
$$
This uses the fact that  $x$  and  $z$  differ in at most the addition or removal of  $k$  data points. Thus  $V_{t + k - 1,x} \leq V_{t - 1,z}$ , and no point's score increases by more than  $k$  from  $V_{t - 1,z}$  to  $V_{t + k - 1,x}$ . Since their numerator upper bound is  $V_{t - k - 1,x} \cdot e^{\varepsilon (t + 1)}$  (note that the 2 is dropped here because approximate Tukey depth is monotonic; see Section 7.3 of [2] for details), the result follows.
10 Extended Experiment Results
Task ID	NonDP	BAS	Tukey	L-BAS	L-Tukey	K-BAS	K-Tukey
361072	7.3e-01	7.8e-01	-∞	-2.7e-02	1.0e-01	2.1e-01	1.0e-01
361073	4.6e-01	-3.4e+00	-∞	-4.6e-01	-2.3e-02	-4.8e-01	-4.3e-01
361074	8.0e-01	-4.5e+04	-6.6e+02	-8.2e+02	-1.7e+02	-5.6e+05	3.1e-01
361075	6.2e-01	-3.6e+02	-∞	3.2e-03	2.2e-02	5.2e-03	7.1e-02
361076	2.8e-01	-3.4e+00	-∞	-5.6e-01	-2.5e-01	-4.3e+00	8.5e-02
361077	8.2e-01	-2.1e+08	-∞	-1.0e+07	3.7e-01	-1.9e+08	6.6e-01
361078	6.5e-01	-1.1e+03	-6.4e+00	-7.4e-01	-1.2e+00	-6.1e+02	-9.5e-01
361079	2.6e-01	-5.4e+00	-1.0e+01	-1.3e+01	-2.e+00	-9.4e+00	-6.8e-01
361085	3.3e-01	-2.e+01	3.3e-01	-1.7e+01	2.8e-01	-1.3e+01	3.7e-01
361087	7.2e-01	-3.1e+03	-8.8e+04	-4.7e+00	-1.9e+02	-2.8e+00	6.6e-01
361088	7.3e-01	5.4e-02	-∞	1.4e-01	3.e-01	2.1e-01	3.6e-01
361089	6.2e-01	-3.e+00	-1.5e+01	-1.3e+00	-5.9e+00	-5.9e+04	-2.6e-01
361091	2.4e-01	-3.0e+04	-1.9e+00	-3.0e+04	3.8e-02	-3.1e+04	4.9e-02
361092	2.0e-02	-4.7e+05	-∞	-4.1e+04	-1.4e+08	-1.3e+05	-9.3e+08
361093	4.3e-01	-5.6e-02	-∞	-4.6e-02	-∞	-4.9e-02	-∞
361094	8.3e-01	4.2e-01	-3.8e+05	-1.0e+00	-4.2e+05	-2.4e+00	-3.0e+05
361095	2.2e-01	-1.0e+00	-1.2e+09	-8.0e-01	2.e-01	1.5e-01	2.1e-01
361096	9.7e-01	-9.1e+02	-1.e+08	-4.9e+00	9.1e-01	7.1e-01	8.8e-01
361098	8.6e-01	-2.9e+01	-∞	-3.9e+00	-3.1e+00	-2.2e-01	-5.6e+00
361099	4.0e-01	-4.4e-01	-4.8e+10	-4.8e-01	-9.8e-03	-4.2e-01	3.2e-01
361100	1.2e-01	-6.3e+01	-∞	-1.1e+00	-2.8e-02	-4.6e-01	-1.7e-01
361101	3.3e-01	-9.3e+01	-5.9e+08	3.6e-01	2.e-01	3.0e-01	2.1e-01
361102	7.6e-01	-4.1e+02	-1.4e+15	6.8e-02	-2.6e-01	-4.0e+00	-5.3e+00
361103	5.3e-01	4.3e-01	-6.3e+07	5.e-01	4.9e-01	3.3e-01	4.8e-01
361104	6.8e-01	-3.7e+03	-6.4e+07	-1.5e+02	-2.6e+05	-8.3e-01	-2.9e+01
Figure 4:  $R^2$  values for  $k = 5$ . Each entry is the median test value from 10 trials. The top two private values for each dataset are bolded.
Task ID	NonDP	BAS	Tukey	L-BAS	L-Tukey	K-BAS	K-Tukey
361072	7.4e-01	5.3e-01	-∞	3.5e-01	1.8e-01	7.3e-01	8.2e-02
361073	4.6e-01	-8.2e-01	-∞	-4.4e-01	-1.7e+00	-4.8e-01	-1.6e+01
361074	8.1e-01	-2.3e+06	-1.3e+03	-5.6e+06	-3.4e+02	-7.2e+04	-1.4e+02
361075	6.2e-01	-1.7e+02	-∞	7.5e-02	-9.7e-01	3.1e-02	-2.8e+00
361076	2.8e-01	-2.4e+02	-8.2e+04	-2.7e+01	-3.5e+03	-2.4e+01	-1.4e+03
361077	7.9e-01	-1.1e+08	-∞	-1.4e+07	-3.1e+01	-2.1e+08	-1.7e+02
361079	2.4e-01	-1.8e+00	-3.7e+01	-1.2e+01	-8.1e-01	-4.1e+00	-1.5e+00
361087	7.1e-01	-3.7e+01	-1.5e+04	-7.2e+00	-4.8e+03	-8.5e+01	-2.e+01
361088	7.3e-01	-1.9e+00	-∞	1.2e-01	9.2e-02	1.2e-01	7.1e-02
361091	2.4e-01	-3.0e+04	-3.4e+00	-3.0e+04	6.8e-02	-3.1e+04	5.7e-02
361092	4.0e-02	-2.4e+06	-∞	-8.7e+05	-7.6e+09	-7.5e+05	-3.7e+10
361093	4.3e-01	-3.2e-02	-∞	-5.e-01	-∞	-1.1e-01	-∞
361095	2.2e-01	-1.9e+01	-1.2e+09	-8.6e-01	-2.4e+07	-1.4e+00	2.1e-01
361096	9.8e-01	-3.7e+02	-1.0e+08	-3.4e+03	-1.8e+00	-1.0e+03	4.2e-01
361098	8.6e-01	-1.8e+02	-∞	-4.1e-01	-1.3e+07	-9.5e-01	-5.8e-01
361099	4.0e-01	-4.3e-01	-2.3e+10	-4.9e-01	-2.4e+09	-4.8e-01	-2.5e+08
361100	1.2e-01	-9.2e+01	-∞	-1.6e+00	-8.2e-02	-3.5e+00	-5.2e-01
361101	3.4e-01	-3.3e+03	-1.7e+08	-8.1e-01	-1.4e+02	-9.5e-01	-4.7e+05
361102	7.5e-01	-9.5e+02	-1.7e+15	-1.3e+00	-1.5e+15	-2.5e+01	-1.4e+05
361103	5.4e-01	-2.6e+00	-1.2e+08	4.9e-01	5.1e-01	3.5e-01	4.9e-01
361104	6.8e-01	-9.e+02	-1.0e+08	-2.9e+03	-3.5e+07	-1.4e+03	-6.6e+05
Figure 5: This figure records the same information as Figure 4, but for  $k = 10$ .