SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our experiments use SubLasso [21] as a baseline "plug-and-play" private feature selection method. At a high level, the algorithm randomly partitions its data into  $m$  subsets, computes a non-private Lasso regression model on each, and then privately aggregates these models to select  $k$  significant features. The private aggregation process is simple; for each subset's learned model, choose the  $k$  features with largest absolute coefficient, then apply private top-  $k$  to compute the  $k$  features most selected by the  $m$  models. Kifer et al. [21] introduced and analyzed SubLasso; we collect its relevant properties in Lemma 4.1. Pseudocode appears in Algorithm 2.
Algorithm 2 SubLasso  $(D,k,m,\varepsilon)$
1: Input: Examples  $D = \{(X_i,Y_i)\}_{i=1}^n$ , number of selected features  $k$ , number of models  $m$ , privacy parameter  $\varepsilon$   
2: Randomly partition  $D$  into  $m$  equal-size subsets  $S_1, \ldots, S_m$   
3: for  $i = 1, \ldots, m$  do  
4: Compute Lasso model  $\theta_i$  on  $S_i$   
5: Compute set  $C_i$  of the  $k$  indices of  $\theta_i$  with largest absolute value  
6: Compute binary vector  $v_i \in \{0,1\}^d$  where  $v_{i,j} = \mathbb{1}_{j \in C_i}$   
7: Compute  $V \in \mathbb{R}^d = \sum_{i=1}^n v_i$   
8: Return Peel  $(V,k,1,\varepsilon)$
Lemma 4.1. SubLasso is  $\varepsilon$ -DP and runs in time  $O(d^{2}n)$ .
Proof. The privacy guarantee is immediate from that of Peel (Lemma 2.4). Solving Lasso on  $n / m$  data points with  $d$ -dimensional features takes time  $O\left(\frac{d^2n}{m}\right)$  [12]. Multiplying through by  $m$  produces the final result, since Peel only takes time  $O(dm)$ .
Finally, we briefly discuss the role of the intercept feature in SubLasso. As with all algorithms in our experiments, we add an intercept feature (with a constant value of 1) to each vector of features. Each Lasso model is trained on data with this intercept. However, the intercept is removed before the private voting step,  $k$  features are chosen from the remaining features, and the intercept is added back afterward. This ensures that privacy is not wasted on the intercept feature, which we always include.
4.2 Comparison Algorithms
We evaluate seven algorithms:
1. NonDP is a non-private baseline running generic ordinary least-squares regression.
2. BAS runs Boosted AdaSSP [30] without feature selection. We imitate the parameter settings used by Tang et al. [30] and set feature and gradient clipping norms to 1 and the number of boosting rounds to 100 throughout.
3. Tukey runs the Tukey mechanism [2] without feature selection. We introduce and use a tighter version of the propose-test-release (PTR) check given by Amin et al. [2]. This reduces the number of models needed for PTR to pass. The proof appears in Section 9 in the Appendix and may be of independent interest. To privately choose the number of models  $m$  used by the Tukey mechanism, we first privately estimate a  $1 - \eta$  probability lower bound on the number of points  $n$  using the Laplace CDF,  $\tilde{n} = n + \operatorname{Lap}\left(\frac{1}{\varepsilon'}\right) - \frac{\ln(1/2\eta)}{\varepsilon'}$ , and then set the number of models to  $m = \lfloor \tilde{n}/d \rfloor$ . Tukey spends  $5\%$  of its  $\varepsilon$  privacy budget estimating  $m$  and the remainder on the Tukey mechanism.
4. L-BAS runs spends  $5\%$  of its  $\varepsilon$  privacy budget choosing  $m = \lfloor \tilde{n} /k\rfloor$  for SubLasso,  $5 \%$  of  $\varepsilon$  running SubLasso, and then spends the remainder to run BAS on the selected features.
5. L-Tukey spends  $5\%$  of its  $\varepsilon$  privacy budget choosing  $m = \lfloor \tilde{n} / k \rfloor$  for SubLasso,  $5\%$  running SubLasso, and the remainder running Tukey on the selected features using the same  $m$ .
6. K-BAS spends  $5\%$  of its  $\varepsilon$  privacy budget running DPKendall and then spends the remainder running BAS on the selected features.
7. K-Tukey spends  $5\%$  of its  $\varepsilon$  privacy budget choosing  $m = \lfloor \tilde{n} /k\rfloor$ ,  $5\%$  running DPKendall, and then spends the remainder running the Tukey mechanism on the selected features.
Figure 1: Plots of rank data for each private method.
Figure 2: Plots of the number of datasets with positive  $R^2$  for each private method.
All experiments use  $(\ln (3),10^{-5})$  -DP. Where applicable,  $5 \%$  of the privacy budget is spent on private feature selection,  $5 \%$  on choosing the number of models, and the remainder is spent on private regression. Throughout, we use  $\eta = 10^{-4}$  as the failure probability for the lower bound used to choose the number of models. For each algorithm and dataset, we run 10 trials using random 90-10 train-test splits and record the resulting test  $R^2$  values. Tables of the results at  $k = 5$  and  $k = 10$  appear in Section 10 in the Appendix. A condensed presentation appears below. At a high level, we summarize the results in terms of relative and absolute performance.
4.3.1 Relative Performance
First, for each dataset and method, we compute the median  $R^2$  of the final model across the 10 trials and then rank the methods, with the best method receiving a rank of 1. Figure 1 plots the number of times each method is ranked first or second.
At  $k = 5$  (left), K-Tukey performs best by a significant margin: it ranks first on  $48\%$  of datasets, twice the fraction of any other method. It also ranks first or second on the largest fraction of datasets  $(68\%)$ . In some contrast, L-BAS obtains the top ranking on  $24\%$  of datasets, whereas K-BAS only does so on  $8\%$ ; nonetheless, the two have nearly the same number of total first or second rankings. At  $k = 10$  (right), no clear winner emerges among the feature selecting methods, as L-BAS, K-BAS, and K-Tukey are all first or second on around half the datasets<sup>6</sup>, though the methods using SubLasso have a higher share of datasets ranked first.
Note that  $k = 10$  only uses 21 datasets. This is because 4 of the 25 datasets used for  $k = 5$  have  $d < 10$ .
4.3.2 Absolute Performance
Second, we count the number of datasets on which the method achieves a positive median  $R^2$  (Figure 2), recalling that  $R^2 = 0$  is achieved by the trivial model that always predicts the mean label. The  $k = 5$  (left) setting again demonstrates clear trends: K-Tukey attains  $R^2 > 0$  on  $56\%$  of datasets, L-Tukey does so on  $40\%$ , K-BAS does so on  $28\%$ , and L-BAS on  $20\%$ . DPKendall therefore consistently demonstrates stronger performance than SubLasso. As in the rank data, at  $k = 10$  the picture is less clear. However, K-Tukey is still best by some margin, with the remaining feature selecting methods all performing roughly equally well.
A few trends are apparent from Section 4.3. First, DPKendall generally achieves stronger final utility than SubLasso, particularly for the Tukey mechanism; the effect is similar but smaller for  $k = 10$ ; and feature selection generally improves the performance of private linear regression.
Comparing Feature Selection Algorithms. A possible reason for DPKendall's improvement over SubLasso is that, while SubLasso takes advantage of the stability properties that Lasso exhibits in certain data regimes [21], this stability does not always hold in practice. Another possible explanation is that the feature coefficients passed to Peel scale with  $O(n)$  for DPKendall and  $m = O\left(\frac{n}{d}\right)$  or  $m = O\left(\frac{n}{k}\right)$  for SubLasso. Both algorithms's invocations of Peel add noise scaling with  $O\left(\frac{k}{\varepsilon}\right)$ , so DPKendall's larger scale makes it more robust to privacy-preserving noise. Finally, we emphasize that DPKendall achieves this even though its  $O(dkn\log (n))$  runtime is asymptotically smaller than the  $O(d^{2}n)$  runtime of SubLasso in most settings.
Choosing k. Next, we examine the decrease in performance from  $k = 5$  to  $k = 10$ . Conceptually, past a certain point adding marginally less informative features to a private model may worsen utility due to the privacy cost of considering these features. Moving from  $k = 5$  to  $k = 10$  may cross this threshold for many of our datasets; note from Figure 4 and Figure 5 in the Appendix that, of the 21 datasets used for  $k = 10$ ,  $86\%$  witness their highest private  $R^2$  in the  $k = 5$  setting<sup>7</sup>. Moreover, from  $k = 5$  to  $k = 10$  the total number of positive  $R^2$  datasets across methods declines by more than  $50\%$  from 41 to 19, with all methods achieving positive  $R^2$  less frequently at  $k = 10$  than  $k = 5$ . We therefore suggest  $k = 5$  as the more relevant setting, and a good choice in practice.
The Effect of Private Feature Selection. Much work aims to circumvent generic lower bounds for privately answering queries by taking advantage of instance-specific structure [6, 16, 33, 5]. Similar works exist for private optimization, either by explicitly incorporating problem information [35, 18] or showing that problem-agnostic algorithms can, under certain conditions, take advantage of problem structure organically [24]. We suggest that this paper makes a similar contribution: feature selection reduces the need for algorithms like Boosted AdaSSP and the Tukey mechanism to "waste" privacy on computations over irrelevant features. This enables them to apply less obscuring noise to the signal contained in the selected features. The result is the significant increase in utility shown here.
We briefly discuss DPKendall's limitations. First, it requires an end user to choose the number of features  $k$  to select; we suggest  $k = 5$  as a reasonable first cut. Second, DPKendall's use of Kendall rank correlation may struggle when ties are intrinsic to the data's structure, e.g., when the data is categorical, as a monotonic relationship between feature and label becomes less applicable. Finally, Kendall rank correlation may fail to distinguish between linear and nonlinear monotonic feature-label relationships, even though the former is more likely to be useful for linear regression. Unfortunately, it is not obvious how to incorporate relationships more sophisticated than simple monotonicity without sacrificing rank correlation's low sensitivity. Answering these questions may be an interesting avenue for future work.
Nonetheless, the results of this paper demonstrate that DPKendall expands the applicability of plug-and-play private linear regression algorithms while providing more utility in less time than the current state of the art. We therefore suggest that DPKendall presents a step forward for practical private linear regression.
The exceptions are datasets 361075, 361091, and 361103.
[20] Maurice G Kendall. A new measure of rank correlation. Biometrika, 1938.
[21] Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory (COLT), 2012.
[22] William R Knight. A computer method for calculating Kendall's tau with ungrouped data. Journal of the American Statistical Association, 1966.
[23] Matt J Kusner, Yu Sun, Karthik Sridharan, and Kilian Q Weinberger. Private causal inference. In Artificial Intelligence and Statistics (AISTATS), 2016.
[24] Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta. When Does Differentially Private Learning Not Suffer in High Dimensions? Neural Information Processing Systems (NeurIPS), 2022.
[25] Frank McSherry and Ilya Mironov. Differentially private recommender systems: Building privacy into the netflix prize contenders. In Knowledge Discovery and Data Mining (KDD), 2009.
[26] Donald Bruce Owen. A table of normal integrals. Communications in Statistics-Simulation and Computation, 1980.
[27] Jayshree Sarathy, Sophia Song, Audrey Emma Haque, Tania Schlatter, and Salil Vadhan. Don't look at the data! How differential privacy reconfigures data subjects, data analysts, and the practices of data science. In Conference on Human Factors in Computing Systems (CHI), 2023.
[28] Ben Stoddard, Yan Chen, and Ashwin Machanavajjhala. Differentially private algorithms for empirical machine learning. arXiv preprint arXiv:1411.5428, 2014.
[29] Kunal Talwar, Abhradeep Guha Thakurta, and Li Zhang. Nearly optimal private lasso. Neural Information Processing Systems (NIPS), 2015.
[30] Shuai Tang, Sergul Aydore, Michael Kearns, Saeyoung Rho, Aaron Roth, Yichen Wang, Yu-Xiang Wang, and Zhiwei Steven Wu. Improved Differentially Private Regression via Gradient Boosting. arXiv preprint arXiv:2303.03451, 2023.
[31] Abhradeep Guha Thakurta and Adam Smith. Differentially private feature selection via stability arguments, and the robustness of the lasso. In Conference on Learning Theory (COLT), 2013.
[32] John W Tukey. Mathematics and the picturing of data. In International Congress of Mathematicians (IMC), 1975.
[33] Jonathan Ullman. Private multiplicative weights beyond linear queries. In *Principles of Database Systems* (PODS), 2015.
[34] Yu-Xiang Wang. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. In Uncertainty in Artificial Intelligence (UAI), 2018.
[35] Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. International Conference on Learning Representations (ICLR), 2021.
6 Alternative Notions of Correlation
6.1 Pearson
The Pearson correlation between some feature  $X$  and the label  $Y$  is defined by
$$
r (X, Y) := \frac {\operatorname {C o v} (X , Y)}{\sqrt {\operatorname {V a r} (X) \operatorname {V a r} (Y)}}. \tag {2}
$$
Evaluated on a sample of  $n$  points, this becomes
$$
r (X, Y) := \frac {\sum_ {i = 1} ^ {n} (X _ {i} - \bar {X}) (Y _ {i} - \bar {Y})}{\sqrt {\sum_ {i = 1} ^ {n} (X _ {i} - \bar {X}) ^ {2}} \sqrt {\sum_ {i = 1} ^ {n} (Y _ {i} - \bar {Y}) ^ {2}}}.
$$
where  $\bar{X}$  and  $\bar{Y}$  are sample means.
Lemma 6.1.  $r\in [-1,1]$
Proof. This is immediate from Cauchy-Schwarz.
Note that a value of  $-1$  is perfect anticorrelation and a value of 1 is perfect correlation. A downside of Pearson correlation is that it is not robust. In particular, its sensitivity is the same as its range.
Lemma 6.2. Pearson correlation has  $\ell_1$ -sensitivity  $\Delta (r) = 2$
Proof. Consider neighboring databases  $D = \{(-1, -1), (1, 1), (-c, c)\}$ .  $r(D) = 1$ , but
$$
\begin{array}{l} r (D ^ {\prime}) = \frac {(- 1 + \frac {c}{3}) (- 1 - \frac {c}{3}) + (1 + \frac {c}{3}) (1 - \frac {c}{3}) + (- \frac {2 c}{3} \cdot \frac {2 c}{3})}{\sqrt {(- 1 + \frac {c}{3}) ^ {2} + (1 + \frac {c}{3}) ^ {2} + (- \frac {2 c}{3}) ^ {2}} \sqrt {(- 1 - \frac {c}{3}) ^ {2} + (1 - \frac {c}{3}) ^ {2} + (\frac {2 c}{3}) ^ {2}}} \\ \approx \frac {- (\frac {2 c}{3}) ^ {2}}{(\frac {2 c}{3}) ^ {2}} = - 1 \\ \end{array}
$$
where the approximation is increasingly accurate as  $c$  grows.
6.2 Spearman
Spearman rank correlation is Pearson correlation applied to rank variables.
Definition 6.3. Given data points  $X_{1},\ldots ,X_{n}\in \mathbb{R}$ , the corresponding rank variables  $R(X_1),\ldots ,R(X_n)$  are defined by setting  $R(X_{i})$  to the position of  $X_{i}$  when  $X_{1},\ldots ,X_{n}$  are sorted in descending order. Given data  $(X,Y)$ , the Spearman rank correlation is  $\rho (X,Y)\coloneqq r(R(X),R(Y))$
For example, given database  $D = \{(0,1),(1,0),(2,3)\}$ , its Spearman rank correlation is
$$
\rho (D) := r (\{(3, 2), (2, 3), (1, 1) \}).
$$
A useful privacy property of rank is that it does not depend on data scale. Moreover, if there are no ties then Spearman rank correlation admits a simple closed form.
Lemma 6.4.  $\rho(X, Y) = 1 - \frac{6\sum_{i=1}^{n}(R(X_i) - R(Y_i))^2}{n(n^2 - 1)}$ .
If we consider adding a "perfectly unsorted" data point with rank variables  $(1, n + 1)$  to a perfectly sorted database with rank variables  $\{(1, 1), (2, 2), \ldots, (n, n)\}$ ,  $\rho$  changes from 1 to  $1 - \frac{6(n + n^2)}{n(n^2 - 1)}$ . The sensitivity's dependence on  $n$  complicates its usage with add-remove privacy. Nonetheless, both Spearman and Kendall correlation's use of rank makes them relatively easy to compute privately, and as the two methods are often used interchangeably in practice, we opt for Kendall rank correlation for simplicity.
7 Deferred Proofs From Section 3.3
We first restate and prove Lemma 3.5.
Proof. Recall from Definition 3.1 the population formulation of  $\tau$ ,
$$
\tau (X, Y) = \mathbb {P} \left[ \left(X - X ^ {\prime}\right) \left(Y - Y ^ {\prime}\right) > 0 \right] - \mathbb {P} \left[ \left(X - X ^ {\prime}\right) \left(Y - Y ^ {\prime}\right) <   0 \right] \tag {3}
$$
where  $X$  and  $X^{\prime}$  are i.i.d., as are  $Y$  and  $Y^{\prime}$ . In our case, we can define  $Z = \sum_{j\neq j^{*}}\beta_{j}X_{j} + \xi$  and rewrite the first term of Equation 3 for our setting as
$$
\int_ {0} ^ {\infty} f _ {X _ {j ^ {*}} - X _ {j ^ {*}} ^ {\prime}} (t) \cdot [ 1 - F _ {Z - Z ^ {\prime}} (- \beta_ {j ^ {*}} t) ] d t + \int_ {- \infty} ^ {0} f _ {X _ {j ^ {*}} - X _ {j ^ {*}} ^ {\prime}} (t) \cdot F _ {Z - Z ^ {\prime}} (- \beta_ {j ^ {*}} t) d t. \tag {4}
$$
where  $f_{A}$  and  $F_{A}$  denote densities and cumulative distribution functions of random variable  $A$ , respectively. Since the relevant distributions are all Gaussian,  $X_{j^{*}} - X_{j^{*}}^{\prime}\sim N(0,2\sigma_{j^{*}}^{2})$  and  $Z - Z^{\prime}\sim N(0,2[\sum_{j\neq j^{*}}\beta_{j}^{2}\sigma_{j}^{2} + \sigma_{e}^{2}])$ . For neatness, shorthand  $\sigma^2 = 2\sigma_{j^*}^2$  and  $\sigma_{-1}^{2} = 2[\sum_{j\neq j^{*}}\beta_{j}^{2}\sigma_{j}^{2} + \sigma_{e}^{2}]$ . Then if we let  $\phi$  denote the PDF of a standard Gaussian and  $\Phi$  the CDF of a standard Gaussian, Equation 4 becomes
$$
\begin{array}{l} \int_ {0} ^ {\infty} \frac {1}{\sigma} \phi (t / \sigma) \cdot [ 1 - \Phi (- \beta_ {j ^ {*}} t / \sigma_ {- 1}) ] d t + \int_ {- \infty} ^ {0} \frac {1}{\sigma} \phi (t / \sigma) \Phi (- \beta_ {j ^ {*}} t / \sigma_ {- 1}) d t \\ = \frac {1}{\sigma} \left[ \int_ {0} ^ {\infty} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t + \int_ {- \infty} ^ {0} \phi (- t / \sigma) \Phi (- \beta_ {j ^ {*}} t / \sigma_ {- 1}) d t \right] \\ = \frac {2}{\sigma} \int_ {0} ^ {\infty} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t. \\ \end{array}
$$
We can similarly analyze the second term of Equation 3 to get
$$
\begin{array}{l} \int_ {0} ^ {\infty} f _ {X _ {j ^ {*}} - X _ {j ^ {*}} ^ {\prime}} (t) \cdot F _ {Z - Z ^ {\prime}} (- \beta_ {j ^ {*}} t) d t + \int_ {- \infty} ^ {0} f _ {X _ {j ^ {*}} - X _ {j ^ {*}} ^ {\prime}} (t) \cdot [ 1 - F _ {Z - Z ^ {\prime}} (- \beta_ {j ^ {*}} t) ] d t \\ = \int_ {0} ^ {\infty} \frac {1}{\sigma} \phi (t / \sigma) \Phi (- \beta_ {j ^ {*}} t / \sigma_ {- 1}) d t + \int_ {- \infty} ^ {0} \frac {1}{\sigma} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t \\ = \frac {2}{\sigma} \int_ {- \infty} ^ {0} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t. \\ \end{array}
$$
Using both results, we get
$$
\begin{array}{l} \tau (X _ {1}, Y) = \frac {2}{\sigma} \left[ \int_ {0} ^ {\infty} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t - \int_ {- \infty} ^ {0} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t \right] \\ = \frac {2}{\sigma} \left[ \int_ {0} ^ {\infty} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t - \int_ {0} ^ {\infty} \phi (t / \sigma) [ 1 - \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) ] d t \right] \\ = \frac {4}{\sigma} \int_ {0} ^ {\infty} \phi (t / \sigma) \Phi (\beta_ {j ^ {*}} t / \sigma_ {- 1}) d t - 1 \\ = \frac {4}{\sigma} \cdot \frac {\sigma}{2 \pi} \left(\frac {\pi}{2} + \arctan \frac {\beta_ {j ^ {*}} \sigma}{\sigma_ {- 1}}\right) - 1 \\ = \frac {2}{\pi} \cdot \arctan \frac {\beta_ {j ^ {*}} \sigma}{\sigma_ {- 1}}. \\ \end{array}
$$
where the third equality uses  $\int_0^\infty \phi (t / \sigma)dt = \frac{\sigma}{2}$  and the fourth equality comes from Equation 1,010.4 of Owen [26]. Substituting in the values of  $\sigma$  and  $\sigma_{-1}$  yields the claim.
Next is Lemma 3.7.
Proof. Recall from Definition 2.3 that Gumbel  $(b)$  has density  $f(x) = \frac{1}{b} \cdot \exp\left(-\frac{x}{b} - e^{-x / b}\right)$ . Then  $\frac{f(x)}{f(-x)} = \exp\left(-\frac{x}{b} - e^{-x / b} - \frac{x}{b} + e^{x / b}\right)$ . For  $z \geq 0$ , by  $e^z = \sum_{n=0}^{\infty} \frac{z^n}{n!}$ ,
$$
2 z + e ^ {- z} \leq 2 z + 1 - z + \frac {z ^ {2}}{2} = 1 + z + \frac {z ^ {2}}{2} \leq e ^ {z}
$$
so since  $b \geq 0$ ,  $f(x) \geq f(-x)$  for  $x \geq 0$ . Letting  $F(z) = \exp(-\exp(-z / b))$  denote the CDF for Gumbel (b), it follows that for  $z \geq 0$ ,  $1 - F(z) \geq F(-z)$ . Thus the probability that  $\max_{j \in [d]} |X_j|$  exceeds  $t$  is upper bounded by  $2d(1 - F(t))$ . The claim then follows from rearranging the inequality
$$
- b \ln \left(- \ln \left(1 - \frac {\eta}{2 d}\right)\right) \leq t
$$
$$
b \ln \left(\frac {1}{- \ln (1 - \eta / [ 2 d ])}\right) \leq t
$$
$$
\begin{array}{l} 2 d (1 - F (t)) \leq \eta \\ 1 - \frac {\eta}{2 d} \leq F (t) \\ 1 - \frac {\eta}{2 d} \leq \exp (- \exp (- t / b)) \\ \ln \left(1 - \frac {\eta}{2 d}\right) \leq - \exp (- t / b) \\ \end{array}
$$
and using  $-\ln (1 - x) = \sum_{i = 1}^{\infty}x^{i} / i\geq x$
A summary of the datasets used in our experiments appears in Figure 3.
We briefly discuss the role of the intercept in these datasets. Throughout, we explicitly add an intercept feature (constant 1) to each vector. Where feature selection is applied, we explicitly remove the intercept feature during feature selection and then add it back to the  $k$  selected features afterward. The resulting regression problem therefore has dimension  $k + 1$ . We do this to avoid spending privacy budget selecting the intercept feature.
9 Modified PTR Lemma
This section describes a simple tightening of Lemma 3.6 from Amin et al. [2] (which is itself a small modification of Lemma 3.8 from Brown et al. [7]). Tukey uses the result as its propose-test-release (PTR) check, so tightening it makes the check easier to pass. Proving the result will require introducing details of the Tukey algorithm. The following exposition aims to keep this document both self-contained and brief; the interested reader should consult the expanded treatment given by Amin et al. [2] for further details.
Tukey depth was introduced by Tukey [32]. Amin et al. [2] used an approximation for efficiency. Roughly, Tukey depth is a notion of depth for a collection of points in space. (Exact) Tukey depth is evaluated over all possible directions in  $\mathbb{R}^d$ , while approximate Tukey depth is evaluated only over axis-aligned directions.
Definition 9.1 ([32, 2]). A halfspace  $h_v$  is defined by a vector  $v \in \mathbb{R}^d$ ,  $h_v = \{y \in \mathbb{R}^d \mid \langle v, y \rangle \geq 0\}$ . Let  $E = \{e_1, \dots, e_d\}$  be the canonical basis for  $\mathbb{R}^d$  and let  $D \subset \mathbb{R}^d$ . The approximate Tukey depth of
OpenML Task ID	n	d	[n/d]	[n/5]	[n/10]
361072	8192	22	372	1638	819
361073	15000	27	555	3000	1500
361074	16599	17	976	3319	1659
361075	7797	614	12	1559	779
361076	6497	12	541	1299	649
361077	13750	34	404	2750	1375
361078	20640	9	2293	4128	2064
361079	22784	17	1340	4556	2278
361085	10081	7	1440	2016	1008
361087	13932	14	995	2786	1393
361088	21263	80	265	4252	2126
361089	20640	9	2293	4128	2064
361091	515345	91	5663	103069	51534
361092	8885	83	107	1777	888
361093	4052	13	311	810	405
361094	8641	6	1440	1728	864
361095	166821	24	6950	33364	16682
361096	53940	27	1997	10788	5394
361098	10692	18	594	2138	1069
361099	17379	21	827	3475	1737
361100	39644	74	535	7928	3964
361101	581835	32	18182	116367	58183
361102	21613	20	1080	4322	2161
361103	394299	27	14603	78859	39429
361104	241600	16	15100	48320	24160
Figure 3: Parameters of the 25 datasets used in our experiments.
a point  $y \in \mathbb{R}^d$  with respect to  $D$ , denoted  $\tilde{T}_D(y)$ , is the minimum number of points in  $D$  in any of the 2d halfspaces determined by  $E$  containing  $y$ ,
$$
\tilde {T} _ {D} (y) = \min  _ {v \in \pm E s. t. y \in h _ {v}} \sum_ {x \in D} \mathbb {1} _ {x \in h _ {v}}.
$$