SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Travis Dick*
Jennifer Gillenwater†
Existing work on differentially private linear regression typically assumes that end users can precisely set data bounds or algorithmic hyperparameters. End users often struggle to meet these requirements without directly examining the data (and violating privacy). Recent work has attempted to develop solutions that shift these burdens from users to algorithms, but they struggle to provide utility as the feature dimension grows. This work extends these algorithms to higher-dimensional problems by introducing a differentially private feature selection method based on Kendall rank correlation. We prove a utility guarantee for the setting where features are normally distributed and conduct experiments across 25 datasets. We find that adding this private feature selection step before regression significantly broadens the applicability of "plug-and-play" private linear regression algorithms at little additional cost to privacy, computation, or decision-making by the end user.
Differentially private [10] algorithms employ carefully calibrated randomness to obscure the effect of any single data point. Doing so typically requires an end user to provide bounds on input data to ensure the correct scale of noise. However, end users often struggle to provide such bounds without looking at the data itself [27], thus nullifying the intended privacy guarantee. This has motivated the development of differentially private algorithms that do not require these choices from users.
To the best of our knowledge, two existing differentially private linear regression algorithms satisfy this "plug-and-play" requirement: 1) the Tukey mechanism [2], which combines propose-test-release with an exponential mechanism based on Tukey depth, and 2) Boosted AdaSSP [30], which applies gradient boosting to the AdaSSP algorithm introduced by Wang [34]. We refer to these methods as, respectively, Tukey and BAS. Neither algorithm requires data bounds, and both feature essentially one chosen parameter (the number of models  $m$  for Tukey, and the number of boosting rounds  $T$  for BAS) which admits simple heuristics without tuning. In contrast, AdaSSP requires a user to provide bounds on the data's feature and label norms, and DPSGD requires a user to configure hyperparameters including learning rate, clipping norms, batch size, and number of epochs. Both algorithms produce much weaker utility when these parameters are even moderately misconfigured [2].
Tukey obtains strong empirical results when the number of data points  $n$  greatly exceeds the feature dimension  $d$  [2], while BAS obtains somewhat weaker utility on a larger class of datasets [30]. Nonetheless, neither algorithm provides generally strong performance on its own. Evaluated over a collection of 25 linear regression datasets taken from Tang et al. [30], Tukey and BAS obtain coefficient of determination  $R^2 > 0$  on only four (see Section 4); for context, a baseline of  $R^2 = 0$  is achieved by the trivial constant predictor, which simply outputs the mean label. These results suggest room for improvement for practical private linear regression.
*tdick@google.com, Google Research
$\dagger$ jgillenw@gmail.com, work done while at Google Research
$\ddagger$  mtjoseph@google.com, Google Research
We extend existing work on private linear regression by adding a preprocessing step that applies private feature selection. At a high level, this strategy circumvents the challenges of large feature dimension  $d$  by restricting attention to  $k \ll d$  carefully selected features. We initiate the study of private feature selection in the context of "plug-and-play" private linear regression and make two concrete contributions:
1. We introduce a practical algorithm, DPKendall, for differentially private feature selection (Section 3). DPKendall uses Kendall rank correlation [20] and only requires the user to choose the number  $k$  of features to select. It satisfies  $\varepsilon$ -DP and, given  $n$  samples with  $d$ -dimensional features, runs in time  $O(dkn\log(n))$  (Theorem 3.4). We also provide a utility guarantee when the features are normally distributed (Theorem 3.8).
2. We conduct experiments across 25 datasets (Section 4), with  $k$  fixed at 5 and 10. These compare Tukey and BAS without feature selection, with SubLasso feature selection [21], and with DPKendall feature selection. Using  $(\ln(3), 10^{-5})$ -DP to cover both private feature selection and private regression, we find at  $k = 5$  that adding DPKendall yields  $R^2 > 0$  on  $56\%$  of the datasets. Replacing DPKendall with SubLasso drops the rate to  $40\%$  of datasets, and omitting feature selection entirely drops it further to  $16\%$ .
In summary, we suggest that DPKendall significantly expands the applicability and utility of practical private linear regression.
The focus of this work is practical private feature selection applied to private linear regression. We therefore refer readers interested in a more general overview of the private linear regression literature to the discussions of Amin et al. [2] and Tang et al. [30].
Several works have studied private sparse linear regression [21, 31, 17, 29]. However, Jain and Thakurta [17] and Talwar et al. [29] require an  $\ell_{\infty}$  bound on the input data, and the stability test that powers the feature selection algorithm of Thakurta and Smith [31] requires the end user to provide granular details about the optimal Lasso model. These requirements are significant practical obstacles. An exception is the work of Kifer et al. [21]. Their algorithm first performs feature selection using subsampling and aggregation of non-private Lasso models. This feature selection method, which we call SubLasso, only requires the end user to select the number of features  $k$ . To the selected features, Kifer et al. [21] then apply objective perturbation to privately optimize the Lasso objective. As objective perturbation requires the end user to choose parameter ranges and provides a somewhat brittle privacy guarantee contingent on the convergence of the optimization, we do not consider it here. Instead, our experiments combine SubLasso feature selection with the Tukey and BAS private regression algorithms. An expanded description of SubLasso appears in Section 4.2.
We now turn to the general problem of private feature selection. There is a significant literature studying private analogues of the general technique of principal component analysis (PCA) [25, 15, 8, 19, 11, 1]. Unfortunately, all of these algorithms assume some variant of a bound on the row norm of the input data. Stoddard et al. [28] studied private feature selection in the setting where features and labels are binary, but it is not clear how to extend their methods to the non-binary setting that we consider in this work. SubLasso is therefore the primary comparison private feature selection method in this paper. We are not aware of existing work that studies private feature selection in the specific context of "plug-and-play" private linear regression.
Finally, private rank correlation has previously been studied by Kusner et al. [23]. They derived a different, normalized sensitivity bound appropriate for their "swap" privacy setting and applied it to privately determine the causal relationship between two random variables.
Throughout this paper, a database  $D$  is a collection of labelled points  $(x,y)$  where  $x\in \mathbb{R}^d$ ,  $y\in \mathbb{R}$ , and each user contributes a single point. We use the "add-remove" form of differential privacy.
Definition 2.1 ([10]). Databases  $D$ ,  $D'$  from data domain  $\mathcal{D}$  are neighbors  $D \sim D'$  if they differ in the presence or absence of a single record. A randomized mechanism  $\mathcal{M}: \mathcal{D} \to \mathcal{O}$  is  $(\varepsilon, \delta)$ -differentially private ( $DP$ ) if for all  $D \sim D' \in \mathcal{D}$  and any  $S \subseteq \mathcal{O}$
$$
\mathbb {P} _ {\mathcal {M}} \left[ \mathcal {M} (D) \in S \right] \leq e ^ {\varepsilon} \mathbb {P} _ {\mathcal {M}} \left[ \mathcal {M} \left(D ^ {\prime}\right) \in S \right] + \delta .
$$
We use basic composition to reason about the privacy guarantee obtained from repeated application of a private algorithm. More sophisticated notions of composition exist, but for our setting of relatively few compositions, basic composition is simpler and suffers negligible utility loss.
Lemma 2.2 ([10]). Suppose that for  $j \in [k]$ , algorithm  $\mathcal{A}_j$  is  $(\varepsilon_j, \delta_j)$ -DP. Then running all  $k$  algorithms is  $(\sum_{j=1}^{k} \varepsilon_j, \sum_{j=1}^{k} \delta_j)$ -DP.
Both SubLasso and DPKendall use a private subroutine for identifying the highest count item(s) from a collection, known as private top- $k$ . Several algorithms for this problem exist [4, 9, 13]. We use the pure DP "peeling mechanism" based on Gumbel noise [9], as its analysis is relatively simple, and its performance is essentially identical to other variants for the relatively small  $k$  used in this paper.
Definition 2.3 ([9]). A Gumbel distribution with parameter  $b$  is defined over  $x \in \mathbb{R}$  by  $\mathbb{P}[x; b] = \frac{1}{b} \cdot \exp \left( -\frac{x}{b} - e^{-x / b} \right)$ . Given  $c = (c_1, \ldots, c_d) \in \mathbb{R}^d$ ,  $k \in \mathbb{N}$ , and privacy parameter  $\varepsilon$ ,  $\text{Peel}(c, k, \Delta_{\infty}, \varepsilon)$  adds independent Gumbel  $\left( \frac{2k\Delta_{\infty}}{\varepsilon} \right)$  noise to each count  $c_j$  and outputs the ordered sequence of indices with the largest noisy counts.
Lemma 2.4 ([9]). Given  $c = (c_{1},\dots ,c_{d})\in \mathbb{R}^{d}$  with  $\ell_{\infty}$  sensitivity  $\Delta_{\infty}$ ,  $\mathrm{Peel}(c,k,\Delta_{\infty},\varepsilon)$  is  $\varepsilon$  -DP.
The primary advantage of Peel over generic noise addition is that, although users may contribute to  $d$  counts, the added noise only scales with  $k$ . We note that while Peel requires an  $\ell_{\infty}$  bound, neither DPKendall nor SubLasso needs user input to set it: regardless of the dataset, DPKendall's use of Peel has  $\ell_{\infty} = 3$  and SubLasso's use has  $\ell_{\infty} = 1$  (see Algorithms 1 and 2).
3 Feature Selection Algorithm
This section describes our feature selection algorithm, DPKendall, and formally analyzes its utility. Section 3.1 introduces and discusses Kendall rank correlation, Section 3.2 describes the full algorithm, and the utility result appears in Section 3.3.
3.1 Kendall Rank Correlation
The core statistic behind our algorithm is Kendall rank correlation. Informally, Kendall rank correlation measures the strength of a monotonic relationship between two variables.
Definition 3.1 ([20]). Given a collection of data points  $(X,Y) = \{(X_{1},Y_{1}),\ldots ,(X_{n},Y_{n})\}$  and  $i < i^{\prime}$ , a pair of observations  $(X_{i},Y_{i}),(X_{i^{\prime}},Y_{i^{\prime}})$  is discordant if  $(X_{i} - X_{i^{\prime}})(Y_{i} - Y_{i^{\prime}}) < 0$ . Given data  $(X,Y)$ , let  $d_{X,Y}$  denote the number of discordant pairs. Then the empirical Kendall rank correlation is
$$
\hat {\tau} (X, Y) := \frac {n}{2} - \frac {2 d _ {X , Y}}{n - 1}.
$$
For real random variables  $X$  and  $Y$ , we can also define the population Kendall rank correlation by
$$
\tau (X, Y) = \mathbb {P} \left[ (X - X ^ {\prime}) (Y - Y ^ {\prime}) > 0 \right] - \mathbb {P} \left[ (X - X ^ {\prime}) (Y - Y ^ {\prime}) <   0 \right].
$$
Kendall rank correlation is therefore high when an increase in  $X$  or  $Y$  typically accompanies an increase in the other, low when an increase in one typically accompanies a decrease in the other, and close to 0 when a change in one implies little about the other. We typically focus on empirical Kendall rank correlation, but the population definition will be useful in the proof of our utility result.
Before discussing Kendall rank correlation in the context of privacy, we note two straightforward properties. First, for simplicity, we use a version of Kendall rank correlation that does not account for ties. We ensure this in practice by perturbing data by a small amount of continuous random noise. Second,  $\tau$  has range  $[-1, 1]$ , but (this paper's version of)  $\hat{\tau}$  has range  $[-n/2, n/2]$ . This scaling does not affect the qualitative interpretation and ensures that  $\hat{\tau}$  has low sensitivity<sup>4</sup>.
In particular, without scaling,  $\hat{\tau}$  would be  $\approx \frac{1}{n}$ -sensitive, but  $n$  is private information in add-remove privacy.
Lemma 3.2.  $\Delta (\hat{\tau}) = 3 / 2$
Proof. At a high level, the proof verifies that the addition or removal of a user changes the first term of  $\hat{\tau}$  by at most  $1/2$ , and the second term by at most 1.
In more detail, consider two neighboring databases  $(X,Y)$  and  $(X',Y')$ . Without loss of generality, we may assume that  $(X,Y) = \{(X_1,Y_1),\ldots,(X_n,Y_n)\}$  and  $(X',Y') = \{(X_1',Y_1'),\ldots,(X_{n+1}',Y_{n+1}')\}$  where for all  $i \in [n]$  we have that  $X_i' = X_i$  and  $Y_i' = Y_i$ . First, we argue that the number of discordant pairs in  $(X',Y')$  cannot be much larger than in  $(X,Y)$ . By definition, we have that  $d_{X',Y'} - d_{X,Y} = \sum_{j=1}^{n} \mathbb{1}_{(X_j - X_{n+1}') (Y_j - Y_{n+1}') < 0}$ . In particular, this implies that  $d_{X',Y'} - d_{X,Y} \in [0,n]$ .
We can rewrite the difference in Kendall correlation between  $(X,Y)$  and  $(X^{\prime},Y^{\prime})$  as follows:
$$
\begin{array}{l} \hat {\tau} (X, Y) - \hat {\tau} \left(X ^ {\prime}, Y ^ {\prime}\right) = \frac {n}{2} - \frac {2 d _ {X , Y}}{n - 1} - \frac {n + 1}{2} + \frac {2 d _ {X ^ {\prime} , Y ^ {\prime}}}{n} \\ = 2 \left(\frac {d _ {X ^ {\prime} , Y ^ {\prime}}}{n} - \frac {d _ {X , Y}}{n - 1}\right) - \frac {1}{2} \\ = 2 \left(\frac {d _ {X ^ {\prime} , Y ^ {\prime}} - d _ {X , Y}}{n} + \frac {d _ {X , Y}}{n} - \frac {d _ {X , Y}}{n - 1}\right) - \frac {1}{2} \\ = 2 \frac {d _ {X ^ {\prime} , Y ^ {\prime}} - d _ {X , Y}}{n} + 2 d _ {X, Y} \left(\frac {1}{n} - \frac {1}{n - 1}\right) - \frac {1}{2} \\ = 2 \frac {d _ {X ^ {\prime} , Y ^ {\prime}} - d _ {X , Y}}{n} - \frac {d _ {X , Y}}{\binom {n} {2}} - \frac {1}{2}, \\ \end{array}
$$
where the final equality follows from the fact that  $2\left(\frac{1}{n} - \frac{1}{n-1}\right) = -1 / \binom{n}{2}$ . Using our previous calculation, the first term is in the range  $[0, 2]$  and, since  $d_{X,Y} \in [0, \binom{n}{2}]$ , the second term is in the range  $[-1, 0]$ . It follows that
$$
- \frac {3}{2} = 0 - 1 - \frac {1}{2} \leq \hat {\tau} (X, Y) - \hat {\tau} (X ^ {\prime}, Y ^ {\prime}) \leq 2 - 0 - \frac {1}{2} = \frac {3}{2},
$$
and therefore  $|\hat{\tau}(X, Y) - \hat{\tau}(X', Y')| \leq 3/2$ .
To show that the sensitivity is not smaller than  $3/2$ , consider neighboring databases  $(X,Y)$  and  $(X',Y')$  such that  $d_{X,Y} = 0$  and  $(X',Y')$  contains a new point that is discordant with all points in  $(X,Y)$ . Then  $d_{X',Y'} = n$  while  $d_{X,Y} = 0$ . Then  $\hat{\tau}(X,Y) - \hat{\tau}(X',Y') = 2 - 0 - 1/2 = 3/2$ .
Turning to privacy, Kendall rank correlation has two notable strengths. First, because it is computed entirely from information about the relative ordering of data, it does not require an end user to provide data bounds. This makes it a natural complement to private regression methods that also operate without user-provided data bounds. Second, Kendall rank correlation's sensitivity is constant, but its range scales linearly with  $n$ . This makes it easy to compute privately. A contrasting example is Pearson correlation, which requires data bounds to compute covariances and has sensitivity identical to its range. An extended discussion of alternative notions of correlation appears in Section 6.
Finally, Kendall rank correlation can be computed relatively quickly using a variant of merge sort.
Lemma 3.3 ([22]). Given collection of data points  $(X,Y) = \{(X_1,Y_1),\ldots ,(X_n,Y_n)\}$ ,  $\hat{\tau} (X,Y)$  can be computed in time  $O(n\log (n))$
3.2 DPKendall
Having defined Kendall rank correlation, we now describe our private feature selection algorithm, DPKendall. Informally, DPKendall balances two desiderata: 1) selecting features that correlate with the label, and 2) selecting features that do not correlate with previously selected features. Prioritizing only the former selects for redundant copies of a single informative feature, while prioritizing only the latter selects for features that are pure noise.
In more detail, DPKendall consists of  $k$  applications of Peel to select a feature that is correlated with the label and relatively uncorrelated with the features already chosen. Thus, letting  $S_{t}$  denote the set
of  $t - 1$  features already chosen in round  $t$ , each round attempts to compute
$$
\max  _ {j \notin S _ {t}} \left(| \hat {\tau} \left(X _ {j}, Y\right) | - \frac {1}{t - 1} \sum_ {j ^ {\prime} \in S _ {t}} | \hat {\tau} \left(X _ {j}, X _ {j ^ {\prime}}\right) |\right). \tag {1}
$$
The  $\frac{1}{t - 1}$  scaling ensures that the sensitivity of the overall quantity remains fixed at  $\frac{3}{2}$  in the first round and 3 in the remaining rounds. Note that in the first round we take second term to be 0, and only label correlation is considered.
Algorithm 1 DPKendall  $(D,k,\varepsilon)$
1: Input: Examples  $D = \{(X_i,Y_i)\}_{i=1}^n$ , number of selected features  $k$ , privacy parameter  $\varepsilon$   
2: for  $j = 1,\ldots,d$  do  
3: Compute  $\hat{\tau}_j^Y = |\hat{\tau}(X_j,Y)|$   
4: Initialize  $S = \emptyset$   
5: Initialize  $\hat{\tau} = \hat{\tau}^Y \in \mathbb{R}^d$   
6: Initialize  $\hat{\tau}^X = 0 \in \mathbb{R}^d$   
7: for  $t = 1,\ldots,k$  do  
8: Set  $\Delta_{\infty} = \frac{3}{2} + \frac{3}{2} \cdot \mathbb{1}_{t > 1}$   
9: Set  $s_t = \text{Peel}\left(\hat{\tau}^Y + \frac{\hat{\tau}^X}{t - 1}, 1, \Delta_{\infty}, \frac{\varepsilon}{k}\right)$   
10: Expand  $S = S \cup s_t$   
11: Update  $\hat{\tau}_{s_t}^Y = -\infty$   
12: for  $j \notin S$  do  
13: Update  $\hat{\tau}_j^X = \hat{\tau}_j^X - |\hat{\tau}(X_j,X_{s_t})|$   
14: Return  $S$
Pseudocode for DPKendall appears in Algorithm 1. Its runtime and privacy are easy to verify.
Theorem 3.4. DPKendall runs in time  $O(dkn\log (n))$  and satisfies  $\varepsilon$ -DP.
Proof. By Lemma 3.3, each computation of Kendall rank correlation takes time  $O(n \log(n))$ , so Line 2's loop takes time  $O(dn \log(n))$ , as does each execution of Line 12's loop. Each call to Peel requires  $O(d)$  samples of Gumbel noise and thus contributes  $O(dk)$  time overall. The loop in Line 7 therefore takes time  $O(dkn \log(n))$ . The privacy guarantee follows from Lemmas 2.4 and 3.2.
For comparison, standard OLS on  $n$  samples of data with  $d$  features requires time  $O(d^{2}n)$ ; DPKendall is asymptotically no slower as long as  $n \leq O(2^{d / k})$ . Since we typically take  $k \ll d$ , DPKendall is computationally "free" in many realistic data settings.
3.3 Utility Guarantee
The proof of DPKendall's utility guarantee combines results about population Kendall rank correlation (Lemma 3.5), empirical Kendall rank correlation concentration (Lemma 3.6), and the accuracy of Peel (Lemma 3.7). The final guarantee (Theorem 3.8) demonstrates that DPKendall selects useful features even in the presence of redundant features.
We start with the population Kendall rank correlation guarantee. Its proof, and all proofs for uncited results in this section, appears in Section 7 in the Appendix.
Lemma 3.5. Suppose that  $X_{1},\ldots ,X_{k}$  are independent random variables where  $X_{j}\sim N(\mu_{j},\sigma_{j}^{2})$  Let  $\xi \sim N(0,\sigma_e^2)$  be independent noise. Then if the label is generated by  $Y = \sum_{j = 1}^{k}\beta_{j}X_{j} + \xi$  for any  $j^{*}\in [k]$
$$
\tau (X _ {j ^ {*}}, Y) = \frac {2}{\pi} \cdot \arctan \frac {\beta_ {j ^ {*}} \sigma_ {j ^ {*}}}{\sqrt {\sum_ {j \neq j ^ {*}} \beta_ {j} ^ {2} \sigma_ {j} ^ {2} + \sigma_ {e} ^ {2}}}.
$$
To interpret this result, recall that  $\tau \in [-1,1]$  and  $\arctan$  has domain  $\mathbb{R}$ , is odd, and has  $\lim_{x\to \infty}\arctan x = \pi /2$ . Lemma 3.5 thus says that if we fix the other  $\sigma_{j}$  and  $\sigma_{e}$  and take  $\sigma_{j^{*}}\rightarrow \infty$ ,  $\tau (X_{j^{*}},Y)\to \mathrm{sign}(\beta_{j^{*}})$  as expected. The next step is to verify that  $\hat{\tau}$  concentrates around  $\tau$ .
Lemma 3.6 (Lemma 1 [3]). Given  $n$  observations each of random variables  $X$  and  $Y$ , with probability  $1 - \eta$ ,
$$
| \hat {\tau} (X, Y) - \frac {n}{2} \cdot \tau (X, Y) | \leq \sqrt {8 n \ln (2 / \eta)}.
$$
Finally, we state a basic accuracy result for the Gumbel noise employed by Peel.
Lemma 3.7. Given i.i.d. random variables  $X_{1},\ldots ,X_{d}\sim \mathrm{Gumbel}(b)$ , with probability  $1 - \eta$
$$
\max  _ {j \in [ d ]} | X _ {j} | \leq b \ln \left(\frac {2 d}{\eta}\right).
$$
We now have the tools necessary for the final result.
Theorem 3.8. Suppose that  $X_{1}, \ldots, X_{k}$  are independent random variables where each  $X_{j} \sim N(\mu_{j}, \sigma_{j}^{2})$ . Suppose additionally that of the remaining  $d - k$  random variables, for each  $j \in [k]$ ,  $n_{j}$  are copies of  $X_{j}$ , where  $\sum_{j=1}^{k} n_{j} \leq d - k$ . For each  $j \in [k]$ , let  $S_{j}$  denote the set of indices consisting of  $j$  and the indices of its copies. Then if the label is generated by  $Y = \sum_{j=1}^{k} \beta_{j} X_{j} + \xi$  where  $\xi \sim N(0, \sigma_{e}^{2})$  is independent random noise, if
$$
n = \Omega \left(\frac {k \cdot \ln (d k / \eta)}{\varepsilon \cdot \min  _ {j ^ {*} \in [ k ]} \left\{\left| \arctan \frac {\beta_ {j ^ {*}} \sigma_ {j ^ {*}}}{\sqrt {\sum_ {j \neq j ^ {*}} \beta_ {j} ^ {2} \sigma_ {j} ^ {2} + \sigma_ {e} ^ {2}}} \right| \right\}}\right),
$$
then with probability  $1 - O(\eta)$ , DPKendall correctly selects exactly one index from each of  $S_{1}, \ldots, S_{k}$ .
Proof. The proof reduces to applying the preceding lemmas with appropriate union bounds. Dropping the constant scaling of  $\eta$  for neatness, with probability  $1 - O(\eta)$ :
1. Feature-label correlations are large for informative features and small for uninformative features: by Lemma 3.5 and Lemma 3.6, each feature in  $j^{*} \in \cup_{j \in [k]} S_{j}$  has
$$
\hat {\tau} (X _ {j ^ {*}}, Y) \geq \frac {n}{\pi} \cdot \arctan \frac {\beta_ {j ^ {*}} \sigma_ {j ^ {*}}}{\sqrt {\sum_ {j \neq j ^ {*}} \beta_ {j} ^ {2} \sigma_ {j} ^ {2} + \sigma_ {e} ^ {2}}} - \sqrt {8 n \ln (d / \eta)})
$$
and any  $j^{*}\notin \cup_{j\in [k]}S_{j}$  has  $\hat{\tau} (X_{j^*},Y)\leq \sqrt{8n\ln(d / \eta)}$
2. Feature-feature correlations are large between copies of a feature and small between independent features: by Lemma 3.5 and Lemma 3.6, for any  $j \in [k]$  and  $j_1, j_2 \in S_j$ ,
$$
\hat {\tau} \left(X _ {j _ {1}}, X _ {j _ {2}}\right) \geq \frac {n}{2} - \sqrt {8 n \ln (d / \eta)}
$$
and for any  $j_1, j_2$  such that there exists no  $S_j$  containing both,  $\hat{\tau}(X_{j_1}, X_{j_2}) \leq \sqrt{8n \ln(d / \eta)}$ .
3. The at most  $dk$  draws of Gumbel noise have absolute value bounded by  $\frac{k}{\varepsilon} \ln \left( \frac{dk}{\eta} \right)$ .
Combining these results, to ensure that DPKendall's  $k$  calls to Peel produce exactly one index from each of  $S_{1},\ldots ,S_{k}$ , it suffices to have
$$
n \cdot \min _ {j ^ {*} \in [ k ]} \left\{\left| \arctan \frac {\beta_ {j ^ {*}} \sigma_ {j ^ {*}}}{\sqrt {\sum_ {j \neq j ^ {*}} \beta_ {j} ^ {2} \sigma_ {j} ^ {2} + \sigma_ {e} ^ {2}}} \right| \right\} = \Omega \left(\left[ \sqrt {n} + \frac {k}{\varepsilon} \right] \ln (\frac {d k}{\eta})\right)
$$
which rearranges to yield the claim.
This section collects experimental evaluations of DPKendall and other methods on 25 of the 33 datasets<sup>5</sup>. Descriptions of the relevant algorithms appear in Section 4.1 and Section 4.2. Section 4.3 discusses the results. Experiment code may be found on Github [14].
<sup>5</sup>We omit the datasets with OpenML task IDs 361080-361084 as they are restricted versions of other included datasets. We also exclude 361090 and 361097 as non-private OLS obtained  $R^2 \ll 0$  on both. Details for the remaining datasets appear in Figure 3 in the Appendix.
4.1 Feature Selection Baseline