Title: Better Private Linear Regression Through Better Private Feature Selection
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

linear regression

sparse

feature selection

kendall
Abstract: Existing work on differentially private linear regression typically assumes that end users can precisely set data bounds or algorithmic hyperparameters. End users often struggle to meet these requirements without directly examining the data (and violating privacy). Recent work has attempted to develop solutions that shift these burdens from users to algorithms, but they struggle to provide utility as the feature dimension grows. This work extends these algorithms to higher-dimensional problems by introducing a differentially private feature selection method based on Kendall rank correlation. We prove a utility guarantee for the setting where features are normally distributed and conduct experiments across 25 datasets. We find that adding this private feature selection step before regression significantly broadens the applicability of ``plug-and-play'' private linear regression algorithms at little additional cost to privacy, computation, or decision-making by the end user.

Better Private Linear Regression Through Better Private Feature Selection

Abstract

Existing work on differentially private linear regression typically assumes that end users can precisely set data bounds or algorithmic hyperparameters. End users often struggle to meet these requirements without directly examining the data (and violating privacy). Recent work has attempted to develop solutions that shift these burdens from users to algorithms, but they struggle to provide utility as the feature dimension grows. This work extends these algorithms to higher-dimensional problems by introducing a differentially private feature selection method based on Kendall rank correlation. We prove a utility guarantee for the setting where features are normally distributed and conduct experiments across 25 datasets. We find that adding this private feature selection step before regression significantly broadens the applicability of 'plug-and-play' private linear regression algorithms at little additional cost to privacy, computation, or decision-making by the end user.

1 Introduction

Differentially private [10] algorithms employ carefully calibrated randomness to obscure the effect of any single data point. Doing so typically requires an end user to provide bounds on input data to ensure the correct scale of noise. However, end users often struggle to provide such bounds without looking at the data itself [27], thus nullifying the intended privacy guarantee. This has motivated the development of differentially private algorithms that do not require these choices from users.

To the best of our knowledge, two existing differentially private linear regression algorithms satisfy this 'plug-and-play' requirement: 1) the Tukey mechanism [2], which combines propose-test-release with an exponential mechanism based on Tukey depth, and 2) Boosted AdaSSP [30], which applies gradient boosting to the AdaSSP algorithm introduced by Wang [34]. We refer to these methods as, respectively, Tukey and BAS . Neither algorithm requires data bounds, and both feature essentially one chosen parameter (the number of models m for Tukey , and the number of boosting rounds T for BAS ) which admits simple heuristics without tuning. In contrast, AdaSSP requires a user to provide bounds on the data's feature and label norms, and DPSGD requires a user to configure hyperparameters including learning rate, clipping norms, batch size, and number of epochs. Both algorithms produce much weaker utility when these parameters are even moderately misconfigured [2].

Tukey obtains strong empirical results when the number of data points n greatly exceeds the feature dimension d [2], while BAS obtains somewhat weaker utility on a larger class of datasets [30]. Nonetheless, neither algorithm provides generally strong performance on its own. Evaluated over a collection of 25 linear regression datasets taken from Tang et al. [30], Tukey and BAS obtain coefficient of determination R 2 > 0 on only four (see Section 4); for context, a baseline of R 2 = 0 is achieved by the trivial constant predictor, which simply outputs the mean label. These results suggest room for improvement for practical private linear regression.

1.1 Our Contributions

We extend existing work on private linear regression by adding a preprocessing step that applies private feature selection. At a high level, this strategy circumvents the challenges of large feature dimension d by restricting attention to k glyph[lessmuch] d carefully selected features. We initiate the study of private feature selection in the context of 'plug-and-play' private linear regression and make two concrete contributions:

We introduce a practical algorithm, DPKendall , for differentially private feature selection (Section 3). DPKendall uses Kendall rank correlation [20] and only requires the user to choose the number k of features to select. It satisfies ε -DP and, given n samples with d -dimensional features, runs in time O ( dkn log( n )) (Theorem 3.4). We also provide a utility guarantee when the features are normally distributed (Theorem 3.8).

We conduct experiments across 25 datasets (Section 4), with k fixed at 5 and 10 . These compare Tukey and BAS without feature selection, with SubLasso feature selection [21], and with DPKendall feature selection. Using (ln(3) , 10 -5 ) -DP to cover both private feature selection and private regression, we find at k = 5 that adding DPKendall yields R 2 > 0 on 56% of the datasets. Replacing DPKendall with SubLasso drops the rate to 40% of datasets, and omitting feature selection entirely drops it further to 16%.

In summary, we suggest that DPKendall significantly expands the applicability and utility of practical private linear regression.

1.2 Related Work

The focus of this work is practical private feature selection applied to private linear regression. We therefore refer readers interested in a more general overview of the private linear regression literature to the discussions of Amin et al. [2] and Tang et al. [30].

Several works have studied private sparse linear regression [21, 31, 17, 29]. However, Jain and Thakurta [17] and Talwar et al. [29] require an glyph[lscript] ∞ bound on the input data, and the stability test that powers the feature selection algorithm of Thakurta and Smith [31] requires the end user to provide granular details about the optimal Lasso model. These requirements are significant practical obstacles. An exception is the work of Kifer et al. [21]. Their algorithm first performs feature selection using subsampling and aggregation of non-private Lasso models. This feature selection method, which we call SubLasso , only requires the end user to select the number of features k . To the selected features, Kifer et al. [21] then apply objective perturbation to privately optimize the Lasso objective. As objective perturbation requires the end user to choose parameter ranges and provides a somewhat brittle privacy guarantee contingent on the convergence of the optimization, we do not consider it here. Instead, our experiments combine SubLasso feature selection with the Tukey and BAS private regression algorithms. An expanded description of SubLasso appears in Section 4.2.

We now turn to the general problem of private feature selection. There is a significant literature studying private analogues of the general technique of principal component analysis (PCA) [25, 15, 8, 19, 11, 1]. Unfortunately, all of these algorithms assume some variant of a bound on the row norm of the input data. Stoddard et al. [28] studied private feature selection in the setting where features and labels are binary, but it is not clear how to extend their methods to the non-binary setting that we consider in this work. SubLasso is therefore the primary comparison private feature selection method in this paper. We are not aware of existing work that studies private feature selection in the specific context of 'plug-and-play' private linear regression.

Finally, private rank correlation has previously been studied by Kusner et al. [23]. They derived a different, normalized sensitivity bound appropriate for their 'swap' privacy setting and applied it to privately determine the causal relationship between two random variables.

2 Preliminaries

Throughout this paper, a database D is a collection of labelled points ( x, y ) where x ∈ R d , y ∈ R , and each user contributes a single point. We use the 'add-remove' form of differential privacy.

Definition 2.1 ([10]) . Databases D,D ′ from data domain D are neighbors D ∼ D ′ if they differ in the presence or absence of a single record. A randomized mechanism M : D → O is ( ε, δ ) -differentially private (DP) if for all D ∼ D ′ ∈ D and any S ⊆ O

We use basic composition to reason about the privacy guarantee obtained from repeated application of a private algorithm. More sophisticated notions of composition exist, but for our setting of relatively few compositions, basic composition is simpler and suffers negligible utility loss.

Lemma 2.2 ([10]) . Suppose that for j ∈ [ k ] , algorithm A j is ( ε j , δ j )-DP. Then running all k algorithms is ( ∑ k j =1 ε j , ∑ k j =1 δ j ) -DP.

Both SubLasso and DPKendall use a private subroutine for identifying the highest count item(s) from a collection, known as private topk . Several algorithms for this problem exist [4, 9, 13]. We use the pure DP 'peeling mechanism' based on Gumbel noise [9], as its analysis is relatively simple, and its performance is essentially identical to other variants for the relatively small k used in this paper.

Definition 2.3 ([9]) . A Gumbel distribution with parameter b is defined over x ∈ R by P [ x ; b ] = 1 b · exp ( -x b -e -x/b ) . Given c = ( c 1 , . . . , c d ) ∈ R d , k ∈ N , and privacy parameter ε , Peel ( c, k, ∆ ∞ , ε ) adds independent Gumbel ( 2 k ∆ ∞ ε ) noise to each count c j and outputs the ordered sequence of indices with the largest noisy counts.

Lemma 2.4 ([9]) . Given c = ( c 1 , . . . , c d ) ∈ R d with glyph[lscript] ∞ sensitivity ∆ ∞ , Peel ( c, k, ∆ ∞ , ε ) is ε -DP.

The primary advantage of Peel over generic noise addition is that, although users may contribute to d counts, the added noise only scales with k . We note that while Peel requires an glyph[lscript] ∞ bound, neither DPKendall nor SubLasso needs user input to set it: regardless of the dataset, DPKendall 's use of Peel has glyph[lscript] ∞ = 3 and SubLasso 's use has glyph[lscript] ∞ = 1 (see Algorithms 1 and 2).

3 Feature Selection Algorithm

This section describes our feature selection algorithm, DPKendall , and formally analyzes its utility. Section 3.1 introduces and discusses Kendall rank correlation, Section 3.2 describes the full algorithm, and the utility result appears in Section 3.3.

3.1 Kendall Rank Correlation

The core statistic behind our algorithm is Kendall rank correlation. Informally, Kendall rank correlation measures the strength of a monotonic relationship between two variables.

Definition 3.1 ([20]) . Given a collection of data points ( X,Y ) = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } and i < i ′ , a pair of observations ( X i , Y i ) , ( X i ′ , Y i ′ ) is discordant if ( X i -X i ′ )( Y i -Y i ′ ) < 0 . Given data ( X,Y ) , let d X,Y denote the number of discordant pairs. Then the empirical Kendall rank correlation is

For real random variables X and Y , we can also define the population Kendall rank correlation by

Kendall rank correlation is therefore high when an increase in X or Y typically accompanies an increase in the other, low when an increase in one typically accompanies a decrease in the other, and close to 0 when a change in one implies little about the other. We typically focus on empirical Kendall rank correlation, but the population definition will be useful in the proof of our utility result.

Before discussing Kendall rank correlation in the context of privacy, we note two straightforward properties. First, for simplicity, we use a version of Kendall rank correlation that does not account for ties. We ensure this in practice by perturbing data by a small amount of continuous random noise. Second, τ has range [ -1 , 1] , but (this paper's version of) ˆ τ has range [ -n/ 2 , n/ 2] . This scaling does not affect the qualitative interpretation and ensures that ˆ τ has low sensitivity 4 .

Lemma 3.2. ∆(ˆ τ ) = 3 / 2 .

Proof. At a high level, the proof verifies that the addition or removal of a user changes the first term of ˆ τ by at most 1/2, and the second term by at most 1.

In more detail, consider two neighboring databases ( X,Y ) and ( X ′ , Y ′ ) . Without loss of generality, we may assume that ( X,Y ) = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } and ( X ′ , Y ′ ) = { ( X ′ 1 , Y ′ 1 ) , . . . , ( X ′ n +1 , Y ′ n +1 ) } where for all i ∈ [ n ] we have that X ′ i = X i and Y ′ i = Y i . First, we argue that the number of discordant pairs in ( X ′ , Y ′ ) cannot be much larger than in ( X,Y ) . By definition, we have that d X ′ ,Y ′ -d X,Y = ∑ n j =1 ✶ ( X j -X ′ n +1 )( Y j -Y ′ n +1 ) < 0 . In particular, this implies that d X ′ ,Y ′ -d X,Y ∈ [0 , n ] . We can rewrite the difference in Kendall correlation between ( X,Y ) and ( X ′ , Y ′ ) as follows:

where the final equality follows from the fact that 2( 1 n -1 n -1 ) = -1 / ( n 2 ) . Using our previous calculation, the first term is in the range [0 , 2] and, since d X,Y ∈ [0 , ( n 2 ) ] , the second term is in the range [ -1 , 0] . It follows that

and therefore | ˆ τ ( X,Y ) -ˆ τ ( X ′ , Y ′ ) | ≤ 3 / 2 .

To show that the sensitivity is not smaller than 3 / 2 , consider neighboring databases ( X,Y ) and ( X ′ , Y ′ ) such that d X,Y = 0 and ( X ′ , Y ′ ) contains a new point that is discordant with all points in ( X,Y ) . Then d X ′ ,Y ′ = n while d X,Y = 0 . Then ˆ τ ( X,Y ) -ˆ τ ( X ′ , Y ′ ) = 2 -0 -1 / 2 = 3 / 2 .

Turning to privacy, Kendall rank correlation has two notable strengths. First, because it is computed entirely from information about the relative ordering of data, it does not require an end user to provide data bounds. This makes it a natural complement to private regression methods that also operate without user-provided data bounds. Second, Kendall rank correlation's sensitivity is constant, but its range scales linearly with n . This makes it easy to compute privately. A contrasting example is Pearson correlation, which requires data bounds to compute covariances and has sensitivity identical to its range. An extended discussion of alternative notions of correlation appears in Section 6.

Finally, Kendall rank correlation can be computed relatively quickly using a variant of merge sort.

Lemma 3.3 ([22]) . Given collection of data points ( X,Y ) = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } , ˆ τ ( X,Y ) can be computed in time O ( n log( n )) .

3.2 DPKendall

Having defined Kendall rank correlation, we now describe our private feature selection algorithm, DPKendall . Informally, DPKendall balances two desiderata: 1) selecting features that correlate with the label, and 2) selecting features that do not correlate with previously selected features. Prioritizing only the former selects for redundant copies of a single informative feature, while prioritizing only the latter selects for features that are pure noise.

In more detail, DPKendall consists of k applications of Peel to select a feature that is correlated with the label and relatively uncorrelated with the features already chosen. Thus, letting S t denote the set

of t -1 features already chosen in round t , each round attempts to compute

The 1 t -1 scaling ensures that the sensitivity of the overall quantity remains fixed at 3 2 in the first round and 3 in the remaining rounds. Note that in the first round we take second term to be 0, and only label correlation is considered.

Pseudocode for DPKendall appears in Algorithm 1. Its runtime and privacy are easy to verify.

Theorem 3.4. DPKendall runs in time O ( dkn log( n )) and satisfies ε -DP.

Proof. By Lemma 3.3, each computation of Kendall rank correlation takes time O ( n log( n )) , so Line 2's loop takes time O ( dn log( n )) , as does each execution of Line 12's loop. Each call to Peel requires O ( d ) samples of Gumbel noise and thus contributes O ( dk ) time overall. The loop in Line 7 therefore takes time O ( dkn log( n )) . The privacy guarantee follows from Lemmas 2.4 and 3.2.

For comparison, standard OLS on n samples of data with d features requires time O ( d 2 n ) ; DPKendall is asymptotically no slower as long as n ≤ O (2 d/k ) . Since we typically take k glyph[lessmuch] d , DPKendall is computationally 'free' in many realistic data settings.

3.3 Utility Guarantee

The proof of DPKendall 's utility guarantee combines results about population Kendall rank correlation (Lemma 3.5), empirical Kendall rank correlation concentration (Lemma 3.6), and the accuracy of Peel (Lemma 3.7). The final guarantee (Theorem 3.8) demonstrates that DPKendall selects useful features even in the presence of redundant features.

We start with the population Kendall rank correlation guarantee. Its proof, and all proofs for uncited results in this section, appears in Section 7 in the Appendix.

Lemma 3.5. Suppose that X 1 , . . . , X k are independent random variables where X j ∼ N ( µ j , σ 2 j ) . Let ξ ∼ N (0 , σ 2 e ) be independent noise. Then if the label is generated by Y = ∑ k j =1 β j X j + ξ , for any j ∗ ∈ [ k ] , glyph[negationslash]

To interpret this result, recall that τ ∈ [ -1 , 1] and arctan has domain R , is odd, and has lim x →∞ arctan x = π/ 2 . Lemma 3.5 thus says that if we fix the other σ j and σ e and take σ j ∗ →∞ , τ ( X j ∗ , Y ) → sign ( β j ∗ ) as expected. The next step is to verify that ˆ τ concentrates around τ .

Lemma 3.6 (Lemma 1 [3]) . Given n observations each of random variables X and Y , with probability 1 -η ,

Finally, we state a basic accuracy result for the Gumbel noise employed by Peel .

Lemma 3.7. Given i.i.d. random variables X 1 , . . . , X d ∼ Gumbel ( b ) , with probability 1 -η ,

We now have the tools necessary for the final result.

Theorem 3.8. Suppose that X 1 , . . . , X k are independent random variables where each X j ∼ N ( µ j , σ 2 j ) . Suppose additionally that of the remaining d -k random variables, for each j ∈ [ k ] , n j are copies of X j , where ∑ k j =1 n j ≤ d -k . For each j ∈ [ k ] , let S j denote the set of indices consisting of j and the indices of its copies. Then if the label is generated by Y = ∑ k j =1 β j X j + ξ where ξ ∼ N (0 , σ 2 e ) is independent random noise, if glyph[negationslash]

then with probability 1 -O ( η ) , DPKendall correctly selects exactly one index from each of S 1 , . . . , S k .

Proof. The proof reduces to applying the preceding lemmas with appropriate union bounds. Dropping the constant scaling of η for neatness, with probability 1 -O ( η ) :

Feature-label correlations are large for informative features and small for uninformative features: by Lemma 3.5 and Lemma 3.6, each feature in j ∗ ∈ ∪ j ∈ [ k ] S j has

glyph[negationslash]

and any j ∗ glyph[negationslash]∈ ∪ j ∈ [ k ] S j has ˆ τ ( X j ∗ , Y ) ≤ √ 8 n ln( d/η ) .

Feature-feature correlations are large between copies of a feature and small between independent features: by Lemma 3.5 and Lemma 3.6, for any j ∈ [ k ] and j 1 , j 2 ∈ S j ,

and for any j 1 , j 2 such that there exists no S j containing both, ˆ τ ( X j 1 , X j 2 ) ≤ √ 8 n ln( d/η ) .

3. The at most dk draws of Gumbel noise have absolute value bounded by k ε ln ( dk η ) .

Combining these results, to ensure that DPKendall 's k calls to Peel produce exactly one index from each of S 1 , . . . , S k , it suffices to have glyph[negationslash]

which rearranges to yield the claim.

4 Experiments

This section collects experimental evaluations of DPKendall and other methods on 25 of the 33 datasets 5 . Descriptions of the relevant algorithms appear in Section 4.1 and Section 4.2. Section 4.3 discusses the results. Experiment code may be found on Github [14].

4.1 Feature Selection Baseline

Our experiments use SubLasso [21] as a baseline 'plug-and-play' private feature selection method. At a high level, the algorithm randomly partitions its data into m subsets, computes a non-private Lasso regression model on each, and then privately aggregates these models to select k significant features. The private aggregation process is simple; for each subset's learned model, choose the k features with largest absolute coefficient, then apply private topk to compute the k features most selected by the m models. Kifer et al. [21] introduced and analyzed SubLasso ; we collect its relevant properties in Lemma 4.1. Pseudocode appears in Algorithm 2.

Algorithm 2 SubLasso ( D,k,m,ε )

1: Input: Examples D = { ( X i , Y i ) } n i =1 , number of selected features k , number of models m , privacy parameter ε

2: Randomly partition D into m equal-size subsets S 1 , . . . , S m

3: for i = 1 , . . . , m do

4: Compute Lasso model θ i on S i

5: Compute set C i of the k indices of θ i with largest absolute value

6: Compute binary vector v i ∈ { 0 , 1 } d where v i,j = ✶ j ∈ C i 7: Compute V ∈ R d = ∑ n i =1 v i 8: Return Peel ( V, k, 1 , ε )

Proof. The privacy guarantee is immediate from that of Peel (Lemma 2.4). Solving Lasso on n/m data points with d -dimensional features takes time O ( d 2 n m ) [12]. Multiplying through by m produces the final result, since Peel only takes time O ( dk ) .

Finally, we briefly discuss the role of the intercept feature in SubLasso . As with all algorithms in our experiments, we add an intercept feature (with a constant value of 1) to each vector of features. Each Lasso model is trained on data with this intercept. However, the intercept is removed before the private voting step, k features are chosen from the remaining features, and the intercept is added back afterward. This ensures that privacy is not wasted on the intercept feature, which we always include.

4.2 Comparison Algorithms

We evaluate seven algorithms:

NonDP is a non-private baseline running generic ordinary least-squares regression.

BAS runs Boosted AdaSSP [30] without feature selection. We imitate the parameter settings used by Tang et al. [30] and set feature and gradient clipping norms to 1 and the number of boosting rounds to 100 throughout.

Tukey runs the Tukey mechanism [2] without feature selection. We introduce and use a tighter version of the propose-test-release (PTR) check given by Amin et al. [2]. This reduces the number of models needed for PTR to pass. The proof appears in Section 9 in the Appendix and may be of independent interest. To privately choose the number of models m used by the Tukey mechanism, we first privately estimate a 1 -η probability lower bound on the number of points n using the Laplace CDF, ˜ n = n + Lap ( 1 ε ′ ) -ln(1 / 2 η ) ε ′ , and then set the number of models to m = glyph[floorleft] ˜ n/d glyph[floorright] . Tukey spends 5% of its ε privacy budget estimating m and the remainder on the Tukey mechanism.

L -BAS runs spends 5% of its ε privacy budget choosing m = glyph[floorleft] ˜ n/k glyph[floorright] for SubLasso , 5% of ε running SubLasso , and then spends the remainder to run BAS on the selected features.

L -Tukey spends 5% of its ε privacy budget choosing m = glyph[floorleft] ˜ n/k glyph[floorright] for SubLasso , 5% running SubLasso , and the remainder running Tukey on the selected features using the same m .

K -BAS spends 5% of its ε privacy budget running DPKendall and then spends the remainder running BAS on the selected features.

K -Tukey spends 5% of its ε privacy budget choosing m = glyph[floorleft] ˜ n/k glyph[floorright] , 5% running DPKendall , and then spends the remainder running the Tukey mechanism on the selected features.

Tukey

L-BAS

L-Tukey

K-BAS

K-Tukey

method

0

2

4

6

8

10

12

14

16

number of datasets

k

= 5

rank 1

rank 2

Tukey

L-BAS

L-Tukey

K-BAS

K-Tukey

method

0

2

4

6

8

10

number of datasets

k

= 10

rank 1

rank 2

Tukey

L-BAS

L-Tukey

K-BAS

K-Tukey

method

0

5

10

15

# datasets with

R

2 >0

4

1

5

10

7

14

k

= 5

Tukey

L-BAS

L-Tukey

K-BAS

K-Tukey

method

0

5

10

15

# datasets with

R

2 >0

1

0

4

4

4

6

k

= 10

4.3 Results

All experiments use (ln(3) , 10 -5 ) -DP. Where applicable, 5% of the privacy budget is spent on private feature selection, 5% on choosing the number of models, and the remainder is spent on private regression. Throughout, we use η = 10 -4 as the failure probability for the lower bound used to choose the number of models. For each algorithm and dataset, we run 10 trials using random 90-10 train-test splits and record the resulting test R 2 values. Tables of the results at k = 5 and k = 10 appear in Section 10 in the Appendix. A condensed presentation appears below. At a high level, we summarize the results in terms of relative and absolute performance.

4.3.1 Relative Performance

First, for each dataset and method, we compute the median R 2 of the final model across the 10 trials and then rank the methods, with the best method receiving a rank of 1. Figure 1 plots the number of times each method is ranked first or second.

At k = 5 (left), K -Tukey performs best by a significant margin: it ranks first on 48% of datasets, twice the fraction of any other method. It also ranks first or second on the largest fraction of datasets (68%). In some contrast, L -BAS obtains the top ranking on 24% of datasets, whereas K -BAS only does so on 8%; nonetheless, the two have nearly the same number of total first or second rankings. At k = 10 (right), no clear winner emerges among the feature selecting methods, as L -BAS , K -BAS , and K -Tukey are all first or second on around half the datasets 6 , though the methods using SubLasso have a higher share of datasets ranked first.

4.3.2 Absolute Performance

Second, we count the number of datasets on which the method achieves a positive median R 2 (Figure 2), recalling that R 2 = 0 is achieved by the trivial model that always predicts the mean label. The k = 5 (left) setting again demonstrates clear trends: K -Tukey attains R 2 > 0 on 56% of datasets, L -Tukey does so on 40%, K -BAS does so on 28%, and L -BAS on 20%. DPKendall therefore consistently demonstrates stronger performance than SubLasso . As in the rank data, at k = 10 the picture is less clear. However, K -Tukey is still best by some margin, with the remaining feature selecting methods all performing roughly equally well.

4.4 Discussion

A few trends are apparent from Section 4.3. First, DPKendall generally achieves stronger final utility than SubLasso , particularly for the Tukey mechanism; the effect is similar but smaller for k = 10 ; and feature selection generally improves the performance of private linear regression.

Comparing Feature Selection Algorithms . A possible reason for DPKendall 's improvement over SubLasso is that, while SubLasso takes advantage of the stability properties that Lasso exhibits in certain data regimes [21], this stability does not always hold in practice. Another possible explanation is that the feature coefficients passed to Peel scale with O ( n ) for DPKendall and m = O ( n d ) or m = O ( n k ) for SubLasso . Both algorithm's invocations of Peel add noise scaling with O ( k ε ) , so DPKendall 's larger scale makes it more robust to privacy-preserving noise. Finally, we emphasize that DPKendall achieves this even though its O ( dkn log( n )) runtime is asymptotically smaller than the O ( d 2 n ) runtime of SubLasso in most settings.

Choosing k . Next, we examine the decrease in performance from k = 5 to k = 10 . Conceptually, past a certain point adding marginally less informative features to a private model may worsen utility due to the privacy cost of considering these features. Moving from k = 5 to k = 10 may cross this threshold for many of our datasets; note from Figure 4 and Figure 5 in the Appendix that, of the 21 datasets used for k = 10 , 86% witness their highest private R 2 in the k = 5 setting 7 . Moreover, from k = 5 to k = 10 the total number of positive R 2 datasets across methods declines by more than 50%, from 41 to 19, with all methods achieving positive R 2 less frequently at k = 10 than k = 5 . We therefore suggest k = 5 as the more relevant setting, and a good choice in practice.

The Effect of Private Feature Selection . Much work aims to circumvent generic lower bounds for privately answering queries by taking advantage of instance-specific structure [6, 16, 33, 5]. Similar works exist for private optimization, either by explicitly incorporating problem information [35, 18] or showing that problem-agnostic algorithms can, under certain conditions, take advantage of problem structure organically [24]. We suggest that this paper makes a similar contribution: feature selection reduces the need for algorithms like Boosted AdaSSP and the Tukey mechanism to 'waste' privacy on computations over irrelevant features. This enables them to apply less obscuring noise to the signal contained in the selected features. The result is the significant increase in utility shown here.

5 Conclusion

We briefly discuss DPKendall 's limitations. First, it requires an end user to choose the number of features k to select; we suggest k = 5 as a reasonable first cut. Second, DPKendall 's use of Kendall rank correlation may struggle when ties are intrinsic to the data's structure, e.g., when the data is categorical, as a monotonic relationship between feature and label becomes less applicable. Finally, Kendall rank correlation may fail to distinguish between linear and nonlinear monotonic feature-label relationships, even though the former is more likely to be useful for linear regression. Unfortunately, it is not obvious how to incorporate relationships more sophisticated than simple monotonicity without sacrificing rank correlation's low sensitivity. Answering these questions may be an interesting avenue for future work.

Nonetheless, the results of this paper demonstrate that DPKendall expands the applicability of plugand-play private linear regression algorithms while providing more utility in less time than the current state of the art. We therefore suggest that DPKendall presents a step forward for practical private linear regression.

References

Kareem Amin, Travis Dick, Alex Kulesza, Andres Munoz Medina, and Sergei Vassilvitskii. Differentially private covariance estimation. Neural Information Processing Systems (NeurIPS) , 2019.

Kareem Amin, Matthew Joseph, Mónica Ribero, and Sergei Vassilvitskii. Easy differentially private linear regression. In International Conference on Learning Representations (ICLR) , 2023.

Youssef Anzarmou, Abdallah Mkhadri, and Karim Oualkacha. The Kendall interaction filter for variable interaction screening in high dimensional classification problems. Journal of Applied Statistics , 2022.

Raghav Bhaskar, Srivatsan Laxman, Adam Smith, and Abhradeep Thakurta. Discovering frequent patterns in sensitive data. In Knowledge Discovery and Data Mining (KDD) , 2010.

Jaroslaw Błasiok, Mark Bun, Aleksandar Nikolov, and Thomas Steinke. Towards instanceoptimal private query release. In Symposium on Discrete Algorithms (SODA) , 2019.

Avrim Blum, Katrina Ligett, and Aaron Roth. A learning theory approach to non-interactive database privacy. In Symposium on the Theory of Computing (STOC) , 2008.

Gavin Brown, Marco Gaboardi, Adam Smith, Jonathan Ullman, and Lydia Zakynthinou. Covariance-Aware Private Mean Estimation Without Private Covariance Estimation. In Neural Information Processing Systems (NeurIPS) , 2021.

Kamalika Chaudhuri, Anand Sarwate, and Kaushik Sinha. Near-optimal differentially private principal components. Neural Information Processing Systems (NIPS) , 2012.

David Durfee and Ryan M Rogers. Practical differentially private top-k selection with paywhat-you-get composition. Neural Information Processing Systems (NeurIPS) , 2019.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference (TCC) , 2006.

Cynthia Dwork, Kunal Talwar, Abhradeep Thakurta, and Li Zhang. Analyze gauss: optimal bounds for privacy-preserving principal component analysis. In Symposium on Theory of Computing (STOC) , 2014.

Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. Least angle regression. Annals of Statistics , 2004.

Jennifer Gillenwater, Matthew Joseph, Andres Munoz Medina, and Monica Ribero Diaz. A Joint Exponential Mechanism For Differentially Private Topk . In International Conference on Machine Learning (ICML) , 2022.

Google. private_kendall. https://github.com/google-research/google-research/ tree/master/private_kendall , 2023.

Moritz Hardt and Aaron Roth. Beating randomized response on incoherent matrices. In Symposium on Theory of Computing (ICML) , 2012.

Moritz Hardt and Guy N Rothblum. A multiplicative weights mechanism for privacy-preserving data analysis. In Foundations of Computer Science (FOCS) , 2010.

Prateek Jain and Abhradeep Guha Thakurta. (Near) dimension independent risk bounds for differentially private learning. In International Conference on Machine Learning (ICML) , 2014.

Peter Kairouz, Monica Ribero Diaz, Keith Rush, and Abhradeep Thakurta. (Nearly) Dimension Independent Private ERM with AdaGrad Rates via Publicly Estimated Subspaces. In Conference on Learning Theory (COLT) , 2021.

Michael Kapralov and Kunal Talwar. On differentially private low rank approximation. In Symposium on Discrete Algorithms (SODA) , 2013.

Maurice G Kendall. A new measure of rank correlation. Biometrika , 1938.

Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory (COLT) , 2012.

William R Knight. A computer method for calculating Kendall's tau with ungrouped data. Journal of the American Statistical Association , 1966.

Matt J Kusner, Yu Sun, Karthik Sridharan, and Kilian Q Weinberger. Private causal inference. In Artificial Intelligence and Statistics (AISTATS) , 2016.

Xuechen Li, Daogao Liu, Tatsunori B Hashimoto, Huseyin A Inan, Janardhan Kulkarni, Yin-Tat Lee, and Abhradeep Guha Thakurta. When Does Differentially Private Learning Not Suffer in High Dimensions? Neural Information Processing Systems (NeurIPS) , 2022.

Frank McSherry and Ilya Mironov. Differentially private recommender systems: Building privacy into the netflix prize contenders. In Knowledge Discovery and Data Mining (KDD) , 2009.

Donald Bruce Owen. A table of normal integrals. Communications in Statistics-Simulation and Computation , 1980.

Jayshree Sarathy, Sophia Song, Audrey Emma Haque, Tania Schlatter, and Salil Vadhan. Don't look at the data! How differential privacy reconfigures data subjects, data analysts, and the practices of data science. In Conference on Human Factors in Computing Systems (CHI) , 2023.

Ben Stoddard, Yan Chen, and Ashwin Machanavajjhala. Differentially private algorithms for empirical machine learning. arXiv preprint arXiv:1411.5428 , 2014.

Kunal Talwar, Abhradeep Guha Thakurta, and Li Zhang. Nearly optimal private lasso. Neural Information Processing Systems (NIPS) , 2015.

Shuai Tang, Sergul Aydore, Michael Kearns, Saeyoung Rho, Aaron Roth, Yichen Wang, YuXiang Wang, and Zhiwei Steven Wu. Improved Differentially Private Regression via Gradient Boosting. arXiv preprint arXiv:2303.03451 , 2023.

Abhradeep Guha Thakurta and Adam Smith. Differentially private feature selection via stability arguments, and the robustness of the lasso. In Conference on Learning Theory (COLT) , 2013.

John W Tukey. Mathematics and the picturing of data. In International Congress of Mathematicians (IMC) , 1975.

Jonathan Ullman. Private multiplicative weights beyond linear queries. In Principles of Database Systems (PODS) , 2015.

Yu-Xiang Wang. Revisiting differentially private linear regression: optimal and adaptive prediction & estimation in unbounded domain. In Uncertainty in Artificial Intelligence (UAI) , 2018.

Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee. Bypassing the ambient dimension: Private sgd with gradient subspace identification. International Conference on Learning Representations (ICLR) , 2021.

6 Alternative Notions of Correlation

6.1 Pearson

The Pearson correlation between some feature X and the label Y is defined by

Evaluated on a sample of n points, this becomes

where ¯ X and ¯ Y are sample means.

Proof. This is immediate from Cauchy-Schwarz.

Note that a value of -1 is perfect anticorrelation and a value of 1 is perfect correlation. A downside of Pearson correlation is that it is not robust. In particular, its sensitivity is the same as its range.

Lemma 6.2. Pearson correlation has glyph[lscript] 1 -sensitivity ∆( r ) = 2 .

where the approximation is increasingly accurate as c grows.

6.2 Spearman

Spearman rank correlation is Pearson correlation applied to rank variables.

Definition 6.3. Given data points X 1 , . . . , X n ∈ R , the corresponding rank variables R ( X 1 ) , . . . , R ( X n ) are defined by setting R ( X i ) to the position of X i when X 1 , . . . , X n are sorted in descending order. Given data ( X,Y ) , the Spearman rank correlation is ρ ( X,Y ) := r ( R ( X ) , R ( Y )) .

For example, given database D = { (0 , 1) , (1 , 0) , (2 , 3) } , its Spearman rank correlation is

A useful privacy property of rank is that it does not depend on data scale. Moreover, if there are no ties then Spearman rank correlation admits a simple closed form.

If we consider adding a 'perfectly unsorted' data point with rank variables (1 , n +1) to a perfectly sorted database with rank variables { (1 , 1) , (2 , 2) , . . . , ( n, n ) } , ρ changes from 1 to 1 -6( n + n 2 ) n ( n 2 -1) . The sensitivity's dependence on n complicates its usage with add-remove privacy. Nonetheless, both Spearman and Kendall correlation's use of rank makes them relatively easy to compute privately, and as the two methods are often used interchangeably in practice, we opt for Kendall rank correlation for simplicity.

7 Deferred Proofs From Section 3.3

We first restate and prove Lemma 3.5.

Lemma 3.5. Suppose that X 1 , . . . , X k are independent random variables where X j ∼ N ( µ j , σ 2 j ) . Let ξ ∼ N (0 , σ 2 e ) be independent noise. Then if the label is generated by Y = ∑ k j =1 β j X j + ξ , for any j ∗ ∈ [ k ] , glyph[negationslash]

Proof. Recall from Definition 3.1 the population formulation of τ ,

glyph[negationslash]

where X and X ′ are i.i.d., as are Y and Y ′ . In our case, we can define Z = ∑ j = j ∗ β j X j + ξ and rewrite the first term of Equation 3 for our setting as

glyph[negationslash]

where f A and F A denote densities and cumulative distribution functions of random variable A , respectively. Since the relevant distributions are all Gaussian, X j ∗ -X ′ j ∗ ∼ N (0 , 2 σ 2 j ∗ ) and Z -Z ′ ∼ N (0 , 2[ ∑ j = j ∗ β 2 j σ 2 j + σ 2 e ]) . For neatness, shorthand σ 2 = 2 σ 2 j ∗ and σ 2 -1 = 2[ ∑ j = j ∗ β 2 j σ 2 j + σ 2 e ] . Then if we let φ denote the PDF of a standard Gaussian and Φ the CDF of a standard Gaussian, Equation 4 becomes glyph[negationslash]

We can similarly analyze the second term of Equation 3 to get

Using both results, we get

where the third equality uses ∫ ∞ 0 φ ( t/σ ) dt = σ 2 and the fourth equality comes from Equation 1,010.4 of Owen [26]. Substituting in the values of σ and σ -1 yields the claim.

Next is Lemma 3.7.

Lemma 3.7. Given i.i.d. random variables X 1 , . . . , X d ∼ Gumbel ( b ) , with probability 1 -η ,

Proof. Recall from Definition 2.3 that Gumbel ( b ) has density f ( x ) = 1 b · exp ( -x b -e -x/b ) . Then f ( x ) f ( -x ) = exp ( -x b -e -x/b -x b + e x/b ) . For z ≥ 0 , by e z = ∑ ∞ n =0 z n n ! ,

so since b ≥ 0 , f ( x ) ≥ f ( -x ) for x ≥ 0 . Letting F ( z ) = exp( -exp( -z/b )) denote the CDF for Gumbel ( b ) , it follows that for z ≥ 0 , 1 -F ( z ) ≥ F ( -z ) . Thus the probability that max j ∈ [ d ] | X j | exceeds t is upper bounded by 2 d (1 -F ( t )) . The claim then follows from rearranging the inequality

and using -ln(1 -x ) = ∑ ∞ i =1 x i /i ≥ x .

8 Datasets

A summary of the datasets used in our experiments appears in Figure 3.

We briefly discuss the role of the intercept in these datasets. Throughout, we explicitly add an intercept feature (constant 1) to each vector. Where feature selection is applied, we explicitly remove the intercept feature during feature selection and then add it back to the k selected features afterward. The resulting regression problem therefore has dimension k +1 . We do this to avoid spending privacy budget selecting the intercept feature.

9 Modified PTR Lemma

This section describes a simple tightening of Lemma 3.6 from Amin et al. [2] (which is itself a small modification of Lemma 3.8 from Brown et al. [7]). Tukey uses the result as its propose-testrelease (PTR) check, so tightening it makes the check easier to pass. Proving the result will require introducing details of the Tukey algorithm. The following exposition aims to keep this document both self-contained and brief; the interested reader should consult the expanded treatment given by Amin et al. [2] for further details.

Tukey depth was introduced by Tukey [32]. Amin et al. [2] used an approximation for efficiency. Roughly, Tukey depth is a notion of depth for a collection of points in space. (Exact) Tukey depth is evaluated over all possible directions in R d , while approximate Tukey depth is evaluated only over axis-aligned directions.

Definition 9.1 ([32, 2]) . A halfspace h v is defined by a vector v ∈ R d , h v = { y ∈ R d | 〈 v, y 〉 ≥ 0 } . Let E = { e 1 , ..., e d } be the canonical basis for R d and let D ⊂ R d . The approximate Tukey depth of

a point y ∈ R d with respect to D , denoted ˜ T D ( y ) , is the minimum number of points in D in any of the 2 d halfspaces determined by E containing y ,

At a high level, Lemma 3.6 from Amin et al. [2] is a statement about the volumes of regions of different depths. The next step is to formally define these volumes.

Definition 9.2 ([2]) . Given database D , define S i,D = { y ∈ R d | ˜ T D ( y ) ≥ i } to be the set of points with approximate Tukey depth at least i in D and V i,D = vol ( S i,D ) to be the volume of that set. When D is clear from context, we write S i and V i for brevity. We also use w D ( V d,D ) := ∫ S d,D exp( ε · ˜ T D ( y )) dy to denote the weight assigned to V d,D by an exponential mechanism whose score function is ˜ T D .

Amin et al. [2] define a family of mechanisms A 1 , A 2 , . . . , where A t runs the exponential mechanism to choose a point of approximately maximal Tukey depth, but restricted to the domain of points with Tukey depth at least t . Since this domain is a data-dependent quantity, they use the PTR framework to select a safe depth t . We briefly recall the definitions of 'safe' and 'unsafe' databases given by Brown et al. [7], together with the key PTR result from Amin et al. [2].

Definition 9.3 (Definitions 2.1 and 3.1 [7]) . Two distributions P , Q over domain W are ( ε, δ ) -indistinguishable , denoted P ≈ ε,δ Q , if for any measurable subset W ⊂ W ,

Database D is ( ε, δ, t ) -safe if for all neighboring D ′ ∼ D , we have A t ( D ) ≈ ε,δ A t ( D ′ ) . Let Safe ( ε,δ,t ) be the set of safe databases, and let Unsafe ( ε,δ,t ) be its complement.

We can now restate Lemma 3.6 from Amin et al. [2]. Informally, it states that if the volume of an 'outer' region of Tukey depth is not much larger than the volume of an 'inner' region, the difference

in depth between the two is a lower bound on the distance to an unsafe database. For the purpose of this paper, Tukey applies this result by finding such a k , adding noise for privacy, and checking that the resulting distance to unsafety is large enough that subsequent steps will be privacy-safe.

Lemma 9.4. Define M ( D ) to be a mechanism that receives as input database D and computes the largest k ∈ { 0 , . . . , t -1 } such that there exists g > 0 where

or outputs -1 if the inequality does not hold for any such k . Then for arbitrary D

M is 1-sensitive, and

for all z ∈ Unsafe ( ε, 4 e ε δ,t ) , d H ( D,z ) > M ( D ) .

We provide a drop-in replacement for Lemma 9.4 that slightly weakens the requirement placed on k . Lemma 9.5. Define M ( D ) to be a mechanism that receives as input database D and computes the largest k ∈ { 0 , . . . , t -1 } such that

or outputs -1 if the inequality does not hold for any such k . Then for arbitrary D

M is 1-sensitive, and

for all z ∈ Unsafe ( ε, 4 e ε δ,t ) , d H ( D,z ) > M ( D ) .

The new result therefore replaces the denominator V t + k + g +1 ,D · e εg/ 2 with denominator w D ( V t + k -1 ,D ) e ε ( t + k +1) . Every point in V t + k -1 ,D of depth at least t + k + g + 1 has score at least t + k + g + 1 , so w D ( V t + k -1 ,D ) e ε ( t + k +1) ≥ V t + k + g +1 ,D · e εg , so V t + k + g +1 ,D · e εg/ 2 ≤ w D ( V t + k + g +1 ,D ) e ε ( t + k +1) and the check for the new result is no harder to pass. To see that it may be easier, note that only the new result takes advantage of the higher scores of deeper points in V t + k -1 ,D .

Proof of Lemma 9.5. First we prove item 1. Let D and D ′ be any neighboring databases and suppose WLOG that D ′ = D ∪ { x } . We want to show that | M ( D ) -M ( D ′ ) | ≤ 1 .

First we prove relationships between the points with approximate Tukey depth at least p and p -1 in datasets D and D ′ . From the definition of approximate Tukey depth, together with the fact that D ′ contains one additional point, for any point y , we are guaranteed that ˜ T D ( y ) ≤ ˜ T D ′ ( y ) ≤ ˜ T D ( y ) + 1 . Recall that S p,D is the set of points with approximate Tukey depth at least p in D . This implies that S p +1 ,D ⊂ S p,D . Next, since for every point y we have ˜ T D ′ ( y ) ≥ ˜ T D ( y ) , we have that S p,D ⊂ S p,D ′ . Finally, since ˜ T D ( y ) ≥ ˜ T D ′ ( y ) -1 , we have that S p,D ′ ⊂ S p -1 ,D . Taken together, we have

Next, since the unnormalized exponential mechanism density y ↦→ exp( y ˆ T D ( y )) is non-negative, we have that w D ( V p +1 ,D ) = ∫ S p +1 ,D exp( glyph[epsilon1] ˜ T D ( y )) dy ≤ ∫ S p,D exp( glyph[epsilon1] ˜ T D ( y )) dy = w D ( V p,D ) . Using the fact that ˜ T D ( y ) ≤ ˜ T D ′ ( y ) , we have that w D ( V p,D ) = ∫ S p,D exp( glyph[epsilon1] ˜ T D ( y )) dy ≤ ∫ S p,D ′ exp( glyph[epsilon1] ˜ T D ′ ( y )) dy = w D ′ ( V p,D ′ ) . Finally, using the fact that ˜ T D ′ ( y ) ≤ ˜ T D ( y ) + 1 , we have w D ′ ( V p,D ′ ) = ∫ S p,D ′ exp( ε ˜ T D ′ ( y )) dy ≤ ∫ S p -1 ,D exp( glyph[epsilon1] ˜ T D ( y ) + glyph[epsilon1] ) dy = e glyph[epsilon1] w D ( V p -1 ,D ) . Together, this gives

Now suppose there exists k ∗ D ′ ≥ 0 such that V t -k ∗ D ′ -1 ,D ′ w D ′ ( V t + k ∗ D ′ +2 ,D ′ ) · e ε ( t + k ∗ D ′ +1) ≤ δ . Then by Equation 5, V t -k ∗ D ′ -1 ,D ′ ≥ V t -k ∗ D ′ ,D , and by Equation 6 w D ′ ( V t + k ∗ D ′ +2 ,D ′ ) ≤ w D ( V t + k ∗ D ′ +1 ,D ) · e ε , so

It follows that

V t -k ∗ D ′ ,D w D ( V t + k ∗ D ′ +1 ,D ) · e ε ( t + k ∗ D ′ ) ≤ δ and then k ∗ D ≥ k ∗ D ′ -1 . Similarly, if there exists k ∗ D ≥ 0 such that

As in Lemma 9.4, item 2 is a consequence of Lemma 3.8 from Brown et al. [7] and the fact that k ∗ = -1 is a trivial lower bound on distance. The only change made to the proof of Lemma 3.8 of Brown et al. [7] is, in its notation, to replace its denominator lower bound with

This uses the fact that x and z differ in at most the addition or removal of k data points. Thus V t + k -1 ,x ≤ V t -1 ,z , and no point's score increases by more than k from V t -1 ,z to V t + k -1 ,x . Since their numerator upper bound is V t -k -1 ,x · e ε ( t +1) (note that the 2 is dropped here because approximate Tukey depth is monotonic; see Section 7.3 of [2] for details), the result follows.

10 Extended Experiment Results

Task ID

NonDP

Tukey

L

-

L

-

Tukey

K

-

K

-

Tukey

361072

7.4e-01

5.3e-01

-∞

3.5e-01

1.8e-01

7.3e-01

8.2e-02

361073

4.6e-01

-8.2e-01

-∞

-4.4e-01

-1.7e+00

-4.8e-01

-1.6e+01

361074

8.1e-01

-2.3e+06

-1.3e+03

-5.6e+06

-3.4e+02

-7.2e+04

-1.4e+02

361075

6.2e-01

-1.7e+02

-∞

7.5e-02

-9.7e-01

3.1e-02

-2.8e+00

361076

2.8e-01

-2.4e+02

-8.2e+04

-2.7e+01

-3.5e+03

-2.4e+01

-1.4e+03

361077

7.9e-01

-1.1e+08

-∞

-1.4e+07

-3.1e+01

-2.1e+08

-1.7e+02

361079

2.4e-01

-1.8e+00

-3.7e+01

-1.2e+01

-8.1e-01

-4.1e+00

-1.5e+00

361087

7.1e-01

-3.7e+01

-1.5e+04

-7.2e+00

-4.8e+03

-8.5e+01

-2.e+01

361088

7.3e-01

-1.9e+00

-∞

1.2e-01

9.2e-02

1.2e-01

7.1e-02

361091

2.4e-01

-3.4e+00

6.8e-02

-3.1e+04

5.7e-02

361092

4.0e-02

-2.4e+06

-∞

-8.7e+05

-7.6e+09

-7.5e+05

-3.7e+10

361093

4.3e-01

-3.2e-02

-∞

-5.e-01

-∞

-1.1e-01

-∞

361095

2.2e-01

-1.9e+01

-1.2e+09

-8.6e-01

-2.4e+07

-1.4e+00

2.1e-01

361096

9.8e-01

-3.7e+02

-1.0e+08

-3.4e+03

-1.8e+00

-1.0e+03

4.2e-01

361098

8.6e-01

-1.8e+02

-∞

-4.1e-01

-1.3e+07

-9.5e-01

-5.8e-01

361099

4.0e-01

-4.3e-01

-2.3e+10

-4.9e-01

-2.4e+09

-4.8e-01

-2.5e+08

361100

1.2e-01

-9.2e+01

-∞

-1.6e+00

-8.2e-02

-3.5e+00

-5.2e-01

361101

3.4e-01

-3.3e+03

-1.7e+08

-8.1e-01

-1.4e+02

-9.5e-01

-4.7e+05

361102

7.5e-01

-9.5e+02

-1.7e+15

-1.3e+00

-1.5e+15

-2.5e+01

-1.4e+05

361103

5.4e-01

-2.6e+00

-1.2e+08

4.9e-01

5.1e-01

3.5e-01

4.9e-01

361104

6.8e-01

-9.e+02

-1.0e+08

-2.9e+03

-3.5e+07

-1.4e+03

-6.6e+05

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe submission proposed a private feature selection algorithm that can be directly applied as a preprocessing step before any private linear regression algorithm. The algorithm itself is based on private top-k selection with the Kendall rank correlation, which has a constant sensitivity. Further experiments on 25 regression datasets demonstrate the effectiveness of the proposed algorithm compared to a different feature selection algorithm.\n\nSTRENGTHS:\n1. the choice of using Kendall rank correlation as the scoring function for features is smart, and it has a constant sensitivity. With an increasing number of data samples, the algorithm becomes more and more accurate.\n\n2. the experimental execution is thorough, and experimental results strongly supports the effectiveness of the algorithm.\n\nWEAKNESSES:\n1. the proposed private feature selection indeed improves both algorithms, but it also introduces a new hyperparameter. It would be great if the authors could provide recommendations on when to use the proposed feature selection and how to set the number of features to maintain (the submission indeed points out that 5 would be a good starting point, but I wonder if the authors could comment more).\n\n2. it seems that the performance of BAS by itself is worse than what Tang et al reported in their work. Perhaps the authors implemented the algorithm themselves, but since BAS is based on AdaSSP, I do recommend the authors check the implementation provided by Yu-Xiang Wang here: https://github.com/yuxiangw/autodp/blob/master/tutorials/tutorial_AdaSSP_vs_noisyGD.ipynb. A few additional cares are taken in the AdaSSP's implementation.\n\n3. Tukey itself is based on subsample and aggregate, so I think it might be interesting to integrate the feature selection step into Tukey as a whole algorithm rather than having two parts.\n\nQUESTIONS:\nplease see the weaknesses above.\n\nLIMITATIONS:\nplease see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies a differentially private feature selection method based on Kendall rank correlation as plug-and-play to make differentially private linear regression more suitable to high-dimensional problems without requiring users to set data bounds or algorithmic hyperparameters. This work provides e a utility guarantee for such private feature selection when features are normally distributed. This work conducts experiments across 25 datasets by applying DP-Kendall to Turkey and BAS to show the effectiveness of DP-Kendall.\n\nSTRENGTHS:\n1. This work makes the existing differentially private linear regression more suitable for high-dimension problems by a feature selection based on Kendall rank correlation. This work provides an analysis of how to make Kendall rank correlation DP and is also well-motivated why choosing Kendall rank correlation. \n\n2. DP-Kendall only incurs a little additional cost to privacy and computation.\n\n3. This work provides the utility guarantee for such DP feature selection when features are normally distributed. \n\n4. The experiments across 25 datasets show that DP-Kendall is necessary when the low-dimension assumption does not hold.  \n\n5. The paper is well-organized and easy to follow.\n\nWEAKNESSES:\n1. DPKendall requires an end user to choose the number of features k to select. The experiment result of $k=5$ and $k=10$ shows that the value of $k$ significantly changes the results, which indicates that it may not be easy to select a proper $k$.\n\n2. DPKendall's use of Kendall rank correlation may struggle when ties are intrinsic to the data’s structure.\n\nMinor:\n1. The experiment set-up considers $(\\epsilon, \\delta)=(\\ln 3, 1e-5)$, while some evaluated datasets have $n>100,000$ samples, and $1e-5$ is not smaller than such $1/n$.\n\nQUESTIONS:\n1. The current version of the paper provides the computation cost of DP-Kendall, DP-Lasso, and standard OLS. I wonder if it is possible to compare the utility guarantee with previous works.\n\n2. For the experiment part, it seems that the value of $k$ should not change the performance of baseline BAS and Turkey as these two do not include the feature selection step. Is the difference between these two methods in Figure 2 for $k=5$ and $k=10$ a result of random runs?\n\nLIMITATIONS:\nThe paper states the following limitations:\n\n1. DPKendall requires an end user to choose the number of features k to select. \n\n2. DPKendall's use of Kendall rank correlation may struggle when ties are intrinsic to the data’s structure.\n\n3. Kendall rank correlation may fail to distinguish between a feature with a strong linear monotonic relationship with the label and a feature with a strong nonlinear monotonic relationship with the label.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper under consideration delves into the challenge of performing linear regression under differential privacy, especially in scenarios where the data is unbounded. This issue arises primarily due to the current private linear regression mechanisms that require tuning a set of hyperparameters for unbounded data, thereby degrading data privacy. To navigate this, the paper utilizes two \"plug-and-play\" mechanisms—namely the Turkey and the BoostedAda (BAS) mechanisms—which circumvent the necessity of parameter tuning. Nonetheless, their limitation lies in their inadequate scalability for high-dimensional data.\n\nTo address this, the paper innovates a differentially private technique for feature selection, DPKendall, intended for high dimensional unbounded data. It’s goal is to choose the most important features—those significantly correlated to the dependent variable in linear regression. These features can then be used with either the Turkey or BAS mechanism.\n\nThe distinctiveness of DPKendall lies in its utilization of Kendall rank correlation to identify correlations between features. The advantages of this method is that it does not require knowledge of data range—relying solely on the relative ordering of the data—and its bounded sensitivity, which is the measure of how much the correlation computation changes in response to a single individual's input data changing. Bounded sensitivity is crucial for upholding differential privacy.  Then, the DPKendall method privately selects k features, seeking to optimize the Kendall rank correlation with the dependent variable while minimizing its correlation with previously chosen features. The latter objective aids in eliminating redundancy.\n\n\nAlthough principal component analysis (PCA) is a prevalent approach for dimension reduction, its applicability is restricted due to most DP methods' necessity for bounded data. Therefore, the paper compares DPKendall with another private feature selection technique, SubLasso.\n\nThe paper substantiates the efficacy of the DPKendall method through experiments on 25 datasets, each comparing different settings that apportion the privacy budget between a private feature selection method and a private linear regression method. The experimental results lend credence to the superiority of DPKendall over SubLasso in performing private linear regression, thereby highlighting its promise for this task. Moreover, the paper provides theoretical claims that the DPKendall methods is able to choose good features in the presence of many redundant features.\n\nSTRENGTHS:\nThe paper demonstrates the applicability of the Kendall rank correlation statistical technique under differential privacy, enabling the computation of correlations within unbounded data in a privacy-preserving manner, which is of significant utility. This approach is leveraged for private feature selection, serving as a dimension reduction technique aimed at facilitating private linear regression.\n\nThe paper offers both theoretical insights and empirical evidence for their proposed method. Theoretically, it presents a novel contribution by proving that DPKendall effectively identifies features that are correlated with the dependent variable. Empirically, it substantiates the practicality of DPKendall, demonstrating that its application for feature selection culminates in superior outcomes in linear regression.\n\nWEAKNESSES:\nThe experimental outcomes indicate that the DPKendall approach exhibits enhanced performance compared to the alternative SubLasso method. However, DPKendall does not invariably outperform, and the experimental specifications remain somewhat ambiguous. Consequently, it would help to discern under which conditions DPKendall delivers superior results. Addressing questions such as the impact on performance as data dimensions escalate, or which method excels at varying parameter levels could provide valuable insights.\n\nQUESTIONS:\n1) Could you provide a precise definition of what constitutes a \"plug-and-play\" algorithm? The proposed method necessitates the selection of parameter k, and also the budget split. How does this aspect impact the method's practical applicability?\n\n2) In Section 4.2, could you elucidate how the number of gradient boosting rounds T was determined for the BAS method? Were there instances where baselines had to allocate privacy budget for determining the parameter T for the BAS method?\n\n3) Would you clarify the execution of the K-BAS method, specifically referring to bullet 6 under Section 4.2? It is mentioned that 5% of the privacy budget was allocated to DPKendall and 95% to private linear regression. Is this accurate, or was there an allocation of 5% for model selection in this method as well?\n\n4) Has the Kendall rank correlation been employed in previous differential privacy-related work?\n\n5_ Are there any other potential downstream applications for private rank correlations? For instance, could it be utilized for Principal Component Analysis (PCA)?\n\n6) How does the optimal selection of k vary depending on the dataset?\n\n7) Could you explain the rationale behind the chosen split (5%) of the privacy budget in the conducted experiments?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of feature selection for linear regression subject to differential privacy constraints.  One of the benefits of this work is that the feature selection does not require setting bounds on features a priori, thus allowing for fewer hyper-parameters which are difficult to set in practice.  By developing a DP version of the Kendall rank correlation, they show that first doing DP feature selection can significantly improve the quality of the DP linear regression model.  Existing DP linear regression also do not require bounds on features, so together with this DP feature selection, it provides a practical way to do DP linear regression with feature selection.\n\nSTRENGTHS:\nLinear regression is a fundamental problem in data analytics, so having a simple DP version of it will help practitioners in various applications.  Feature selection is an important subroutine in linear regression and reducing the hyper parameters can be beneficial.  The application of the exponential mechanism which balances the two desiderata (features that correlate with label and features that do not correlate with already selected features) is nice.  Can other works for feature selection (e.g. PCA) be summarized in a similar way?\n\nWEAKNESSES:\nThis work claims that their approach will help expand the applicability of private linear regression, but it does not seem like existing approaches were too bad without feature selection.  Was a better private linear regression really the bottleneck for deploying DP linear regression in practice, especially when DP parameters are typically large in practice?  Although this is a nice result, it seems incremental.  The main contributions that I see include a sensitivity analysis of the empirical Kendall rank correlation and a utility guarantee under certain assumptions that I am not sure how the result would be used in practice as it depends on the population parameters.  I think that developing DP algorithms with as few hyper parameters is important, but the feature selection still introduces the $k$ parameter without a clear way to set it.  \n\nFurther, I would like to better understand the connection between Kusner et al [22] and this work.  Is the neighboring relation between “swap” and”add-remove” technically important?  Seems like there are a few nuisances with the “add-remove” neighboring relation, e.g. the scaling of $\\hat{\\tau}$ and the estimation of $\\tilde{n}$ in the comparison of algorithms.  I would be interested in how the various approaches in the experiments compare with as few hyper parameters as possible.  For instance, there is some budget allocated for computing $\\tilde{n}$ and an $\\eta$ parameter which seem arbitrary and can be avoided with the “swap” neighboring relation.  \n\nMinor:\n- is the $R^2 = 0$ approach really trivial?  Don’t you need to set some bound on the labels to compute the mean?\n\n- Line 297 “alls”\n\n- Line 7 in Algorithm 2, should the sum be to $m$ not $n$?\n\n- I might be misunderstanding the pseudocode in Algorithm 1, but it seems like $\\hat{\\tau}^X$ will always be negative.\nThe sensitivity of the rank correlation reminds me of work from DP non-parametric hypothesis testing from Couch, Kazan, Shi, Bray, Grace ’19.  How does the Kendal rank correlation compare with the Kruskal-Wallis rank statistic? \n\n- In the introduction, it is not clear what is meant by “plug-and-play” requirement.  Maybe there is a missing paragraph or sentence?\n\nQUESTIONS:\nHow important is the \"add-remove\" neighboring relation compared with \"swap\"?\n\nRather than using “basic” DP composition to compare the various $k$ parameters in the feature selection, can you instead frame this part under zCDP?  In the end you are quoting an approximate DP guarantee, due to the Tukey mechanism being approximate DP, so framing the feature selection as zCDP should be straightforward.  The reason is because as you increase $k$, you are suffering a larger impact on privacy loss than necessary, due to basic composition, which can also influence the general guidance you gave of using $k = 5$.  I would be interested to see the performance of the various tests using zCDP composition with $k$ being swept over several choices.\n\nLIMITATIONS:\nLimitations were stated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}