Title: Robust group and simultaneous inferences for high-dimensional single index model
Venue: NeurIPS 2024
Accepted: yes
Keywords: FDR control

high-dimensional inference

honest test

outliers

robustness
Abstract: The high-dimensional single index model (SIM), which assumes that the response is independent of the predictors given a linear combination of predictors, has drawn attention due to its flexibility and interpretability, but its efficiency is adversely affected by outlying observations and heavy-tailed distributions. This paper introduces a robust procedure by recasting the SIM into a pseudo-linear model with transformed responses. It relaxes the distributional conditions on random errors from sub-Gaussian to more general distributions and thus it is robust with substantial efficiency gain for heavy-tailed random errors. Under this paradigm, we provide asymptotically honest group inference procedures based on the idea of orthogonalization, which enjoys the feature that it does not require the zero and nonzero coefficients to be well-separated. Asymptotic null distribution and bootstrap implementation are both established. Moreover, we develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the false discovery rate asymptotically. Numerical results indicate that the new procedures can be highly competitive among existing methods, especially for heavy-tailed errors.

Robust group and simultaneous inferences for high-dimensional single index model

Abstract

The high-dimensional single index model (SIM), which assumes that the response is independent of the predictors given a linear combination of predictors, has drawn attention due to its flexibility and interpretability, but its efficiency is adversely affected by outlying observations and heavy-tailed distributions. This paper introduces a robust procedure by recasting the SIM into a pseudo-linear model with transformed responses. It relaxes the distributional conditions on random errors from sub-Gaussian to more general distributions and thus it is robust with substantial efficiency gain for heavy-tailed random errors. Under this paradigm, we provide asymptotically honest group inference procedures based on the idea of orthogonalization, which enjoys the feature that it does not require the zero and nonzero coefficients to be well-separated. Asymptotic null distribution and bootstrap implementation are both established. Moreover, we develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the false discovery rate asymptotically. Numerical results indicate that the new procedures can be highly competitive among existing methods, especially for heavy-tailed errors.

1 Introduction

With fast development of information technologies, high-dimensional data are frequently collected in various scientific disciplines. To efficiently analyze high-dimensional data, various regularization methods were developed and thoroughly investigated in both theory and methodology. For a comprehensive review, see [19]. Though significant progress has been made about the theory of estimation and variable selection, statistical inference for a high-dimensional model remains largely unexplored until the seminal works of [58], [47] and [29]. They developed debiased estimators that correct the bias introduced by LASSO and established the estimators' asymptotic normality. [40]

proposed a decorrelated score method which is a general procedure and is applicable to a large family of penalized M-estimators. [45] introduced a recursive online-score estimation for high-dimensional generalized linear model. [46] proposed the nuisance penalized regression which does not penalize the parameters of interest. [21] and [20] studied inference problems for the high-dimensional Cox model and longitudinal data, respectively. Logistic regression model was investigated in [36]. Other recent developments include [44], [50], [5], and [19].

Most of current inference procedures focus on parametric regression models. To address the potential mis-specification issue of parametric models, the single index model (SIM) which is one of the most popular semiparametric modelling techniques, has received extensive attention and in-depth research in the past decade. Let Y ∈ R be the response variable along with predictor vector X = ( X 1 , · · · , X p ) ⊤ ∈ R p . We consider a general SIM,

where β = ( β 1 , · · · , β p ) ⊤ , the link function g ( · ) is unknown and ⊥ means independence. Equivalently, we have

The above statement is that, given X ⊤ β which is called index, the response Y is independent of the predictor vector X . Clearly, model (1.1) covers linear models, generalized linear models and also the classical SIM [26], that is, Y = g 1 ( X ⊤ β ) + ϵ with g 1 ( · ) being an unknown function. It relaxes restrictive assumptions on parametric models and is flexible enough to capture complex relationship between the response and the predictors.

For high-dimensional SIMs, variable selection has been considered by many authors. Examples include [30], [60], [57], [1], [51], [42], [52], [41] and [43]. Our primary interest is to detect whether a set of predictors contributes to the response Y or not given the other predictors, that is testing the following group inference problem:

̸

where G is a prespecified subset of { 1 , 2 , . . . , p } with p 0 = |G| . Such a hypothesis naturally arises in the high-dimensional setting. For example, researchers may want to test whether a gene pathway, consisting of multiple genes for the same biological functions, is important for certain clinical outcome The above group inference problem also includes global significance testing as a special case when we set G = { 1 , 2 , . . . , p } and p 0 = p . The group inference problem is more difficult than purely global significance testing. Actually, for group inference problem, we are dealing with a high-dimensional interested parametric vector with a high-dimensional nuisance parameter. Such a group inference problem was considered by [59] for high-dimensional linear models. See also [16], and [23]. All of [59], [16], and [23] did not establish asymptotical honesty of their procedures, which is critical for reliable inference in high-dimensional setting.

For SIM, due to the existence of the unknown link function g ( · ) , the group inference problem is more difficult. Actually, the parameter of interest and the nonparametric function are bundled together, which makes the inference being challenging. For the model (1.1), [17] developed a debiased LASSO procedure for individual coefficient, while [53] considered simultaneous confidence interval for optimal treatment regimes under the classical SIM [26]. Their procedures, as well as many other existing works, rely heavily on the sub-Gaussian assumption about the error term. In practice, data with heavy-tailed distribution or in the presence of outliers are very common [18], and the efficiency of those procedures developed under sub-Gaussian assumption would be largely deteriorated. Recently [25] investigated the robust inference problem of high-dimensional SIM by adopting the Huber loss with original response. However, their procedure still requires bounded first moment condition of the error term and needs to select a suitable robustification parameter. In this paper, we aim to develop robust group inference methods for high-dimensional SIM without any moment condition on the error term and without choosing robustification parameter.

Interestingly, under a mild condition on the predictors, which is called linearity condition in the literature of sufficient dimension reduction [31], we find that the SIM can be recast into a pseudo-linear model with transformed response, allowing us to make robust inference in a simple fashion. With a specific 'response-distribution" transformation, the sub-Gaussian response assumption required in existing methods is totally avoided, and accordingly our proposed procedures are robust to outliers or heavy-tailed error distributions.

Besides group inference of a prespecified subset of predictors, we are also interested in identifying relevant predictors. To this aim, large-scale simultaneous hypotheses are considered

̸

Apart from identifying as many nonzero β j as possible, to obtain results with uncertainty quantification, we would like to control the false discovery rate (FDR) which is an extremely popular tool to maintain the ability to reliably detect true alternatives without excessive false positive results when large-scale hypotheses are simultaneously tested [4]. [28], [20], [36] and [9] considered FDR control in high-dimensional regression models. We follow this line and develop a FDR control procedure for high-dimensional SIM.

Our major contributions are listed from the following three aspects.

We extend the rank-LASSO procedure in [43] to include both convex and non-convex penalties and establish error bound of any local optimum of the empirical objective. These theoretical results are summarized in subsection 3.1.

In Section 2, we provide asymptotically honest group inference procedures based on the idea of orthogonalization for testing the joint effect of many predictors, which enjoys the feature that it does not require the zero and nonzero coefficients to be well-separated. We demonstrate the superiority of our test procedures both theoretically and empirically. Please see Section 3 for theoretical justification and Section 4 for numerical studies respectively.

We develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the FDR asymptotically. To this end, we develop suitable multiple testing procedures and show that the proposed methods can control the false discovery rate (FDR, [4]) both theoretically and empirically. We refer the readers to Appendix A.3 and A.6 for details.

̸

Notation. For a d -dimension vector U , we write ∥ U ∥ r = ( ∑ d k =1 U r k ) 1 /r and ∥ U ∥ ∞ = max 1 ≤ k ≤ d | U k | to denote l r and l ∞ norms of U . Further we define ∥ U ∥ 0 = # { k : U k = 0 } . A random variable X is sub -Gaussian if the moment generating function (MGF) of X 2 is bounded at some point, namely E exp( X 2 /K 2 ) ≤ 2 , where K is a positive constant. A random variable Y is sub -Exponential if the MGF of | Y | is bounded at some point, namely E exp( | Y | /K ′ ) ≤ 2 , where K ′ is a positive constant. For a, b ∈ R , we write a ∨ b = max { a, b } .

2 Group inference based on distribution transformation

Without loss of generality, assume that E ( X ) = 0 , and Σ = Var ( X ) > 0 . Now let σ h = Cov { X , h ( Y ) } for a given transformation function h ( · ) of the response. Define β h = Σ -1 σ h . We then have the following result.

Proposition 2.1. Assume that E ( X | X ⊤ β ) is a linear function of X ⊤ β . Then β h is proportional to β , that is, β h = κ h × β for some constant κ h .

̸

The above proposition follows directly from Theorem 2.1 in [32]. The assumption in above proposition is known as linearity condition (LC) for predictors. It is satisfied when X has an elliptical distribution and widely assumed in the sufficient dimension reduction literature [31, 14]. [24] showed that in high-dimensional setting, the LC holds to a reasonable approximation. Throughout of the paper, we assume that κ h = 0 . This assumption is mild. In fact, when h ( · ) is monotone and g ( · , · ) is monotone with respect to the first argument, this assumption is satisfied.

From the definition of β h , we can write

where the error term e must satisfy E ( e X ) = 0 . Different from the existing literature which usually imposes independence between the regression error and the predictors, the regression error e in the transformed model is only uncorrelated with the predictors. This implies that under LC, we can recast the general SIM into a pseudo-linear model with transformed response h ( Y ) . Therefore, by Proposition 2.1, testing H 0 , G is equivalent to

̸

This reformulation is important, allowing us to circumvent the issue of estimating the unknown link function g ( · , · ) and to make inference of β hj , j ∈ G in a linear regression model with transformed response instead of β j in SIM. As we show later, this reformulation greatly facilitates the construction of our test statistic and simplifies the computation as well.

In practice, we need to choose a suitable transformation function h ( · ) . We note that [17]'s procedure also relies on the above proposition and they essentially work with h ( Y ) = Y . For robustness consideration, throughout this paper, we consider the distribution function of Y , denoted by F ( Y ) as the transformation function. Actually with the equation (2.3), given the widely imposed subgaussian assumption on the predictors, any bounded transformation function h ( Y ) would lead the transformed error term e being subgaussian, even if the original error term ϵ in the single index model Y = g ( X ⊤ β , ϵ ) comes from Cauchy distribution. Further as noted by [43], in the empirical distribution function, the term ∑ n j =1 I ( Y j ≤ Y i ) is the rank of Y i . Since statistics with ranks such as Wilcoxon test and the Kruskall-Wallis ANOVA test, are well-known to be robust, this then intuitively explains why our procedures with response-distribution transformation are robust with respect to outliers in response. Moreover the distribution function is very easy to estimate and thus our approach is straightforward to implement and understand.

Our test statistic for group inference of H ′ 0 , G relies on individual inference of H ′ 0 j : β hj = 0 . Now we first consider individual hypothesis H ′ 0 j . Let Z j be the subvector of X without X j , and γ j be the subvector of β h without β hj . Suppose that { X i , Y i } n i =1 is a random sample from the population ( X , Y ) . Similarly, we denote Z ij as the sample of Z j . Define

where β h is the regression coefficient of the model (2.3) with h = F .

Our approach is based on the idea of orthogonalization. The main idea of orthogonalization is to construct a statistic for target parameter which is locally insensitive to the nuisance parameters. Dealing with high-dimensional models, it plays an important role to make the statistic of target parameter immune to the bias from the estimators of high-dimensional nuisance parameters, which in turn enables the statistical inference of parameter of interest. For relevant references, see for example, [3], [40] and [2].

As pointed by [53], the adoption of orthogonalization is nontrivial for high-dimensional semiparametric setting, particularly for index model, where the challenge of bundled parameter arises. Fortunately, with the Proposition 2.1, the SIM can be recast into a pseudo-linear model with transformed response (2.3). Note that under the null hypothesis H 0 j

Further the above equation has the orthogonality property

This then motivates us to consider the following quantity

In practice, we need to use suitable estimates of F , γ j and θ j in (2.4) as those quantities are unknown. Naturally, F n ( y ) = n -1 ∑ n i =1 I ( Y i ≤ y ) , the empirical distribution of Y 1 , . . . , Y n , can be used to estimate F ( Y ) . For θ j , we estimate it by the penalized least-squares method

where p λ X ( · ) is a penalty function with a tuning parameter λ X . Similarly, for γ j , we adopt the following penalized least-squares

where L n ( β h ) = (2 n ) -1 ∑ n i =1 { F n ( Y i ) -1 / 2 -X ⊤ i β h } 2 and p λ Y ( · ) is a penalty function with a tuning parameter λ Y . The ˆ γ j is then set as the subvector of ˆ β h without ˆ β hj . Accordingly, for each individual hypothesis H ′ 0 j : β hj = 0 we define the standardized test statistic

where

ˆ e ij = F n ( Y i ) -1 / 2 -Z ⊤ ij ˆ γ j and ˆ m j ( y ) = n -1 ∑ n i =1 ( X ij -Z ⊤ ij ˆ θ j ) { I ( Y i ≥ y ) -F n ( Y i ) } . Note that F n ( Y i ) = n -1 ∑ n j =1 I ( Y j ≤ Y i ) . Then given predictors X i 's being fixed, perturbations in the responses would not make the value of ˜ T nj change as long as the ranks of Y i 's remain unchanged.

Denote ˜ T n, G = ( ˜ T nj ) j ∈G . To test the null hypothesis H 0 , G , we consider test statistic based on the max norm of ˜ T n, G . That is,

Based on the limiting null distribution obtained in subsection 3.2, we can reject null hypothesis H 0 , G at the significant level α if and only if M n, G ≥ c G ( α ) , where c G ( α ) = 2 log p 0 -log log p 0 + q α and q α is the 1 -α quantile of the Gumbel distribution with the cumulative distribution function exp {-1 √ π exp( -x/ 2) } , that is,

Our inference procedure is summrized in the Algothrim 1 as follows.

Algorithm 1: Group inference based on distribution transformation

Input:

Covariates data { X i } n i =1 , response data { Y i } n i =1 .

Output:

Testing results for group inference problem (1.2).

1 Compute the penalized least-square estimator ˆ β h defined in (2.6);

2 for j ∈ G do

3 compuate estimator ˆ γ j and ˆ θ j defined in (2.5);

4 calculate the standard statistic ˜ T nj defined in (2.7);

5 calculate the test statistic M n, G defined in (2.9) ;

6 if M n, G ≥ c G ( α ) , reject H 0 , G ; otherwise accept H 0 , G .

3 Theoretical properties

3.1 Estimation error bound

The theoretical analysis of ˆ β h requires a substantial modification of the proof technique as compared to existing works on high-dimensional inference. It is related to the fact that empirical distribution F n ( Y i ) are dependent, and thus L n ( β h ) is a sum of dependent random variables. Note that [43] considered a similar penalized procedure with LASSO penalty. While [60] and [51] considered nonconvex penalties such as SCAD and MCP to reduce estimation bias incurred by LASSO. However,

their results only allow the dimension to be polynomial order of the sample size and thus cannot work in ultrahigh-dimensional setting. Further their results only provide guarantees for global optima. In this paper, we consider both convex and non-convex penalties and establish error bound of any local optimum of the empirical objective.

To be specific, assume that ˆ β h satisfies the first-order necessary condition to be a local minimum of 2.6, that is,

Let s Y be the sparsity level for β h , i.e., s Y = ∥ β h ∥ 0 . For any vector ˆ β h satisfying the condition (3.11), we have the following result.

Theorem 3.1. Under Assumptions A.1 and A.2 in Appendix A.8, the ˆ β h defined in (3.11) satisfies

with probability at least 1 -c 1 exp( -c 2 log p ) . Here, ( c, c ′ , c 1 , c 2 ) are universal constants and λ X , λ Y ≍ √ log p/n .

In Theorem 3.1 we establish error bound of penalized least-squares estimators with empirical distribution function of the response. For consistency in L 2 -loss, we require the sparsity level s Y satisfy that s Y = o ( n/ log p ) . While in terms of L 1 -loss, the limitation becomes s Y = o ( √ n/ log p ) . Clearly the sparsity level is allowed to be diverging. Our results unify both convex [43] and nonconvex penalties [60, 51]. Compared with the results in [60] and [51], the dimension can be exponential order of the sample size, and no minimal signal condition is imposed to obtain those error bounds. In the proof, we modify Hoeffding's inequality by a probability inequality for U -statistic to handle tail probability in dependent case, which may be interesting in their own rights.

3.2 Asymptotic null distribution

In this subsection, we derive the asymptotic null distribution for statistic defined in (2.9). Denote Ω = { β h ∈ R p : ∥ β h ∥ 0 ∨ max 1 ≤ j ≤ p ∥ θ j ∥ 0 ≤ s } and β h G = ( β hj ) j ∈G . We consider the following parameter space for H 0 , G

Theorem 3.2. Suppose that Assumptions A.1-A.6 in Appendix A.8 and LC condition hold. If s = o ( √ n/ (log p log p 0 )) , then for given t ∈ R we have

Theorem 3.2 implies that the type I error of the proposed test statistic M n, G converges to any pre-specified significance level uniformly over β h ∈ Ω 0 G . The hypothesis test with such uniform convergence property is called honest test. The advantage of honest test is that the limiting distribution of our procedure is uniformly valid over s -sparse high-dimensional models despite the possible imperfect model selection via estimator ˆ β h . An immediate implication is that it relaxes the assumption on signal strength and does not require the zero and nonzero effects to be well-separated. In particular, this procedure does not require the initial estimator to select zero and nonzero signals perfectly, which is nearly impossible in practice. Due to these excellent statistical properties, honest test has recently drawn lots of attention. See [47], [29], [3], [2], [11], [53] for further examples.

On the basis of Theorem 3.2 , we can reject null hypothesis H 0 , G at the significant level α if and only if M n, G ≥ c G ( α ) , where c G ( α ) = 2 log p 0 -log log p 0 + q α and q α is defined in (2.10).

3.3 Power analysis

We next to consider the asymptotic power analysis of the M n, G . In this section, we show that our test statistic is powerful in the sense that the separation rate is of order √ (2 + ϵ 0 ) log p 0 /n for some ϵ 0 > 0 when p 0 →∞ . At the beginning we define the following parameter space for H 1 , G

-1

where c 0 is a positive constant and δ j = E ( X 2 j ) -E ( X j Z ⊤ j ) E ( Z j Z ⊤ j ) E ( Z j X j ) .

Theorem 3.3. Suppose that conditions in Theorem 3.2 are satisfied, we have for some ϵ 0 > 0 ,

Theorem 3.3 implies that our test procedure can still be powerful even when there exists few components of β G with a magnitude being larger than √ (2 + ϵ 0 ) log p 0 /n . Therefore our testing procedure is powerful against 'sparse" alternative aforementioned. This separation rate is widely discussed in the literature, such as [8], [59] and [36]. Theorem 3.3 shows our tests based on distribution transformation can also achieve this lower bound.

Remark 3.1. In the proof of Theorem 3.3, we show that

where the M 0 G has the type I extreme limiting distribution, which is mentioned in Theorem 3.2. Then the power is largely determined by the second term

and it can be explained as the maximum of the signal-to-noise ratio for j ∈ G . The power of the test based on M n, G increases with the growth of SNR( M n, G ) . When SNR( M n, G ) ≥ (2 + ϵ 0 ) log p 0 /n , the asymptotic power is tending to 1.

In this paper, we focus on the distribution transformation of the response. In the following we make some comparisons between this choice and another natural one. Actually, same to [17], one may consider use h ( Y ) = Y to conduct the following test statistic

Here ˆ γ + j is the subvector of ˆ β + h without j -th component, and ˆ β + h is a penalized least-squares estimator of β + h = E ( XX ⊤ ) -1 E ( X Y ) as follows:

ˆ θ j is estimated by (2.5). And ˆ σ +2 j is an appropriate estimator of σ +2 j := E { ( X j -Z ⊤ j θ j ) 2 ( Y -Z ⊤ j γ + j ) 2 } . For the group inference problem (1.2), it's natural to consider the following testing statistic:

Now we discuss the power properties of M + n, G . By similar arguments as Theorem 3.3, the power of the test based on M + n, G is determined by

For the convenience of illustration, we consider the classical SIM, that is, Y = g 1 ( X ⊤ β ) + ϵ and ϵ is independent of X . To compare the theoretical power of M n, G and M + n, G , it suffices to compare the SNRs aforementioned. In the classical SIM case, the ratio of SNR can be simplified as

where c 3 is a constant depending on ( X , Y ) . The detailed form of c 3 is presented in the Appendix A.2. When Var( ϵ ) > c -1 3 , M n, G is more powerful than M + n, G . Actually when ϵ follows a heavy-tail distribution, Var( ϵ ) can be very large even be infinite.

The above results illustrate the robustness of our test statistic M n, G . In fact, with distribution transformation, no moment assumption on ϵ is required. Even if the ϵ comes from Cauchy distribution, our test procedure still works well but the M + n, G requires sub-Gaussian assumption on the error term and thus in this situation would fail.

4 Numerical studies

In this section, extensive simulation studies are carried out to evaluate the numerical performance of the proposed methods for the global inference problem described in (1.2). We consider the high-dimensional SIM in equation (1.1) and generate the data from the following models:

Here, the predictors X i , i = 1 , · · · , n are generated from multivariate normal distribution N p ( 0 p , Σ ) . The covariance matrix Σ ∈ R p × p is block diagonal, that is, Σ = diag ( Σ 1 , · · · , Σ 10 ) , where Σ 1 is p 10 × p 10 dimensional identity matrix, and each Σ k is p 10 × p 10 dimensional with an AR (1) correlation structure, that is ( Σ k ) ij = (0 . 1 k -0 . 1) | i -j | , k = 2 , . . . , 10 ; i, j = 1 , · · · , p/ 10 . We consider two different error distributions for ϵ which is independent of X : (1) standard normal distribution N (0 , 1) ; (2) Cauchy distribution or equivalently Student's t distribution with 1 degree of freedom, t (1) . Without loss of generality, we set the active set be { 1 , 2 , . . . , 6 } . The regression coefficients β are generated from an arithmetic sequence from 0 . 1 to 2 , that is β j = 0 . 2 + 0 . 038( j -1) for 1 ≤ j ≤ 6 and β j = 0 otherwise. We consider n = 200 , 500 and p = 800 . We use Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz.

In order to explore the robustness behaviors of our proposed statistics, we add outliers to pollute the observations: p out of the responses are picked at random and increased by m out-times maximum of original responses, shorted as p out + m out · max( responses ) . Here p out is the proportion of outliers and m out is a pre-set constant standing for outlier strength. Throughout the simulation study, we analyze the results with both the original data (shorted as Ori.) and the data with outliers (shorted as Out.).

We assess the empirical type I error and the empirical power of group inference with the nominal level α = 0 . 05 based on 500 simulation runs. We consider the following five different choices for G to test the hypothesis (1.2):

Note that G 1 and G 2 are small groups, G 3 and G 4 are large groups. G 1 and G 3 consist of only zero coefficients, while G 2 and G 4 includes nonzero elements.

Then we report the numerical results of empirical rejection rate (ERR) for different scenarios, where ERR is the proportion of rejected hypotheses among the total 500 simulations. For G 1 and G 3 , the ERR is the empirical type I error; For G 2 and G 4 , the ERR is the empirical power.

From the numerical results for all scenarios displayed in table 1, we have the following findings. Firstly, when there are outliers or heavy errors, our methods still have very high powers and control empirical sizes well. Secondly, for both linear model and nonlinear model, small groups and large groups, our procedures have similar satisfactory performance in this experiment. Thirdly, we find that the dimensionality of all predictors p does not effect the empirical performances. Our test procedures with LASSO penalty perform similarly to those with SCAD or MCP. Therefore, we only present the results of test statistics with LASSO penalty. We use the R-package ncvreg [6].

Moreover, we compare the proposed methods with the three-step testing procedure based on the studentized statistics in [59], denoted as ST. Here, we use R package SILM [59] to implement the three-step testing procedure, set the number of bootstrap replications as 500 and choose the splitting proportion of 30% for screening.

Table 2 summarizes the results of empirical type I errors and powers with α = 0 . 05 for different methods and models based on 500 simulation runs (due to the computation limit, we only run 200 times for ST). Here we consider ( n, p ) = (200 , 400) , and other settings are same as before. It is obvious that our methods outperform the ST. Firstly, when there are no outliers and heavy errors, all methods have empirical power 1 for linear model. But for non-linear model, our test procedures T (1) α and T (2) α are more powerful than ST. For instance, under non-linear model with original data and ϵ ∼ N (0 , 1) , the empirical powers of ST for G 2 and G 4 are 0.785 and 0.430, while the corresponding

empirical powers of our methods are all 1. Secondly, the outliers and heavy errors have dramatic effect on the power performances of ST, which has very low powers even as low as the nominal level. For example, under linear model with outliers, the empirical power of ST for G 4 is 0.050 and that of our proposal is 1; under non-liner model with Cauchy error, the corresponding empirical powers of ST and our proposal are 0.040 and 1, respectively. Thirdly, all methods can control type I errors reasonably. Overall our methods have better powers and are robust with respect to outliers and heavy errors, whereas ST in [59] can completely fail when outliers exist.

5 Discussions

In this paper, we investigate robust inference for high-dimensional single index model (SIM). Under the linearity condition about the predictors, we recast the SIM into pseudo-linear model with transformed response. A response-distribution transformation is considered. This transformation choice avoids the sub-Gaussian assumption for the response. Our introduced procedures are thus robust with respect to outliers or heavy-tailed errors. We develop asymptotically honest group inference procedures. Asymptotic distribution and bootstrap implementation are both established. For testing the individual coefficients simultaneously, multiple testing procedures are proposed and shown to control the false discovery rate asymptotically. Our numerical studies illustrate the robustness of our procedures against outliers or heavy errors.

̸

In this paper, the linearity condition about the predictors plays a very important role in the methodology development. How to make inference for high-dimensional SIM without this condition is of great importance and interest. The second-order Stein's method with score function-based corrections investigated in [56] and [55] would be a powerful alternative. The condition κ h = 0 excludes even link functions, and in particular the problem of sparse phase retrieval. [39] introduced a novel procedure to deal with the problem of sparse phase retrieval when κ h = 0 . It would be of interest to extend their approach to make robust group inference. Further our paper doesn't provide quantitative bounds on

robustness. It is also of interest to make inference for partially linear single-index regression model [15]. We will pursue these challenging problems in near future.

References

Pierre Alquier and Gérard Biau. Sparse single-index model. Journal of Machine Learning Research , 14(1):243-280, 2013.

Alexandre Belloni, Victor Chernozhukov, Denis Chetverikov, and Ying Wei. Uniformly valid post-regularization confidence regions for many functional parameters in z-estimation framework. Annals of Statistics , 46(6B):3643 - 3675, 2018.

Alexandre Belloni, Victor Chernozhukov, and Kengo Kato. Uniform post-selection inference for least absolute deviation regression and other z-estimation problems. Biometrika , 102(1):77-94, 2015.

Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological) , 57(1):289-300, 1995.

Jelena Bradic, Gerda Claeskens, and Thomas Gueuning. Fixed effects testing in highdimensional linear mixed models. Journal of the American Statistical Association , 115(532):1835-1850, 2020.

Patrick Breheny and Jian Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Annals of Applied Statistics , 5(1):232-253, 2011.

Peter Bühlmann, Markus Kalisch, and Lukas Meier. High-dimensional statistics with a view toward applications in biology. Annual Review of Statistics and Its Application , 1(1):255-278, 2014.

TCai, Weidong Liu, and Yin Xia. Two-sample test of high dimensional means under dependence. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76(2):349-372, 2014.

T Tony Cai, Zijian Guo, and Rong Ma. Statistical inference for high-dimensional generalized linear models with binary outcomes. Journal of the American Statistical Association , pages 1-14, 2021.

Tianxi Cai, T Tony Cai, Katherine Liao, and Weidong Liu. Large-scale simultaneous testing of cross-covariance matrices with applications to phewas. Statistica Sinica , 29:983-1005, 2019.

Mehmet Caner and Anders Bredahl Kock. Asymptotically honest confidence regions for high dimensional parameters by the desparsified conservative lasso. Journal of Econometrics , 203(1):143-168, 2018.

Xiaohui Chen. Gaussian and bootstrap approximations for high-dimensional U-statistics and their applications. Annals of Statistics , 46(2):642 - 678, 2018.

Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. Annals of Statistics , 41(6):2786 - 2819, 2013.

R Dennis Cook and Liqiang Ni. Sufficient dimension reduction via inverse regression: A minimum discrepancy approach. Journal of the American Statistical Association , 100(470):410428, 2005.

Shijie Cui, Xu Guo, and Zhe Zhang. Estimation and inference in ultrahigh dimensional partially linear single-index models. arXiv preprint arXiv:2404.04471 , 2024.

Ruben Dezeure, Peter Bühlmann, and Cun-Hui Zhang. High-dimensional simultaneous inference with the bootstrap. Test , 26(4):685-719, 2017.

Hamid Eftekhari, Moulinath Banerjee, and Yaacov Ritov. Inference in high-dimensional singleindex models under symmetric designs. Journal of Machine Learning Research , 22(27):1-63, 2021.

Jianqing Fan, Quefeng Li, and Yuyan Wang. Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 79(1):247-265, 2017.

Jianqing Fan, Runze Li, Cun-Hui Zhang, and Hui Zou. Statistical foundations of data science . Chapman and Hall/CRC, 2020.

Ethan X Fang, Yang Ning, Runze Li, et al. Test of significance for high-dimensional longitudinal data. Annals of Statistics , 48(5):2622-2645, 2020.

Ethan X Fang, Yang Ning, and Han Liu. Testing and confidence intervals for high dimensional proportional hazards models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 79(5):1415-1437, 2017.

Zhe Fei, Ji Zhu, Moulinath Banerjee, and Yi Li. Drawing inferences for high-dimensional linear models: A selection-assisted partial regression and smoothing approach. Biometrics , 75(2):551-561, 2019.

Zijian Guo, Claude Renaux, Peter Bühlmann, and Tony Cai. Group inference in high dimensions with applications to hierarchical testing. Electronic Journal of Statistics , 15(2):6633-6676, 2021.

Peter Hall and Ker-Chau Li. On almost linearity of low dimensional projections from high dimensional data. Annals of Statistics , 21(2):867-889, 1993.

Dongxiao Han, Miao Han, Jian Huang, and Yuanyuan Lin. Robust inference for highdimensional single index models. Scandinavian Journal of Statistics , 50(4):1590-1615, 2023.

Wolfgang Hardle, Peter Hall, and Hidehiko Ichimura. Optimal smoothing in single-index models. Annals of Statistics , 21(1):157-178, 1993.

Oliver Hines, Oliver Dukes, Karla Diaz-Ordaz, and Stijn Vansteelandt. Demystifying statistical learning based on efficient influence functions. The American Statistician , pages 1-13, 2022.

Adel Javanmard and Hamid Javadi. False discovery rate control via debiased lasso. Electronic Journal of Statistics , 13(1):1212-1253, 2019.

Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for highdimensional regression. Journal of Machine Learning Research , 15(1):2869-2909, 2014.

Efang Kong and Yingcun Xia. Variable selection for the single-index model. Biometrika , 94(1):217-229, 2007.

Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association , 86(414):316-327, 1991.

Ker-Chau Li and Naihua Duan. Regression analysis under link violation. Annals of Statistics , 17(3):1009-1052, 1989.

Runze Li, Wei Zhong, and Liping Zhu. Feature screening via distance correlation learning. Journal of the American Statistical Association , 107(499):1129-1139, 2012.

Weidong Liu. Gaussian graphical model estimation with false discovery rate control. Annals of Statistics , 41(6):2948-2978, 2013.

Po-Ling Loh and Martin J Wainwright. Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima. Journal of Machine Learning Research , 16(1):559-616, 2015.

Rong Ma, T Tony Cai, and Hongzhe Li. Global and simultaneous hypothesis testing for high-dimensional logistic regression models. Journal of the American Statistical Association , 116(534):984-998, 2021.

Nicolai Meinshausen, Lukas Meier, and Peter Bühlmann. P-values for high-dimensional regression. Journal of the American Statistical Association , 104(488):1671-1681, 2009.

Sahand N Negahban, Pradeep Ravikumar, Martin J Wainwright, and Bin Yu. A unified framework for high-dimensional analysis of m -estimators with decomposable regularizers. Statistical Science , 27(4):538-557, 2012.

Matey Neykov, Zhaoran Wang, and Han Liu. Agnostic estimation for misspecified phase retrieval models. Journal of Machine Learning Research , 21:1-39, 2020.

Yang Ning and Han Liu. A general theory of hypothesis tests and confidence regions for sparse high dimensional models. Annals of Statistics , 45(1):158-195, 2017.

Yaniv Plan and Roman Vershynin. The generalized lasso with non-linear observations. IEEE Transactions on Information Theory , 62(3):1528-1537, 2016.

Peter Radchenko. High dimensional single index models. Journal of Multivariate Analysis , 139:266-282, 2015.

Wojciech Rejchel and Małgorzata Bogdan. Rank-based lasso-efficient methods for highdimensional robust model selection. Journal of Machine Learning Research , 21(244):1-47, 2020.

Chengchun Shi, Rui Song, Zhao Chen, and Runze Li. Linear hypothesis testing for high dimensional generalized linear models. Annals of Statistics , 47(5):2671-2703, 2019.

Chengchun Shi, Rui Song, Wenbin Lu, and Runze Li. Statistical inference for high-dimensional models via recursive online-score estimation. Journal of the American Statistical Association , 116(535):1307-1318, 2021.

Qiang Sun and Heping Zhang. Targeted inference involving high-dimensional data using nuisance penalized regression. Journal of the American Statistical Association , 116(535):14721486, 2021.

Sara Van de Geer, Peter Bühlmann, Yaacov Ritov, and Ruben Dezeure. On asymptotically optimal confidence regions and tests for high-dimensional models. Annals of Statistics , 42(3):11661202, 2014.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Jingshen Wang, Xuming He, and Gongjun Xu. Debiased inference on treatment effect in a high-dimensional model. Journal of the American Statistical Association , 115(529):442-454, 2020.

Tao Wang, Pei-Rong Xu, and Li-Xing Zhu. Non-convex penalized estimation in highdimensional models with single-index structure. Journal of Multivariate Analysis , 109:221-235, 2012.

Tao Wang and LiXing Zhu. A distribution-based lasso for a general single-index model. Science China Mathematics , 58(1):109-130, 2015.

Yunan Wu, Lan Wang, and Haoda Fu. Model-assisted uniformly honest inference for optimal treatment regimes in high dimension. Journal of the American Statistical Association , pages 1-10, 2021.

Yin Xia, Tianxi Cai, and T Tony Cai. Two-sample tests for high-dimensional linear regression with an application to detecting interactions. Statistica Sinica , 28:63-92, 2018.

Zhuoran Yang, Krishna Balasubramanian, Zhaoran Wang, and Han Liu. Learning non-gaussian multi-index model via second-order stein's method. Advances in Neural Information Processing Systems , 30:6097-6106, 2017.

Zhuoran Yang, Krishnakumar Balasubramanian, and Han Liu. High-dimensional non-gaussian single index models via thresholded score function estimation. In International conference on machine learning , pages 3851-3860. PMLR, 2017.

Peng Zeng, Tianhong He, and Yu Zhu. A lasso-type approach for estimation and variable selection in single index models. Journal of Computational and Graphical Statistics , 21(1):92109, 2012.

Cun-Hui Zhang and Stephanie S Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76(1):217-242, 2014.

Xianyang Zhang and Guang Cheng. Simultaneous inference for high-dimensional linear models. Journal of the American Statistical Association , 112(518):757-768, 2017.

Li-Ping Zhu and Li-Xing Zhu. Nonconcave penalized inverse regression in single-index models with high dimensional predictors. Journal of Multivariate Analysis , 100(5):862-875, 2009.

A Appendix / supplemental material

Notation. For functions f ( n ) and g ( n ) , we write f ( n ) ≲ g ( n ) to mean that f ( n ) ≤ cg ( n ) for some universal constant c ∈ (0 , ∞ ) , and similarly, f ( n ) ≳ g ( n ) when f ( n ) ≥ c ′ g ( n ) for some universal constant c ′ ∈ (0 , ∞ ) . We write f ( n ) ≍ g ( n ) when f ( n ) ≲ g ( n ) and f ( n ) ≳ g ( n ) hold simultaneously. For a function h : R p → R , we write ∇ h and ∇ k h to denote a gradient or subgradient and its k -th component, if they exist. The sub -Gaussian norm of a sub -Gaussian random variable X is defined as ∥ X ∥ ψ 2 :=inf { t > 0 : E exp( X 2 /t 2 ) ≤ 2 } . Similarly, the sub -Exponential norm of a sub -Exponential random variable Y is defined as ∥ Y ∥ ψ 1 :=inf { t > 0 : E exp( | Y | /t ) ≤ 2 } .

A.1 A multiplier bootstrap test procedure

As discussed in Remark 2.1, the critical value obtained from the asymptotic distribution may not work well in practice since this weak convergence is typically slow and assumptions for dependence structure of the covariates. We next describe a simple gaussian multiplier bootstrap method to obtain an accurate critical value. Let w 1 , . . . , w n be i.i.d. N (0 , 1) random variables that are independent of { X i , Y i } n i =1 and define the gaussian multiplier bootstrap statistic:

̸

where and

is the kernel funtion in T nj . The bootstrap critical value is given by c (2) G ( α ) = inf { t ∈ R : Pr w ( M ♯ n, G ≤ t ) ≥ 1 -α } , where Pr w is the probability measure induced by the multiplier variables { w i } n i =1 holding { X i , Y i } n i =1 fixed. We can reject null hypothesis H 0 , G at the significant level α if and only if M n, G ≥ c (2) G ( α ) ,

Theorem A.1. Suppose that Assumptions A.1, A.2, A.7, A.8 and LC condition hold. If s = o ( √ n/ (log p (log np 0 ) 3 / 2 ) ) , we have

Remark A.1. Here we assume s = o ( √ n/ (log p (log np 0 ) 3 / 2 ) ) which is slightly stricter than s = o ( √ n/ (log p log p 0 )) in Theorem 3.2. Suppose ζ 1 is a sequence satisfying ζ 1 √ 1 ∨ log( p 0 /ζ 1 ) → 0 as n → ∞ . This assumption has two purposes. Firstly, making the difference between ∥ ˜ T n, G ∥ ∞ and ∥ ˜ S n, G ∥ ∞ bounded by ζ 1 . Secondly, making Pr w (∣ ∣ √ M ♯ n, G -√ L ♯ n, G ∣ ∣ ≥ ζ 1 ) → 0 in probability, where L ♯ n, G is the ideal version of M ♯ n, G and it is defined in (1.27) in the Appendix A.9.

Theorem A.2. Suppose that conditions in Theorem A.1 are satisfied, we have for some ϵ 0 >

0

,

Where Ω 1 G ( · ) is defined in subsection 3.3.

The theoretical guarantee of the bootstrap procedure is based on the gaussian approximation theory for nondegenerate U -statistic. Symmetrization of the kernel function ˆ h j ( · ) plays an important role in theory. Thus ˆ h sym j ( · ) is necessary in our gaussian multiplier bootstrap procedure. Meanwhile, Theorem A.1 holds whether p 0 is divergent or not. While the limiting null distribution established in Theorem 3.2 asks p 0 be divergent.

A.2 The detailed form of c 3

By the definition of SNR and Lemma A.8 in the Supplementary Material, for a given β ∈ R p we have

Similarly, we have

Thus the ratio of SNR( M n, G ) and SNR( M + n, G ) can be simplified as

where j ′ = arg j ∈G max ( β j δ j σ + j ) 2 . In the case of classical SIM, that is, Y = g 1 ( X ⊤ β ) + ϵ and ϵ is independent of X . We can derive

Thus we have

and

A.3 False discovery rate control

In the previous section, we consider joint significance testing of a prespecified subset of predictors. In applications, however, the parameter of interest may not be specified in advance. Denote H 0 = { j : β j = 0 , j = 1 , . . . , p } , and H 1 = { j : β j = 0 , j = 1 , . . . , p } . It is of interest to identify the elements in H 1 . To this aim, we consider simultaneous testing of the following hypotheses

̸

̸

In this section, we aim to develop data-driven procedures to control the FDR.

Recall that in Section 2, for each individual hypothesis H 0 j : β j = 0 we define the standardized statistic ˜ T nj = ˆ σ -1 j T nj defined in (2.7). At a given threshold level t > 0 , H 0 j is rejected if | ˜ T nj | ≥ t . For each t , let R 0 ( t ) = ∑ j ∈H 0 I ( | ˜ T nj | ≥ t ) , and R ( t ) = ∑ p j =1 I ( | ˜ T nj | ≥ t ) be the total number of false discoveries and the total number of discoveries, respectively. Accordingly, the false discovery proportion (FDP) and FDR are

Let G 0 ( t ) be the proportion of the nulls falsely rejected by the procedure among all the true nulls at the threshold level t , namely, G 0 ( t ) = q -1 0 ∑ j ∈H 0 I ( | ˜ T nj | ≥ t ) , where q 0 = |H 0 | . In practice, it is reasonable to assume that the true alternatives are sparse, that is, p -q 0 =: q 1 = o ( p ) . If the sample size is large, we can use the tails of normal distribution G ( t ) = 2 -2Φ( t ) to approximate G 0 ( t ) . In fact, it will be shown that, for b p = √ 2 log p -log log p ,

in probability as ( n, p ) → ∞ . To summarize, we have the following multiple testing procedure controlling the FDR and FDP at a pre-specified level 0 < α < 1 , which is summarized in Algorithm A.1.

Algorithm 2: multiple testing procedure

1 Let 0 < α < 1 , b p = √ 2 log p -log log p and define

If ˆ t in (1.18) does not exist, then let ˆ t = √ 2 log p . We reject H 0 j whenever | ˜ T j | ≥ ˆ t .

Next we show that under mild conditions, our proposed multiple testing procedure control the FDR asymptotically. Recall that Ω = { β h ∈ R p : ∥ β h ∥ 0 ∨ max 1 ≤ j ≤ p ∥ θ j ∥ 0 ≤ s } . The following theorem shows that ˜ T nj is uniformly asymptotically normal distributed and G 0 ( t ) is well approximated by G ( t ) .

Theorem A.3. Suppose that Assumptions A.1, A.2, A.8 in Appendix A.8 and LC condition hold. If p = O ( n c ) for some constant c > 0 and s = o ( √ n/ (log p ) 2 ) . Then as ( n, p ) →∞ , for any β h ∈ Ω we have

If in addition Assumption A.9 holds, then

in probability.

In the results given in Section 2, we could allow p to grow exponentially fast in n . However, in order to control FDR, we only allow p = O ( n c ) to apply the moderate derivation result in [34]. Further we assume s = o ( √ n/ (log p ) 2 ) which is slightly stricter than s = o ( √ n/ (log p log p 0 )) to make the difference between ˜ T nj and its ideal version ˜ S nj uniformly bounded with the rate o (1 / √ log p ) . Note that [28] imposed a similar condition. The following theorem provides the asymptotic FDR control of our procedure.

Theorem A.4. Under the assumptions in Theorem A.3, for ˆ t defined in our multiple testing procedure, we have

A.4 Discussions of robustness

In this subsection, we carefully discuss the robustness of our inference procedure based on the efficient influence function. We find that our statistics are robust with respect to perturbations in the responses. The details are given below. Recall that our test procedure is inspired by the quantity

Next we derive the efficient influence function (EIF) of I . We rewrite I as

where P is distribution of ( X j , Z ⊤ j , Y ) ⊤ . Consider the following parametric submodel indexed by t , i.e.

where t ∈ [0 , 1] , and ˜ P is a point mass at a single observation ˜ o = (˜ x j , ˜ z ⊤ j , ˜ y ) ⊤ . As mentioned in [27], the EIF for I at observation ˜ o is

where Ψ( P t ) = E P t [ { F P t ( Y ) -1 / 2 -Z ⊤ j γ j } ( X j -Z ⊤ j θ j )] . Denote o = ( x j , z ⊤ j , y ) ⊤ and O = ( X j , Z ⊤ j , Y ) ⊤ . Let m t ( o ) = { F P t ( y ) -1 / 2 -z ⊤ j γ j } ( x j -z ⊤ j θ j ) and Ψ( P t ) = E P t [ m t ( O )] . Further, the operator, ∂ t , applied to an arbitrary function g ( t ) , is defined as

Some calculation entails that

Note that

Thus we derive that and

As m 0 (˜ o ) = { F P (˜ y ) -1 / 2 -˜ z ⊤ j γ j } (˜ x j -˜ z ⊤ j θ j ) , by some calculations we derive

Recall that

where m j ( y ) = E [( X j -Z ⊤ j θ j ) { I ( Y ≥ y ) -F ( Y ) } ] . Here T nj is an individual test statistic corresponding to the parameter I . While S nj is an individual quantity corresponding to the EIF of I . From our proof, we know that T nj = S nj + o p (1) .

By the formula of ϕ (˜ o, P ) , given ( x j , z ⊤ j ) ⊤ , it can shown that ϕ (˜ o, P ) is bounded for any ˜ y ∈ R . Thus our test statistics are robust with respect to the perturbations in the responses.

A.5 Additional simulation results for group inference

In this subsection, we conduct simulation studies to compare our procedure with other methods based on different transformation functions. The simulation settings are summarized as follows. We consider the following two models:

Model 1: Linear model: Y = X ⊤ β + ϵ .

Model 2: Non-linear model: Y = exp( X ⊤ β + ϵ ) .

The regression coefficients β = ( β 1 , β 2 , . . . , β p ) ⊤ are generated as β j = δ for j = 1 , . . . , 6 and β j = 0 otherwise, where δ can be regarded as a signal strength parameter. We generate the error term from the standard normal distribution. We add outliers to pollute the observations: p out of the responses are picked at random and increased by m out -times maximum of original responses. Specifically, the detailed settings for the above parameters are as follows. Firstly, we consider the sample size n = 500 and the dimension p = 800 . Secondly, we set the signal strength parameter δ to vary from { 0 . 1 , 0 . 3 , 0 . 5 } . Thirdly, we fix m out to 10. Lastly, we vary p out from 0 to 0 . 5 in increments of 0 . 1 . For more complete comparisons, we consider three transformation procedures: (1) h ( Y ) = F ( Y ) (Our method); (2) h ( Y ) = Y ; (3) h ( Y ) = sigmoid ( Y ) = 1 / { 1 + exp( -Y ) } . Other simulation settings are the same as described in section 4 of the main text.

Figure 1 summarizes the results of empirical type I error and empirical power with the significant level of α = 0 . 05 for different methods when the error term follows the standard normal distribution. Firstly, under the null hypothesis ( G 1 ), h ( Y ) = sigmoid ( Y ) cannot control the type I error for Model 2, while other procedures perform well. Secondly, under the alternative hypothesis ( G 2 ), the empirical powers of other procedures decrease rapidly with the increase of p out , while the powers of our procedure remain stable, which is particularly noticeable when δ = 0 . 5 . This finding indicates that our method has strong robustness when the responses are polluted. Thirdly, our method performs well for both Model 1 and Model 2, indicating that our method is robust across different single-index models.

Model

1

G 1

Model

1

G 2

Model

2

G 1

Model

2

G 2

d

=

0.1

d

=

0.3

d

=

0.5

0

0.1

0.2

0.3

0.4

0.5

0

0.1

0.2

0.3

0.4

0.5

0

0.1

0.2

0.3

0.4

0.5

0

0.1

0.2

0.3

0.4

0.5

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

pout

Empirical Rejection Rate

Transformation F(Y) (Our) Y sigmoid(Y)

A.6 Additional simulation results for false discovery rate control

In this subsection, we examine the empirical FDR results of our proposed procedure. We also make comparisons with two alternatives. Let BHV n denote the BH procedure [4] with p -values obtained from V n . BHW n is similarly defined. Here V n and W n denote the debiased lasso procedure introduced by [17] and the decorrelated score method proposed by [40], respectively. Here, we use the Rfunction p.adjust(...,method="BH") to implement the BH procedure. Wesimultaneously test H 0 j : j ∈ H 0 for all j = 1 , · · · , p . We set the dimension p = 1000 and sample size n = 1000 , 1500 . The sparsity level |H 1 | = q 1 is set as 4 or 8. The data is generated from the following linear and nonlinear models:

where β = β 1 with δ 0 = 1 , that is β = ( s 0 ︷ ︸︸ ︷ 1 , · · · , 1 , 0 , · · · , 0) ⊤ and ϵ comes from N (0 , 1) or t (1) .

We evaluate the empirical FDRs and the empirical powers of our test statistic T n and two existing high-dimensional inference procedures BHV n and BHW n . Here the power is defined as the number of correctly discovered variables divided by the number of truly active variables. We set the desired FDR level as α = 0 . 2 and the results are calculated based on 500 replications.

We illustrate our proposed procedure under different models, error distributions, sample sizes, and sparsity levels. Table 3 shows the simulation results. We have the following important observations. Firstly our introduced procedure controls the FDR well, even when the random error comes from Cauchy distribution. Secondly, our method has power 1, indicating that the proposed method can detect all important predictors. Thirdly, the other two methods lose their powers when random error follows Cauchy distribution and sometimes fail to control FDR.

A.7 Real data analysis

In this subsection, we illustrate the practical applications of our procedures by a real data analysis. The group inference is helpful to decide whether a group of predictors are important or not for the response. If we find a group of predictors are important, we would like to know which specific predictors in the group are significant. For this aim, our developed multiple testing procedure is useful. For instance, researchers may aim to test whether a gene pathway, consisting of high-dimensional

genes for the same biological function, is important for a certain clinical outcome, given the other high-dimensional genes. When determining that a certain gene pathway is important, researchers need to further identify specific genes within the pathway which are important for a certain clinical outcome.

Weapply our methods on a dataset about riboflavin (vitamin B2) production rate with Bacillus Subtilis. This dataset is made publicly by [7] and has been analyzed by many authors, for instance [37], [47], [29], and [22]. The dataset riboflavin can be obtained from the R package hdi . It consists of n = 71 observations of strains of Bacillus Subtilis and p = 4088 covariates, measuring the log-expression levels of 4088 genes. The response variable is the logarithm of the riboflavin production rate.

Our goal is to detect which genes are associated with riboflavin production rate. Like most existing studies, we first reduce ultrahigh-dimension to a moderate high-dimension. Here we pick out first 300 genes by distance correlation based screening [33]. We first conduct global testing on these 300 genes. The p -value of our group inference procedure is 1 . 286 e -04 , indicating that the null hypothesis is rejected and the selected 300 genes are influential for riboflavin production rate. Next, we further use FDR control procedure to select the important genes in these 300 genes.

By implementing our proposed FDR control procedure with the FDR level of 0.1, we identify 10 genes that are significantly associated with the response. That is G I = {YTGB_at, YCKE_at, YXLE_at, YXLD_at, YJCJ_at, XHLA_at, xepA_at, YCGO_at, RPLP_at, XKDS_at}. If the FDR level is set as 0.2, 5 more genes will be selected. That is G II = G I ∪ {SPOIISA_at, YHCB_at, XKDI_at, YJCF_at, XHLB_at}.

We further conduct group inference on the selected subsets G I , G II and their complement sets G c I , G c II . As expected, our group inference procedure finds again that G I , G II are significant while G c I , G c II are not. The corresponding p -values of G I , G c I , G II and G c II are 6.748e-06, 7.257e-01, 9.334e-06 and 9.370e-01, respectively. These results suggest that the genes selected by the FDR control procedure are really influential.

We compare these selected genes with other methods. For example, the multi-sample-splitting method proposed in [37] identified YXLD_at; [47] did not select any gene using the de-sparsified Lasso; [29] only selected two genes: YXLD_at and YXLE_at and [22] claimed YCKE_at, XHLA_at, YXLD_at, YDAR_at and YCGN_at as significant. From G I and G II , we can see clearly that the gene YXLD_at is detected by not only [37], [29], [22], but also by our procedure. Besides, the genes YCKE_at, YXLE_at and XHLA_at which are detected by [29] and [22], are also found by our method. Further, our procedure detects some additional important genes.

A.8 Technical assumptions

Assumption A.1. We assume that

(i) X j , X ⊤ β h and Z ⊤ j θ j , j = 1 , . . . , p are all sub-Gaussian with uniformly bounded subGaussian norms.

(iii) Σ = Cov( X ) > 0 .

The above conditions are mild and widely assumed in the high-dimensional literature. Condition (i) is used to obtain l ∞ norm of 1 √ n ∑ n i =1 ( X ij -Z ⊤ ij θ j ) Z ij in the proof. Sub-Gaussian assumption is a standard condition while working with random predictors in high-dimensional models [49]. However, we should emphasize here that we do not make sub-Gaussian assumption for the error term (or equivalently for the response), which is widely assumed in the literature of high-dimensional inference [40, 17]. Actually we even do not assume the existence of E ( ϵ ) .

Assumption A.2. We assume that

(i) The function p λ satisfies p λ (0) = 0 and is symmetric around zero (i.e., p λ ( t ) = p λ ( -t ) , for all t ∈ R ).

(ii) On the nonnegative real line, the function p λ is nondecreasing.

(iii) For t > 0 , the function t → p λ ( t ) t is nonincreasing in t .

̸

(iv) The function p λ is differentiable for all t = 0 and subdifferentiable at t = 0 , with lim t → 0 + p ′ λ ( t ) = λL for some L > 0 .

(v) There exists µ > 0 such that p µ λ ( t ) := p λ ( t ) + µ 2 t 2 is convex.

Many commonly used penalty functions p λ satisfy all the conditions in Assumption A.2, such as LASSO, SCAD and MCP penalties. Proofs can be found in [35]. By the result in [35], we have the following proposition for ˆ θ j .

Assumption A.3. The RSC condition in [35] is

where the α 1 , α 2 are strictly positive constants, the τ 1 , τ 2 are nonnegative constants and the ∆ ∈ R p is a p -dimension column vector.

Here we denote

and ˜ S nj = σ -1 j S nj , where m j ( y ) = E [( X j -Z ⊤ j θ j ) { I ( Y ≥ y ) -F ( Y ) } ] . It could be shown that ˜ S nj = ˜ T nj + o p (1) and Var( ˜ S nj ) = 1 . Denote ˜ S n = ( ˜ S nj ) p j =1 and Λ = Var( ˜ S n ) , where Λ jk = Cov( ˜ S nj , ˜ S nk ) is ( j, k ) component of Λ . And denote ˜ S n, G = ( ˜ S nj ) j ∈G and Λ G = Var( ˜ S n, G ) correspondingly. Further denote Ω = { β h ∈ R p : ∥ β h ∥ 0 ∨ max 1 ≤ j ≤ p ∥ θ j ∥ 0 ≤ s } and β h G = ( β hj ) j ∈G .

Assumption A.4. { log( np 0 ) } 7 /n = o (1) .

Assumption A.5. C -1 0 ≤ λ min ( Λ ) ≤ λ max ( Λ ) ≤ C 0 for some constant C 0 > 0 .

̸

Assumption A.6. max 1 ≤ j = k ≤ p | Λ jk | ≤ c 0 < 1 for some constant c 0 > 0 .

Assumptions A.4-A.6 are mild and commonly used in the high-dimensional settings. Assumption A.4 is the technical condition to bound the difference between ∥ ˜ S n, G ∥ ∞ and ∥ N G ∥ ∞ , where N G ∼ N p 0 ( 0 , Λ G ) . Specially, Assumption A.4 imposes suitable restrictions on the growth rate of p 0 and it is commonly used in the group inference literature, such as [59] and [16]. Assumption A.5 and A.6 are used to establish the limiting distribution of ∥ N G ∥ ∞ . Assumptions A.5 requires that the eigenvalues of Λ G are bounded. Assumption A.6 is widely used to establish the limiting distribution of the max type statistic in the high-dimensional settings, see [8], [54], and [36] for further examples.

The validity of the above bootstrap method requires the following assumptions.

Assumption A.7. { log( np 0 ) } 8 /n = o (1) .

Assumption A.8. min 1 ≤ j ≤ p E ( ˜ S 2 nj ) ≥ C min , where C min is a positive constant.

Assumption A.7 and A.8 are the technical assumptions needed for the gaussian multiplier bootstrap method. Assumption A.7 is imposed for controlling the estimation error of gaussian multiplier bootstrap statistic M ♯ n, G . Theorem A.1 below establishes the honest property of the bootstrap procedure for the statistic M n, G .

̸

Assumption A.9. Suppose that for some b > 0 and q > 0 ,

Remark A.2. Recall Λ jk is the covariance between ˜ S nj and ˜ S nk . Thus, A ( b ) contains all the strongly correlated pairs and Assumption A.9 requires that the number of these pairs cannot be too large. If | Λ jk | ≤ a 0 for some constant 0 < a 0 ≤ 1 , then this assumption holds under |A ( b ) | = O ( p 2 1+ a 0 -q / (log p ) 2 ) for some b > 0 and q > 0 . Similar assumptions were considered in [10], [20] and [36].

A.9 Proofs of theorems

The proof of Theorem 3.1: We denote

where L n ( β h ) is defined in equation (2.6). By triangle inequality of l 1 and l 2 norms, it suffices to show that the l 1 and l 2 norms of ˆ β h -˜ β h and ˜ β h -β h satisfy the requirements in this theorem respectively.

Firstly we calculate l 1 and l 2 norms of difference between ˆ β h and ˜ β h . The results are presented in Lemma A.2.

Secondly, by Lemma A.8 in Supplementary Material and Theorem 1 in [43], we can show that ˜ β h = ( n -1) β h /n , and thus

By the sparse assumption, ∥ β h ∥ 2 /n = o ( √ s Y log p/n ) and ∥ β h ∥ 1 /n = o ( s Y √ log p/n ) . Thus Theorem 3.1 is proved.

The proof of Theorem 3.2: By Lemma A.3, the distribution of max j ∈G | ˜ T nj | can be approximated by ∥ N G ∥ ∞ with N G ∼ N p 0 ( 0 , Λ G ) . By Lemma 6 of [8], we obtain

Note that all the universal constants do not depend on n, p, β h and θ j , j = 1 , . . . , p . Thus this theorem is proved.

The proof of Theorem A.1: For the convenience of the proof, we introduce some notations. Denote U nj as

̸

where and

Denote ˜ U nj =: σ -1 j U nj . U nj plays an important role in the proof of this theorem.

Let w 1 , . . . , w n be i.i.d. N (0 , 1) random variables that are independent of { X i , Y i } n i =1 and denote L ♯ n, G as :

̸

where U nj and h sym j ( · ) are defined in (1.24) and (1.25). The proof follows from the general results for the multiplier bootstrap of U -statistic in [12]. Here, we extend these results for approximate high-dimensional U -statistic by results in [13] and rewrite it in a suitable form for our analysis. It is now presented in Lemma A.22 in the Supplementary Material. Suppose ζ 1 is a sequence satisfying ζ 1 √ 1 ∨ log( p 0 /ζ 1 ) → 0 as n →∞ . According to Lemma A.22, to prove this theorem, it suffices

ζ 1 = C (log( np 0 )) -1 / 2 , where C is a sufficient large constant. The proofs of these results are presented in Lemmas A.17, A.18 in the Supplementary Material.

With Lemmas A.17, A.18 and A.22 in the Supplementary Material, we obtain

when H 0 , G holds. Note that all the universal constants in the proof do not depend on n, p, β h and θ j , j = 1 , . . . , p . We thus have

The proof of Theorem 3.3: Using the inequality 2 a 1 a 2 ≤ ϵ -1 a 2 1 + ϵa 2 2 for any ϵ > 0 , we have

Denote η G = | max j ∈G ˆ σ -2 j ( T nj - √ nβ hj δ j ) 2 -max j ∈G σ -2 j ( S nj - √ nβ hj δ j ) 2 | , and η G = o p ((log p 0 ) -1 / 2 ) by Lemma A.16 in the Supplementary Material. We can derive

The last equality holds by Lemma A.19, A.20 in the Supplementary Material and Lemma 6 of [8]. Denote t = log log p 0 / 2 , it implies that

For any β h ∈ Ω 1 G (2 + ϵ 0 ) , applying Lemma A.14 in the Supplementary Material, we derive

holds with probability 1 as ( n, p 0 ) → 1 .

Substitute (1.29) and (1.30) into (1.28), we derive that

Now we prove the validity of c ( i ) G ( α ) , i = 1 , 2 . Note that c (1) G ( α ) ≤ 2 log p 0 by the definition of c (1) G ( α ) . By the properties of Gaussian multiplier bootstrap statistic(Lemma A.22), c (2) G ( α ) is equal to the (1 -α ) th quantile of max j ∈G ˆ σ -2 j ( T nj - √ nβ hj δ j ) 2 asymptotically. Thus lim ( n,p 0 ) →∞ inf β h ∈ Ω 1 G (2+ ϵ 0 ) Pr ( 2 log p 0 > c (2) G ( α ) ) = 1 by (1.29). By choosing ( ϵ 0 , ϵ ) satisfying 2+ ϵ 0 1+ ϵ -2 ϵ ≥ 2 ,

can be proved.

The proof of Theorem A.3: At first we calculate the uniform bound of the difference between constructed statistic ˜ T nj and its ideal version ˜ S nj . The result is presented in Lemma A.4. For (1.19), by Lemma 6.1 in [34], we have

for some constant 0 < γ 1 < 1 / 2 . So (1.19) follows from Lemma A.4, and the fact that G ( t + o ((log p ) -1 / 2 )) /G ( t ) = 1 + o (1) uniformly in 0 ≤ t ≤ √ 2 log p .

For (1.20), it suffices to show that

Let z 0 < z 1 < . . . < z d p ≤ 1 and t i = G -1 ( z i ) , where z 0 = G ( b p ) , z i = c p /p + c 2 / 3 p e i ν /p with c p = pG ( b p ) , and d p = { log ( p -c p ) c 2 / 3 p } 1 /ν and 0 < ν < 1 , which will be specified later. We have G ( t i ) /G ( t i +1 ) = 1 + o (1) uniformly in i , and t 0 / √ 2 log ( p/c p ) = 1 + o (1) . Note that uniformly for 1 ≤ j ≤ m,G ( t i ) /G ( t i -1 ) → 1 as p →∞ . The proof of (1.32) reduces to show that

In fact, for each ϵ > 0 , we have

Set I ( t ) = ∑ j ∈H 0 { I ( | ˜ S nj |≥ t ) -Pr( | ˜ S nj |≥ t ) } q 0 G ( t ) . By Markov's inequality Pr( | I ( t i ) | ≥ ϵ/ 2) ≤ E { I ( t i ) } 2 ϵ 2 / 4 , and it suffices to show ∑ d p j =0 E { I ( t i ) } 2 = o (1) . To see this, by (1.31),

̸

For ( j, k ) ∈ A ( b ) c with j, k ∈ H 0 , applying Lemma 6 . 1 in [34], we have I 12 ( t ) ≤ C (log p ) -1 -ξ for some ξ > 0 uniformly in 0 < t < √ 2 log p . By Lemma 6 . 2 in [34], for ( j, k ) ∈ A ( b ) with j, k ∈ H 0 , we have

So that

Note that for 0 ≤ t ≤ b p , we have G ( t ) ≥ G ( b p ) = c p /p , so that by assumption A.9 it follows that for some b, q > 0 ,

By the above inequalities, we can prove (1.33) by choosing 0 < ν < 1 so that

The proof of Theorem A.4: We first consider the case when ˆ t , given by (1.18), doesn't exist. In this case, ˆ t = √ 2 log p and we consider the event F 0 = { ∑ j ∈H 0 I ( | ˜ T nj | ≥ √ 2 log p ) ≥ 1 } , which mean at least one false positive. In order to show the FDR/FDP can be controlled in this case, we show that

And we have

For any ϵ > 0 , we can bound the first term by

By Lemma A.4, we know that Pr ( max j ∈H 0 | ˜ T nj -˜ S nj | ≥ ϵ ) → 0 . For simplify, we rewrite ˜ S nj = ∑ n i =1 ξ ij / √ n . Since E ( ξ ij ) = 0 , Var( ξ ij ) = 1 and { ξ ij } n i =1 is a i.i.d. sequence, by Lemma 6.1 of [34], we have sup 0 ≤ t ≤ 2 √ log p ∣ ∣ ∣ Pr( | ˜ S nj |≥ t ) G ( t ) -1 ∣ ∣ ∣ ≤ C (log p ) -1 . Now let t = √ 2 log p -ϵ , we have

for j ∈ H 0 uniformly. Hence p max j ∈H 0 Pr ( ˜ S nj ≥ √ 2 log p -ϵ ) ≤ ( C + 1) pG ( √ 2 log p -ϵ ) , which goes to zero as ( n, p ) → ∞ . By symmetry, we know that Pr( F 2 ) in (1.35) also goes to 0. Therefore (1.34) is proved.

Now consider the case when 0 ≤ ˆ t ≤ b p holds. We have

The proof is complete if A p → 0 in probability, which has been shown by Theorem A.3.

A.10 Some technical lemmas

For the convenience of the proof, we introduce some definitions. Denote U nj as

̸

where

And denote ˜ U nj =: σ -1 j U nj . U nj plays an important role in the proof of main results. Let s X be the sparsity level for parameter θ j as s X = max 1 ≤ j ≤ p ∥ θ j ∥ 0 .

Lemma A.1. Under Assumptions A.1 and A.2, the ˆ θ j defined in (2.5) satisfies

with probability at least 1 -c 1 exp( -c 2 log p ) . Here, ( c, c 1 , c 2 ) are universal constants.

Lemma A.2. Under assumptions in Theorem 3.1, ˆ β h defined in (3.11) satisfies

with probability at least 1 -c 1 exp( -c 2 log p ) . Here ˜ β h = arg min β h ∈ R p E L n ( β h ) with L n ( β h ) defined in equation (2.6) .

Proof. The main part of the proof is to verify RSC condition in assumption A.3 and get exponential inequality of ∥∇L n ( β h ) ∥ ∞ . Based on these results, we use Lemma A.5 to prove theorem A.2.

Thus, we focus on RSC condition at the beginning. Note that for ∆ ∈ R p ,

From Proportion 2 in [38], we then have

with probability at least 1 -c 1 exp {-c 2 n } for an appropriate choice of α 1 . By the arithmetric mean-geometric mean inequality,

and consequently,

which establishes (1.22) in assumption A.3. As square loss function is convex, condition (1.23) then follows via Lemma 8 in [35]. So we verify RSC condition completely.

Next, we construct exponential inequalities of ∇L n ( β h ) . By conditions of lemma A.5, it suffices to show that there exist universal constants ( c, c 1 , c 2 ) such that

By lemma A.7, we can choose proper ( c, c 1 , c 2 ) such that both RSC condition and inequality (1.38) are satisfied. As λ Y is proportional to √ log p n , the claimed l 1 and l 2 bounds then follow directly from Lemma A.5 by choosing R proportional to 1 λ Y .

Lemma A.3. Under the assumptions in Theorem 3.2,

where N G ∼ N p 0 ( 0 , Λ G ) .

for some constant C , with probability 1 as ( n, p ) → ∞ . And q ( n, p ) is a sequence satisfying q ( n, p ) = o ((log p 0 ) -1 / 2 ) . This implies that

By sub -Gaussian condition in Assumption A.1 and Assumption A.8, lim sup n →∞ sup t ∈ R I 1 ≤ 0 . By the Gaussian anti-concentration inequality in Lemma A.20, we derive that sup t ∈ R I 2 ≤ Cq ( n, p ) √ 1 ∨ log( p 0 /q ( n, p )) for some constant C . As q ( n, p ) = o ((log p 0 ) -1 / 2 ) , we derive that

Similarly,

This completes the proof.

Lemma A.4. Under the assumptions in Theorem A.3,

Proof. The proof of this Lemma is similar to Lemma A.16. We only need replace G with H 0 and replace p 0 with p .

Lemma A.5. (Theorem 1 in [35]) Suppose the regularizer ρ λ satisfies Assumption A.2, the empirical loss L n satisfies the RSC condition with 3 4 µ < α 1 , and β ∗ is feasible for the objective. Consider any choice of λ such that

and suppose n ≥ 16 R 2 max( τ 2 1 ,τ 2 2 ) α 2 2 log p . Then any vector ˜ β satisfying the first -order necessary condition (3.11) satisfies the error bounds

where k = ∥ β ∗ ∥ 0 .

Lemma A.6. Consider a U -statistics

̸

with a kernel h based on i.i.d. random variables Z 1 , . . . , Z n . Then for ∀ s ∈ R ,

where N = ⌊ n 2 ⌋ , which represents the greatest integer less than or equal to n 2 .

Proof. It can be seen in the proof of Lemma 14 in [43].

Lemma A.7. Under the Assumptions A.1 and A.2, there exist universal constants ( c, c 1 , c 2 ) such that

where ∇L n ( ˜ β h ) is the gradient of L n ( ˜ β h ) , which is a p -dimension column vector.

Proof. Suppose ∇ k L n ( ˜ β h ) is the k -th component of ∇L n ( ˜ β h ) . Note that

̸

where I ( · ) is an indicator function. So we decompose ∇ k L n ( ˜ β h ) into three parts:

where A k is a U -statistics with kernel h k ( y 1 , x 1 k ; y 2 , x 2 k ) = 1 2 [ x 1 k { I ( y 2 ≤ y 1 ) -F ( y 1 ) } + x 2 k { I ( y 1 ≤ y 2 ) -F ( y 2 ) }] . Then

with t = c 3 √ log p n .

Firstly, for probability (1.42). For ∀ s ∈ R

where C 1 is an absolute constant, K = max 1 ≤ k ≤ p ∥ X 1 k ∥ ψ 2 and N = ⌊ n 2 ⌋ . Note that (1) holds by Markov inequality, (2) holds by Lemma A.6. Denote s = Nt C 1 K 2 , we have

Secondly, for probability (1.43). As both { X ⊤ i ˜ β h - { F ( Y i ) -1 2 } } n i =1 and { X ik } n i =1 are sub -Gaussian random variables, { X ik [ X ⊤ i ˜ β h -{ F ( Y i ) -1 2 } ]} n i =1 is a zero mean sub -Exponential random variable sequence with sub -Exponential norm ∥ ∥ X ik [ X ⊤ i ˜ β h -{ F ( Y i ) -1 2 } ]∥ ∥ ψ 1 ≤ KK ′ , where K ′ = ∥ X ⊤ 1 ˜ β h -{ F ( Y 1 ) -1 2 }∥ ψ 2 . Then by inequality for r.v. with sub -exponential sum (Corollary 2.8.3 in [49]),

where C 2 is an absolute constant.

Thirdly, for probability (1.44). Using general Hoeffding's inequality (Theorem 2.6.3 in [49]) directly,

where C 3 is an absolute constant. As exp( -n 3 ) = o (exp( -n )) , probability (1.44) is trivial compared to other two terms. Combining inequalities (1.45), (1.46) and (1.47), we have

where C 4 = min { c 2 36 C 1 K 2 , C 2 c 2 9 K 2 K ′ 2 } . For a given constant c , we can choose arbitrary constants c 1 ≥ 4 , c 2 ≤ C 4 -1 and inequality (1.41) is satisfied.

Lemma A.8. Under the assumptions in theorem 3.1, then

Proof. The proof of this lemma is similar to the proof of theorem 2.1 in [32]. Using Jensen's inequality and LC condition, we can get this result. We omit the proof.

Similarly, we have

Lemma A.9. Suppose p λ X satisfies Assumption A.2. And under the Assumption A.1, we have

where ˆ θ j is defined in (2.5) and L is a constant only depend on p λ X .

Proof. By the property of ˆ θ j , we have

with probability 1. Under assumption A.2 and by Lemma 4(a) in [35], all subgradients and derivatives of p λ X are bounded in magnitude by Lλ X , that is,

And this lemma is proved.

Lemma A.10. Under the Assumptions A.1 and A.2, for β h ∈ { β h : ∥ β h ∥ ∞ ≲ √ log p/n } , there exist universal constants ( c, c 1 , c 2 ) such that

If ∥ β h ∥ ∞ = 0 , then I 2 , I 3 = 0 . Thus this Lemma is proved by Lemma A.7. So in the following proof, suppose ∥ β h ∥ ∞ > 0 , that is, there exists 1 ≤ j ≤ p , such that | β hj | > 0 . As ∥ β h ∥ ∞ ≲ √ log p/n , there exist a constant c 3 > 0 such that ∥ β h ∥ ∞ ≤ c 3 √ log p/n . Denote c ′ = c/ 3 , c ′ 1 = c 1 / 3 , we have

So it suffices to prove Pr( ∥ I k ∥ ∞ ≥ c ′ √ log p/n ) ≤ c ′ 1 exp( -c 2 log p ) , k = 1 , 2 , 3 .

For k = 1 , this inequality holds by lemma A.7 and choosing proper constants.

For k = 2 , by sub-Gaussian Assumption in Assumption A.1 and applying Bernstein inequality(Theorem 2.8.2 in [49]), we derive

where c 4 is a constant independent of β h and K = max 1 ≤ k ≤ p ∥ X j Z jk ∥ ψ 1 . Thus this inequality holds as n ≫ log p .

For k = 3 . As X j , j = 1 , . . . , p are all sub-Gaussian with uniform bounded sub-Gaussian norm, then it follows that ∥ I 3 ∥ ∞ ≤ Cc 3 √ log p/n . In summary, this Lemma is proved.

Lemma A.11. Let X 1 , . . . , X n be i.i.d mean zero random variables. If there exist constants L 1 and L 2 , such that Pr( | X i | ⩾ x ) ⩽ L 1 exp( -L 2 x r ) for some r > 0 , then for x ⩾ √ 8 E ( X 2 i ) /n

Proof. Details of the proof can be seen in [40].

Lemma A.12. For a sequence η ( n, p ) →∞ as ( n, p ) →∞ , we have

Proof. We can get this result by using DKW inequality([48]) directly.

Lemma A.13. Under the Assumption A.1 and A.2, max i,j ( X ij -Z ⊤ ij θ j ) 2 , max i,j e 2 i,j and max i,j ∣ ∣ ( X ij -Z ⊤ ij θ j ) e i,j ∣ ∣ are all O p (log( np )) . Both max j 1 n ∑ n i =1 { Z ⊤ ij ( θ j -ˆ θ j ) } 2 and max j 1 n ∑ n i =1 (ˆ e ij -e ij ) 2 are O p ( s log p n ) . We abbreviate 1 ≤ i ≤ n, j ∈ G as i, j respectively.

Proof. For a sufficient large constant C , by sub-Gaussian assumption we have

where K 1 is a constant independent of the choice of i, j . So max i,j ( X ij -Z ⊤ ij θ j ) 2 = O p (log( np )) is proved. By same argument, we can prove both max i,j e 2 i,j and max i,j ∣ ∣ ( X ij -Z ⊤ ij θ j ) e i,j ∣ ∣ are O p (log( np )) . Applying Theorem 2 in [35], it follows directly that max j 1 n ∑ n i =1 [ Z ⊤ ij ( θ j -ˆ θ j ) ] 2 = O p ( s log p n ) . But for max j 1 n ∑ n i =1 (ˆ e ij -e ij ) 2 , there is something different. Note that for a sufficient large constant C > 0 ,

The first term of last inequality is o (1) by lemma A.12 with η ( n, p ) = √ log p . The second term is also o (1) by applying Theorem 2 in [35] again. In summary, this lemma is proved.

Lemma A.14. Under the Assumptions A.1 and A.2,

where

Proof. To prove this lemma, we define

where e ij = F ( Y i ) -1 / 2 -Z ⊤ ij γ j , m j ( Y i ) = E β h [( X j -Z ⊤ j θ j ) { I ( Y ≥ Y i ) -F ( Y ) } ] and ˜ σ 2 j is an unbiased estimator of σ 2 j . It follows by the triangle inequality that max j ∈G ∣ ∣ σ 2 j -ˆ σ 2 j ∣ ∣ ≤ max j ∈G ∣ ∣ σ 2 j -˜ σ 2 j ∣ ∣ +max j ∈G ∣ ∣ ˜ σ 2 j -ˆ σ 2 j ∣ ∣ . So it suffices to prove both terms are o p ( (log p 0 ) -1 ) .

We begin with estimation of the rate of max j ∈G ∣ ∣ σ 2 j -˜ σ 2 j ∣ ∣ . Note that

Since Pr β h (max j ∈G ∣ ∣ σ 2 j -˜ σ 2 j ∣ ∣ ≥ t ) ≤ p 0 max j ∈G Pr β h ( ∣ ∣ σ 2 j -˜ σ 2 j ∣ ∣ ≥ t ) and for ∀ t ≥ σ 2 j ,

By sub-Gaussian assumption, ( X ij -Z ⊤ ij θ j ) e ij + m j ( Y i ) is a sub -Exponential random variable with uniform bounded sub -Exponential norm. When t is large enough, by the definition of sub -Exponential random variable, we could find some constants L 1 , L 2 such that Pr β h (∣ ∣ ( X ij -Z ⊤ ij θ j ) e ij + m j ( Y i ) ∣ ∣ ≥ √ t -σ 2 j ) ≤ L 1 exp( -L 2 √ t ) . Thus it follows by Lemma A.11 for some t ≥ C √ (log np 0 ) 5 /n with sufficient large constant C ,

for some constants C 1 and C 2 . Thus, max j ∈G ∣ ∣ σ 2 j -˜ σ 2 j ∣ ∣ = O p (√ (log np 0 ) 5 /n ) .

Next we estimate the rate of max j ∈G ∣ ∣ ˜ σ 2 j -ˆ σ 2 j ∣ ∣ . By definitions,

So it suffices to estimate the converge rate of I i , i = 1 , 2 , 3 respectively. Decomposing I 1 , it follows that

where I 1 ν is given by

Then we deal with I 1 ν , ν = 1 , . . . , 6 respectively. For simplify, in the rest of proof, we abbreviate 1 ≤ i ≤ n, j ∈ G as i, j respectively.

For I 11 , we have I 11 ≤ max i,j ( X ij -Z ⊤ ij θ j ) 2 max j 1 n ∑ n i =1 (ˆ e ij -e ij ) 2 = O p (log( np )) O p ( n -1 s log p ) = O p ( n -1 s log p log( np )) by lemma A.13. Similarly, we have I 12 ≤ max i,j e 2 ij max j 1 n ∑ n i =1 { Z ⊤ ij ( θ j -ˆ θ j ) } 2 = O p ( n -1 s log p log( np )) .

For I 13 , by Cauchy-Schwartz inequality and lemma A.13, it holds that

Similarly, we have

Next we deal with I 2 . By definition and Hoeffding's inequality we have

In summary, we have that I 1 + I 2 = O p ( r ( n, p, s )) , where

Similarly, we can show that I 3 = O p ( √ r ( n, p, s )) , it follows from Cauchy-Schwartz inequality that I 3 ≤ √ max j ˜ σ 2 j √ 2( I 1 + I 2 ) .

Thus this lemma is proved.

Lemma A.15. Under the Assumptions A.1,

Proof. For simplify, we rewrite S nj - √ nβ hj δ j = ∑ n i =1 ξ ij / √ n . Under assumption A.1, it's easy to show that { ξ ij } n i =1 is an i.i.d. zero mean sub -Exponential r.v. sequence. So we can prove this lemma by using Bernstein's inequality and Bonferroni's inequality.

Lemma A.16. Under the Assumptions A.1 and A.2,

uniformly for β h ∈ Ω ∩ { β h : ∥ β h ∥ ∞ ≲ √ log p/n } with

Where

and r 2 ( n, p, p 0 , s ) is defined in (1.48) .

Next we deal with max j ∈G | A ij | , i = 1 , 2 respectively. We handle max j ∈G | A 1 j | at first. We decompose A 1 j as

where U nj is defined in (1.24). So it suffices to prove max j ∈G | I ij | = O p ( r 1 ( n, p, p 0 , s )) , i = 1 , 2 , 3 , 4 , 5 respectively. For a sufficient small constant c ′ > 0 and a sufficient large constant C > 0 , we have following results.

Step 1, for i = 1 . max j ∈G | I ij | = O p ( r 1 ( n, p, p 0 , s )) holds by sub-Gaussian Assumption and Bernstein inequality. The proof is similar to step 2 in the proof of Lemma A.10, we omit it.

Step 2, for i = 2 . By Proposition A.1 and Lemma A.10, we have

where ( c, c 1 , c 2 ) are constants defined in these lemma and independent of the choice of j . We can derive that √ nλ X λ Y s X = O ( s log p/ √ n ) by the fact that λ X , λ Y ≍ √ log p/n

Step 3, for i = 3 . It's similar to step 2 by using Theorem 3.1 and Lemma A.9. We omit it.

Step 4, for i = 4 . Note that I 4 j = n -1 / 2 ( U nj -β hj δ j ) + n -1 / 2 β hj δ j and n -1 / 2 max j ∈G | β hj δ j | = O ( n -1 √ log p ) . It suffices to deal with n -1 / 2 ( U nj -β hj δ j ) . We begin the proof in the same way as we argue in the proof of lemma A.7. Applying lemma A.6, for ∀ s ∈ R we have

where N = ⌊ n/ 2 ⌋ and t ∈ R + . Denote W ij = h j ( Y i , X i ; Y N + i , X N + i ) -β hj δ j , i = 1 . . . . , N . By the sub -Gaussian assumption made in assumption A.1, { W ij } N i =1 is an i.i.d. sub -Exponential sequence and max 1 ≤ j ≤ p ∥ W · j ∥ ψ 1 < ∞ . Thus by the same way used in the proof of Bernstein inequality([49]), we derive

where c is a constant. Denote t = Cr 1 ( n, p, p 0 , s ) , we prove it as n ≫ t -2 .

Step 5, for i = 5 . By the theory of U -statistics, U nj -S nj / √ n is a two-order canonical U -statistic. Thus max j ∈G | I 5 j | = O p ( { n -1 (log p 0 ) 3 / 2 log( np 0 )+ n -1 / 2 log p 0 } ) by Lemma A.21. In summary, we derive that max j ∈G | A 1 j | = O p ( r 1 ( n, p, p 0 , s )) since ˆ σ j ≥ C min -ψ n ≥ C min / 2 > 0 for n large enough.

Next we prove max j ∈G | A 2 j | = o p ( r 2 ( n, p, p 0 , s )(log p 0 ) 1 / 2 ) . For ∀ c ′ > 0 we have

The last equality holds by Lemma A.14 and A.15. Note that all the universal constants do not depend on n, p, β h and θ j , j = 1 , . . . , p . Thus this lemma is proved.

Lemma A.17. Under the Assumptions A.1 and A.2,

Proof. The proof of this Lemma is similar to Lemma A.16.

Lemma A.18. Under the Assumptions in theorem A.1, if s = o ( √ n/ (log p (log np 0 ) 3 / 2 ) ) , then

where L ♯ n, G is defined in (1.27) in the main text and C is a sufficient large constant.

Proof. By definition,

̸

Thus it suffices to prove Pr w ( I i ≥ C (log p 0 ) -1 / 2 ) → 0 in probability, i = 1 , 2 .

For I 1 , since w i is standard Gaussian r.v., the Hoeffding inequality implies that

where C ′ is a constant. Applying Lemma A.17, Pr w ( I 1 ≥ C (log( np 0 )) -1 / 2 ) = o p (1) is proved. For I 2 , the Hoeffding inequality also implies that

̸

̸

For simplify, we abbreviate ˆ h j ( Y i , X i ; Y ˜ i , X ˜ i ) , h j ( Y i , X i ; Y ˜ i , X ˜ i ) as ˆ h j , h j respectively. In the rest of proof, we abbreviate max 1 ≤ i ≤ n , max ˜ i = i , max j ∈G as max i , max ˜ i , max j respectively. To prove Pr w ( I 2 ≥ C (log( np 0 )) -1 / 2 ) = o p (1) , it suffices to prove that

̸

By the Hölder inequality,

̸

Thus it suffices to prove I 2 k = o p ((log( np 0 )) -1 / 2 ) , k = 1 , 2 respectively. For I 21 , it holds that

where ˆ η ij = K -Z ⊤ ij ˆ γ j , η ij = K -Z ⊤ ij γ j and K = 1 / 2 or -1 / 2 . Note that ˆ η ij -η ij = Z ⊤ ij ( γ j -ˆ γ j ) and

where the last step follows from the s √ log p/n convergence of ∥ γ j -ˆ γ j ∥ 1 and max i,j | Z ij | = O p ( √ log( np 0 )) , due to the sub-Gaussian properties of Z ij . Following the similar arguments, we have

In summary, we have

where the last equality holds by Assumption A.7. For I 22 , it holds that

where the last inequality holds by Lemma A.14 and max i, ˜ i,j | h j | = O p (log( p 0 n )) . In summary, this lemma is proved.

Lemma A.19. Let X 1 , . . . , X n be n i.i.d random vectors, where X i =( X i 1 , . . . , X id ) ⊤ . Assume that there are some constants 0 < c 1 < C 1 such that E ( X 2 ij ) ≥ c 1 and X ij is sub-Exponential with ∥ X ij ∥ ψ 1 ≤ C 1 for all 1 ≤ j ≤ d . If (log( dn )) 7 /n = o (1) , then

where N ∼ N d ( 0 , E ( XX ⊤ ) )

Proof. Lemma H.6 in [40], and this lemma is adapted from [13].

Lemma A.20. Assume that X ∼ N d (0 , Σ ) . Let σ 2 j = Σ jj and define σ min = min j σ j and σ max = max j σ j . Then

where C is a constant depending on σ min and σ max .

Proof. Lemma H.5 in [40], and this lemma is adapted from [13].

̸

Lemma A.21. (A maximal inequality for canonical U-statistics). Let X 1 , . . . , X n be n i.i.d. p -dimension random vectors. Let f : R p × R p → R d be a symmetric and canonical kernel such that max 1 ≤ m ≤ d E { exp ( | f m | /B n )} ≤ 2 , and B n is a sequence of positive reals. Let V n = { n ( n -1) } -1 ∑ 1 ≤ i = j ≤ n f ( X i , X j ) . If 2 ≤ d ≤ exp( bn ) for some constant b > 0 , then there exists a constant C ( b ) > 0 such that

Proof. This lemma is adapted from [12].

Lemma A.22. Let X 1 , . . . , X n be n i.i.d. p -dimension random vectors and h : R p × R p → R d be a symmetric and nondegenerate kernel such that E | h k ( X 1 , X 2 ) | < ∞ for all k = 1 , . . . , d . Consider the U -statistic of order two:

̸

Let T n = √ n ( U n -θ ) / 2 , where θ = E { h ( X 1 , X 2 ) } is the mean of U n . Define g ( x ) = E { h ( X 1 , X 2 ) | X 1 = x } and x ∈ R p . Let e 1 , . . . , e n be i.i.d. N (0 , 1) random variables that are independent of { X i } n i =1 and

̸

Let B n ≥ 1 be a sequence of real numbers possibly tending to infinity. We assume that:

(1) There exists a constant b > 0 such that E { g 2 m ( X 1 ) } ≥ b for all m =1 , . . . , d .

(3) ∥ h m ( X 1 , X 2 ) ∥ ψ 1 ≤ B n for all m = 1 , . . . , d .

Denote a ∥ T ♯ n ∥ ∞ ( α ) as the α th conditional quantile of ∥ T ♯ n ∥ ∞ given X 1 , . . . , X n , where α ∈ (0 , 1) . If (log( dn )) 7 /n = o (1) , then

In additional, assume that there exist statistics ˆ T n and ˆ T ♯ n such that

and for some ξ ( n ) depending on n . If ξ ( n ) √ 1 ∨ log( d/ξ ( n )) = o (1) , then

where a ∥ ˆ T ♯ n ∥ ∞ ( α ) is the α th conditional quantile of ∥ ˆ T ♯ n ∥ ∞ given X 1 , . . . , X n .

Proof. This lemma is adapted from [12] and [13].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction clearly state the paper's contributions and scope. And the claim does match theoretical and experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our paper in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The appendix contains full set of assumptions and a complete proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the information needed is provided in Section 4 and Appendix A.6. Code is avaliable in supplementary materials.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is avaliable in the Supplementary Material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details are provided in Section 4 and Appendix A.6. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This paper focuses on statistical inference, and we provide empirical size and power in numerical studies.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Type of compute workers and memory are provided in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper does conform with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is a foundational research and not tied to particular applications.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors cite the original paper that produced the code package.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the high-dimensional single index model (SIM), which takes the form $Y=g(X^T\beta, \epsilon)$ with $\epsilon$ and $X$ being orthogonal. Although this model has flexibility and interpretability, its efficiency is adversely affected by outlying observations and heavy-tailed distributions. The paper improves this in the following 3 aspects:

(1) they extend the rank-LASSO procedure to include both convex and non-convex penalties and establish error bound of any local optimum of the empirical objective;

(2) they provide asymptotically honest group inference procedures based on the idea of orthogonalization for testing the joint effect of many predictors;

(3) they develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the FDR asymptotically.

STRENGTHS:
Provides new ideas that deal with the inefficiency of single index model due to outlying observations and heavy-tailed distributions.

WEAKNESSES:
I think the paper is quite well-written. The only issue may be that the authors can provide more about the background of the problem,

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a robust group hypothesis testing procedure for a high-dimensional single index model based on a data-driven transformation of the response variable. The key observation is that under the the linearity condition (LC) of the predictors, a wide class of single index models can be equivalently (for testing purposes) converted into a linear model with a transformed response variable. The main contribution of this paper is the introduction of the distribution transformation (a data-driven approach) of the response variable, and derive the asymptotic distribution of the resulting test statistic. Numerical performance of the proposed method is evaluated through simulation studies.\n\nSTRENGTHS:\nThe proposed method is interesting and potentially useful. The paper is well-written and the presentation is clear.\n\nWEAKNESSES:\nThe simulation study can be improved.\n\nQUESTIONS:\n1. The introduction of the distribution function $F_n$ in (2.7) is interesting but also a little arbitrary. I suspect that any other functions that can be written as linear combinations of functions of $Y_i$'s can be used in (2.7), for example, the kernel density estimator, the kernel smoother of $Y_i$'s. The resulting test statistic should still be asymptotically equivalent to a U-statistic. Could the authors comment on this?\n\n2. Whenever the term \"robustness\" is involved, it implies loss of efficiency in some cases. In my opinion, it is important to make a transparent evaluation of the limitations of the proposed procedure. In the simulation study, one can add a comparison to the method with $h(Y)=Y$ as described in section 3.3. It will be informative if one can showcase in which cases which procedure is more powerful, and thus understand which method to use in practice.\n\n3. In the simulation study, I would also suggest a graphical comparison between different methods by gradually increasing $p_{out}$ from $0$ to, say, $0.5$ for a more complete picture.\n\nLIMITATIONS:\nThe simulation study can be improved.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an algorithm for group inference in single-index models with an unknown link function, that is robust to heavy-tailed noise in the responses. The central idea of the approach is based on the property of elliptical distributions that the linear input-output correlations remain along a fixed direction for any transformation of the labels. The algorithm introduces a linear estimation objective for transformed labels resulting in an estimator converging to the true parameters under sparsity assumptions. Crucially, the obtained estimator yields a robust test for group inference satisfying the orthogonality property, which relaxes the requirements of separation between zero and non-zero coefficients. The obtained test further satisfies honesty, namely the uniform convergence of Type 1 error. The work concludes with numerical tests of the proposed approach.\n\nSTRENGTHS:\nThe paper is generally well written and provides an adequate motivation for the problem of group inference under heavy-tailed response, and the idea of orthogonalization. The proposed approach is straightforward to implement and understand. The paper provides an extensive discussion of some important properties of the test such as honesty and power. The estimation error bounds are non-asymptotic and hold for general scalings of p,n. The numerical experiments support the efficiency and robustness of the approach.\n\nWEAKNESSES:\nA few limitations of the setup are not explicitly discussed:\n1) The robustness is restricted to variability in the noise of the response function, not w.r.t general perturbations to the distribution. The paper also doesn’t provide quantitative bounds on robustness.\n2) The condition kappa_h \\neq 0 excludes even link functions and in particular the problem of sparse phase retrieval.\n3) Theorem 3.1 imposes limitations on the sparsity level that should be explicit. The results in the paper seems to be restricted to constant sparsity levels/\n\nThe choice of h=F for robustness appears to be due to F(Y) being uniformly distributed in [0,1] independent of the distribution of Y. This should be more discussed explicitly.\n\nQUESTIONS:\nQuestions:\n\nWhy are λX, λY not directly described in Theorem 3.1?\n\nWhy is the transformation H=F particularly suitable for robustness? Is it because of the distribution of F(Y) being agnostic to the distribution of Y?\n\nFor sub-exponential variables, Lemma A.11 appears to be weaker than Bernstein’s inequality. \n\nCould you clarify the difference w.r.t Bernstein’s inequality and related inequalities for random variables with bounded Orlicz norms?\n\nLIMITATIONS:\nSome missing discussion on limitations is described above under “weaknesses”. The work is primarily of a theoretical nature and has no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}