Robust group and simultaneous inferences for high-dimensional single index model
Authors: Weichao Yang
Hongwei Shi
Xu Guo
Changliang Zou
Venue: NeurIPS 2024 poster (2024)
Keywords: FDR control
high-dimensional inference
honest test
outliers
robustness
Primary area: probabilistic_methods
TL;DR: This paper introduces high-dimensional robust inference procedures by recasting the single index model into a pseudo-linear model with transformed  responses.

Abstract: The high-dimensional single index model (SIM), which assumes that the response is independent of the predictors given a linear combination of predictors, has drawn attention due to its flexibility and interpretability, but its efficiency is adversely affected by outlying observations and heavy-tailed distributions. This paper introduces a robust procedure by recasting the SIM into a pseudo-linear model with transformed responses. It relaxes the distributional conditions on random errors from sub-Gaussian to more general distributions and thus it is robust with substantial efficiency gain for heavy-tailed random errors. Under this paradigm, we provide asymptotically honest group inference procedures based on the idea of orthogonalization, which enjoys the feature that it does not require the zero and nonzero coefficients to be well-separated. Asymptotic null distribution and bootstrap implementation are both established. Moreover, we develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the false discovery rate asymptotically. Numerical results indicate that the new procedures can be highly competitive among existing methods, especially for heavy-tailed errors.

Robust group and simultaneous inferences for high-dimensional single index model
Weichao Yang
Hongwei Shi
School of Statistics
School of Statistics
Beijing Normal University, Beijing, China wht.gweeihaa@amlll... bnu edu.
Beijing Normal University, Beijing, China shihongwei21 @mail bnu edu.
Xu Guo
Changliang Zou
School of Statistics
NITFID, School of Statistics and Data Science, LPMC and KLMDASR and LEBPS, Nankai University, Tianji,, China noucl@nankal. edu. cn
Beijing Normal University, Beijing, China rustat120bml edu. cn
Abstract
The high- nigh-dimensional single index model (SIM), which assumes that the response is independent the predictors given a linear combination of predictors, has drawn attention due its flexibility and interpretability, but its efficiency is adversely affected by outlying observations and heavy-tailed distributions. This paper intro- duces robust procedure recasting the SIM into pseudo-linear model with transformed responses. relaxes the distributional conditions on random errors from
more general distributions and thus it is robust with sub- stantial efficiency gain for heavy-tailed random errors. Under this paradigm, we provide asymptotically honest group inference procedures based on the idea of orthogonalization, which enjoys the feature that it does not require the zero and nonzero coefficients be well-separated. Asymptotic null distribution and boot- strap implementation are both established. Moreover, we develop multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the false discovery rate asymptotically. Numerical results indicate that the new procedures can be highly competitive among existing methods, especially for heavy-tailed errors.
Introduction
With fast development of information technologies, high- sith-dimensional data frequently collected in various scientific disciplines. To efficiently analyze nigh-dimensional data, various regulariza- tion methods were developed and thoroughly investigated in both theory and methodology. For comprehensive review, see [19]. Though significant progress has been made about the theory of estimation and variable selection, statistical inference for high-dimensional model remains largely unexplored until the seminal works of (58), (47 and [29]. They developed debiased estimators that correct the bias introduced by LASSO and established the estimators' asymptotic normality. [40]
"The first two authors contribute equally and co-first authors. Xu Guo is the corresponding author. Guo was supported by National Key National Natural Science Foundation of China Grant (Nos. 12322112. 12071039 1031103), and the Fundamental Research Funds for the Central Universities. Zou was supported by the National Key R & D Program of China (Grant Nos. 2022YFA 1003703, 2022YFA 1003800), the National Natural Science Foundation of China (Grant Nos. 11925106, 1223 1011, 11931001, 12226007. 12326325).
Program of China (Grant Nos. 2023YFA 1008702, 2023YFA101 1A101100 100),
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

proposed decorrelated score method which is a general procedure and is applicable to a large family of penalized M-estimators. 445 introduced a recursive online-score estimation for nigh-dimensional
generalized línear model. [46 proposed the nuisance penalized regression which does not penalize
the parameters of interest. 21 and [20] studied inference problems for the high-dimensional Cox
model and longitudinal data, respectively. Logistic regression model was investigated in [36]. Other
recent developments include [44]. [50]. [5], and [19].
Most of current inference procedures focus on parametric regression models. To address the potential mis-specification issue of parametric models, the single index model (SIM) which is one of the most popular semiparametrie modelling techniques, has received extensive attention and in-depth research in the past decade. Let Y I be the response variable along with predictor vector X = (x Xp) E R". We consider a general SIM,
Y = g( G,c) with c C_X,
(1.1)
where (3 (3,,... lently, we have
BBp) ;p) the link function g()) is unknown and means independence. Equiva-
Y XTB.
The above statement is that, given X B which called index, the response is independent of the predictor vector X. Clearly, model (1.1 covers linear models, generalized linear models and also the classical SIM (26), that is, Y = 91 (X (3) + e with gi (.) being an unknown function, relaxes restrictive assumptions on parametric models and is flexible enough to capture complex relationship between the response and the predictors.
For high-dimensional SIMs, variable selection has been considered by many authors. Examples include [3) 60), 57 M. ST. 421. [52] and 43]. Our primary interest detect whether
set of predictors contributes to the response Y or not given the other predictors, that is testing the following group inference problem:
Ho,g : 33 = for all jeg versus
H1,g By * O for some j E g.
(1.2)
where G is prespecified subset of {1. 21,2.... p} with [g]. Such a hypothesis naturally arises in the gigh-dimensional setting. For example, researchers may want to test whether gene pathway, consisting of multiple genes for the same biological functions, is important for certain clinical outcome The above group inference problem also includes global significance testing as a special case when we set G = {1, ...,......) p} and Po p. The group inference problem is more difficult than purely global significance testing. Actually, for group inference problem, we are dealing with a high-dimensional interested parametric vector with a high-dimensional nuisance parameter. Such a group inference problem was considered by [59] for high-dimensional linear models. See also [16), and [23] All of [59], [6]. and (23) did not establish asymptotical honesty of their procedures, which critical for reliable inference in high-dimensional setting.
For SIM, due to the existence of the unknown link function g()), the group inference problem is more difficult. Actually, the parameter of interest and the nonparametric function are bundled together, which makes the inference being challenging. For the model (I.I. 17 developed a debiased LASSO procedure for individual coefficient, while (53) considered simultaneous confidence interval for optimal treatment regimes under the classical SIM [26]. Their procedures, as well as many other existing works, rely heavily on the sub- Gaussian assumption about the error term. In practice, data with heavy-tailed distribution or in the presence of outliers are very common [18), and the efficiency of those procedures developed under sub-Gaussian assumption would be largely deteriorated. Recently 25 investigated the robust inference problem of high-dimensional SIM by adopting the Huber loss with original response. However, their procedure still requires bounded first moment condition of the error term and needs to select suitable robustification parameter. In this paper, aim to develop robust group inference methods for high-ding dimensioonl SIM without any moment condition on the error term and without choosing robustification parameter.
Interestingly, under a mild condition on the predictors, which called linearity condition in the literature of sufficient dimension reduction [II,, we find that the SIM can be recast into a pseudo-linear model with transformed response, allowing us to make robust inference in simple fashion. With a specific "response-distriee distriiutddttiiittioonn transformation, the sub- mub-Gaussian response assumption required in existing methods is totally avoided, and accordingly our proposed procedures are robust to outliers or heavy-taile error distributions.

---

Besides group inference of a prespecified subset of predictors, we are also interested in identifying relevant predictors. To this aim, large-scale simultaneous hypotheses are considered
Hoj 3,
versus
H11 ;3 00 1 j
Apart from identifying as many nonzero By as possible, to obtain results with uncertainty quantifica- tion, we would like control the false discovery rate (FDR) which is an extremely popular tool to maintain the ability to reliably detect true alternatives without excessive false positíve results when large-scale hypotheses are simultaneously tested 4 (28). 20), 36 and ( considered FDR control in high-dimensional regression models. We follow this line and develop a FDR control procedure for high- sehhdimennional SIM.
Our major contributions are listed from the following three aspects.
We extend the rank-LASSO procedure in 43 to include both convex and non-convex penalties and establish error bound of any local optimum of the empirical objective. These theoretical results summarized in subsection B.I
In Section 2 we provide asymptotically honest group inference procedures based on the idea of orthogonalization for testing the joint effect of many predictors, which enjoys the feature that it does not require the zero and nonzero coefficients to be well-separated. We demonstrate the superiority of our test procedures both theoretically and empirically, Please see Section 3 for theoretical justification and Section 4 for numerical studies respectively,
We develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able control the FDR asymptotically. To this end, we develop suitable multiple testing procedures and show that the proposed methods can control the false discovery rate (FDR, (4) both theoretically and empirically. We refer the readers to Appendix A.3 and A.6 for details.
Notation. For d-dimension vector U, we write ||]|, - DE),U,) and U||
maxigssalUA to denote I, and Ix0 norms of U. Further we define |0/ = # (k UR 0}. A random variable X is sub- Gaussian if the moment generating function (MGF) of X is bounded at some point, namely exp(X2 / I 2, where is positive constant. A random variable Y is sub- Exponenlial if the MGF of YY| is bounded some point, namely E exxp((Yl/K") S 2, where K" is a positive constant. For a, b E R, we write a V = max(a,b) b}.
2 Group inference based on distribution transformation
Without loss of generality, assume that E(X) O, and Σ Var(X) > 0. Now let o Cov(X, h{Y)} for a given transformation function h(-) of the response. Define Br, = E 0%. We then have the following result.
Proposition 2.1. Assume that EX( X (TB) is a linear function X B. Then Bn is proportional to (3, that is. B% = Kr X B for some constant Ky.-
The above proposition follows directly from Theorem 2.1 in 32 The assumption in above propo- sition is known as linearity condition (L.C) for predictors. It is satisfied when X has an elliptical distribution and widely assumed in the sufficient dimension reduction literature 31 14 24 showed that in high-dimensional setting, the LC holds to a reasonable approximation. Throughout of the paper, we assume that Ky + 0. This assumption is mild. In fact, when h(-) is monotone and g(:,)) is monotone with respect to the first argument, this assumption is satisfied.
From the definition of Br, we can write
k(Y)= = 33,, X +
(2.3)
where the error term must satisfy E(ex)) Different from the existing literature which usually imposes independence between the regression error and the predictors, the regression error e in the transformed model only uncorrelated with the predictors. This implies that under LC, we can recast the general SIM into pseudo- model with transformed response h(Y). Therefore, by Proposition 2.1 testing Hu,g equivalent
Ho, Brg 0 for all j G versus H',g: BBg # 0 for some j e G.
3

---

This reformulation is important, allowing us to circumvent the issue of estimating the unknown link function -(:-)) and to make inference of J in linear regression model with transformed response instead of 33 in SIM. As we show later, this reformulation greatly facilitates the construction of our test statistic and simplifies the computation as well.
In practice, we need to choose suitable transformation function h(-). We note that [7'ss procedure also relies on the above proposition and they essentially work with h(Y) YY For robustness consideration, throughout this paper, we consider the distribution function of Y, denoted by F(Y) as the transformation function. Actually with the equation 2.3). given the widely imposed subgaussian assumption on the predictors, any bounded transformation function h(Y) would lead the transformed error term e being subgaussian, even if the original error term in the single index model Y g(X B, e) comes from Cauchy distribution. Further as noted by [43]. in the empirical distribution function, the term Σj=1 r(Y, S Y2) the rank of Y, Since statistics with ranks such as Wilcoxon test and the Kruskall-Wallhi ANOVA test, are well-known be robust, this then intuitively explains why our procedures with iisponse-distriuns in response. Moreover the distribution function is very easy to estimate and thus our approach is straightforward to implement and understand.
transformation are robust with respect to outliers
Our test statistic for group inference of Hl.g relies on individual inference of Hi BBn 0. Now we first consider individual hypothesis Hoy. Let Z, be the subvector of X without Xj. and Y be the subvector of 3r without 3ng. Suppose that (X,, Y: is a random sample from the population (X, Y). Similarly, we denote Zij as the sample of Zj. Define
Br = BX - (E((((()) - 1/2H] and 0, 1-
where Bn is the regression coefficient of the model 2.3 with h = F.
Our approach based on the idea of orthogonalization. The main idea of orthogonalization is to construct statistic for target parameter which is locally insensitive to the nuisance parameters. Dealing with high-dimensional models, it plays an important role to make the statistic of target parameter immune to the bias from the estimators of high-dimensional nuisance parameters, which in turn enables the statistical inference of parameter of interest. For relevant references, see for example, 40 and 22.
3
As pointed by [53], the adoption of orthogonalization nontrivial for nigh-dimensional semiparamet- ric setting, particularly for index model, where the challenge of bundled parameter arises. Fortunately. with the Proposition 2. the SIM can be recast into a pseudo-lines linear model with transformed response Note that under the null hypothesis lilo,
2.3
E [{F(Y)) 1/2 }3 (X, z, 0,] = 0.
Further the above equation has the orthogonality property
[(F()) - 1/2 - -2/ 7) (X, -z -2, E
= (X, - 2J. 0,)1 = O.
Ty This then motivates us to consider the following quantity
T5 Σ 1/2 (Xij (Z,5-17,0)) Z,,,,)
(2.4)
In practice, we need to use suitable estimates of F. Yy and 0, in 2.4 as those quantities are unknown. Naturally, (3) = n Ei= I(Y; S y). the empirical distribution of Yı,..., Yn, can be used to estimate F(Y). For θj. we estimate it by the penalized lquares method
0, arg min 1
Σ
(X,,
2.
Σ PAx (()l)),
(2.5)
0, EREP
2n
where PAx (-) is a penalty function with a tuning parameter Ax. Similarly, for Yi, we adopt the following penalized lquares
Br arg min (a,) + Σ Par (18aa)),
(2.6)
Br ERR

---

where (3,) = (2m) E. ( 1/2- and Pxy (-) is a penalty function with tuning parameter Λy. The 7, is then the subvector of Gr without Ang. Accordingly, for each individual hypothesis 11l6; Pnj = we define the standardized test statistic
Tnj
FL) 1/2 2,541} (Xiy (X.s
ZT,0,),
(2.7)
8j n
where
03 = Σ (Xij - Z, }
(2.8)
GG F,(Y) - 2, and (y) =n that PY((Y) n -nー I(,; Y.). Then given predictors X;'s being fixed, perturbations in the responses would not make the value of Tay change as long as the ranks of Y,'s remain unchanged.
2, 2 3) F.())) Note
Denote Tn (Tns)JEg. To test the null hypothesis Ho,g. we consider test statistic based on the max norm of Tn That is,
Mn.6
max Thj JEG
(2.9)
can reject null hypothesis lf at the significant level a and only if Mn. 2 (g(a),, where ca(a) = 2 log - log log Po + and ga the 1 a quantile of the Gumbel distribution with the cumulative distribution function expl- V= exp(-x/2)), that is,
Based the limiting null distribution obtained in subsection 3.2
qa - -lg(T)) 2 log log - -1-
(2.10)
Our inference procedure is summrized in the Algothrim as follows.
Algorithm 1: Group inference based on distribution transformation
Input: Covariates data (X; ) response data (Y))
Output: Testing results for group inference problem (.2).
Compute the penalized least-square estimator Br. defined in (2.6):
2 for j E G do
compuate estimator 9 and O defined in 2.5
calculate the standard statistic Tnj defined in 2.7
calculate the test statistic Mn,c defined in (2.9
6 Mn.g 2 (a()), reject Hoog; otherwise accept Ho.g-
Remark 2.1. The use of quantiles of limiting null distribution in cg(a) is attractive from computa- tional point of view. On the other hand, the validity of limiting null distribution requires additional assumptions regarding the dependence structure of the components of the covariates X. See Assump- tion A.3 in Appendix(A.8)) slow. solve these problems, we propose a multiplier bootstrap approuch and show its validity in theory, please see Appendix A. for details.
details. Besides, it is well known that this weak convergence is typically
3 Theoretical properties
3.1 Estimation error bound
The theoretical analysis of 3, requires a substantial modification of the proof technique as compared to existing works on high-dimensional inference. is related to the fact that empirical distribution (Y.)) are dependent, and thus C., (Ba) is sum of dependent random variables. Note that 43 considered a similar penalized procedure with LASSO penalty. While (60) and [ST] considered non- convex penalties such as SCAD and MCP to reduce estimation bias incurred by LASSO. However,
5

---

their results only allow the dimension to be polynomial order of the sample size and thus cannot work in altrahighdimeesional setting. Further their results only provide guarantees for global optima. In this paper, we consider both convex and non-convex penalties and establish error bound of any local optimum of the empirical objective.
To be specific, assume that Br, satisfies the first-order necessary condition to be local minimum of 2.6 that is,
(3.) + Vpay (3x), Bn) 2 0, for all feasible 3 E Rp.
(3.11)
Let sy be the sparsity level for Bh, i.c., Sy 13allo- For any vector Br satisfying the condition (3.IT) we have the following result.
Theorem 3.1. Under Assumptions A.7 and A.2 in Appendix A.8 the Br defined in 3.11 satisfies
| Bxll2 S cAy by and 1| - Balli I d'
with probability at least ei exp(-o2 logp). Here, (c, 01 02) are universal constants and Ay x //gp/n.
we establish error bound of penalized least-squares estimators with empirical distribution function of the response. For consistency in L2-loss, require the sparsity level sy satisfy that sy = on/ log p). While in terms of Lı-loss, the limitation becomes sy o( olyn) n/ log p). Clearly the sparsity level is allowed to be diverging. Our results unify both convex 44 and non- be exponential order of the sample size, and no minimal signal condition is imposed to obtain those error
In Theorem 3.1
convex penalties [60 51. Compared with the results in (60) and [51], the dimension
bounds. In the proof, we modify Hoeffding inequality by a probability inequality for U-statistic to handle tail probability in dependent case, which may be interesting in their own rights.
3.2 Asymptotic null distribution
In this subsection, we derive the asymptotic null distribution for statistic defined in 2.9 Denote Ω = (3xx E RR: 1|,3%,lo llo V maxigissl|0 O,olo S 88}
and Bng = (Sng)seg. We consider the following
parameter space for Ho,g
2g (B, E TR? Big 0} 0
Theorem 3.2. Suppose that Assumptions
A. A in Appendix A.8 and LC condition hold. If
o(Vn/(log log Po)), then for given
I we have
lim sup (n.po)->oe B.Eng
Pr Pr(Ma.g 21og po + log log Po 51) exp Vx exp(-1/2) =0.
Theorem 3.2 implies that the type eтror of the proposed test statistic Mn.g converges to any pre-specified significance level uniformly over 3r E 29. The hypothesis test with such uniform convergence property is called honest test. The advantage of honest test that the limiting distribution of our procedure is uniformly valid over s-sparse high-dimensional models despite the possible imperfect model selection via estimator 3),. An immediate implication is that it relaxes the assumption on signal strength and does not require the zero and nonzero effects to be well well-separated In particular, this procedure does not require the initial estimator to select zero and nonzero signals perfectly, which nearly impossible in practice. Due these excellent statistical properties, honest test has recently drawn lots of attention. See [47] 29 3 2
53 for further examples.
we can reject null hypothesis Iolo the significant level a if and only if Mn.g 2 ca(a;) where eg (a) = 2 llog po - log log Po + 4a and Ja is defined in (2.10).
On the basis of Theorem 3.2.
3.3 Power analysis
We next to consider the asymptotic power analysis of the Mn,g. In this section, we show that our test statistic is powerful in the sense that the separation rate is of order /(2 co) log Pa/ for some
60 > when Po + oc. At the beginning we define the following parameter space for lll
(co) { Bn E R.. max J6g
log Po
6o
2
n

---

where co is positive constant and 8, = [ - _( ) "B(,))
((),))) Theorem 3.3. Suppose that conditions in Theorem 3.2 are satisfied, we have for some eo > lim +00 (2
inf Pr Pr 2 cg(a)) =1.
(3.12)
Theorem 3.3 implies that test procedure can still be powerful
when there exists few components of Bg with magnitude being larger than V co) log po/n. Therefore testing procedure is powerful against "sparse" alternative aforementioned. This separation rate widely discussed in the literature, such as [ [59] and [36]. Theorem 3.3
shows our tests based
distribution transformation can also achieve this lower bound.
Remark 3.1. In the proof of Theorem
3.3 we show that
Bhjoj
Mn.g x MM +
IDaX
JEB
Jj
where the M2 has the type I extreme limiting distribution, which is mentioned in Theorem 3.2 Then the power is largely determined by the second term
Bryoz
SNR(M max
(3.13)
JEG
and it can be explained as the maximum of the signal-to-noise ratio for j E G. The power of the test based on Ma.g increases with the growth of SNR(Mn.g). When SNR SNR(Ma.). 2 (2 + eu) log po/n. n,, the asymptotic power is tending to 1.
In this paper, we focus on the distribution transformation of the response. In the following we make some comparisons between this choice and another natural one. Actually, same to [7]. one may consider use hYY) Y to conduct the following test statistic
T - (x,))xx,, -Z,,,)
Eo..
(3.14)
==1
Here 7, is the subvector of 3) without j-th component, and 3 is penalized least-squares estimator of 1 ( as follows:
/ arg min Eo. 3
- Σ PAv (19
Elo 2n
t=1
0, is estimated by 2.5 And 0/2
22 is an appropriate estimator of a 0,)0,
zJJ )2), For the group inference problem (.2).. it's natural to consider the following testing statistic:
M max (T,))
JEG
Now we discuss the power properties of g. By similar arguments as Theorem 3.3 the power of the test based on M is determined by
SNR(M R. = max
(3.15)
JEG
For the convenience of illustration, we consider the classical SIM, that is, Y- (X (3 + and is independent of X. To compare the theoretical power of Mn.9 and M gt it suffices compare the SNRs aforementioned. In the classical SIM case, the ratio of SNR can be simplified
SNR(Mn.t) 2 c3 Var(c), SNR(M,t))
where C3 is constant depending on (X,Y). The detailed form of c3 is presented in the Appendix A.2 When Var(e) > oz Ma.g is more powerful than M n.g- Actually when follows a heavy-tail distribution, Var(e) can be very large even be infinite.
The above results illustrate the robustness of our test statistic Mn,g. In fact, with distribution transformation, no moment assumption on e is required. Even the comes from Cauchy distribution, test procedure still works well but the M requires uub-Gausiia assumption the error term and thus in this situation would fail.
7

---

4 Numerical studies
In this section, extensive simulation studies are carried out to evaluate the numerical performance of the proposed methods for the global inference problem described in (1.2) We consider the high-dimensional SIM in equation [1.1] and generate the data from the following models:
Model 1: Linear model: Y = x c.
Model 2: Non-linear model: Y = exp(X 3+c).
Here, the predictors X;, i = 1, n are generated from multivariate normal distribution (pp )). The covariance matrix Σ E Rexp is block diagonal, that is, Σ diag (Ei,-. Σi0), where Σ, is to x to dimensional identity matrix, and each Σ₁ is to x to dimensional with an AR(1) correlation structure, that is (Ex) = (0.1k 0.1) = 2..... 10; i,j 1, ..... PP/1. We consider two different error distributions for e which is independent of X: (1) standard normal distribution NO, 1); (2) Cauchy distribution or equivalently Student's distribution with degree of freedom, t(1). Without loss of generality, we set the active set be {1, 2.... 6}. The regression coefficients are generated from an arithmetic sequence from 0.1 to that is 8g 0.2 + 0.038(j 1) for 1 1 j S 6 and 3, = 0) otherwise. We consider 200, 500 and p = 800. We use Intel(R) Xeon(R) Silver 4208 CPU @ 2.GOGHz.
.....,......
In order to explore the robustness behaviors of our proposed statistics, we add outliers to pollute the observations: Pou of the responses are picked random and increased by moa- maximum of original responses, shorted as Pout + Mout max (responsss,ss) Here Pout is the proportion of outliers and Mout is a pre-set constant standing for outlier strength. Throughout the simulation study, we analyze the results with both the original data (shorted as Ori.) and the data with outliers (shorted as Out.).
We assess the empirical type eπrrr and the empirical power of group inference with the nominal level a = 0.05 based on 500 simulation runs. We consider the following five different choices for g to test the hypothesis
2.p G {10, 11, 4-p 3,p 3.p-2.p 1.p).
G2 = {3....66]U
G3 {10, (10...., .0.....,99,7-- 149.....
.0, ,P]
G1 =
Note that In and G2 are small groups, G3 and G4 are large groups 91 and G3 consist of only zero coefficients, while G2 and G4 includes nonzero elements.
Then we report the numerical results empirical rejection rate (ERR) for different scenarios, where ERR is the proportion of rejected hypotheses among the total 500 simulations. For G1 and G3. the ERR is the empirical type I error; For G2 and G1, the ERR is the empirical power.
From the numerical results for all scenarios displayed in table I we have the following findings. Firstly, when there are outliers or heavy errors, our methods still have very high powers and control empirical sizes well. Secondly, for both linear model and nonlinear model, small groups and large groups, our procedures have similar satisfactory performance this experiment. Thirdly, we find that the dimensionality of all predictors does not effect the empirical performances. Our test procedures with LASSO penalty perform similarly to those with SCAD or MCP. Therefore, we only present the results of test statistics with LASSO penalty. We use the R-package novreg [6].
Moreover, we compare the proposed methods with the three-step testing procedure based on the studentized statistics in [59], denoted as ST. Here, we use R package SILM [59] to implement the threc-step testing procedure, set the number of bootstrap replications as 500 and choose the splitting proportion of 30% for screening.
Table 2 summarizes the results of empirical type errors and powers with a 0.05 for different methods and models based on 500 simulation runs (due to the computation limit, we only run 200 times for ST). Here we consider (n.p) = (200, 400), and other settings are same as before. It is obvious that our methods outperform the ST. Firstly, when there are no outliers and heavy errors, all methods have empirical power 1 for linear model. But for non-linear model, our test procedures T((1) and (2(2) are more powerful than ST. For instance, under non-linear model with original data and N (0, 1), the empirical powers of ST for G2 and G3 are 0.785 and 0.430, while the corresponding
8

---

Table 1: Simulation results for the group inference problems
<~ N(0, 1)
é < t(1)
G1
G2
Gs
G4
G1
G2
G
Y2
Ori.
Out.
Ori.
Out.
Ori.
Out.
Ori.
Out.
Ori.
Ori.
Ori.
Ori.
Y
X
B +
200
0.028
0.034
1.000
1,000
0.026
0.046
1.000
1.000
0.026
1.000
0.058
1.000
500
0.018
0.028
1.000
1,000
0.024
0.040
1.000
1.000
0.032
1.000
0.054
1.000
Y exp(X
3 e)
200
0.018
0.024
1.000
1.000
0.030
0.052
1.000
1.000
0.044
1.000
0.084
1.000
500
0.034 0.040
1,000
1.000
0.026
0.040 1.000 1.000
0.030
1.000
0.028
1.000
empirical powers of our methods are all 1. Secondly, the outliers and heavy errors have dramatic effect the power performances of ST, which has very low powers For example, under linear model with outliers, the empirical power of ST for 94 is 0.050 and that of our proposal is I; under non-liner model with Cauchy error, the corresponding empirical powers of ST and our proposal are 0.040 and I, respectively. Thirdly, all methods can control type errors reasonably. Overall our methods have better powers and are robust with respect outliers and heavy errors, whereas ST in [59] can completely fail when outliers exist.
low as the nominal level.
Table Simulation results for our method and ST.
(0,
f(1)
G1
G2
Go
G2
91
G2
Gs
Gi
Method
Ori.
Out.
Ori.
Out.
Ori.
Out.
Ori.
Out.
Ori.
Ori.
Ori.
Ori.
Y
X B B+c
Our
0.028
0.030
1.000
1.000
0.038
0.044
1.000
1.000
0.012
1.000
0.050
1.000
ST
0.045
0.040
1.000
0.115
0.015
0.040
1.000
0,050
0.055
0.370
0.055
0.250
Y exp(X
c)
Our
0.032
0.054
1.000
1.000
0.034
0.070
1.000
1,000
0.042
1.000
0.054
1.000
ST
0.025
0.040
0.785
0.045
0.060
0.075
0.430
0.080
0.045
0.085
0.035
0.040
5 Discussions
In this paper, we investigate robust inference for high-dimensione single index model (SIM). Under the linearity condition about the predictors, we recast the SIM into pseudo-lines model with trans- formed response. A response-diss distribution transformation is considered. This transformation choice avoids the sub-Gaussian assumption for the response. Our introduced procedures are thus robust with respect outliers or heavy-tailed errors. We develop asymptotically honest group inference procedures. Asymptotic distribution and bootstrap implementation are both established. For testing the individual coefficients simultancously, multiple testing procedures are proposed and shown to control the false discovery rate asymptotically. Our numerical studies illustrate the robustness of our procedures against outliers or heavy errors.
In this paper, the linearity condition about the predictors plays a very important role in the methodol- ogy development. How to make inference for nigh-dimensional SIM without this condition is of great importance and interest. The second-order Stein's 's method with score function-basee corrections in-
vestigated in [56] and (55) would be a powerful alternative. The condition Ki,, * excludes even link functions, and in particular the problem of sparse phase retrieval. [39 introduced novel procedure to deal with the problem of sparse phase retrieval when Ka It would be of interest extend their
approach to make robust group inference. Further paper doesn't 't provide quantitative bounds on

---

robustness. It also of interest to make inference for partially linear single-index regression model We will pursue these challenging problems in near future.
[5].
References
[1] Pierre Alquier and Gérard Biau. Sparse single-index model. Journal of Machine Learning Research, 14(1):243- 2013.
[2] Alexandre Belloni, Victor Chernozhukov, Denis Chetverikov, and Ying Wei. Uniformly valid post-regularization confidence regions for many functional parameters in z-estimation frame- work. Annals of Statistics, 46(6B):3643 3675, 2018.
[3] Alexandre Belloni, Victor Chernozhukov, and Kengo Kato. Uniform post-selection inference for least absolute deviation regression and other z-estimation problems. Biometrika, 102(1):77-94, 2015.
[4] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1))289-300, 1995.
[5] Jelena Bradic, Gerda Claeskens, and Thomas Gueuning. Fixed effects testing in high- dimensional linear mixed models. Journal of the American Statistical Association, 1156532):1835-1850 2020.
[6] Patrick Breheny and Jian Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications biological feature selection. Annals of Applied Statistics, 5(1):232-255) 253, 2011.
[7] Peter Bühlmann, Markus Kalisch, and Lukas Meier. High-dimensional statistics with view toward applications in biology. Annual Review of Statistics and Its Application, 1(1):255-278, 2014.
[8] T Cai, Weidong Liu, and Yin Xia. Two-sample test of high dimensional means under dependence. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 76(2):349-372, 2014.
[9] Tony Cai, Zijian Guo, and Rong Ma. Statistical inference for high-dimensional generalized linear models with binary outcomes. Journal of the American Statistical Association, pages 1-14, 2021.
[10] Tianxi Cai, Tony Cai, Katherine Liao, and Weidong Liu. Large-scale simultaneous testing of cross-covariance matrices with applications phewas. Statistica Sinica, 29:983-1005, 2019.
[11] Mehmet Caner and Anders Bredahl Kock. Asymptotically honest confidence regions for high dimensional parameters by the desparsified conservative lasso. Journal of Econometrics, 203(1):143-165 2018
[12] Xiaohui Chen. Gaussian and bootstrap approximations for high-dimensional U-statistics and their applications. Annals of Statistics, 46(2):642 678, 2018.
[13] Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Gaussian approximations and multiplier bootstrap for maxima of sums of high-dimensional random vectors. Annals of Statistics, 41(6):2786 2819, 2013.
[14] R Dennis Cook and Liqiang Ni. Sufficient dimension reduction via inverse regression: A minimum discrepancy approach. Journal of the American Statistical Association, 100(470):40- 428, 2005.
[15] Shijie Cui, Xu Guo, and Zhe Zhang. Estimation and inference in ultrahigh dimensional partially linear single-index models. arXiv preprint arXiv:2404. 04471, 2024.
[16] Ruben Dezeure, Peter Bühlmann, and Cun-Hui Zhang. High-dimensional simultaneous infer- ence with the bootstrap. Test, 26(4):685-719, 2017. 2017.
10

---

[17] Hamid Eftekhari, Moulinath Banerjee, and Yaacov Ritov. Inference in gigh-dimensional single- index models under symmetric designs. Journal of Machine Learning Research, 22(27):1-63, 2021.
[18] Jianqing Fan, Quefeng Li, and Yuyan Wang. Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 79(1):247-265 2017.
[19] Jianqing Fan, Runze Li, Cun-Hui Zhang, and Hui Zou. Statistical Statistical of data science. Chapman and Hal//CRC, 2020.
[20] Ethan X Fang. Yang Ning, Runze Li, et Test of significance for high-dimensional longitudinal data. Annals of Statistics, (6(5):2622-2645 2020.
[21] Ethan X Fang, Yang Ning, and Han Liu. Testing and confidence intervals for high dimensional proportional hazards models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 7965):1415-1437, 2017.
[22] Zhe Fei, Ji Zhu, Moulinath Banerjee, and Yi Li. Drawing inferences for high-dimensional linear models: A dneectio-assistedd partial regression and smoothing approach. Biometrics, 75(2):551-56))
2019.
[23] Zijian Guo, Claude Renaux, Peter Bühlmann, and Tony Cai. Group inference in high dimensions with applications to hierarchical testing, Electronic Journal of Statistics, 15()):6633-6676 2021.
[24] Peter Hall and Ker-Chau Li. On almost linearity of low dimensional projections from high dimensional data. Annals of Statistics, 21(2):867-889, 1993.
[25] Dongxiao Han, Miao Han, Jian Huang, and Yuanyuan Lin. Robust inference for high- dimensional single index models. Scandinavian Journal of Statistics, 50(4):1590-1615, 2023.
[26] Wolfgang Hardle, Peter Hall, and Hidehiko Ichimura. Optimal smoothing in single-index models. Annals of Statistics, 21(1):157-178, 1993.
[27] Oliver Hines, Oliver Dukes, Karla Diaz-Ordaz, and Stijn Vansteelandt. Demystifying statistical learning based on efficient influence functions. The American Statistician, pages 1-13, 2022.
[28] Adel Javanmard and Hamid Javadi. False discovery rate control via debiased lasso. Electronic Journal of Statistics, (((1):2212-225) 2019.
[29] Adel Javanmard and Andrea Montanari. Confidence intervals and hypothesis testing for high- dimensional regression. Journal of Machine Learning Research, 15(1):2869-9009, 2014.
[30] Efang Kong and Yingeun Xia. Variable selection for the single-index model. Biometrika, 94(1):217-2) 2007.
[31] Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414):316-22) 1991.
[32] Ker-Chau Li and Naihua Duan. Regression analysis under link violation. Annals of Statistics, 1009:1009-1002, 1989.
[33] Runze Li, Wei Zhong, and Liping Zhu. Feature screening via distance correlation learning. Journal of the American Statistical Association, 1074499):122--111 2012.
[34] Weidong Liu. Gaussian graphical model estimation with false discovery rate control. Annals of Statistics, (1(6):2948-299 2978, 2013.
[35] Po-Ling Loh and Martin J Wainwright. Regularized m-estimators with nonconvexity: Statistical and algorithmic theory for local optima. Journal of Machine Learning Research, 16(1):559-616. 2015.

---

[36] Rong Ma, T Tony Cai, and Hongzhe Li. Global and simultaneous hypothesis testing for gigh--imensional logistic regression models. Journal of the American Statistical Association, 1 116(534):944-99) 2021.
[37] Nicolai Meinshausen, Lukas Meier, and Peter Bühlmann. P-values for high-dimensional regression, Journa of the American Statistical Association, 104(488):1611-1681, 2009.
[38] Sahand N Negahban, Pradeep Ravikumar, Martin J Wainwright, and Bin Yu. A unified framework for high-dimensional analysis of m-estimators with decomposable regularizers. Statistical Science, 27(4):538- 557, 2012.
[39] Matey Neykov, Zhaoran Wang, and Han Liu. Agnostic estimation for misspecified phase retrieval models. Journal of Machine Learning Research, 21:1-39, 2020.
[40] Yang Ning and Han Liu. A general theory of hypothesis tests and confidence regions for sparse high dimensional models. Annals of Statistics, 45(1):158- 195, 2017.
[41] Yaniv Plan and Roman Vershynin. The generalized lasso with non-linear observations. IEEE Transactions on Information Theory, 62(3):1528-1553 2016.
[42] Peter Radchenko. High dimensional single index models. Journal of Multivariate Analysis, 139:266-28 282, 2015.
[43] Wojciech Rejchel and Malgorzata Bogdan, Rank-based lasso-efficient methods for high- dimensional robust model selection. Journal of Machine Learning Research, 21(244):1-47, 2020.
[44] Chengchun Shi, Rui Song, Zhao Chen, and Runze Li. Linear hypothesis testing for high dimensional generalized linear models. Annals of Statistics, 47(5):2671-2703, 2019.
[45] Chengchun Shi, Rui Song, Wenbin Lu, and Runze Li. Statistical inference for high- ligh-dimensional models via recursive online-score estimation. Journal of the American Statistical Association, 116(535): 1307-1318, 2021.
[46] Qiang Sun and Heping Zhang. Targeted inference involving high-dimensional data using nuisance penalized regression. Journal of the American Statistical Association, 116(535):1472- 1486, 2021.
[47] Sara Van de Geer, Peter Bühlmann, Yaacov Ritov, and Ruben Dezeure. On asymptotically opti- mal confidence regions and tests for high-dimensional models. Annals of Statistics, 42(3):1166- 1202, 2014.
[48] Aad W Van der Vaart. Asymptotic statistics, volume Cambridge university press, 2000.
[49] Roman Vershynin. High-dimenssioall probability: An introduction with applications data science, volume 47. Cambridge university press, 2018
[50] Jingshen Wang, Xuming He, and Gongjun Xu. Debiased inference treatment effect in high-dimensional model. Journal of the American Statistical Association, 115(529):442-444 4554 2020.
[51] Tao Wang, Pei-Rong Xu, and Li-Xing Zhu. Non-convex penalized estimation in high- dimensional models with single-index structure. Journal of Multivariate Analysis, 109:221-235, 2012.
[52] Tao Wang and LiXing Zhu. A distribution- based lasso for a general single-index model. Science China Mathematics, 58(1): 130, 2015.
[53] Yunan Wu, Lan Wang, and Haoda Fu. Model-assisted uniformly honest inference for optimal treatment regimes in high dimension. Journal of the American Statistical Association, pages 1-10, 2021.
[54] Yin Xia, Tianxi Cai, and T Tony Cai. Two-sample tests for high-dimensional linear regression with application to detecting interactions. Statistica Sinica, 28:63-92, 2018.
12

---

[55] Zhuoran Yang. Krishna Balasubramanian, Zhaoran Wang, and Han Liu. Learning non gaussian multi-index model via second-order stein's method. Advances in Neural Information Processing Systems, 30:6097-61066 2017.
[56] Zhuoran Yang, Krishnakumar Balasubramanian, and Han Liu. High-dimensional non con-gaussian single index models via thresholded score function estimation. In International conference on machine learning, pages 3851-3860. PMLR, 2017.
[57] Peng Zeng, Tianhong He, and Yu Zhu. A lasso-type approach for estimation and variable selection in single index models. Journal of Computational and Graphical Statistics, 21(1):92- 109, 2012.
[58] Cun-Hui Zhang and Stephanie Zhang. Confidence intervals for low dimensional parameters in high dimensional linear models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 761122217 2014.
[59] Xianyang Zhang and Guang Cheng. Simultaneous inference for nigh-dimensional linear models. Journal of the American Statistical Association, 112(518):757-765 2017.
[60] Li-Ping Zhu and Li-Xing Zhu. Nonconcave penalized inverse regression in single-index models with high dimensional predictors. Journal of Multivariate Analysis, 100(5):622-87,
2009. 2009.
A Appendix/ / supplemental material
Notation. For functions (() and g(n), we write f(n) S g(n) to mean that f(n) S cg(n) for
some universal constant E (0, oo), and similarly, ((n) Z g(n) when (n) 2 c'g(n) for some universal constant é' E (0, oo). We write ['n) x (()) when f(n) S g(n) and ((n) 2 g(n) hold simultaneously. For a function h R² -> R. write Vh and Vah to denote a gradient subgradient and its k-th component, they exist. The sub- Gaussian norm of a sub- Gaussian random variable X is defined as || inf 0 IE exp((2//t) < 2}. Similarly, the sub- Exponential norm of a sub- Eotp-ppoonnttalll random variable is defined as | := inf(t > 0: E Eexp((Y S 2}.
A.1 multiplier bootstrap test procedure
As discussed in Remark 2.1 the critical value obtained from the asymptotic distribution may not work well in practice since this weak convergence is typically slow and assumptions for dependence structure of the covariates. We next describe a simple gaussian multiplier bootstrap method to obtain an accurate critical value. Let ....... Wn be i.i.d. N(0 1) random variables that are independent of (X,, Vi and define the gaussian multiplier bootstrap statistic:
1
Mf = max
h 5ym (Y, X.), - vn Tas uis
JEG
where
1
h h (Y,, Y, X3)
2 (h,
and
h,,Y, X, Xx) = { 10V SY)
1 - 21,73/ -
Xij X.y--Z1,00
is the kernel funtion in Tay. The bootstrap critical value is given by (a) inf e R a. where Pr., the probability measure induced by the multiplier variables
Pric (M n S 1) 2
)a holding {X,,Y:) Y fixed. We can reject null hypothesis Hoo at the significant level a
and only if Mn. 2 Cg (a).
(2)
Theorem A.1. Suppose that Assumptions A.l
A.7
A.8
and LC condition hold. If s =
of ///////oo p(log 3/2)), we have
lim sup sup n no eneng aE(0.1)
P(Mn.
(2) (a)) - -al = 0. cg
(1.16)
13

---

Remark A.1. Here we assume s s= = p(log (1oogppp))3)33) which is slightly stricter than o(vn/(log plog Po)) in Theorem 3.2 Suppose ÇI is a sequence satisfying /ivIV log(pu/5) /G) + cc. as n oc. This assumption has two purposes. Firstly, making the difference between | lloe
and [|Sm,glls lloo bounded by Çt. Secondly, making Pr M (lyMM,. VLi.ol (1) + 0 in probability, where I the ideal version of M ng and is defined (1277) in the Appendix A.9
Theorem A.2. Suppose that conditions in Theorem A.1 are satisfied, we have for some cu > 0,
lim
inf
Pr Pr(Mn.g 2 co(a)) =
(1.17)
(N.Pa) +00 BA (2E
Where (.) defined in subsection 3.3
The theoretical guarantee of the bootstrap procedure is based the gaussian approximation theory for nondegenerate U-statistic. Symmetrization of the kernel function h-(-) plays important role in theory. Thus i (-) is necessary in our gaussian multiplier bootstrap procedure. Meanwhile, Theorem A. holds whether po is divergent or not. While the limiting null distribution established in
Theorem 3.2 asks po be divergent.
A.2
The detailed form of c3
By the definition of SNR and Lemma A.8 in the Supplementary Material, for given B E R2 we have
Coo X
B3
3,5 2
SNR(M,ng)
(BT 211²
max JEG
Jj
Similarly, we have
SNR(M = Cou2 xT, (3 Σ13)
Bgoj
max JE9
Thus the ratio of (NR(M,..) and SNR(S (t) be simplified
SNR(M,,c) (Fo) (a SNR(M
2 Cov (Y, oz
where j' = argjeg max( (2d In the case of classical SIM, that is, = gX 3) + e and e is
independent of X. We can derive
(at)² E((X, =
-2-P88Y2)-)))122 E(xy 2T.0yy (ar(c).
Thus have
SNR(M SNR(M
33 XX,, - 0z, Var(e),
CooP(Y, X)03.
and
2z0
C3
Coc (Y, (Y,33
A.3 False discovery rate control
In the previous section, we consider joint significance testing of a prespecified subset of predictors. In applications, however, the parameter of interest may be specified in advance. Denote Ho [j: = 0, j = 1,...,,7 .P), and Hi = (j : 33 * 0, j = 1, I,...,p). It is of interest to identify the elements in H1. To this aim, we consider simultaneous testing of the following hypotheses
Hoj B3
versus
H1; 3, + O, 1 15 p.
In this section, we aim develop data-driven procedures to control the FDR.
14

---

Recall that in Section 2, for each individual hypothesis Hloj : 53 = 0 we define the standardized
statistic Tag = , TTn defined in (2.7) At a given threshold level 0, Hoj is rejected if [Tail z t. For each let Ro(t) SEH [[Tny] 2 t). and R(t) -ド / ([Zny] 2 t) be the total number of false discoveries and the total number of discoveries, respectively. Accordingly, the false discovery proportion (FDP) and FDR are
ΣjEHo [(TTngl 20 max (E T(TT 2 1)
(tP(
(D) ((FDP()))
Let (t) be the proportion of the nulls falsely rejected by the procedure among all the true nulls the threshold level I, namely, Go(() = %o EjEN, T(TT,5] 2 1), where ạo |Pool- In practice, reasonable to assume that the true alternatives are sparse, that is, p 40 =: 91 = o(p). If the sample
size is large, we can use the tails of normal distribution G(t) = 2 2b)(t approximate Gt(t). In
fact, it will be shown that, for by = /2logp log log p.
Gt
sup 05t5b,
+0
G(t)
in probability (n,p) To summarize, have the following multiple testing procedure controlling the FDR and FDP at a pre-specified level 0 < a < 1, which summarized in Algorithm
A.
Algorithm 2: multiple testing procedure
Let 0 < a 1, bp /2 log log p and define
pG(t) i i= inl 0 max E I((Tug) 2 1)
S saj.
(1.18)
If in (1.18) does not exist, then let f V2 We reject Ho, whenever [Ty 2f.
Next we show that under mild conditions, our proposed multiple testing procedure control the FDR asymptotically, Recall that S {B,, E RD 11,9% llo m/xxis/spll0 10ylo
S s}. The following theorem
shows that Taj is uniformly asymptotically normal distributed and Go is well approximated by G(t).
Theorem A.3. Suppose that Assumptions 4.7
A.2
in Appendix A.8 and LC condition hold. If (log 2) Then as (n. p) + 0o, for any Br, E S2
A.8
p O(n"), for some constant and 0 v n we have
Pr([T mygl 2 f)
sup jE osts Vo
- 0.
(1.19)
2 24t(t)
If in addition Assumption
A.9 holds, then
Go(t) G(l)
sup SSt
+0
(1.20)
in probability.
In the results given in Section 2, we could allow p to grow exponentially fast in n. However, in order to control FDR, we only allow p O(n") to apply the moderate derivation result in (4. Further which is slightly stricter than = of /n/ (log p log logPo)) to make the
we assume = o( (ol///ooop2 difference between Tn5 and its ideal version S.y uniformly bounded with the rate o(1/ /v/ Vlog p). Note that (28) imposed a similar condition. The following theorem provides the asymptotic FDR control of our procedure.
Theorem A.4. Under the assumptions in Theorem A.3 for f. defined in our multiple testing procedure, have
lim FDR(f) S
(1.21)
(no
15

---

A.4 Discussions of robustness
In this subsection, we carefully discuss the robustness of our inference procedure based on the efficient influence function. We find that our statistics are robust with respect to perturbations in the responses. The details are given below. Recall that our test procedure inspired by the quantity
I : B(((()) - 1/2-1222 - 2j 7x)(3;; 2j 0,)l-
Next derive the efficient influence function (EIF) of We rewrite as
I = (P) Ex((F>(X))
2j 7x 2, 0,),
where P is distribution of (X,, 2x Y) Consider the following parametric submodel indexed by t, L.e.
P. = tp + (1 t)P,
where / E [0, 1l, and P is point mass at single observation ō (øj, 2J .T) As mentioned in 27.. the EIF for at observation is
(P((
$ P)
dt 1=
2, 0z), Denote o = (zj;,z]00) ,y) and = (X,, Y)) Let m.( = (z), - 2/ - 2, 02) and (P,)) = Ep, (m))]
where ((())) - Ep, [1
Further, the operator, 0g applied to an arbitrary function g(t), is defined as
dg(t)
Oeg(t) dl 1=0"
Some calculation entails that
a, (PP) = / am)(()) mm(O)l==od0.P
=El00.m + m (PP)).
Note that
my (O) = (tI(Y 2 + (1- 1/2 2T,;;((x 2,
Thus we derive that
(0) (I(Y 0) Fx(Y))(X; -270,).
and
(pp00m.(()) = Ep((TT( 2 0) - FP(())(X, - 2T0,1]
As (0) (Fr(0) 1/2 2T - 0z), by some caleulations derive
Φ(ö, P) (Fr(i) - 1/2 13) 2T00,) + Ep((T( 2 5) Fx())) 2,0,11 - V(P).
Recall that
Thy vn Σ
(Y.)
1 - 2778 } (Kg-22701 2
i=1
"
{ ((Y)) 2 23/33 } (xis - 21,00 + my (Y3)
Sns vn Σ
where my(v) = EX( zJ (TO > y) F(Y))]. Here Túy is an individual test statistic corresponding to the parameter T. While Sny is individual quantity corresponding to the EIF of T. From our proof, we know that Txj = Snj + (1).
By the formula of (0ā P), given (zy, =T),, can shown that 66,, P) is bounded for any ÿ E lR.
Thus our test statistics are robust with respect to the perturbations in the responses.
16

---

A.5 Additional simulation results for group inference
In this subsection, we conduct simulation studies compare our procedure with other methods based on different transformation functions. The simulation settings are summarized as follows. We consider the following two models:
Model 1: Linear model: Y == - B
Model 2: Non-linear model: Y exp(X B+
The regression coefficients 3 (31, . 3p) are generated as 3 = for 1 6 and 8j = 0 otherwise, where 5 can be regarded as a signal strength parameter. We generate the error term from the standard normal distribution. We add outliers to pollute the observations: Pout of the responses are picked at random and increased by Mout-times maximum of original responses. Specifically, the detailed settings for the above parameters 800. Secondly, we set the signal strength parameter
follows. Firstly, consider the
sample size = 500 and the dimension ó vary from {0.1, 0.3, 0.5). Thirdly, we fix Mout 10. Lastly, we vary Pout from 0.5 in increments of 0.1. For more complete comparisons, we consider three transformation procedures: (1) h(Y) F(Y) (Our method); (2) h(Y) = Y: (3) h(Y) sigmoid(Y) 1/(1 + exp(-Y)). Other simulation settings are the same as described in section 4 of the main text.
Figure summarizes the results of empirical type I error and empirical power with the significant level of a = 0.05 for different methods when the error term follows the standard normal distribution. Firstly, under the null hypothesis (G1), h(Y) sigmoid(Y) cannot control the type error for Model 2, while other procedures perform well. Secondly, under the alternative hypothesis (G2), the empirical powers of other procedures decrease rapidly with the increase of Poats while the powers of our procedure remain stable, which is particularly noticeable when å = 0.5. This finding indicates that our method has strong robustness when the responses are polluted. Thirdly, our method performs well for both Model and Model 2. indicating that our method is robust across different single-inde index models.
Model
Model
Model
M Model
G
G
G
1.00
0.25
Empirical Rejection Rate
0.00
1.00
0.75
0.50
0.25
0.00
1.00
0.75
0.50
0.25
0.00
0.1
0.2
Pour
Transformation F(Y) (Our) sigmoid(Y)
Figure 1: Under different settings of the generated model, the testing group and the signal strength, simulated results of the proposed method the proportion of outliers Pout from 0 to 0.5 increments of 0.1 when the error term follows the standard normal distribution.
17

---

A.6 Additional simulation results for false discovery rate control
In this subsection, we examine the empirical FDR results of our proposed procedure. We also make comparisons with two alternatives. Let BH-V, denote the BH procedure [4] with p-values obtained from n,, BH-W,, is similarly defined. Here V2 and W,, denote the debiased lasso procedure introduced by 17 and the decorrelated score method proposed by [40], respectively. Here, we use the implement the BH procedure. We simultaneously test 1000 and sample size 1000, 1500. The sparsity level [H1] = q1 is set as 4 or 8. The data is generated from the following linear and nonlinear models:
function p. addjjus(( (.... method="BH")
Hloj J E Ho for all j = 1, P. We set the dimension
Y x B +
Y sin (0.5X (3 exp(X 1+1)+ tc,
where 3 = 31 with 6n 1, that is 3 (1,... 0, .0T and e comes from N 1) or t(1).
We evaluate the empirical FDRs and the empirical powers of our test statistic T, and two existing high-dimensional inference procedures BH-V,, and BH-W,, Here the power is defined as the number of correctly discovered variables divided by the number of truly active variables. We set the desired FDR level as a = 0.2 and the results are calculated based on 500 replications.
We illustrate our proposed procedure under different models, error distributions, sample sizes, and sparsity levels. Table shows the simulation results. We have the following important observations. Firstly our introduced procedure controls the FDR well, even when the random error comes from Cauchy distribution. Secondly, our method has power indicating that the proposed method can detect all important predictors. Thirdly, the other two methods lose their powers when random error follows Cauchy distribution and sometimes fail to control FDR.
Table 3: Empirical FDRs and powers of proposed method, BH-V, and BH-W,, Here the dimension = 1000 and nominal FDR level a = 0.2.
Linear model
Non-lineas model
1000
1500
1000
1500
Sparsity
Method
FDR
Power
FDR
Power
FDR
Power
FDR
Power
((0,
1)
Proposed
0.188
1.000
0.187
1.000
0.197
1.000
0.203
1.000
BH-V,
0.245
1.000
0.235
1.000
0.198
0.773
0.206
0.730
BH-W,
0.133
1.000
0.122
1.000
0.125
0.851
0.125
0.879
8 Proposed
0.194
1.000
0.198
1.000
0.211
1.000
0.211
1.000
BH-V, 0.243
1.000
0.230
1,000
0.458
0.011
0.248
0.005
BH-W,,
0.138
1.000
0.142
1.000
0.097
0.028
0.122
0.041
((1)
4 Proposed BHIVV,
0.191
1.000
0.196
1.000
0.196
1.000
0.213
1.000
0.328
0.017
0.283
0.013
0.249
0.448
0.204
0.446
BH-W, W.. 0.121
0.028
0.104
0.032
0.127
0.628
0.134
0.664
Proposed
0.204
1.000
0.206
1.000
0.212
1.000
0.202
1.000
BH-V,
0.299
0.019
0.244
0.013
0.460
0.014
0.245
0.006
BH-W,
0.087
0.022
0.113
0.031
0.114
0.024
0.093
0.050
A.7 Real data analysis
In this subsection, we illustrate the practical applications of our procedures by a real data analysis. The group inference is helpful to decide whether group of predictors are important or not for the response. If we find group of predictors are important, we would like to know which specific predictors in the group are significant. For this aim, our developed multiple testing procedure useful. For instance, researchers may aim to test whether gene pathway, consisting of high-dimmensioal
18

---

genes for the same biological function, is important for certain clinical outcome, given the other nigh-dimensional genes. When determining that certain gene pathway is important, researchers need to further identify specific genes within the pathway which are important for a certain clinical outcome.
We apply our methods on dataset about riboflavin (vitamin B2) production rate with Bacillus Subtilis. This dataset is made publicly by 7 and has been analyzed by many authors, for instance (37), 47
[2]] and 22] The dataset riboflavin can be obtained from the R package hdi. It consists of n 71 observations of strains of Bacillus Subtilis and p = 4088 covariates, measuring the exgeeessssooon levels of 4088 genes. The response variable is the logarithm of the riboflavin production rate.
Our goal is to detect which genes associated with riboflavin production rate. Like most existing studies, we first reduce ultrahigh--imension to a moderate high-dimension. Here we pick out first 300 genes by distance correlation based screening [33] We first conduct global testing on these 300 genes. The p-value of our group inference procedure 1.286e 04, indicating that the null hypothesis is rejected and the selected 300 genes are influential for riboflavin production rate. Next, we further use FDR control procedure to select the important genes in these 300 genes.
By implementing our proposed FDR control procedure with the FDR level of 1. we identify 10 genes that are significantly associated with the response. That is 9r YTGG at, YCKE_ at, YXLE at, YXLD at, YJCJ XHLA at, xepA at, YCGO RPLP at, XKDS at). If the FDR level is set as 0.2, 5 more genes will be selected. That is G1 = | at, YHCB at, XKDI at, YJCF at, XHIBBat))
We further conduct group inference on the selected subsets Gr. 911 and their complement sets gj. 9ir- As expected, our group inference procedure finds again that Gr, Grr are significant while 9j. 9j, are not. The corresponding p- values Gr, 9j. G5r and Gi are 6. 6.748e-06, 7.257e-01, 9.334e-06 and 9.370e-01, respectively. These results suggest that the genes selected by the FDR control procedure are really influential.
We compare these selected genes with other methods, For example, the multi multi-samples sample-splitinng method proposed in (77 identified YXLD at; [47] did not select any gene using the de-sparsified Lasso; [29 only selected two genes: YXLD at and YXLE at and [22] claimed YCKE at. XHLA at, YXLD at, YDAR_a at and YCGN at as significant. From 9, and 9I1, we can see clearly that the gene YXLD is detected by not only [37, (29. 222. but also by our procedure. Besides, the genes YCKE_ at, YXLE_ and XHLA which are detected by [29] and [22), are also found by our method. Further, procedure detects some additional important genes.
A.8 Technical assumptions
Assumption A.1. We assume that
(i) Xz. x Br and 0z j = 1.....P are all sub-Gaussian with uniformly bounded sub- Gaussian norms.
(ii) c/ loge S Ax S CVV logg for some constants 0 < C. n
(iii) Σ Cov O.
The above conditions are mild and widely assumed in the high-dimensional literature. Condition (i) 0,),, in the proof. Sub-Gaussian assumption
used to obtain (oo norm of Vn 1 Em,(X,, is a standard condition while working with random predictors in nigh-dimensional models [49]. However, should emphasize here that do not make sub-Gausee Gaussann assumption for the error term (or equivalently for the response), which is widely assumed in the literature of high-dimensional inference 440 [7]. Actually we even do not assume the existence of IE(e).
Assumption A.2. We assume that
(i) The function px satisfies px (0) 0 and is symmetric around zero (i.e. px(t) px(- ((t)). for all t E R.).
(i) On the nonnegative real line, the functiinnnnnnn px is nondecreasing.
(iii) For t > 0, the function (x(t) is nonincreasing in t.
19

---

(iv) The function Px is differentiable for all t o 0 and subdifferentiable at t 0, with lim + (t) for some L
(v) There exists μ 0 such that (t) px(t) + 2t2 is convex.
Many commonly used penalty functions PA satisfy all the conditions in Assumption A.2 such as LASSO, SCAD and MCP penalties. Proofs can be found in [35]. By the result in [35] wc have the following proposition for 9. Assumption A.3. The RSC condition in (35 is
log A I
an 01
V||A|22 S 1, (1.22)
TI
(B + A) VB,
log
02|| V2\
||
v|A 2 2 1,
(1.23)
where the a1 02 are strictly positive constants, the T1: T2 are nonnegative constants and the A E IR a p-dimension column vector.
Here denote
-1//733 {xxx
[xy- X.j - 2.J0,) + my)
Snj =
F(Y)) 2
and Sng oj ooj'Sns where m2( E[(x; 2, (I(( 2 v) - F(Y)) It could be shown that Snj = Tnj + (1) and Var(Sng) = 1. Denote 5, = (5n)) and A Var(S,). where Ays= = COv(Sns, Snk) is (j, k) component of A. And denote Sn.g = (Snz jee and Ag = Var Sn.g) correspondingly. Further denote 3A E Rº : 11B% lo V max1s/5p 10, llo S s} and Bng = (Bnj)
Assumption A.4. (log(npo))7/n< o(1).
Assumption A.S. Co < Amin(A) S Amax (A) S Co for some constani Co > O.
Assumption A.6. max1gjfk$p |Ajal I cu < 1 for some constant > 0.
Assumptions A.6 mild and commonly used in the high-dimensional settings. Assumption is the technical condition to bound the difference between ||S,.g loo and ||Ngllz where Ng Npo (0, Ag). Specially, Assumption A.4 imposes suitable restrictions on the growth rate of pu and it is commonly used in the group interence literature, such (59) and [16]. Assumption [A.5 and A.6 are used to establish the limiting distribution of ||o. Assumptions A.5 requires that the eigenvalues of Ag are bounded. Assumption A.6 is widely used to establish the limiting distribution
of the max type statistic in the high-dimensional settings, see [ [54] and (36] for further examples.
The validity of the above bootstrap method requires the following assumptions. Assumption A.7. {log(mpo)) 8 /n o(1).
Assumption A.8. min SJ B( 2 Cmin- where Cmin is positive constant.
Assumption A.7 and A.8 are the technical assumptions needed for the gaussian multiplier bootstrap method. Assumption A.7 is imposed for controlling the estimation error of gaussian multiplier
bootstrap statistic M Theorem A.1 below establishes the honest property of the bootstrap
procedure for the statistic Mn,g-
Denote B(aa) = kG |Ayal 2 a, i,k: Ho.j + k) and A(b) = (((ogg))-2-
Assumption A.9. Suppose that for some b > 0 and > 0,
ta
Σ
Ask
= =0 =0² (p>> (logp)2).
p
Remark A.2. Recall Ask is the covariance between Snz and Suk- Thus, A(b) contains all the strongly correlated pairs and Assumption A.9 requires that the number of these pairs cannot be
large. If [A sul & 0g for some constant O < ag $ 1, then this assumption holds under AA(b)
(pp 2 -9 19/000//00p /(log)))) for some b > 0 and g > 0. Similar assumptions were considered
200
and (36].
20

---

A.9
Proofs of theorems
The proof of Theorem 3.1 We denote
Gr arg min E.,, (3x)) B,ER*
where (Bh) is defined in equation (2.6). By triangle inequality of and norms, it suffices to show that the 1 and norms of Br 3r and Br 3r, satisfy the requirements in this theorem respectively.
Firstly we calculate I and la norms of difference between 3r and 3r. The results are presented in LemmalA.2
Secondly, by Lemma A.8 in Supplementary Material and Theorem 1 in 43 we can show that B, (n - 1)5a/n, n, and thus
1 Ball2 - -11,/112. and 11, - Bhlli -113/111.
By the sparse assumption, Theorem is proved.
113xll2/n - oorer logp/n) and [/alll/n = olsv /lggp/n). Thus
The proof of Theorem by ||N loo with Ng
3.2
By Lemma A.3 the distribution of maxjeg [Tnjl can be approximated (0, Ag). By Lemma 6 of R we obtain
Npp
1 1 exp(-x1/0
lim (v. 700
Pr(Mu. 2logro Po + log lo <) exp
V
Note that all the universal constants do depend on n, p, Bn. and 02, j -11, I,..,,P. Thus this theorem is proved.
The proof of Theorem A.1 For the convenience of the proof, we introduce some notations. Denote Ung
Unj n(n
h (Y, (Y, X:),
(1.24)
Isidtin
where
1f
h5 11500 (Y, X3) hy
(1.25)
and
- ( (.,021,,0)
hy Y3 X:) 1(%
(1.26)
Unj- Let wi...., wn be i.i.d. N(0, 1) random variables that are independent of (X,, Y.l and denote L L. as:
Denote
Ung Unj- Unj plays important role in the proof of this theorem.
n
L1 max n.g
h sym (Y, X,; X.: Yy,
Uni ym
(1.27)
JEG
where Unj and he""() (.) are defined in 1.24 and (.25). The proof follows from the general results for the multiplier bootstrap of U-statistic in [2] Here. we extend these results for approximate high-dimensional U-statistic by results in [ and rewrite it in suitable form for our analysis. It is now presented in Lemma A.22 in the Supplementary Material. Suppose Çi is sequence satisfying / v log(po/(1) 0 as n + s0. According to Lemma A.22 prove this theorem, suffices to prove
||
||S (p() and Pr (1 Mi T 2 ()) - 0. We denote
21

---

S1 ((log(npo)) 1/2 where C' is sufficient large constant. The proofs of these results are presented in Lemmas A.17 A.18 in the Supplementary Material.
With Lemmas A.17 A.18 18
and A.22 in the Supplementary Material, we obtain
| > &e (a)) a o(1)
sup ac(0.1)
when looo,g holds. Note that all the universal constants in the proof do not depend on n, Br and f. j ...... p. We thus have
sup sup P(Mn. > (a) a o(1)
Breng oE(0.))
The proof of Theorem 3.3 Using the inequality 20102 E aî ca for any > have
max jEG
o,) s (1 c)Mngg +1 + E - max
of (Tnj VnBrzo8)²
(1.28)
JEG
Denote no eax oj (Tnj vn maxjec of 2 (S,j o, and no
Op (o)oogg)) by Lemma A.16 in the Supplementary Material. We can derive
lim inf Pr Prmmax oj (Tin v(iBnoo,)² 2 loggpo Po + loglogpt Po S St)
(n.po) Ba (2+ (2+c0)
jES
lim inf
Pr(max oj [Snj dn 2logro + loglogpo 26-10 s6-1G) jEg
(n Ba (2+(2))
lim (n.po) +00 (2+co)
inf
Pr(max(o) - 83) vn 2log + log log Po S t - ng)
jEG
1
= exp
exp( exxp(tt/2) VT
The last equality holds by Lemma A.19 A.20 in the Supplementary Material and Lemma 6 of (8]
Denote t = log log Po/2, it implics that
lim (n.po) 9, (2+
inf
Pr(max o, (Tny VnBn5d,)2 < log log JEg
Po/2) logloogp)))))) 1. (1.29)
For any Br E (2 + co). applying Lemma A.14 the Supplementary Material, we derive
max
O6 1 On) = (1 + (1)) max(o, V(Bnno,)) > (2 + 60) log Po JEg
(1.30)
jEg
holds with probability (n, Po) +
Substitute (129 and 1.30 into 1.288. we derive that
1
Mn. 2
max (0_ m VnBrz55)2
max 3 VnBr55,)² jeG
+e JEG
(2+ co) log
2 Po
log log Po
+
1te
2e
Now prove the validity of Cg
(2) (a), 1,2. Note that og (a) 2 log Po by the def-
inition of of (a). By the properties of Gaussian multiplier bootstrap statistic(Lemma A.22). C (a) is equal to the (1 - a)t quantile of maxjeg 0j 2 (Tng - Vnßrzōō)²² asymptotically. Thus inf a. (2+co)) Pr(2 log > (2) (a)) = by (1.99) By choosing (eo, e) satisfying
(i. inl
2t.
Ite 3 2,
lim (2 (2+co))))
inf
Pr(Mn,
c3 (t) (a) 1, 1=2
can be proved.
22

---

The proof of Theorem A.33 At first we calculate the uniform bound of the difference between constructed statistic Taj and its ideal version Snz. The result is presented in Lemma A.4
For (1.19.. by Lemma 6.1 in 34 we have
Pr( 21) G(t)
max 1S 0S152
sup
-1 S C(logp)
2)
(1.31)
for some constant 0 < 71 < 1/2. So (.199) follows from Lemma A.4 and the fact that Git 1)/))) = 1 + o(1) uniformly in s t 22
For 1.20 it suffices to show that
ESERO [[15n]] 2
sup 05t5t,
+0 in probability.
(1.32)
G()
Let 20 < 21 < 2d, S 1 and t; = (zi), where 20 G = = e// /p with
1/2
C pG (6p). and dp { log (p-cp) cp/a
and 0 < v < 1, which will be specified later. We have
(1.)) /G (1++1) + (11 uniformly in and to/ / 22log log (p/cp) = + o(1) Note that uniformly for S S m, G (t.) (G (t - as + oc. The proof of (1.32) reduces to show that
Σ HjE 1(1kng] 2t) 1 + 0 in probability.
max d
(1.33)
(00((4)
In fact, for each >> O, have
EjEno [1([5no] 2 - G((t))
E ESEno (1(i 2 (4) G(4)}
Pr max (SiSd
26 5 Pr
2 2</2)
(tG()))
goG(tt)
Set [(1)
EEEN,
By Markov's inequality (r( 2 e/2) S
B(1
(a((()
e/4
and it suffices to show
Σラ= [[T(2 = o(1). To see this, by (1311.
(t) Ejeno [Pr([Sns] 21) - Pr (15mg 2 2)}
(t)
Σ EjkeHo (Pr((Sng) 2 t, Bmdl 21) - PP(([ 2 2)) PI 2 t)}
C
Pr([5ngl 2 1, 1,15 21)
Σ
Got
G2
2 1,11, | 1,sk 21) -1}
Σ
+
(2(()
C
[n(()) Ii((t).
G(t
For (J,b) A(b) with j. k E Ho, applying Lemma 6.1 in 34
we have 112(t) < C(logp) -I-E
-1-- for
some { > 0 uniformly in 0 < t < V2logp. p. By Lemma 6.2 in 34 for (j,k) E A(b) with j. k E Ho.
we have
t²
Pr([Sng) 2 1, 13mm 21) S C((+1) exp
I AA
So that
t2
2IA
(n()) cc
Σ
(L 1 exp
G-2(1) S c
Σ
G(L)
1+1
T AAjk
%
23

---

Note that for 0 I - bp. we have G(t) 2 G (bp) = cp/P. so that by assumption A.9 it follows that for some b, q >
P|Agul +
1n(t) SC
E
p T
= =0(1)((og)))
By the above inequalities, we can prove
1.33) by choosing 0 < < 1 so that
4
d
(1T(
& C
Σ
(pG(t)))
+ Cdp 1- + (logp) }}
dp
$ CC)
+ o(1)
i= Cp +
2/3 Cp
c
=1).
The proof of Theorem A.4:
We first consider the case when f, given by (1.18), doesn't 't exist. In this case, i = 2 2ogg p and we consider the event Fo = <(jjH [[[Ty] 2 v2 2 1}, which mean at least one false positive. In order to show the FDR/FDP can be controlled this case, we show that
Pr(Fo) 0,
as (n,p) + oc.
(1.34)
And we have
Pr(Fo) s Pr
Σ I[Taj 2 22 21)+
+ Pr Σ
I[Tng S -2 21)
jEHo
jeHo
=: Pr(F)) + Pr(F2).
(1.35)
For any e O, we can bound the first term by
Pr(Fi) = Pr
Σ
(SS.. + Tnj - Snj 2 22log)) 21)
jEHo
Σ
I(Smj 2 22lugp-0 -c) -c)21 + +Pr
max jEHo
[Tnj Snyl 2
JEHo
max Pr
3nj /2logp-64 Pr
max Tri -Snl 20) JENU
JEHu
By Lemma A.4 we know that Pr(maxjeeM [Tnj Snjl 2 s) - 0. For simplify, we rewrite
S.; Σm= Eij/ vn. Since E(En))
0, Var(Si) = 1 and (E,) a sequence, by Lemma
Pr( (2t)) S C(logp) Now let 22logp - c, we
6.1 of 33), we have suposisa
G(t)
have
Pr(S.n 2 /2logp - <) S G(V2logg t) C G G(V2logg -)
log
for j E Ho uniformly. Hence p maa Pr(5nn 2 2logg c) S (C + 1pp((22oog Pog c). which goes to zero as (n. p) + cc. By symmetry, we know that Pr(F2) in [.35) also goes to 0. Therefore (.34) proved.
Now consider the case when 0 S i < bp holds. We have
FDP(1) = Σjeno I(Tnjl 2
G0)(1) max EE T(T 2 1), 1] max 1E/= T([Tny) 2 1), 1}
(1 + Ap),
(1)))),
where Ap = supogesh,
Ejeno I(fTm1129) 121) (o)(()
Note that by definition max
gn G(E)
S 900
(1TTl120,111
The proof is complete if Ap + 0 in probability, which has been shown by Theorem A.3

---

A.10
Some technical lemmas
For the convenience of the proof, we introduce some definitions. Denote Ung as
Uni n(n 1) 1Sifisn
1
Σ
where
1
( 1-/j73 ( N6z-21500
NYYY, 2
(1.36)
{1XX- S Yj)
1 4
(
-2,
And denote Unj a; Unj. Uni plays important role in the proof of main results. Let ax be the sparsity level for parameter 0 sx maxisisp 0yllo-
Lemma A.1. Under Assumptions A.1
and A.2
the 0, defined in 2.5 satisfies
110, 0ill1 I I cAxsx
with probability at least 1 cı exp(-c2 log p). Here, (c, (1, 02) are universal constants.
Lemma A.2. Under assumptions in Theorem 3.1 Bn defined (3.11) satisfies
||,; B 1 S / and || Ba S dorrs
with probability at least ai exp( 02 p). Here Br = arg ming, eBe (B,)((k) with (3,)
defined in equation 2.6
Proof. The main part of the proof is to verify RSC condition in assumption A.3 and get exponential inequality of IIV (Bx)ll Based these results, use Lemma A.5 prove theorem A.2
Thus, we focus on RSC condition at the beginning. Note that for E TR.
E, (
[V (B, + A) VL, (3,)) A
1 x A)
From Proportion 2 in 38 we then have
log P |
E, (A) 2 a 51
v|Al2 S 1
(1.37)
n
with probability at least 1 Ci exp( for appropriate choice of ar By the arithmetric mean-geometric mean inequality,
log P
a1 2
Ti log
|
2a
n
and consequently,
7
log
(A)
A1
A
201
which establishes (1.2 in assumption A.3
As square loss function is convex, condition (1.23) then follows via Lemma 8 in 35 So we verify RSC condition completely.
Next, construct exponential inequalities of V (Br). ). By conditions of lemma A.5 it suffices to show that there exist universal constants (c, C1, 02) such that
log
Pr (VC.. 2 2/
< ci exp(-c2 log p).
(1.38)
n
By lemma A.7 we can choose proper (c, C1, such that both RSC condition and inequality (1.38
satisfied. As Ay proportional to log the claimed 1 and lg bounds then follow directly from Lemma A.5 by choosing R proportional to
25

---

We only need replace G with Ho and Lemma A.3. Under the assumptions in Theorem 3.2
lim sup suppPr(maa) Brpaxr(max]7T s 1) PPr(INNolso s1)
n=000 Breng tell 5E9
where Ng ~ No Ag)
Proof. By Lemma A.16 we obtain
mmxx SE9
Tayl
max Snjll S max JE9
max TTg - 3nyl & Cq(n,p)
jE9
for some constant C, with probability as (n,p) + xc. And ((n,)) is sequence satisfying
q(n, p) ((log Po) 1//2). This implies that
Pr(max Tnjl = - Pr(lNsl 1) SPr <Prmmax|S St+ Cg(n,))) Pr([INg St Cq(n,p)) jE9
+ +Pr(IN + Cq(n, - Pr( S
=hh + 12.
By sub-Gaussian condition in Assumption A.1 and Assumption A.8 limsup SUPIER 11 S
By the Gaussian anti-concentration inequality in Lemma [A.20] we derive that supteR 12 S Cq(n, / (n,,, (n) for some constant C. As q(n, p) = of (log (ologpo)-10 we derive that
lim sup (Pr(( mmx Tayl = Pr([|Nglle +1) s 0.
n-000 IE jEg
Similarly,
lim inf inf (Pr (max
Tnyl 5t - Pr(l1Nslo St)) 20.
EER
JEG
This completes the proof.
Lemma A.4. Under the assumptions in Theorem A.3
[Tjj
-Sngl Opl
max
jEHo
Vlog"'
Proof. The proof of this Lemma is similar to Temma A.16 replace Po with p.
Lemma A.5. (Theorem 1 in (35)) Suppose the regularizer Px satisfies Assumption A.2 the empirical loss Cn satisfies the RSC condition with 1 < an, and 3* feasible for the objective. Consider any choice of X such that
4 max (V llac 021
Togp
02
<
<
(1.39)
L
6RL
and suppose n 2 condition (3.11 satisfies the error bounds
16 maxt 73-22) oz
log p. Then any vector B satisfying the first order necessary
6ALvk
24ALk 4ar
||3 - 12 _ 4x 3u¹
and ||3 - 30 <
(1.40)
3μ
where & = 11
26

---

Lemma A.6. Consider U stattistisss
U
h(Zi,Zz)
n(n 1) i7j
with kernel h based on i.i.d. random variables Zi...., Zn Then for Vss E R.
N
E Exp IBU) S E Exp
Σ N
(MiZi, Zi+N) BU}
where = [ which represents the greatest integer less than or equal to 2"
Proof. It can be seen in the proof of Lemma 14 in
Lemma A.7. Under the Assumptions A.j and A.2
there exist universal constants (c, C1 02) such that
Pr
( 2 eo
log S c exp(-c2 logp):
(1.41)
where V (3k) is the gradient of f (Bx), which is p-dimension column vector.
Proof. Suppose (aLn((n)) is the k-th component (G.... Note that
n
Nik x 3n (Fa 111
(3h))
n
X B3 (F(M) 3
X
X,x
F(Y.)
X1h
X Br - (F(X))
X T(, S Y,) F(Y))
2
1=1
X1 X Br (F(Y))
311
n1
n-1
Σ
Xa S XY) -
n
n(n - 1)
1stAisn
Xa(1-F(M))
n2
where I(.) is an indicator function. So we decompose ((C, into three parts:
(B = X X (F(Y) - } n-1
A
X (1 - F(Y)),
n 1=1
where Ag is a Ustatiistics with kernel la (11, 21 x22) = ] [(12 I sn) F(m)) +
122 ((1n < y2) F(zz)}) Then
Pr (n 2 cv log
< Pr
[Ak 21
(1.42)
7
1
- -
Pr
Xix [T Br (F(())
(1.43)
Xa (T((1-F())) F(Y3)/21)
Pr
Σ
(1.44)
n
1=1
with = logg
27

---

Firstly, for probability 1.42
For Vs E IR
Pr(AA 2 1)
exp( exp(-st)]E{(e (exp((x(ss))
hx Xux YN+o X X(rssN)]]
S exp(-st)E
exp
N
I exxp(=st)p
exp
(Xi
2N
CiK2
2²
I exp
st +
2N
where C1 is an absolute constant,
max; sksp pp XI*l| and N 15J. Note that (1) holds by
Markov inequality, (2) holds by Lemma A.6
Denote 8 KIKT.
Nr
we have
N
Pr(A+ 2 S exp
K2
Similarly, we have
Nt
Pr([Ap] 1) 2exp
(1.45)
K2
Secondly, for probability (1.43). As both (x. Br.
}))² and {X are sub- Gaussian random variables, {X1x [ Bn - (F(Y) })]} is zero mean sub- Eo--Eppponnntalll random variable sequence with sub-Euubueeo Exponential norm | 3, (F()) - 11, - KK', where K' = | Br (F(Y)) - H1 Then by inequality for r.v. with sub-erponential sum (Corollary 2.8.3 in 49
(F(Y,)
Pr n Σ
N
X 3n (F(V) 11 21) s 22c0p( -C2 min
K2K KK
nt? nl
)} (1.46)
where C2 is an absolute constant.
Thirdly, for probability (..44). Using general Hoeffding's inequality (Theorem 2.6.3 in 49 directly,
Pr
Σ X 1
n
F()) X 2 S Pr
Xik
2 21) exp -Oss
n2t2
(1.47)
n
n2
K2
where C3s is an absolute constant. As exp( nn = exp (n)) probability (I.44 is trivial compared to other two terms. Combining inequalities (I.45). (..66) and (.47p. have
Pr [VVC, 2 201
logg
log P n
Pr
(a 2 ov IskSp
max
n
log P m
$ p max Pr ( cv
IskSp
< Cpexpp-Caa logr).
where C₁ = min 360, kz 9K ). For given constant c, we can choose arbitrary constants
C2c²
Ci 2 4, C2 S C4 and inequality (I.4T) is satisfied.
Lemma A.8. Under the assumptions in theorem 3.7 then
G Cov(F ( X)
B
ETE
Proof. The proof of this lemma similar to the proof of theorem 2.1 inequality and LC condition, we can get this result. We omit the proof.
32 Using Jensen's
D
28

---

Lemma A.9. Suppose PAx satisfies Assumption A.2
And under the Assumptioo A.7
we have
Zn
S LAx,
where 0g is defined in 2.5 and L is a constant only depend on pxx
Proof. By the property 0, we have
(Xn -2,0,) = Vpxx (0,k)
with probability 1. Under assumption A.2 and by Lemma 4(a) in [35 all subgradients and derivatives of pxx are bounded in magnitude by LAx, that is,
max Vpx (0,x) S LAX. ISAS
And this lemma is proved.
Lemma A.10. Under the Assumptions A.1 and A.2 for Br E (Bx, 11 2 vlog p/n], there exist universal constants (c, C12 C2) such That
logp
Pr
Zug (Y,) - 1/2 -.
2 c
S c1 exp((c2llog))
n2
Proof. At first we decompose
- ZijtFa(Yi) - 1/2 - 2 733 as
n
Ziy (F)() 1/2 21,7,} Vc,((3) 1 n
Σ
Bhg Xzy
72
(3,( + 1
Bug (Xig E(X,2,)) B(X,
=: 11 + 12 + 13.
If |1 0lx then I2, I3 Thus this Lemma is proved by Lemma A.7 So in the following proof, suppose ||,r > O, that is, there exists 1 S j S p. such that Bhyl > O. As [1 S Vlog p/ n, there exist a constant cg > 0 such that ||,x |ox S ca /og./g. Denote = c/3, d = ci /3, we have
Pr
Σ ==1
(.) 1/2 ZT
log cV
Pr
[ 2 zdv
log
n
So suffices to prove Pr([ 2 d /logp//) S eg exp(-02 logp), k 1,
For = 1, this inequality holds by lemma A.7 and choosing proper constants.
For k = 2, by sub-Gaussian Assumption in Assumption A.1 and applying Bernstein inequal- ity(Theorem 2.8.2 in 49 ), we derive
logp
Pr 1121l00 2 d1.
(t(1
log p n
S p max Pr
X Zigk EXX
c
ISkSp
m
c logg
Cvlogg
< 2p max exp 1kkSp
c1 min
Bh K2n
18ng KK vn
m
d'
d
2pexp
C4 min
n
K
29

---

where is a constant independent of Br and K = 25 Z. Thus this inequality holds > logp.
For = 3. As Xj.J ,....,, are all sub- Gaussian with uniform bounded sub-Gaussian norm, then it follows that [|/3 00c I CC3///ol p/n. In summary, this Lemma is proved.
Lemma A.II. Let X1, 1,....; Xa be i.idd mean zero random variables. If there exist constants La and L2, such that Pr (1
2 x) Li exp (-L2a²) ( some O, then for x 2 /SE (X²) /n
Pr
X
2
<4 <xexp
{ 22 } + 4nI1 exp
{
1
L2nt/(2t))
2
1=1
Proof. Details of the proof can be seen in 44
Lemma A.12. For sequence n/(2, p) oo as (n.p) + co, we have
n((.p)
Pr
sup IER
(t) F(t)
o(1).
vn
Proof. We can get this result by using DKW inequality( 48) directly.
Lemma A.13. Under the Assumption A.I and A.2 max, , (Xij 2, max, e and
max,,) (Xi, De,ssl are all (pllg(mp)). Both max, n (Z.= (0, 0,) and
maxj E (Eig eng)² are Op(s logg) We abbreviate 1 S S 9 as i, respectively.
Proof. For a sufficient large constant C, by sub-Gaussian assumption have
Pr(max( Xij 2.,0,)² > C2 log(np))
-Pr(max Xij 2, C CVlog(np)
Snpmax Pr Xij Z, > CVloo(np)
S2mpexpl - Ca log (mp) kR] o(1),
where K, is constant independent of the choice of j. So max;,; (Xi, 2TJ0,)² = (Og((o(())) proved. By same argument, we can prove both max;, et, and max:, (Xi; 0,)0,)ess are
Op (log(ri)) Applying Theorem in [35] it follows directly that max; 1.2 Σi [,,(0, 0,)
(p(s ll But for max, 1 -i= (ēig eij )2, there is something different. Note that for a sufficient
large constant O,
n (éis - e21)² 12 40
logg
Pr max
n
1
(F,(Y) F(X)) -Z,, ~3))}² 40 &logg
Pr =
max-
n
(Y)))) 2 (2,5(ts -r;)}² > 20 $ log
<Pr
max
1=1
logy
>>>>> C." logg
<Pr
sup IER
(t( - F(O)
+ Pr max -
Σ
|TT((95 (90 -m5)}
n 1=1
The first term last inequality is o(1) by lemma A.12 with ()n, p) vlogp. The second term
also o(1) by applying Theorem 2 in (35 again. In summary, this lemma is proved.
D
30

---

Lemma A.14. Under the Assumptions A.7 and A.2
max JEG
= =.) 0f
where
(,, - / (log (lognip) repo
s log p (og(pp)
V (slog //0/2//o/// np
(slog
n
(1.48)
Proof. To prove this lemma, we define
1 [ -Z,0,) es + +x)()))?
o3
where Egy = F(Y) 1/2 Z, 7, mY(Y,) = Ex, 2, 0,) 2 Y,) F(Y)) and o3 is an unbiased estimator of a² It follows by the triangle inequality that max 031 S maxjed 03 oj + maxjeg o3 1.1l So suffices to prove both terms are op polggoo) -1)
We begin with estimation of the rate of maxjes o3 -o3l. Note that
max 00 - 53 max jEg
((x - 2TS0,) ey + (V))² o3
jEg
Since Pro, (maxjeglo) -ō91 2 1) S maxjeg Præ,
531 2 1) and for VI 2 o3. (103 Proa(
Pre.
((x,, (0,y--,,00)) eij + mm,(())))))))
zt
((Xi) -2,,0) eij + m, (Y:))
<Pra,
2
=Prø,
((xg) - (X.j-Z,j0)) eig mY)
2 Vl-
By sub-Gaussian assumption, (X.--Z,50) eij mY,()) sub-Exponential random vari- able with uniform bounded sub- sub-Exponential norm. When is large enough, by the defi- nition of sub- Exponential random variable, we could find some constants L1, L2 such that Pro, (X(,,, 0,) eig + mY,)) vt-o3) S L1 exp(-Lavt). Thus it follows by Lemma (log r5p0 /n with sufficient large constant C, A.11 for some 2 C. Volog
Po max Pro, (107 - 831 2 < 4po exp((n2//3//5 + 4npoC1 expppieepp(CCnn/555ppp = Op ((ppo)-1) jEG
for some constants C₁ and Cg. Thus, maxyegl - 531 (lo)//))5/))
31

---

Next we estimate the rate of max 031. By definitions,
max J59
3 - -631
1
Σ (Xis -1,00)es - mY)))))) - {(X1 ),s0 + (Y))) 2
max /E9
1=1
Σ (Xi, -,,05)ess + (V)) - (Xig- - 21j05)ess-no - -Y)( }
S max ,E9
1=1
2
Σ (Xx, Z.j0,)ess+ + (X)) ((X,, 2,j0)Jas + m,, - (Xis -2,j,)es - mX)(3
+ max jEG 7
2
(Xy, 2, - (Xy, - 2,
Σ (M, (x))²
max JEG
+
max
jEG
1=1
2
+ max jEG
(Xig - 2,,0,es + + (X)) - Z,,0,)ess + m,(Y)) - (Xij - - -mx)))
1=1
=:2(h + 1 + Tz).
So suffices to estimate the converge rate of J, i = 1,2,3 respectively. Decomposing J, it follows that
6
1 S Σ
J
where L is given by
In = max-
1
Σ
(Xi,
(e, - -ey)?
JEG i=1
112 = max -
1
Σ
n
(0, - -0))
jEG n i=1
113 max- jE9 71
1
Σ
n
(Z,, 0,)3 - -
2 Σ (X -
T14 = max lEG
-es)
1=1
Σ (Xij - -0,-0,,, -, -ew)2
1 = max JEG
n
max lE9 n
Σ (Z) - e,)) en)
en
Then we deal with Iie, == 1,...,6 respectively. For simplify, in the rest of proof, we abbreviate 1 sigh n, j E G as i.j respectively.
For In we have In (n slgg) O,,n loogllog( (oogon))) by lemma A.13 112 < maxi, og max, 1 [
m 2T, max, Σ "e
e;y)2
Similarly, we have
(Z,,(0,, - -0))² Op(n log p log (og()))).
32

---

For 113. by Sauchy-Schwarle inequality and lemma A.13
it holds that
113
_
1
max
(0, - -0,)²²
max Σ (Eis - ctj)
12
1=1
1=1
1
(1) 0,,} max Σ (eis e
max )
1=1
logp
(s (slogp)
Op
11
= Op
Similarly, have
I4 S mmax 0,)0,,
max
1 (0; 0
max
1
(eas cij) )2
==1
1=1
= =0p(n - s log plog(np)),
1
11 S max Xis - 2,50,1
max
Z (0, 0,)} max
Σ
(ei - e2y)
1=1
1=1
(s 3//2 /lo(npp)
=0p
n
R
I16 S 2//mmxx
eisl max
2 (0, - -0,)]²
max Σ j
(ēis cij) )2
i=1
=O, (s (slogp) 3/2 log(np)
71
Next deal with Iz. By definition and Hoeffding's inequality we have
1
2,,6 2 Y2) Fn(()) - -Y,()))
1z = max
) n2
i=1
1
12
(1(, Y, - -F.())) 1
max
Σ
(Xij
) n
i=1
3m1
1
2 [ (Xi, 2,,0,1 }
< max
)
= logg)) p).
In summary, we have that 11 + = (r(r(n, (,,,,,,))),, where
r(n,p, slog log(np)
(s (slogp)312 Aognp (s ()gg))2 log
V
V
n Similarly, we can show that 13 = Op( ((n,,, s)), it follows from Cauchy-Schwarde incquality that
n
I3 S max /2(11 + 12).
Thus we have max,eglø? - 831 = 0p( Op( (r(,,rr(ppppp..p s)) and totally,
max 00 - -39=
= =1
(log npo)
s)(n,p,s)
jeg
n
-0p (v (log
log p log np)
V / V (slogp)3/2
(s log p)2 n1
n
n
Op (r2(n, P. Po. s))
Thus this lemma is proved.
33

---

Lemma A.15, Under the Assumptions A.1
max JEG
Sag
Vn = (1p(0opo))
Proof. For simplify, we rewrite Saz - Vn = Σ Sig/ vn. Under assumption A. it's easy to show that [im V==1 is an i..d. zero mean sub- Exponential n.v. sequence. So we prove this lemma by using Bernstein's inequality and Bonferroni's inequality.
Lemma A.16. Under the Assumptions A.7 and A.2
max ((Tng - 03- Vrlhhjog) (Snj - of Vn (r( JEG
uniformly for Br on (3h |B 2 Vlogp/n) with
(in.p.Po.88 = ri p. Po. 8) V (2(n, po. 8) (log Po)
(1.49)
Where
Ti (n, P., s) logp
(log po 3/22 30) log(n)
log Po vnl
n
and (2 P. P. s) is defined in 1.48
Proof.
First decompose (Inj oj vm -(5nj -0 VriBngog) as: (Ins 07 V) - (5nn -0 VVV300800 = (Tni Snz) + (0, -oj (S,s vrg3ngog)
A15 + A2j.
Next we deal with maxjed AAgli = 1,2 respectively. We handle maxjed AAil at first. We decompose A1f as
n
[X)T))(.,; - -ZJ0,) + 00, 0 T
A1j
Σ (F) - 1/2--2,,1
3/22
2
Vn
i=1
(
TT)
(Xi - -Z,00)) 1/2U, (VTUng Smi)
Vn
-1 + T2j + Taj + Taj + Tsj)
where Ung is defined in 1.244. So suffices to prove maxjeg [Ly Op(r1 p. Po. s)), i 1, 2, 3, 4,5 respectively. For a sufficient small constant e' > 0 and sufficient large constant C 0, we have following results.
Step 1, for = 1. maxjeg M,y = (r (n, Po. 8)) holds by sub-Gaussian Assumption and Bernstein inequality. The proof similar step in the proof of Lemma A.10 we omit it.
Po,s)
Step 2, for = 2. By Proposition A.I and Lemma A. we have
Pr(M/yy cv S exp(-c2p).
where (c, c1, 02) are constants defined in these lemma and independent the choice of j. We can derive that /nxxxyysx = ((s log p/ /n) by the fact that Ax, dy x Vlogp/n
Step 3, for = 3. It's similar to step 2 by using Theorem 3.1 and Lemma A.9 We omit it.
Step 4, for i 4. Note that 44 = n 1/2 1/ 3n n 1/2 Brjo and n 1/2 maxjed 1Brrdy
O(n 1 /log p). suffices to deal with n 1/2 (Un, - 82,8.). ). We begin the proof in the same way as we argue in the proof of lemma A.7 Applying lemma A.6 for Vs E R we have
NN
hY, Y XN+1) Bm58 }}. (1.50)
Pr(Uns Snyd, 2 t) < exp(-
exp
N
34

---

where N = [n/2] and t E R*. Denote W53 h (Y, x;; YN+,, Xx+1) - Бルjグ5,
NN.
By the sub-Gaussian assumption made in assumption A.1 (Wy)i= is an i..d. sub-E sub-Erponential sequence and maxigs/spll Wlyl 00. Thus by the same way used in the proof of Bernstein inequality( 49),, we derive
6
2) S max Pr([Un; 2 JE9
2exp -emin (maxigispl W. ,lo,,, )² maxisisp [ 1l )
where c is a constant. Denote = Cri(n, P. po: 8), we prove it as n > 1
Step 5, for i 5. By the theory of U-statistics, Unj Snz/ /n two-order canonical U-statistic.
Thus maxjeg |ayl Op (n (log po) log(npo) +n 1amnm1// log pol) by Lemma A.21 In summary, we derive that maxxjeg |Aı5l = Or(ri 11,4,00 po, since ōj 2 Cmin V/ 2 Cmin 2 >> 0 for large enough.
Next prove maxjes A2z (p(r2(n, Po: s)oggpo)) For Aa > 0 we have
4 po. (o)
Pre.
max jEG
A2yl
2
3
< Pra. (max JEG
o3 03] 2 (n P. P.Po 8)) + Pr
(max jEg
Sns
VnBngos 2 C(loggo) (22
=()).
The last equality holds by Lemma A. 14 and A.15 Note that all the universal constants do not depend on n, p. Bh, and 0g. j = 1.....22 p. Thus this lemma is proved.
Lemma A.17. Under the Assumptions A.7 and A.2
max JEg
[Tnj VnUngy = (p(r(1.p.00.))))
Proof. The proof of this Lemma is similar to Lemma A.16
Lemma A.18. Under the Assumptions theorem A. ifs = olVn// (log npo)3/2)), then
Proc (V
in probability, V 2 C(lo((1pa))-1/2
where Th is defined in
1.27 in the main text and C' is sufficient large constant.
Proof. By definition.
V Mho v Iins
S max jE9
(Tnj
VNū mg)
Σ Wi
2
h (V, X,; Y X3) of hy(s, X,; Yg X:) }os)
+ max JEG
0
Vniln
1=1
=: 11 + 12.
Thus it suffices to prove Pr. (1: Cllog 1/23) + 0 in probability, 1,
For 4, since W; is standard Gaussian r.v., the Hoeffding inequality implies that C'n(log(rupo))
(1 2 (11) < 2exp (max TTn /nūnng)
where C' a constant. Applying Lemma A.17
Pr 2 ((log(upo) -1/2) = o1 is proved.
For Iz, the Hoeffding inequality also implies that
C'n(n - (1(oo_))(log(ppp))
Pr (12 2 C(log(1po))-113) < 2exp
maxjes En Lissl 6, h1 oj - 5),1))²
35

---

Y, For simplify, we abbreviate (Y, X;; X.), h (Y, X5; Yg X:) as hy, hy respectively. In the rest of proof, we abbreviate mixigign max maxjeg as max,, max;, max, respectively. To prove Prw (12 2 C(log(npo)) -1/2) oz (1), it suffices to prove that
1 n(n 1)2²
6o h2 o. 0j/h1) } op ((lo((1po)))
i
By the Hölder inequality,
1
{o h2 h1
max
n(n 1)2
JEG
i=1
ipti
S max 1,6,j
|0 'ng -0j -ojhh) 'n1
max 1,8.5
10 113 - ,511زa 11r1 + max
10-1113 h1 - oj 111
5,2,8
: 121 + I22.
Thus suffices to prove 12x (p((og(ppoo)) h 1, 2 respectively. For I21, it holds that
I21 max
1(x; 2,j0,) (Xij 2, M))b
= =max w (Xx; 0,, -nss]l + max |Z] - -0;) - mT)) + max [0, -0,) [.
1.3
where fig = K 2ig 9zg Tki = K -- -2/l and K = -1/2 or -1/2. Note that fjj Mii 2i7s(7 - -9;) and
Zail 117,1111111 8/111 = -0, (,(01/ log(mpp) log
max
Mii Miyl max f.j
n where the last step follows from the s s/log p/n convergence of 1lo; S ll and max,, |zy (o(Vlog(npo)) due to the sub-Gaussian properties of Zıj. Following the similar arguments, we have
i.j
(log(1pa) )2 log P
max i,j
(Xij (TT0)) - mh)) S max Xx, Z.J0, b,j
max his Tisl = (p(() $1
2.3
n x2 log(npo) logg
max i.j
| (0; ()) - -ms)) < |0 - -,111 max
Zagl max Miss - Tkyl = Opp
(0/ (log(npu)) PPlogg
max max IT, (0, 0, )nsl - 1|0, 0
max 2,j 1.j
Zzg
mmsl = 0p 0/
4,
In summary, we have
(log(npo)
logg
221 Op $V
= Op (1og((n))))
n
where the last equality holds by Assumption
A.7
For 122, it holds that
In2 S maxlo_ of
max Iyy op (log(npp)))
(i.jj
where the last inequality holds by Lemma A.14 and max,,,, |hyl = Op Op(oo((pon)). In summary, this lemma is proved.
Lemma A.19. Let XI...., Xn be n i..d random vectors, where X, = (X....., Xea) Assume that there are some constants < e1 < Cr such that (X2) 2 er and Xiş sub- Exbpexponentaal with [x,jly, I for all 1 I $ I d. If (log(dn))? /n = o(1), then
(1 Σ X
lim sup P
$t)
Pr St) =0,
n0 (ER
80
where N ~ N(xX,
36

---

Proof. Lemma H.6 in [40,, and this lemma is adapted from [3].
Lemma A.20. Assume that X Na(0, Let of زوك and define miin = miny σj and
max
max; jj Then
Pr( S(+e) 11 S0) VVVVl//
sup
EE
where C/ is constant depending min and max
Proof. Lemma H.5 [40], and this lemma is adapted from
Lemma A.21. (A maximal inequality for canonical U-statistics). Let X....., X,, be i.i.d. p- dimension random vectors. Let f Rº x R2 + æ be symmetric and canonical kernel such that maxismsd (xxep((ff /Bn) S and Bn is a sequence of positive reals. Let Vn. (n(n - 1)} Eisisf5n f (X,, Xz). 1f2 S I exp(bn) for some constant b > O, then there exists a constant C(b) > 0 such that
( (.)b)) C(b) (n- log d)3/2 log(nd) +n log d).
That is, V (Bx log d) log(nd) + n log d)).
Proof. This lemma is adapted from 12
Lemma A.22. Let X..... Xn be iidd p-dimension random vectors and h RR? x Rp + Rd be symmetric and nondegenerate kernel such that B./. (Xll)/((,,,) X2)1 do for all I,....d. d. Consider the Utstatistic of order two:
Un n(n -1
Σ
h h(,,,
15iff5n
Let T,, VTU - -2)/2, where (2))(X1 is the mean of U.. Define g(x) = B(h(X,,, X2) X1 x} and E RR. Let c1...., en be iid. N(0,1) 1) random variables that
are independent of [X.)= and
T M 1-
1
Un
Let B, be sequence of real numbers possibly tending infinity. We assume that:
(1) There exists a constant b such that [g (Xi)) 2 b for all m = I,...,d. dd
(2) (E{ hm (2)
2+ S B for 1,2 and for all m I....,d. (3) I [X, X²) 11 I B, for all = 1.....d .d.
Denote a prri (a) as the ath conditional quantile of [ given X where a E 10. (log(dn))? o(1), then
lim sup n0000 eE(0.1)
|n S " (ar) - a =0.
(1,51)
In additional, assume that there exist statistics T,, and TV. such that
Pr Pr[[[[Tll - || 2 E(n) + O,
and
Pre P Il - 1| E(n)) 0 in probability-
for some E(n) depending on n. Iff(n) log(d/E(m)) o(1) then
lim sup 11000 aE(0.1)
(P <apit (a:) a
(1.52)
where aprit 1ao (a) is the oth conditional quantile of | given X,,..., X
Proof. This lemma is adapted from and 13
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction clearly state the paper's contributions and scope. And the claim does match theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of paper in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
38

---

Justification:
The appendix contains full set of assumptions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the information needed is provided in Section and Appendix A.6 is avaliable in supplementary materials.
Code
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
39

---

Answer: [Yes]
Justification:
Code is avaliable in the Supplementary Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee //nips.com htte:/ubbl///liiiee////eeeueassl ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The experimental details are provided in Section and Appendix A.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: This paper focuses statistical inference, and we provide empirical size and power in numerical studies.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
40

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Type of compute workers and memory are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:: http://ps.//.pppc..poo/oo//////a
Answer: [Yes]
Justification: The research conducted in the paper does conform with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper is foundational research and not tied to particular applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
41

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The authors cite the original paper that produced the code package.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
42

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "ehX0i5jkzy", "reviewer": "Reviewer_wCvo", "summary": "The paper proposes an algorithm for group inference in single-index models with an unknown link function, that is robust to heavy-tailed noise in the responses. The central idea of the approach is based on the property of elliptical distributions that the linear input-output correlations remain along a fixed direction for any transformation of the labels. The algorithm introduces a linear estimation objective for transformed labels resulting in an estimator converging to the true parameters under sparsity assumptions. Crucially, the obtained estimator yields a robust test for group inference satisfying the orthogonality property, which relaxes the requirements of separation between zero and non-zero coefficients. The obtained test further satisfies honesty, namely the uniform convergence of Type 1 error. The work concludes with numerical tests of the proposed approach.", "questions": "Questions:\n\nWhy are λX, λY not directly described in Theorem 3.1?\n\nWhy is the transformation H=F particularly suitable for robustness? Is it because of the distribution of F(Y) being agnostic to the distribution of Y?\n\nFor sub-exponential variables, Lemma A.11 appears to be weaker than Bernstein’s inequality. \n\nCould you clarify the difference w.r.t Bernstein’s inequality and related inequalities for random variables with bounded Orlicz norms?", "limitations": "Some missing discussion on limitations is described above under “weaknesses”. The work is primarily of a theoretical nature and has no potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well written and provides an adequate motivation for the problem of group inference under heavy-tailed response, and the idea of orthogonalization. The proposed approach is straightforward to implement and understand. The paper provides an extensive discussion of some important properties of the test such as honesty and power. The estimation error bounds are non-asymptotic and hold for general scalings of p,n. The numerical experiments support the efficiency and robustness of the approach.", "weaknesses": "A few limitations of the setup are not explicitly discussed:\n1) The robustness is restricted to variability in the noise of the response function, not w.r.t general perturbations to the distribution. The paper also doesn’t provide quantitative bounds on robustness.\n2) The condition kappa_h \\neq 0 excludes even link functions and in particular the problem of sparse phase retrieval.\n3) Theorem 3.1 imposes limitations on the sparsity level that should be explicit. The results in the paper seems to be restricted to constant sparsity levels/\n\nThe choice of h=F for robustness appears to be due to F(Y) being uniformly distributed in [0,1] independent of the distribution of Y. This should be more discussed explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable comments.\n## Weakness 1\nAs noticed by you, in our paper, we say our procedure is robust since our methods do not need any moment condition for the error term in the single-index model. For your concern, we further discuss the robustness of our procedure based on the tool of efficient influence function. Recall that our test procedure is inspired by the quantity $$I = \\Psi(P) = E_{P}[\\\\{F_{P}(Y) - 1/2 - Z_j^\\top\\gamma_j\\\\}(X_j - Z_j^\\top\\theta_j)],$$where $P$ is distribution of $(X_j,Z_j^\\top,Y)^\\top$. Next we derive the efficient influence function (EIF) of $I$. Consider $$P_t=t\\tilde P+(1-t)P,$$where $t\\in[0,1]$, and $\\tilde P$ is a point mass at a single observation $\\tilde o  = (\\tilde x_j,\\tilde z_j^\\top,\\tilde y)^\\top$. By some calculation, the EIF for $I$ at observation $\\tilde o$ is$$\\phi(\\tilde o,P)=\\frac{d\\Psi(P_t)}{dt}\\vert_{t=0}=\\\\{F_P(\\tilde y)-1/2-\\tilde z_j^\\top\\gamma_j\\\\}(\\tilde x_j-\\tilde z_j^\\top\\theta_j)+E_P[\\\\{I(Y\\geq\\tilde y)-F_p(Y)\\\\}(X_j-Z_j^\\top\\theta_j)]-\\Psi(P). $$ Since $I(Y\\geq\\tilde y)$ and $F_P(\\tilde y)$ are bounded, then given $(x_j,z_j^\\top)^\\top$, $\\phi(\\tilde o,P)$ is bounded for any $\\tilde y\\in R$. In terms of the EIF, our test statistics are robust with respect to the perturbations in the responses.\n## Weakness 2\nWe agree with you. The condition $\\kappa_h\\neq0$ does exclude even link functions and in particular the problem of sparse phase retrieval. Neykov et al. (2020) introduced a novel procedure to deal with the problem of sparse phase retrieval when $\\kappa_h=0$. We will incorporate their insights in the reivision.\n## Weakness 3\nFor consistency in $L_2$-loss, we require the sparsity level satisfying $s_Y=o(n/\\log p)$. For $L_1$-loss, the restriction becomes $s_Y=o(\\sqrt{n/\\log p})$. The sparsity level is allowed to be diverging with $n$ and $p$.\n## Weakness 4\nWith the equation (2.3), given the widely imposed sub-gaussian assumption on the predictors, any bounded transformation function $h(Y)$ would lead the transformed error term $e$ being sub-gaussian, even if the original error term $\\epsilon$ comes from Cauchy distribution. However, the response-distribution transformation is preferred due to the following additional reasons. \n * As noted by Rejchel and Bogdan (2020), in the empirical distribution function, the term $\\sum_{j=1}^n I(Y_j \\leq Y_i)$ is the rank of $Y_i$. Since statistics with ranks such as Wilcoxon test and the Kruskall-Wallis ANOVA test, are well-known to be robust, this then intuitively explains why our procedures with distribution function are robust with respect to outliers in response.\n* The distribution function is very easy to estimate, and thus, our approach is straightforward to implement and understand. \n* Lastly, besides being robust, the choice of $h(Y) = F(Y)$ would also have relatively high efficiency. Specifically, we conduct detailed simulations to illustrate this point. The simulation settings are summarized in global response, and the simulated results are displayed in the attached pdf file. Besides distribution function, we also consider $h(Y) = Y$ and $h(Y) = \\text{sigmoid}(Y) = 1/\\\\{1+\\exp(-Y)\\\\}$. Compared to other functions, our procedure has high power performance under nearly all the settings. The better performance compared to $\\text{sigmoid}(Y)$ demonstrates that the superiority of our procedure is not merely due to the boundedness of the transformation function.\n## Question 1\nTo simplify the presentation, all main assumptions are given in the A.5 of the Appendix. In Assumption A.1 (ii), we assume that $c\\sqrt{\\log p/n}\\leq \\lambda_{X},\\lambda_{Y}\\leq C\\sqrt{\\log p/n}$ for some constants $0<c\\leq C$.\n## Question 2\nPlease refer to response to weakness 4 for details.\n## Question 3\nAfter some careful calculation, we agree with you that for sub-Exponential variables, Lemma A.11 is weaker than Bernstein’s inequality. Let $X_1,\\ldots,X_n$ be independent, mean zero, sub-Exponential random variables. Then for $t\\geq 0$, Bernstein inequality implies that$$P\\\\{\\vert\\frac{1}{n}\\sum_{i=1}^{n}X_{i}\\bigr\\vert\\geq t\\\\}\\leq 2\\exp\\\\{-L\\min(\\frac{t^{2}}{K^{2}},\\frac{t}{K})n\\\\},$$where $L$ is a constant and $K = \\max_{1\\leq i\\leq n}\\lVert X_{i}\\rVert_{\\psi_1}$. And Lemma A.11 implies that$$P\\\\{\\vert\\frac{1}{n}\\sum_{i=1}^{n}X_i\\vert\\geq t\\\\}\\leq 4\\exp\\\\{-\\frac{1}{8}n^{1/3}t^{2/3}\\\\} + 4n L_{1}\\exp\\\\{-\\frac{L_{2}}{2}n^{1/3}t^{2/3}\\\\}$$for $t\\geq \\sqrt{8E(X_i^2)/n}$, where $L_{1}$ and $L_{2}$ are constants. For ease of comparsion, suppose that $t \\asymp n^{-c}$ for $c\\in(0,1/2]$. Note that $$2\\exp\\\\{-\\frac{Lt^{2}n}{K^{2}}\\\\} \\ll  4n L_{1}\\exp\\\\{-\\frac{L_{2}}{2}n^{1/3}t^{2/3}\\\\}$$when $n$ is sufficiently large. That is, the bound of Bernstein's inequality is sharper than the bound of Lemma A.11. \n## Question 4\nLet $g:[0,\\infty)\\rightarrow[0,\\infty)$ be a non-decreasing convex function with $g(0)=0$. The ``$g$-Orlicz'' norm of a random variable $X$ is $\\Vert X\\Vert_{g} := \\inf\\\\{\\eta>0:E[g(\\vert X\\vert/\\eta)]\\leq 1\\\\}$. \n\nSpecifically, let $\\psi_r(x)=\\exp(x^r) - 1$. The sub-Weibull norm of a r.v. $X$ for any $r>0$ is defined as $\\Vert X\\Vert_{\\psi_{r}}$. The r.v. $X$ follows sub-Weibull distribution for $r>0$ is equivalent to $\\Vert X\\Vert_{\\psi_r}$ is bounded.\nWhen $r=1$, $X$ is sub-Exponential. In this case, Bernstein inequality is applicable when $r=1$. While Lemma A.11 is applicable for $r>0$. Therefore, Lemma A.11 is a generalization of Bernstein's inequality for general sub-Weibull distributions.\n\nIn this article, Lemma A.11 is adopted in the proof of Lemma A.14. In this case, we need to derive the bound of $\\max_{j\\in\\mathcal{G}}\\vert\\sigma_{j}^{2} - \\tilde\\sigma_j^2\\vert$. For $j\\in\\mathcal{G}$, $\\sigma_j^2-\\tilde\\sigma_j^2$ can be represented as $\\sigma_j^2-\\tilde\\sigma_j^2=\\frac{1}{n}\\sum_{i=1}^{n}Z_i$, where $Z_1,\\ldots,Z_n$ are zero mean, i.i.d. variables with bounded sub-Weibull norm for $r=1/2$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5taO1BbOz", "reviewer": "Reviewer_BqNs", "summary": "This paper proposes a robust group hypothesis testing procedure for a high-dimensional single index model based on a data-driven transformation of the response variable. The key observation is that under the the linearity condition (LC) of the predictors, a wide class of single index models can be equivalently (for testing purposes) converted into a linear model with a transformed response variable. The main contribution of this paper is the introduction of the distribution transformation (a data-driven approach) of the response variable, and derive the asymptotic distribution of the resulting test statistic. Numerical performance of the proposed method is evaluated through simulation studies.", "questions": "1. The introduction of the distribution function $F_n$ in (2.7) is interesting but also a little arbitrary. I suspect that any other functions that can be written as linear combinations of functions of $Y_i$'s can be used in (2.7), for example, the kernel density estimator, the kernel smoother of $Y_i$'s. The resulting test statistic should still be asymptotically equivalent to a U-statistic. Could the authors comment on this?\n\n2. Whenever the term \"robustness\" is involved, it implies loss of efficiency in some cases. In my opinion, it is important to make a transparent evaluation of the limitations of the proposed procedure. In the simulation study, one can add a comparison to the method with $h(Y)=Y$ as described in section 3.3. It will be informative if one can showcase in which cases which procedure is more powerful, and thus understand which method to use in practice.\n\n3. In the simulation study, I would also suggest a graphical comparison between different methods by gradually increasing $p_{out}$ from $0$ to, say, $0.5$ for a more complete picture.", "limitations": "The simulation study can be improved.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method is interesting and potentially useful. The paper is well-written and the presentation is clear.", "weaknesses": "The simulation study can be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable comments.\n## Weaknesses\nWe have added more simulations to illustrate our procedures and compare them with other methods. The simulation settings are summarized in global response, and the simulated results are displayed in the attached pdf file.\n## Question 1\nAs noted by you, there are many possible choices for the transformation function $h(Y)$. There are several reasons for us to choose the distribution function as the transformation function:\n* Firstly, the response-distribution transformation function is bounded. Actually with the equation (2.3), given the widely imposed sub-gaussian assumption on the predictors, any bounded transformation function $h(Y)$ would lead the transformed error term $e$ being sub-gaussian, even if the original error term $\\epsilon$ in the single index model comes from Cauchy distribution. \n* As noted by Rejchel and Bogdan (2020), in the empirical distribution function, the term $\\sum_{j=1}^n I(Y_j \\leq Y_i)$ is the rank of $Y_i$. Since statistics with ranks such as Wilcoxon test and the Kruskall-Wallis ANOVA test, are well-known to be robust, this then intuitively explains why our procedures with distribution function are robust with respect to outliers in response.\n* The distribution function is very easy to estimate and thus our approach is straightforward to implement and understand. While the kernel density estimator, the kernel smoother of $Y_i$’s require additionally tuning bandwidth.\n\n## Question 2\nAccording to your valuable suggestion, we have conducted detailed comparisons with the method based on $h(Y) = Y$ as described in section 3.3. The simulation settings are summarized in global response, and the corresponding numerical results are displayed in Figure R.1 in the attached pdf file.\n\n* In Figure R.1, we compare our method with the procedure based on $h(Y)=Y$ when the error term follows the standard normal distribution. In the standard case (normal distribution error, linear model and no response polluted), the performance of test statistic with $h(Y)=Y$ is slightly better than our method with $h(Y)=F(Y)$. However, in other settings, our method with $h(Y)=F(Y)$ performs much better than method with $h(Y)=Y$. \n\nAlthough $h(Y)=Y$ performs well in the standard case, one cannot know the distribution of the error term or the formula of the model in practice. Thus compared to $h(Y)=Y$, we recommend using our procedure in practice.\n## Question 3\nWe agree that a graphical comparison between different methods by gradually increasing $p_{out}$ from 0 to 0.5 would provide a more complete picture. According to your valuable suggestion, we have conducted detailed simulations for comparisons. The simulation settings are summarized in global response, and the results are displayed in Figure R.1 of the attached pdf file.\n\nWe consider three transformation procedures: (1) $h(Y) = F(Y)$ (Our method); (2) $h(Y) = Y$; (3) $h(Y) = \\text{sigmoid}(Y) = 1/\\\\{1+\\exp(-Y)\\\\}$. We vary $p_{out}$ from $\\\\{0, 0.1, 0.2, 0.3, 0.4, 0.5\\\\}$ and the simulation results are shown in Figure R.1. \n* Firstly, under the null hypothesis ($G_1$), $h(Y) = \\text{sigmoid}(Y)$ cannot control the type I error for Model 2, while other procedures perform well. \n* Secondly, under the alternative hypothesis ($G_2$), the empirical powers of other procedures decrease rapidly with the increase of $p_{out}$, while the powers of our procedure remain stable, which is particularly noticeable when $\\delta = 0.5$. This finding indicates that our method has strong robustness when the responses are polluted. \n* Thirdly, our method performs well for both Model 1 and Model 2, indicating that our method is robust across different single-index models.\n\n## Limitations\nFollowing your suggestions, we have added more simulations to illustrate our procedures and compare them with other methods. The simulation settings are summarized in global response, and the simulated results are displayed in Figure R.1 of the attached pdf file.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BKJkNIxzUB", "reviewer": "Reviewer_Tg8m", "summary": "This paper studies the high-dimensional single index model (SIM), which takes the form $Y=g(X^T\\beta, \\epsilon)$ with $\\epsilon$ and $X$ being orthogonal. Although this model has flexibility and interpretability, its efficiency is adversely affected by outlying observations and heavy-tailed distributions. The paper improves this in the following 3 aspects:\n\n(1) they extend the rank-LASSO procedure to include both convex and non-convex penalties and establish error bound of any local optimum of the empirical objective;\n\n(2) they provide asymptotically honest group inference procedures based on the idea of orthogonalization for testing the joint effect of many predictors;\n\n(3) they develop a multiple testing procedure for determining if the individual coefficients are relevant simultaneously, and show that it is able to control the FDR asymptotically.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Provides new ideas that deal with the inefficiency of single index model due to outlying observations and heavy-tailed distributions.", "weaknesses": "I think the paper is quite well-written. The only issue may be that the authors can provide more about the background of the problem,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your valuable comments.\n## Weaknesses\n\nThe group inference is helpful to decide whether a group of predictors are important or not for the response. If we find a group of predictors are important, we would like to know which specific predictors in the group are significant. For this aim, our developed multiple testing procedure is useful. For instance, researchers may aim to test whether a gene pathway, consisting of high-dimensional genes for the same biological function, is important for a certain clinical outcome, given the other high-dimensional genes. When determining that a certain gene pathway is important, researchers need to further identify specific genes within the pathway which are important for a certain clinical outcome.\n\nAccording to your helpful suggestion, we will add a real data analysis in the appendix of the revision. For your convenience, we also copy the real data analysis in the following:\n\nWe apply our methods on a dataset about riboflavin (vitamin B2) production rate with Bacillus Subtilis. This dataset is made publicly by Buhlmann et al. (2014) and has been analyzed by many authors, for instance Meinshausen et al. (2009), Van de Geer et al. (2014), Javanmard and Montanar (2014), and Fei et al. (2019). The dataset riboflavin can be obtained from the R package $\\texttt{hdi}$. It consists of $n = 71$ observations of strains of Bacillus Subtilis and $p = 4088$ covariates, measuring the log-expression levels of 4088 genes. The response variable is the logarithm of the riboflavin production rate.\n\nOur goal is to detect which genes are associated with riboflavin production rate. Like most existing studies, we first reduce ultrahigh-dimension to a moderate high-dimension. Here we pick out first 300 genes by distance correlation based screening Li et al. (2012). We first conduct global testing on these 300 genes. The $p$-value of our group inference procedure is 1.29e-04, indicating that the null hypothesis is rejected and the selected 300 genes are influential for riboflavin production rate. Next, we further use FDR control procedure to select the important genes in these 300 genes.\n\nBy implementing our proposed FDR control procedure with the FDR level of 0.1, we identify 10 genes that are significantly associated with the response. That is \n\n$G_{I}$ = {YTGB\\_at, YCKE\\_at, YXLE\\_at, YXLD\\_at, YJCJ\\_at, XHLA\\_at, xepA\\_at, YCGO\\_at, RPLP\\_at, XKDS\\_at}.\n\nIf the FDR level is set as 0.2, 5 more genes will be selected. That is \n\n$G_{II}$ = $G_I$ $\\cup$ {SPOIISA\\_at, YHCB\\_at, XKDI\\_at, YJCF\\_at, XHLB\\_at}.\n\nWe further conduct group inference on the selected subsets $G_{I}, G_{II}$ and their  complement sets $G_I^c, G_{II}^c$. As expected, our group inference procedure finds again that $G_{I}, G_{II}$ are significant while $G_{I}^c, G_{II}^c$ are not.\nThe corresponding $p$-values of $G_{I}$, $G_{I}^c$, $G_{II}$ and $G_{II}^c$ are 6.75e-06, 7.26e-01, 9.33e-06 and 9.37e-01, respectively. These results suggest that the genes selected by the FDR control procedure are really influential.\n\nWe compare these selected genes with other methods. For example, the multi-sample-splitting method proposed in Meinshausen et al. (2009) identified YXLD\\_at; Van de Geer et al. (2014) did not select any gene using the de-sparsified Lasso; Javanmard and Montanar (2014) only selected two genes: YXLD\\_at and YXLE\\_at and Fei et al. (2019) claimed YCKE\\_at, XHLA\\_at, YXLD\\_at, YDAR\\_at and YCGN\\_at as significant. From $G_I$ and $G_{II}$, we can see clearly that the gene YXLD\\_at is detected by not only Meinshausen et al. (2009), Javanmard and Montanar (2014), Fei et al. (2019), but also by our procedure. Besides, the genes YCKE\\_at, YXLE\\_at and XHLA\\_at which are detected by Javanmard and Montanar (2014) and Fei et al. (2019), are also found by our method. Further, our procedure detects some additional important genes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chairs, Senior Area Chairs, Area Chairs and Reviewers,

Thank all of you for your insightful comments and valuable suggestions, which have significantly enhanced the quality of this work. In this response, all the comments have been carefully addressed and accommodated. A summary of response is as follows.

* Discussions of robustness with respect to perturbations in the responses. In the revision, we carefully discuss the robustness of our inference procedure based on the tool of efficient influence function. Please see the response to weakness 1 of Reviewer wCvo for details.
* Discussions of the response-distribution transformation function. In the revision, we discuss the reasons why the response-distribution transformation is preferred. Please see responses to weakness 4 of Reviewer wCvo and question 1 of Reviewer BqNs for details.
* Relationships between Lemma A.11 and Bernstein' inequality. Please see responses to questions 3 and 4 of Reviewer wCvo for details.
* Additional simulation studies. In the revision, we have conducted additional simulation studies to illustrate the robustness of our procedure. Please refer to the attached pdf file for details.
    
    * **The simulation settings are summarized as follows.** We consider the following two models:
          
         * Model 1: Linear model: $Y = X^\top\beta + \epsilon$.

         * Model 2: Non-linear model: $Y = \exp(X^\top\beta + \epsilon)$.

         The regression coefficients $\beta = (\beta_{1},\beta_{2},\ldots,\beta_{p})^\top$ are generated as $\beta_{j} = \delta$ for $j = 1,\ldots,6$ and $\beta_{j}=0$ otherwise, where $\delta$ can be regarded as a signal strength parameter. We generate the error term from the standard normal distribution. We add outliers to pollute the observations: $p_{out}$ of the responses are picked at random and increased by $m_{out}$-times maximum of original responses. Specifically, the detailed settings for the above parameters are as follows. Firstly, we consider the sample size $n=500$ and the dimension $p=800$. Secondly, we set the signal strength parameter $\delta$ to vary from $\\{0.1,0.3,0.5\\}$. Thirdly, we fix $m_{out}$ to 10. Lastly, we vary $p_{out}$ from $0$ to $0.5$ in increments of $0.1$. For more complete comparisons, we consider three transformation procedures: (1) $h(Y) = F(Y)$ (Our method); (2) $h(Y) = Y$; (3) $h(Y)=\text{sigmoid}(Y)=1/\\{1+\exp(-Y)\\}$. Other simulation settings are the same as described in section 4 of the main text. The simulation results are summarized in Figure R.1 of the attached pdf file. 

    * Figure R.1 summarizes the results of empirical type I error and empirical power with the significant level of $\alpha=0.05$ for different methods when the error term follows the standard normal distribution. Firstly, under the null hypothesis ($G_1$), $h(Y) = \text{sigmoid}(Y)$ cannot control the type I error for Model 2, while other procedures perform well. Secondly, under the alternative hypothesis ($G_2$), the empirical powers of other procedures decrease rapidly with the increase of $p_{out}$, while the powers of our procedure remain stable, which is particularly noticeable when $\delta = 0.5$. This finding indicates that our method has strong robustness when the responses are polluted. Thirdly, our method performs well for both Model 1 and Model 2, indicating that our method is robust across different single-index models.

* Discussion about the background of the problem. In the response, we discuss the practical background of the problem and demonstrate the practical value by a real data analysis. Please see the response to weaknesses of Reviewer Tg8m for details.
* All the other comments from the Program Chairs, Senior Area Chairs, Area Chairs and Reviewers are also carefully addressed.

Thank you very much for giving us an opportunity to revise the paper.

Meta-review: The focus of the paper is an algorithm for group inference in single-index models which is robust under heavy-tailed noise in high dimension: under some sparsity hypothesis, an equivalent linear estimation objective allows to obtain a robust estimator converging to the true parameters.

The paper received positive feedback from all reviewers because of its clarity and the proposed ideas to deal with the problem of group inference under heavy tails in the response in single-index models. The results are further supported by numerical evidence. During the rebuttal discussion, limitations in the analyzed set-up were addressed and clarified in the resubmitted version, and further numerical evidence was added to the manuscript. I share positive evaluations of the Reviewers and recommend therefore acceptance.
Decision: Accept (poster)