Rejoinder

We thank the editors for inviting this discussion and especially the discussants for their kind remarks and thoughtful contributions, which in some cases are so thorough as to verge on articles in their own right! Before turning to specific comments, we set the stage by noting that this article is a reflection of the Ph.D. dissertation of Lu, awarded in 2002. Its chief, original purpose was to provide the first rigorous proof of the inconsistency phenomenon, reviewed in section 2 of our article, and to propose a simple algorithm that demonstrates that exploitation of sparsity allows one to recover consistency. There have been many subsequent developments during the years that this article has been in publication review. In particular, as the various references and the current discussants show, there are now several algorithms and theoretical results that may improve in various respects on the original proposal. For example, already in his Ph.D. dissertation, Paul (2005) recognized that selection of variables based on sample variances alone was inefficient and could only detect components of size (log(n _ p)/n). He developed a two-stage estimator, using information about the correlation between the variables to augment the selected variable set at the second stage, and showed that this successfully selected components of size down to O(log(n _ p)/n). Boaz Nadler’s discussion independently makes a somewhat similar remark about exploiting correlation information, and develops an alternate estimator with properties that appear attractive and worthy of further study. In a technical tour de force, Paul (2005) also developed lower bounds and studied the asymptotic properties of his twostage estimator to show that it achieves (within logarithmic factors) optimal rates of convergence. Nadler also points to the good performance, on our two examples, of the covariance matrix thresholding estimator of Bickel and Levina (2008). Bickel and Levina (2008) were careful to establish convergence properties of their estimator in operator norm, which implies convergence of eigenvectors, and so the attractive performance of the derived eigenvector estimator is, in retrospect (! ), not so surprising. One should caution that further investigation is needed, because our two examples are admittedly at relatively high signal-to-noise ratios. In a recent preprint in the related setting of banded covariance estimation, Cai, Zhang, and Zhou (2008) have shown that threshold choice based on a function of (log p)/n need not be optimal, at least over specific function classes. Nadler provides an interesting analysis of reconstruction error, using what one might call a ‘‘projection oracle’’— namely, an estimator that knows the best subset of k variables to retain. The discussion and his figure 1 suggest that (i) reconstruction error is highly sensitive to incorrect inclusion of noise variables and that (ii) there may be considerable room for reconstruction error improvement. We comment on these in turn. Avoidance of the first point was the reason for inclusion of thresholding as a final stage in our algorithm. No theoretical analysis of thresholding was included in the article, so we attempt some heuristics here. To continue with the notation used by Nadler, suppose that rk is the response vector r restricted to the optimal k variables and r̂k is the eigenvector of sample principal component analysis (PCA) computed only on these k chosen variables. In this relatively high signal-to-noise setting, we may heuristically imagine r̂k as approximately derived from a Gaussian signal plus noise model r̂k 1⁄4 rk þ zk with zk ; Nk(0, I) and 2 1⁄4 1/n (some support for this heuristic can be found in Paul’s (2005) dissertation). Consistent with the proposal in section 4.1 of our original article, let ~ rk denote the result of hard thresholding at lk, where lk 1⁄4 ffiffiffiffiffiffiffiffiffiffiffiffiffi 2 log k p . Bounds for the risk of hard thresholding E k ~ rk rk k # Rð~ rk; rkÞ; may be obtained, for example, from Johnstone (2002, lemma 11.5), and yield Rð~ rk; rkÞ # 2 X