Inference on large-scale structures

The emergence of ‘Big Data’ has driven a new statistical branch which Efron [11] calls ‘Large-Scale Inference’ (LSI). In many LSI problems, the data are structured. Due to proximity in geography, time, etc., the data matrix may contain graphical structures, low-rank structures, clustering structures, etc. These structure pose great challenges as well as great opportunities: by carefully exploiting such structures, we are able to significantly improve the inference. In this dissertation, we illustrate the point by exploiting two types of structures: sparse graphical structures and homogeneous clustering structures. Consider a linear regression model Y = Xβ + , where X ∈ Rn×p is the design matrix and is the noise vector. Our study consists of two parts. The first part is largely motivated by the recent interest in DNA copy number variation (CNV) and long-memory financial data, where the focus is ‘variable selection’: we assume the effects are ‘rare and weak’ in the sense that only a small fraction of the entries of β are nonzero and the nonzeros are individually small; the main interest is to identify this small fraction of nonzeros. We consider the very challenging case where the columns of the design matrix X are heavily correlated. We recognize that in many of such situations, adjacent rows of the Gram matrix G = X ′X are similar, and such structures can be exploited for better variable selection. We develop a new variable selection procedure called ‘Covariate Assisted Screening and Estimation (CASE)’. CASE is a two-stage Screen and Clean method, at the heart of which is a graph-guided multivariate screening procedure. We show that in a broad context, CASE achieves the minimax Hamming selection errors. We have successfully applied CASE to a change-point problem and long-memory time series, and derived explicit forms of the so-called ‘phase diagrams’. The phase diagram is a recent way to visualize variable selection that is especially appropriate for rare and weak effects. iii CASE has advantages over the more well-known L/L-penalization methods and marginal screening. CASE is a flexible idea and is readily extendable to many different settings. The second part of the study is largely motivated by the recent interest on gene network [32] and the study of housing price [16]. We assume the vector β is homogeneous in the sense that its entries cluster into a few groups, and entries in each group share a common value; the main interest is to take advantage of such structures to estimate β. We propose a new method called ‘Clustering Algorithm in Regression via Datadriven Segmentation’ (CARDS). We show that under mild conditions, CARDS successfully recovers the groups of variables and achieves the oracle estimation errors. The study provides additional insights on how to exploit low-dimensional structures in high-dimensional regression problems. CARDS was successfully applied to predicting Polyadenylation signals and the S&P500 stock returns. Compared to methods not taking advantage of homogeneity, CARDS has much smaller estimation errors. Compared to other methods that attempt to exploit homogeneity such as the fused lasso, CARDS has the advantage of not requiring any given ordering or graph on the variables. CARDS is also a flexible idea and can be adapted to many different settings.