covariance regularization by thresholding this paper considers regularizing a covariance matrix of p variables estimated from n observations by hard thresholding we show that the thresholded estimate is consistent in the operator norm as long as the true covariance matrix is sparse in a suitable sense the variables are gaussian or sub gaussian and log p n 0 and obtain explicit rates the results are uniform over families of covariance matrices which satisfy a fairly natural notion of sparsity we discuss an intuitive resampling scheme for threshold selection and prove a general cross validation result that justifies this approach we also compare thresholding to other covariance estimators in simulations and on an example from climate data 1 introduction estimation of covariance matrices is important in a number of areas of statistical analysis including dimension reduction by principal component analysis pca classification by linear or quadratic discriminant analysis lda and qda establishing independence and conditional independence relations in the context of graphical models and setting confidence intervals on linear functions of the means of the components in recent years many application areas where these tools are used have been dealing with very high dimensional datasets and sample sizes can be very small relative to dimension examples include genetic data brain imaging spectroscopic imaging climate data and many others it is well known by now that the empirical covariance matrix for samples of size n from a p variate gaussian distribution np p is not a good estimator of the population covariance if p is large many results in random matrix theory illustrate this from the classical mary law 29 to the more recent work of johnstone and his students on the theory of the largest eigenvalues 12 23 30 and associated eigenvectors 24 however with the exception of a method for estimating the covariance spectrum 11 these probabilistic results do not offer alternatives to the sample covariance matrix alternative estimators for large covariance matrices have therefore attracted a lot of attention recently two broad classes of covariance estimators have emerged those that rely on a natural ordering among variables and assume that variables

ideal spatial adaptation by wavelet shrinkage summary with ideal spatial adaptation an oracle furnishes information about how best to adapt a spatially variable estimator whether piecewise constant piecewise polynomial variable knot spline or variable bandwidth kernel to the unknown function estimation with the aid of an oracle offers dramatic advantages over traditional linear estimation by nonadaptive kernels however it is a priori unclear whether such performance can be obtained by a procedure relying on the data alone we describe a new principle for spatially adaptive estimation selective wavelet reconstruction we show that variable knot spline fits and piecewise polynomial fits when equipped with an oracle to select the knots are not dramatically more powerful than selective wavelet reconstruction with an oracle we develop a practical spatially adaptive method riskshrink which works by shrinkage of empirical wavelet coefficients riskshrink mimics the performance of an oracle for selective wavelet reconstruction as well as it is possible to do so a new inequality in multivariate normal decision theory which we call the oracle inequality shows that attained performance differs from ideal performance by at most a factor of approximately 2 log n where n is the sample size moreover no estimator can give a better guarantee than this within the class of spatially adaptive procedures riskshrink is essentially optimal relying only on the data it comes within a factor log 2 n of the performance of piecewise polynomial and variableknot spline methods equipped with an oracle in contrast it is unknown how or if piecewise polynomial methods could be made to function this well when denied access to an oracle and forced to rely on data alone

modelling multivariate volatilities via conditionally uncorrelated components summary we propose to model multivariate volatility processes on the basis of the newly defined conditionally uncorrelated components cucs this model represents a parsimonious representation for matrix valued processes it is flexible in the sense that each cuc may be fitted separately with any appropriate univariate volatility model computationally it splits one high dimensional optimization problem into several lower dimensional subproblems consistency for the estimated cucs has been established a bootstrap method is proposed for testing the existence of cucs the methodology proposed is illustrated with both simulated and real data sets

covariance matrix selection and estimation via penalised normal likelihood summary we propose a nonparametric method for identifying parsimony and for producing a statistically efficient estimator of a large covariance matrix we reparameterise a covariance matrix through the modified cholesky decomposition of its inverse or the one step ahead predictive representation of the vector of responses and reduce the nonintuitive task of modelling covariance matrices to the familiar task of model selection and estimation for a sequence of regression models the cholesky factor containing these regression coefficients is likely to have many off diagonal elements that are zero or close to zero penalised normal likelihoods in this situation with l1 and l2 penalities are shown to be closely related to tibshirani s 1996 lasso approach and to ridge regression adding either penalty to the likelihood helps to produce more stable estimators by introducing shrinkage to the elements in the cholesky factor while because of its singularity the l1 penalty will set some elements to zero and produce interpretable models an algorithm is developed for computing the estimator and selecting the tuning parameter the proposed maximum penalised likelihood estimator is illustrated using simulation and a real dataset involving estimation of a 102 x 102 covariance matrix

threshold selection for covariance estimation thresholding is a regularization method commonly used for covariance estimation which provides consistent estimators if the population covariance satisfies certain sparsity condition bickel and levina 2008a cai and liu 2011 however the performance of the thresholding estimators heavily depends on the threshold level by minimizing the frobenius risk of the adaptive thresholding estimator for covariances we conduct a theoretical study for the optimal threshold level and obtain its analytical expression a consistent estimator based on this expression is proposed for the optimal threshold level which is easy to implement in practice and efficient in computation numerical simulations and a case study on gene expression data are conducted to illustrate the proposed method

on the increments of partial sum processes with multidimensional indices consider a set xn nd of iid random variables indexed by nd the positive integer d dimensional lattice points the erdos renyi 1970 law of large numbers is generalized to this case and some extensions are given the results include a theorem of chan 1976 for the d parameter brownian motion

on the distribution of the largest eigenvalue in principal components analysis let x 1 denote the square of the largest singular value of an n p matrix x all of whose entries are independent standard gaussian variates equivalently x 1 is the largest principal component variance of the covariance matrix x x or the largest eigenvalue of a p variate wishart distribution on n degrees of freedom with identity covariance consider the limit of large p and n with n p gamma ge 1 when centered by mu p sqrt n 1 sqrt p 2 and scaled by sigma p sqrt n 1 sqrt p 1 sqrt n 1 1 sqrt p 1 3 the distribution of x 1 approaches the tracey widom law of order 1 which is defined in terms of the painleve ii differential equation and can be numerically evaluated and tabulated in software simulations show the approximation to be informative for n and p as small as 5 the limit is derived via a corresponding result for complex wishart matrices using methods from random matrix theory the result suggests that some aspects of large p multivariate distribution theory may be easier to apply in practice than their fixed p counterparts

operator norm consistent estimation of large dimensional sparse covariance matrices estimating covariance matrices is a problem of fundamental importance in multivariate statistics in practice it is increasingly frequent to work with data matrices x of dimension n x p where p and n are both large results from random matrix theory show very clearly that in this setting standard estimators like the sample covariance matrix perform in general very poorly in this large n large p setting it is sometimes the case that practitioners are willing to assume that many elements of the population covariance matrix are equal to 0 and hence this matrix is sparse we develop an estimator to handle this situation the estimator is shown to be consistent in operator norm when for instance we have p n as n oo in other words the largest singular value of the difference between the estimator and the population covariance matrix goes to zero this implies consistency of all the eigenvalues and consistency of eigenspaces associated to isolated eigenvalues we also propose a notion of sparsity for matrices that is compatible with spectral analysis and is independent of the ordering of the variables

regularization of wavelet approximations in this paper we introduce nonlinear regularized wavelet estimators for estimating nonparametric regression functions when sampling points are not uniformly spaced the approach can apply readily to many other statistical contexts various new penalty functions are proposed the hard thresholding and soft thresholding estimators of donoho and johnstone are specific members of nonlinear regularized wavelet estimators they correspond to the lower and upper envelopes of a class of the penalized least squares estimators necessary conditions for penalty functions are given for regularized estimators to possess thresholding properties oracle inequalities and universal thresholding parameters are obtained for a large class of penalty functions the sampling properties of nonlinear regularized wavelet estimators are established and are shown to be adaptively minimax to efficiently solve penalized least squares problems nonlinear regularized sobolev interpolators nrsi are proposed as initial estimators wh

sparse estimation of a covariance matrix we suggest a method for estimating a covariance matrix on the basis of a sample of vectors drawn from a multivariate normal distribution in particular we penalize the likelihood with a lasso penalty on the entries of the covariance matrix this penalty plays two important roles it reduces the effective number of parameters which is important even when the dimension of the vectors is smaller than the sample size since the number of parameters grows quadratically in the number of variables and it produces an estimate which is sparse in contrast to sparse inverse covariance estimation our method s close relative the sparsity attained here is in the covariance matrix itself rather than in the inverse matrix zeros in the covariance matrix correspond to marginal independencies thus our method performs model selection while providing a positive definite estimate of the covariance the proposed penalized maximum likelihood problem is not convex so we use a majorize minimize approach in which we iteratively solve convex approximations to the original nonconvex problem we discuss tuning parameter selection and demonstrate on a flow cytometry dataset how our method produces an interpretable graphical display of the relationship between variables we perform simulations that suggest that simple elementwise thresholding of the empirical covariance matrix is competitive with our method for identifying the sparsity structure additionally we show how our method can be used to solve a previously studied special case in which a desired sparsity pattern is prespecified

better subset regression using the nonnegative garrote a new method called the nonnegative nn garrote is proposed for doing subset regression it both shrinks and zeroes coefficients in tests on real and simulated data it produces lower prediction error than ordinary subset selection it is also compared to ridge regression if the regression equations generated by a procedure do not change drastically with small changes in the data the procedure is called stable subset selection is unstable ridge is very stable and the nn garrote is intermediate simulation results illustrate the effects of instability on prediction error

regularized estimation of large covariance matrices this paper considers estimating a covariance matrix of p variables from n observations by either banding the sample covariance matrix or estimating a banded version of the inverse of the covariance we show that these estimates are consistent in the operator norm as long as logp 2 n 0 and obtain explicit rates the results are uniform over some fairly natural well conditioned families of covariance matrices we also introduce an analogue of the gaussian white noise model and show that if the population covariance is embeddable in that model and well conditioned then the banded approximations produce consistent estimates of the eigenvalues and associated eigenvectors of the covariance matrix the results can be extended to smooth versions of banding and to non gaussian distributions with su ciently short tails a resampling approach is proposed for choosing the banding parameter in practice this approach is illustrated numerically on both simulated and real data

nonconcave penalized likelihood with a diverging number of parameters a class of variable selection procedures for parametric models via nonconcave penalized likelihood was proposed by fan and li to simultaneously estimate parameters and select important variables they demonstrated that this class of procedures has an oracle property when the number of parameters is finite however in most model selection problems the number of parameters should be large and grow with the sample size in this paper some asymptotic properties of the nonconcave penalized likelihood are established for situations in which the number of parameters tends to as the sample size increases under regularity conditions we have established an oracle property and the asymptotic normality of the penalized likelihood estimators furthermore the consistency of the sandwich formula of the covariance matrix is demonstrated nonconcave penalized likelihood ratio statistics are discussed and their asymptotic distributions under the null hypothesis are obtained by imposing some mild conditions on the penalty functions the asymptotic results are augmented by a simulation study and the newly developed methodology is illustrated by an analysis of a court case on the sexual discrimination of salary

improving the efficiency of the log rank test using auxiliary covariates summary under the assumption of proportional hazards the log rank test is optimal for testing the null hypothesis ho 3 0 where b denotes the logarithm of the hazard ratio however if there are additional covariates that correlate with survival times making use of their information will increase the efficiency of the log rank test we apply the theory of semiparametrics to characterize a class of regular and asymptotically linear estimators for b when auxiliary covariates are incorporated into the model and derive estimators that are more efficient the wald tests induced by these estimators are shown to be more powerful than the log rank test simulation studies are used to illustrate the gains in efficiency

adaptive covariance matrix estimation through block thresholding estimation of large covariance matrices has drawn considerable recent attention and the theoretical focus so far has mainly been on developing a minimax theory over a fixed parameter space in this paper we consider adaptive covariance matrix estimation where the goal is to construct a single procedure which is minimax rate optimal simultaneously over each parameter space in a large collection a fully data driven block thresholding estimator is proposed the estimator is constructed by carefully dividing the sample covariance matrix into blocks and then simultaneously estimating the entries in a block by thresholding the estimator is shown to be optimally rate adaptive over a wide range of bandable covariance matrices a simulation study is carried out and shows that the block thresholding estimator performs well numerically some of the technical tools developed in this paper can also be of independent interest

a new approach to cholesky based covariance regularization in high dimensions summary in this paper we propose a new regression interpretation of the cholesky factor of the covariance matrix as opposed to the well known regression interpretation of the cholesky factor of the inverse covariance which leads to a new class of regularized covariance estimators suitable for high dimensional problems regularizing the cholesky factor of the covariance via this regression interpretation always results in a positive definite estimator in particular one can obtain a positive definite banded estimator of the covariance matrix at the same computational cost as the popular banded estimator of bickel levina 2008b which is not guaranteed to be positive definite we also establish theoretical connections between banding cholesky factors of the covariance matrix and its inverse and constrained maximum likelihood estimation under the banding constraint and compare the numerical performance of several methods in simulations and on a sonar data example

large covariance estimation for compositional data via composition adjusted thresholding high dimensional compositional data arise naturally in many applications such as metagenomic data analysis the observed data lie in a high dimensional simplex and conventional statistical methods

model selection and estimation in the gaussian graphical model summary we propose penalized likelihood methods for estimating the concentration matrix in the gaussian graphical model the methods lead to a sparse and shrinkage estimator of the concentration matrix that is positive definite and thus conduct model selection and estimation simultaneously the implementation of the methods is nontrivial because of the positive definite constraint on the concentration matrix but we show that the computation can be done effectively by taking advantage of the efficient maxdet algorithm developed in convex optimization we propose a bic type criterion for the selection of the tuning parameter in the penalized likelihood methods the connection between our methods and existing methods is illustrated simulations and real examples demonstrate the competitive performance of the new methods

optimal sparse volatility matrix estimation for high dimensional ito processes with measurement errors stochastic processes are often used to model complex scientiflc problems in flelds ranging from biology and flnance to engineering and physical science this paper investigates rate optimal estimation of the volatility matrix of a high dimensional it o process observed with measurement errors at discrete time points the minimax rate of convergence is established for estimating sparse volatility matrices by combining the multi scale and threshold approaches we construct a volatility matrix estimator to achieve the optimal convergence rate the minimax lower bound is derived by considering a subclass of it o processes for which the minimax lower bound is obtained through a novel equivalent model of covariance matrix estimation for independent but non identically distributed observations and through a delicate construction of the least favorable parameters in addition a simulation study was conducted to test the flnite sample performance of the optimal estimator and the simulation results were found to support the established asymptotic theory

sparsistency and rates of convergence in large covariance matrix estimation this paper studies the sparsistency and rates of convergence for estimating sparse covariance and precision matrices based on penalized likelihood with nonconvex penalty functions here sparsistency refers to the property that all parameters that are zero are actually estimated as zero with probability tending to one depending on the case of applications sparsity priori may occur on the covariance matrix its inverse or its cholesky decomposition we study these three sparsity exploration problems under a unified framework with a general penalty function we show that the rates of convergence for these problems under the frobenius norm are of order s n log p n n 1 2 where s n is the number of nonzero elements p n is the size of the covariance matrix and n is the sample size this explicitly spells out the contribution of high dimensionality is merely of a logarithmic factor the conditions on the rate with which the tuning parameter n goes to 0 have been made explicit and compared under different penalties as a result for the l 1 penalty to guarantee the sparsistency and optimal rate of convergence the number of nonzero elements should be small sn o pn at most among o pn2 parameters for estimating sparse covariance or correlation matrix sparse precision or inverse correlation matrix or sparse cholesky factor where sn is the number of the nonzero elements on the off diagonal entries on the other hand using the scad or hard thresholding penalty functions there is no such a restriction

wavelet shrinkage asymptopia much recent effort has sought asymptotically minimax methods for recovering infinite dimensional objects curves densities spectral densities images from noisy data a now rich and complex body of work develops nearly or exactly minimax estimators for an array of interesting problems unfortunately the results have rarely moved into practice for a variety of reasons among them being similarity to known methods computational intractability and lack of spatial adaptivity we discuss a method for curve estimation based on n noisy data translate the empirical wavelet coefficients towards the origin by an amount 2 log n n the proposal differs from those in current use is computationally practical and is spatially adaptive it thus avoids several of the previous objections further the method is nearly minimax both for a wide variety of loss functions pointwise error global error measured in l p norms pointwise and global error in estimation of derivatives and for a wide range of smoothness classes including standard holder and sobolev classes and bounded variation this is a much broader near optimality than anything previously proposed we draw loose parallels with near optimality in robustness and also with the broad near eigenfunction properties of wavelets themselves finally the theory underlying the method is interesting as it exploits a correspondence between statistical questions and questions of optimal recovery and information based complexity

positive definite estimators of large covariance matrices using convex optimization we construct a sparse estimator of the covariance matrix that is positive definite and performs well in high dimensional settings a lasso type penalty is used to encourage sparsity and a logarithmic barrier function is used to enforce positive definiteness consistency and convergence rate bounds are established as both the number of variables and sample size diverge an efficient computational algorithm is developed and the merits of the approach are illustrated with simulations and a speech signal classification example copyright 2012 oxford university press

generalized thresholding of large covariance matrices we propose a new class of generalized thresholding operators that combine thresholding with shrinkage and study generalized thresholding of the sample covariance matrix in high dimensions generalized thresholding of the covariance matrix has good theoretical properties and carries almost no computational burden we obtain an explicit convergence rate in the operator norm that shows the tradeoff between the sparsity of the true model dimension and the sample size and shows that generalized thresholding is consistent over a large class of models as long as the dimension p and the sample size n satisfy log p n 0 in addition we show that generalized thresholding has the sparsistency property meaning it estimates true zeros as zeros with probability tending to 1 and under an additional mild condition is sign consistent for nonzero elements we show that generalized thresholding covers as special cases hard and soft thresholding smoothly clipped absolute deviation and adaptive lasso and comp

nonparametric estimation of large covariance matrices of longitudinal data summary estimation of an unstructured covariance matrix is difficult because of its positivedefiniteness constraint this obstacle is removed by regressing each variable on its predecessors so that estimation of a covariance matrix is shown to be equivalent to that of estimating a sequence of varying coefficient and varying order regression models our framework is similar to the use of increasing order autoregressive models in approximating the covariance matrix or the spectrum of a stationary time series as an illustration we adopt fan zhang s 2000 two step estimation of functional linear models and propose nonparametric estimators of covariance matrices which are guaranteed to be positive definite for parsimony a suitable order for the sequence of auto regression models is found using penalised likelihood criteria like aic and bic some asymptotic results for the local polynomial estimators of components of a covariance matrix are established two longitudinal datasets are analysed to illustrate the methodology a simulation study reveals the advantage of the nonparametric covariance estimator over the sample covariance matrix for large covariance matrices