correlated wishart ensembles and chaotic time series we study the correlated wishart ensembles in the context of time series analysis we are interested in the statistics of eigenlevels viz variables associated with independent eigenmodes in the system the motivation of this work is to study the effect of time series correlations on the wishart ensembles in this connection we derive the level density and the two point function for the correlated wishart ensembles by using the binary correlation method using our analytic results we analyze spectra of autocovariance matrices derived from single variable stationary time series we consider the stochastic time series of gaussian variables with exponentially decaying correlations and time series of chaotic maps viz the arnold map the standard map and the stadium billiard map in both cases correlated time series are encountered and analyzed under the framework of random matrix theory it is shown that the eigenlevel statistics for the chaotic maps follow closely those of correlated wishart ensembles it is indicated that the presence of collective modes in the spectra of autocovariance matrices is related to the integrability of the system

correlation functions cluster functions and spacing distributions for random matrices the usual formulas for the correlation functions in orthogonal and symplectic matrix models express them as quaternion determinants from this representation one can deduce formulas for spacing probabilities in terms of fredholm determinants of matrix valued kernels the derivations of the various formulas are somewhat involved in this article we present a direct approach which leads immediately to scalar kernels for the unitary ensembles and matrix kernels for the orthogonal and symplectic ensembles and the representations of the correlation functions cluster functions and spacing distributions in terms of them

eigenvalue significance testing for genetic association genotype eigenvectors are widely used as covariates for control of spurious stratification in genetic association significance testing for the accompanying eigenvalues has typically been based on a standard tracy widom limiting distribution for the largest eigenvalue derived under white noise assumptions it is known that even modest local correlation among markers inflates the largest eigenvalues even in the absence of true stratification in addition a few sample eigenvalues may be extreme creating further complications in accurate testing we explore several methods to identify appropriate null eigenvalue thresholds while remaining sensitive to eigenvalues corresponding to population stratification we introduce a novel block permutation approach designed to produce an appropriate null eigenvalue distribution by eliminating long range genomic correlation while preserving local correlation we also propose a fast approach based on eigenvalue distribution modeling using a simple fit criterion and the general marcenko pastur equation under a simple discrete eigenvalue model block permutation and the model based approach work well for pure simulations and for data resampled from the 1000 genomes project in contrast we find that the standard approach of computing an effective number of markers does not perform well the performance of the methods is also demonstrated for a motivating example from the international cystic fibrosis consortium

outlier robust pca the high dimensional case principal component analysis plays a central role in statistics engineering and science because of the prevalence of corrupted data in real world applications much research has focused on developing robust algorithms perhaps surprisingly these algorithms are unequipped indeed unable to deal with outliers in the high dimensional setting where the number of observations is of the same magnitude as the number of variables of each observation and the dataset contains some arbitrarily corrupted observations we propose a high dimensional robust principal component analysis algorithm that is efficient robust to contaminated points and easily kernelizable in particular our algorithm achieves maximal robustness it has a breakdown point of 50 the best possible while all existing algorithms have a breakdown point of zero moreover our algorithm recovers the optimal solution exactly in the case where the number of corrupted points grows sublinearly in the dimension

a note on universality of the distribution of the largest eigenvalues in certain sample covariance matrices recently johansson and johnstone proved that the distribution of the properly rescaled largest principal component of the complex real wishart matrix x x xtx converges to the tracy widom law as n p the dimensions of x tend to in some ratio n p 0 we extend these results in two directions first of all we prove that the joint distribution of the first second third etc eigenvalues of a wishart matrix converges after a proper rescaling to the tracy widom distribution second of all we explain how the combinatorial machinery developed for wigner random matrices in refs 27 38 and 39 allows to extend the results by johansson and johnstone to the case of x with non gaussian entries provided n p o p1 3 we also prove that max n1 2 p1 2 2 o p1 2 log p a e for general 0

complex gaussian quadrature of oscillatory integrals we construct and analyze gauss type quadrature rules with complex valued nodes and weights to approximate oscillatory integrals with stationary points of high order the method is based on substituting the original interval of integration by a set of contours in the complex plane corresponding to the paths of steepest descent each of these line integrals shows an exponentially decaying behaviour suitable for the application of gaussian rules with non standard weight functions the results differ from those in previous research in the sense that the constructed rules are asymptotically optimal i e among all known methods for oscillatory integrals they deliver the highest possible asymptotic order of convergence relative to the required number of evaluations of the integrand

complete curves in moduli spaces of stable bundles on surfaces in this paper we give a partial answer to a question posed by hirschowitz and hulek in hi hu for moduli spaces of stable rank two vector bundles on the projective plane they asked for the maximal dimension of a complete subvariety of such a moduli space in fact they showed that for the projective plane these moduli spaces contain complete curves in almost every case for many cases they even contain complete surfaces cp hu lp and hu s the main theorem we will prove here is the following cp 3 8

painleve equations nonlinear special functions the six painleve equations pi pvi were first discovered about a hundred years ago by painleve and his colleagues in an investigation of nonlinear second order ordinary differential equations recently there has been considerable interest in the painleve equations primarily due to the fact that they arise as reductions of the soliton equations which are solvable by inverse scattering consequently the painleve equations can be regarded as completely integrable equations and possess solutions which can be expressed in terms of solutions of linear integral equations despite being nonlinear equations although first discovered from strictly mathematical considerations the painleve equations have arisen in a variety of important physical applications including statistical mechanics plasma physics nonlinear waves quantum gravity quantum field theory general relativity nonlinear optics and fibre optics the painleve equations may be thought of a nonlinear analogues of the classical special functions they possess hierarchies of rational solutions and one parameter families of solutions expressible in terms of the classical special functions for special values of the parameters further the painleve equations admit symmetries under affine weyl groups which are related to the associated backlund transformations in this paper i discuss some of the remarkable properties which the painleve equations possess including connection formulae backlund transformations associated discrete equations and hierarchies of exact solutions in particular the second painleve equation pii is used to illustrate these properties and some of the applications of pii are also discussed

algebro geometric analogues of donaldson s polynomials donaldson 5 has introduced polynomials on the second homology of any simply connected closed four manifold m with b odd at least 3 which up to are invariant under orientation preserving diffeomorphisms for every c ibm there is a polynomial 7c m syma h2 s where d c 4c z2 b 1 to define it one considers j c the set which is in a natural way a manifold of gauge equivalence classes of connections on the su 2 bundle on m with c2 c antiself dual with respect to a generic metric on m there is a natural map h2 m h2 j c namely slant product with 88 where p is the universal so 3 principal bundle on m x j c one way of defining 7c m 10 is to construct a compactification of r xc carrying a fundamental class xc one proves that extends to a map h2 m h2 xc then one defines

pca consistency in high dimension low sample size context principal component analysis pca is an important tool of dimension reduction especially when the dimension or the number of variables is very high asymptotic studies where the sample size is fixed and the dimension grows i e high dimension low sample size hdlss are becoming increasingly relevant we investigate the asymptotic behavior of the principal component pc directions hdlss asymptotics are used to study consistency strong inconsistency and subspace consistency we show that if the first few eigenvalues of a population covariance matrix are large enough compared to the others then the corresponding estimated pc directions are consistent or converge to the appropriate subspace subspace consistency and most other pc directions are strongly inconsistent broad sets of sucient conditions for each of these cases are specified and the main theorem gives a catalogue of possible combinations in preparation for these results we show that the geometric representation of hdlss data holds under general conditions which includes a mixing condition and a broad range of sphericity measures of the covariance matrix

partial estimation of covariance matrices a classical approach to accurately estimating the covariance matrix of a p variate normal distribution is to draw a sample of size n p and form a sample covariance matrix however many modern applications operate with much smaller sample sizes thus calling for estimation guarantees in the regime n ll p we show that a sample of size n o m log6p is sufficient to accurately estimate in operator norm an arbitrary symmetric part of consisting of m n nonzero entries per row this follows from a general result on estimating hadamard products m where m is an arbitrary symmetric matrix

threshold selection for covariance estimation thresholding is a regularization method commonly used for covariance estimation which provides consistent estimators if the population covariance satisfies certain sparsity condition bickel and levina 2008a cai and liu 2011 however the performance of the thresholding estimators heavily depends on the threshold level by minimizing the frobenius risk of the adaptive thresholding estimator for covariances we conduct a theoretical study for the optimal threshold level and obtain its analytical expression a consistent estimator based on this expression is proposed for the optimal threshold level which is easy to implement in practice and efficient in computation numerical simulations and a case study on gene expression data are conducted to illustrate the proposed method

the exponentially convergent trapezoidal rule it is well known that the trapezoidal rule converges geometrically when applied to analytic functions on periodic intervals or the real line the mathematics and history of this phenomenon are reviewed and it is shown that far from being a curiosity it is linked with computational methods all across scientific computing including algorithms related to inverse laplace transforms special functions complex analysis rational approximation integral equations and the computation of functions and eigenvalues of matrices and operators

innovated higher criticism for detecting sparse signals in correlated noise higher criticism is a method for detecting signals that are both sparse and weak although first proposed in cases where the noise variables are independent higher criticism also has reasonable performance in settings where those variables are correlated in this paper we show that by exploiting the nature of the correlation performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer indeed it turns out that the case of independent noise is the most difficult of all from a statistical viewpoint and that more accurate signal detection for a given level of signal sparsity and strength can be obtained when correlation is present we characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary the boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is toeplitz

projected principal component analysis in factor models this paper introduces a projected principal component analysis projected pca which employees principal component analysis to the projected smoothed data matrix onto a given linear space spanned by covariates when it applies to high dimensional factor analysis the projection removes noise components we show that the unobserved latent factors can be more accurately estimated than the conventional pca if the projection is genuine or more precisely when the factor loading matrices are related to the projected linear space when the dimensionality is large the factors can be estimated accurately even when the sample size is finite we propose a flexible semi parametric factor model which decomposes the factor loading matrix into the component that can be explained by subject specific covariates and the orthogonal residual component the covariates effects on the factor loadings are further modeled by the additive model via sieve approximations by using the newly proposed projected pca the rates of convergence of the smooth factor loading matrices are obtained which are much faster than those of the conventional factor analysis the convergence is achieved even when the sample size is finite and is particularly appealing in the high dimension low sample size situation this leads us to developing nonparametric tests on whether observed covariates have explaining powers on the loadings and whether they fully explain the loadings the proposed method is illustrated by both simulated data and the returns of the components of the s p 500 index

linear regression for panel with unknown number of factors as interactive fixed effects in this paper we study the least squares ls estimator in a linear panel regression model with unknown number of factors appearing as interactive fixed effects assuming that the number of factors used in estimation is larger than the true number of factors in the data we establish the limiting distribution of the ls estimator for the regression coefficients as the number of time periods and the number of cross sectional units jointly go to infinity the main result of the paper is that under certain assumptions the limiting distribution of the ls estimator is independent of the number of factors used in the estimation as long as this number is not underestimated the important practical implication of this result is that for inference on the regression coefficients one does not necessarily need to estimate the number of interactive fixed effects consistently

asymptotic power of sphericity tests for high dimensional data this paper studies the asymptotic power of tests of sphericity against perturbations in a single unknown direction as both the dimensionality of the data and the number of observations go to infinity we establish the convergence under the null hypothesis and the alternative of the log ratio of the joint densities of the sample covariance eigenvalues to a gaussian process indexed by the norm of the perturbation when the perturbation norm is larger than the phase transition threshold studied in baik et al 2005 the limiting process is degenerate and discrimination between the null and the alternative is asymptotically certain when the norm is below the threshold the process is non degenerate so that the joint eigenvalue densities under the null and alternative hypotheses are mutually contiguous using the asymptotic theory of statistical experiments we obtain asymptotic power envelopes and derive the asymptotic power for various sphericity tests in the contiguity region in particular we show that the asymptotic power of the tracy widom type tests is trivial whereas that of the eigenvalue based likelihood ratio test is strictly larger than the size and close to the power envelope

subsampling for heteroskedastic time series abstract in this article a general theory for the construction of confidence intervals or regions in the context of heteroskedastic dependent data is presented the basic idea is to approximate the sampling distribution of a statistic based on the values of the statistic computed over smaller subsets of the data this method was first proposed by politis and romano 1994b for stationary observations we extend their results to heteroskedastic observations and prove a general asymptotic validity result under minimal conditions in contrast the usual bootstrap and moving blocks bootstrap are typically valid only for asymptotically linear statistics and their justification requires a case by case analysis our general asymptotic results are applied to a regression setting with dependent heteroskedastic errors

on the distribution of the largest eigenvalue in principal components analysis let x 1 denote the square of the largest singular value of an n p matrix x all of whose entries are independent standard gaussian variates equivalently x 1 is the largest principal component variance of the covariance matrix x x or the largest eigenvalue of a p variate wishart distribution on n degrees of freedom with identity covariance consider the limit of large p and n with n p gamma ge 1 when centered by mu p sqrt n 1 sqrt p 2 and scaled by sigma p sqrt n 1 sqrt p 1 sqrt n 1 1 sqrt p 1 3 the distribution of x 1 approaches the tracey widom law of order 1 which is defined in terms of the painleve ii differential equation and can be numerically evaluated and tabulated in software simulations show the approximation to be informative for n and p as small as 5 the limit is derived via a corresponding result for complex wishart matrices using methods from random matrix theory the result suggests that some aspects of large p multivariate distribution theory may be easier to apply in practice than their fixed p counterparts

on the distribution of the length of the longest increasing subsequence of random permutations let sn be the group of permutations of 1 2 n if 7r e sn we say that 7 i1 7f ik is an increasing subsequence in 7r if il i2 ik and 7r ii 7r i2 7r ik let 1n r be the length of the longest increasing subsequence for example if n 5 and 7r is the permutation 5 1 3 2 4 in one line notation thus 7r 1 5 7r 2 1 then the longest increasing subsequences are 1 2 4 and 1 3 4 and n 3 equip sn with uniform distribution

optimal detection of heterogeneous and heteroscedastic mixtures the problem of detecting heterogeneous and heteroscedastic gaussian mixtures is considered the focus is on how the parameters of heterogeneity heteroscedasticity and proportion of non null component influence the difficulty of the problem we establish an explicit detection boundary which separates the detectable region where the likelihood ratio test is shown to reliably detect the presence of non null effect from the undetectable region where no method can do so in particular the results show that the detection boundary changes dramatically when the proportion of nonnull component shifts from the sparse regime to the dense regime furthermore it is shown that the higher criticism test which does not require the specific information of model parameters is optimally adaptive to the unknown degrees of heterogeneity and heteroscedasticity in both the sparse and dense cases

nonparametric detection of signals by information theoretic criteria performance analysis and an improved estimator determining the number of sources from observed data is a fundamental problem in many scientific fields in this paper we consider the nonparametric setting and focus on the detection performance of two popular estimators based on information theoretic criteria the akaike information criterion aic and minimum description length mdl we present three contributions on this subject first we derive a new expression for the detection performance of the mdl estimator which exhibits a much closer fit to simulations in comparison to previous formulas second we present a random matrix theory viewpoint of the performance of the aic estimator including approximate analytical formulas for its overestimation probability finally we show that a small increase in the penalty term of aic leads to an estimator with a very good detection performance and a negligible overestimation probability

high dimensional analysis of semidefinite relaxations for sparse principal components principal component analysis pca is a classical method for dimensionality reduction based on extracting the dominant eigenvectors of the sample covariance matrix however pca is well known to behave poorly in the large p small n setting in which the problem dimension p is comparable to or larger than the sample size n this paper studies pca in this high dimensional regime but under the additional assumption that the maximal eigenvector is sparse say with at most k nonzero components we consider a spiked covariance model in which a base matrix is perturbed by adding a k sparse maximal eigenvector and we analyze two computationally tractable methods for recovering the support set of this maximal eigenvector as follows a a simple diagonal thresholding method which transitions from success to failure as a function of the rescaled sample size dia n p k n k 2 log p k and b a more sophisticated semidefinite programming sdp relaxation which succeeds once the rescaled sample size sdp n p k n klog p k is larger than a critical threshold in addition we prove that no method including the best method which has exponential time complexity can succeed in recovering the support if the order parameter sdp n p k is below a threshold our results thus highlight an interesting trade off between computational and statistical efficiency in high dimensional inference

regularized estimation of linear functionals of precision matrices for high dimensional time series this paper studies a dantzig selector type regularized estimator for linear functionals of high dimensional linear processes explicit rates of convergence of the proposed estimator are obtained and they cover the broad regime from independent identically distributed samples to long range dependent time series and from sub gaussian innovations to those with mild polynomial moments it is shown that the convergence rates depend on the degree of temporal dependence and the moment conditions of the underlying linear processes the dantzig selector estimator is applied to the sparse markowitz portfolio allocation and the optimal linear prediction for time series in which the ratio consistency when compared with an oracle estimator is established the effect of dependence and innovation moment conditions is further illustrated in the simulation study finally the regularized estimator is applied to classify the cognitive states on a real functional magnetic resonance imaging dataset and to portfolio optimization on a financial dataset

shape fluctuations and random matrices we study a certain random growth model in two dimensions closely related to the one dimensional totally asymmetric exclusion process the results show that the shape fluctuations appropriately scaled converges in distribution to the tracy widom largest eigenvalue distribution for the gaussian unitary ensemble gue

testing conditional independence using maximal nonlinear conditional correlation in this paper the maximal nonlinear conditional correlation of two random vectors x and y given another random vector z denoted by 1 x y z is defined as a measure of conditional association which satisfies certain desirable properties when z is continuous a test for testing the conditional independence of x and y given z is constructed based on the estimator of a weighted average of the form nz k 1 f z z k 2 1 x y z z k where f z is the probability density function of z and the z k s are some points in the range of z under some conditions it is shown that the test statistic is asymptotically normal under conditional independence and the test is consistent

on nonoscillating integrals for computing inhomogeneous airy functions integral representations are considered of solutions of the inhomogeneous airy differential equation w z w pm1 pi the solutions of these equations are also known as scorer functions certain functional relations for these functions are used to confine the discussion to one function and to a certain sector in the complex plane by using steepest descent methods from asymptotics the standard integral representations of the scorer functions are modified in order to obtain non oscillating integrals for complex values of z in this way stable representations for numerical evaluations of the functions are obtained the methods are illustrated with numerical results

non parametric detection of the number of signals hypothesis testing and random matrix theory detection of the number of signals embedded in noise is a fundamental problem in signal and array processing this paper focuses on the non parametric setting where no knowledge of the array manifold is assumed first we present a detailed statistical analysis of this problem including an analysis of the signal strength required for detection with high probability and the form of the optimal detection test under certain conditions where such a test exists second combining this analysis with recent results from random matrix theory we present a new algorithm for detection of the number of sources via a sequence of hypothesis tests we theoretically analyze the consistency and detection performance of the proposed algorithm showing its superiority compared to the standard minimum description length mdl based estimator a series of simulations confirm our theoretical analysis

random matrix ensembles with an effective extensive external charge recent theoretical studies of chaotic scattering have encountered ensembles of random matrices in which the eigenvalue probability density function contains a one body factor with an exponent proportional to the number of eigenvalues two such ensembles have been encountered an ensemble of unitary matrices specified by the so called poisson kernel and the laguerre ensemble of positive definite matrices here we consider various properties of these ensembles jack polynomial theory is used to prove a reproducing property of the poisson kernel and a certain unimodular mapping is used to demonstrate that the variance of a linear statistic is the same as in the dyson circular ensemble for the laguerre ensemble the scaled global density is calculated exactly for all even values of the parameter while for random matrices with unitary symmetry the neighbourhood of the smallest eigenvalue is shown to be in the soft edge universality class

groupe de picard des varietes de modules de fibres semi stables sur les courbes algebriques soient x une courbe algebrique projective lisse de genre g 2 sur r d des entiers avec r 2 on note u r d la variete de modules des fibres algebriques semi stables sur x de rang r et de degre d us r d l ouvert de u r d correspondant aux fibres stables on sait que u r d est une variete algebrique projective irreductible et normale si r et d ne sont pas premiers entre eux u r d n est pas lisse sauf dans une exception le cas ou g 2 r 2 et d est pair on supposera dans cet article qu on n est pas dans ce cas on a alors codimu r d u r d us r d 2 et u r d us r d est le lieu des points singuliers de u r d si l est un fibre en droites de degre d sur x on note u r l la sous variete fermee de u r d correspondant aux fibres vectoriels de determinant isomorphe a l on etudie dans cet article les groupes de picard pic u r d et pic u r l on montre d abord que les varietes u r d et u r l sont localement factorielles on calcule ensuite pic u r d et pic u r l qui sont decrits au moyen des diviseurs thetas generalises

distribution functions for random variables for ensembles of positive hermitian matrices distribution functions for random variables that depend on a parameter are computed asymptotically for ensembles of positive hermitian matrices the inverse fourier transform of the distribution is shown to be a fredholm determinant of a certain operator that is an analogue of a wiener hopf operator the asymptotic formula shows that up to the terms of order o 1 the distributions are gaussian

optimal classification in sparse gaussian graphic model consider a two class classification problem where the number of features is much larger than the sample size the features are masked by gaussian noise with mean zero and covariance matrix sigma where the precision matrix omega sigma 1 is unknown but is presumably sparse the useful features also unknown are sparse and each contributes weakly i e rare and weak to the classification decision by obtaining a reasonably good estimate of omega we formulate the setting as a linear regression model we propose a two stage classification method where we first select features by the method of innovated thresholding it and then use the retained features and fisher s lda for classification in this approach a crucial problem is how to set the threshold of it we approach this problem by adapting the recent innovation of higher criticism thresholding hct we find that when useful features are rare and weak the limiting behavior of hct is essentially just as good as the limiting behavior of ideal threshold the threshold one would choose if the underlying distribution of the signals is known if only somewhat surprisingly when omega is sufficiently sparse its off diagonal coordinates usually do not have a major influence over the classification decision compared to recent work in the case where omega is the identity matrix proc natl acad sci usa 105 2008 14790 14795 philos trans r soc lond ser a math phys eng sci 367 2009 4449 4470 the current setting is much more general which needs a new approach and much more sophisticated analysis one key component of the analysis is the intimate relationship between hct and fisher s separation another key component is the tight large deviation bounds for empirical processes for data with unconventional correlation structures where graph theory on vertex coloring plays an important role

a new approach to cholesky based covariance regularization in high dimensions summary in this paper we propose a new regression interpretation of the cholesky factor of the covariance matrix as opposed to the well known regression interpretation of the cholesky factor of the inverse covariance which leads to a new class of regularized covariance estimators suitable for high dimensional problems regularizing the cholesky factor of the covariance via this regression interpretation always results in a positive definite estimator in particular one can obtain a positive definite banded estimator of the covariance matrix at the same computational cost as the popular banded estimator of bickel levina 2008b which is not guaranteed to be positive definite we also establish theoretical connections between banding cholesky factors of the covariance matrix and its inverse and constrained maximum likelihood estimation under the banding constraint and compare the numerical performance of several methods in simulations and on a sonar data example

a diagnostic criterion for approximate factor structure abstract we build a simple diagnostic criterion for approximate factor structure in large panel datasets given observable factors the criterion checks whether the errors are weakly cross sectionally correlated or share at least one unobservable common factor interactive effects a general version allows to determine the number of omitted common factors also for time varying structures the empirical analysis runs on ten thousand us stocks from january 1968 to december 2011 for monthly returns we select time invariant specifications with at least four financial factors and a scaled three factor specification for quarterly returns we cannot select macroeconomic models without the market factor

longest increasing subsequences from patience sorting to the baik deift johansson theorem we describe a simple one person card game patience sorting its analysis leads to a broad circle of ideas linking young tableaux with the longest increasing subsequence of a random permutation via the schensted correspondence a recent highlight of this area is the work of baik deift johansson which yields limiting probability laws via hard analysis of toeplitz determinants

an eigenvector based test for local stationarity applied to array processing in sonar array processing a challenging problem is the estimation of the data covariance matrix in the presence of moving targets in the water column since the time interval of data local stationarity is limited this work describes an eigenvector based method for proper data segmentation into intervals that exhibit local stationarity providing data driven higher bounds for the number of snapshots available for computation of time varying sample covariance matrices application of the test is illustrated with simulated data in a horizontal array for the detection of a quiet source in the presence of a loud interferer

fluctuations of spiked random matrix models and failure diagnosis in sensor networks in this paper the joint fluctuations of the extreme eigenvalues and eigenvectors of a large dimensional sample covariance matrix are analyzed when the associated population covariance matrix is a finite rank perturbation of the identity matrix corresponding to the so called spiked model in random matrix theory the asymptotic fluctuations as the matrix size grows large are shown to be intimately linked with matrices from the gaussian unitary ensemble when the spiked population eigenvalues have unit multiplicity the fluctuations follow a central limit theorem this result is used to develop an original framework for the detection and diagnosis of local failures in large sensor networks for known or unknown failure magnitude

the distribution and moments of the smallest eigenvalue of a random matrix of wishart type abstract given a random rectangular m n matrix with elements from a normal distribution what is the distribution of the smallest singular value to pose an equivalent question in the language of multivariate statistics what is the distribution of the smallest eigenvalue of a matrix from the central wishart distribution in the null case we give new results giving the distribution as a simple recursion this includes the more difficult case when n m is an even integer without resorting to zonal polynomials and hypergeometric functions of matrix arguments with the recursion one can obtain exact expressions for the density and the moments of the distribution in terms of functions usually no more complicated than polynomials exponentials and at worst ordinary hypergeometric functions we further elaborate on the special cases when n m 0 1 2 and 3 and give a numerical table of the expected values for 2 m 25 and 0 n m 25

limits of spiked random matrices ii given a large high dimensional sample from a spiked population the top sample covariance eigenvalue is known to exhibit a phase transition we show that the largest eigenvalues have asymptotic distributions near the phase transition in the rank one spiked real wishart setting and its general analogue proving a conjecture of baik et al ann probab 33 1643 1697 2005 we also treat shifted mean gaussian orthogonal and ensembles such results are entirely new in the real case in the complex case we strengthen existing results by providing optimal scaling assumptions one obtains the known limiting random schrodinger operator on the half line but the boundary condition now depends on the perturbation we derive several characterizations of the limit laws in which appears as a parameter including a simple linear boundary value problem this pde description recovers known explicit formulas at 2 4 yielding in particular a new and simple proof of the painleve representations for these tracy widom distributions

estimating the proportion of false null hypotheses among a large number of independently tested hypotheses we consider the problem of estimating the number of false null hypotheses among a very large number of independently tested hypotheses focusing on the situation in which the proportion of false null hypotheses is very small we propose a family of methods for establishing lower 100 1 a confidence bounds for this proportion based on the empirical distribution of the p values of the tests methods in this family are then compared in terms of ability to consistently estimate the proportion by letting 0 as the number of hypothesis tests increases and the proportion decreases this work is motivated by a signal detection problem that occurs in astronomy

matrix models for beta ensembles this paper constructs tridiagonal random matrix models for general 0 hermite gaussian and laguerre wishart ensembles these generalize the well known gaussian and wishart models for 1 2 4 furthermore in the cases of the laguerre ensembles we eliminate the exponent quantization present in the previously known models we further discuss applications for the new matrix models and present some open problems

properties of higher criticism under strong dependence the problem of signal detection using sparse faint information is closely related to a variety of contemporary statistical problems including the control of false discovery rate and classification using very high dimensional data each problem can be solved by conducting a large number of simultaneous hypothesis tests the properties of which are readily accessed under the assumption of independence in this paper we address the case of dependent data in the context of higher criticism methods for signal detection short range dependence has no first order impact on performance but the situation changes dramatically under strong dependence there although higher criticism can continue to perform well it can be bettered using methods based on differences of signal values or on the maximum of the data the relatively inferior performance of higher criticism in such cases can be explained in terms of the fact that under strong dependence the higher criticism statistic behaves as though the data were partitioned into very large blocks with all but a single representative of each block being eliminated from the dataset

asymptotic behavior of support vector machine for spiked population model for spiked population model we investigate the large dimension n and large sample size m asymptotic behavior of the support vector machine svm classification method in the limit of n m at fixed m n we focus on the generalization performance by analytically evaluating the angle between the normal direction vectors of svm separating hyperplane and corresponding bayes optimal separating hyperplane this is an analogous result to the one shown in paul 2007 and nadler 2008 for the angle between the sample eigenvector and the population eigenvector in random matrix theorem we provide not just bound but sharp prediction of the asymptotic behavior of svm that can be determined by a set of nonlinear equations based on the analytical results we propose a new method of selecting tuning parameter which significantly reduces the computational cost a surprising finding is that svm achieves its best performance at small value of the tuning parameter under spiked population model these results are confirmed to be correct by comparing with those of numerical simulations on finite size systems we also apply our formulas to an actual dataset of breast cancer and find agreement between analytical derivations and numerical computations based on cross validation

distance weighted discrimination high dimension low sample size statistical analysis is becoming increasingly important in a wide range of applied contexts in such situations the popular support vector machine suffers from data piling at the margin which can diminish generalizability this leads naturally to the development of distance weighted discrimination which is based on second order cone programming a modern computationally intensive optimization method

universality at the edge of the spectrum in wigner random matrices we prove universality at the edge for rescaled correlation functions of wigner random matrices in the limit n as a corollary we show that after proper rescaling the 1th 2nd 3rd etc eigenvalues of wigner random hermitian resp real symmetric matrix weakly converge to the distributions established by tracy and widom in g u e g o e cases

robust nearest neighbor methods for classifying high dimensional data we suggest a robust nearest neighbor approach to classifying high dimensional data the method enhances sensitivity by employing a threshold and truncates to a sequence of zeros and ones in order to reduce the deleterious impact of heavy tailed data empirical rules are suggested for choosing the threshold they require the bare minimum of data only one data vector is needed from each population theoretical and numerical aspects of performance are explored paying particular attention to the impacts of correlation and heterogeneity among data components on the theoretical side it is shown that our truncated thresholded nearest neighbor classifier enjoys the same classification boundary as more conventional nonrobust approaches which require finite moments in order to achieve good performance in particular the greater robustness of our approach does not come at the price of reduced effectiveness moreover when both training sample sizes equal 1 our new method can have performance equal to that of optimal classifiers that require independent and identically distributed data with known marginal distributions yet our classifier does not itself need conditions of this type

multivariate analysis and jacobi ensembles largest eigenvalue tracy widom limits and rates of convergence abstract let a and b be independent central wishart matrices in p variables with common covariance and having m and n degrees of freedom respectively the distribution of the largest eigenvalue of a b 1 b has numerous applications in multivariate statistics but is difficult to calculate exactly suppose that m and n grow in proportion to p we show that after centering and scaling the distribution is approximated to second order o p 2 3 by the tracy widom law the results are obtained for both complex and then real valued data by using methods of random matrix theory to study the largest eigenvalue of the jacobi unitary and orthogonal ensembles asymptotic approximations of jacobi polynomials near the largest zero play a central role

painleve formulas of the limiting distributions for nonnull complex sample covariance matrices in a recent study of large non null sample covariance matrices a new sequence of functions gener alizing the gue tracy widom distribution of random matrix theory was obtained this paper derives painleve formulas of these functions and use them to prove that they are indeed distribution functions applications of these new distribution functions to last passage percolation queues in tandem and to tally asymmetric simple exclusion process are also discussed as a part of the proof a representation of orthogonal polynomials on the unit circle in terms of an operator on a discrete set is presented

stable subspace tracking algorithm based on a signed urv decomposition subspace estimation and tracking are of fundamental importance in many signal processing algorithms the class of schur subspace estimators provides a complete parametrization of all principal subspace estimates defined as the column spans of corresponding low rank matrix approximants that lie within a specified 2 norm distance of a given matrix the parametrization is found in terms of a two sided hyperbolic decomposition hyperbolic urv or hurv which can be computed using hyperbolic rotations unfortunately such rotations are commonly associated with numerical instabilities in this paper we present a numerically stable non iterative algorithm to compute the hurv called the signed urv surv algorithm we show that this algorithm implicitly imposes certain constraints on the hurv such that important norm bounds that guarantee stability are satisfied the constraints also restrict the parametrization of the subspace estimate such that it becomes close to the principal subspace provided by the svd which is a special case within this class the complexity of the algorithm is of the same order as that of a qr update updating and downdating are of the same complexity and are both numerically stable surv is proven to provide rank estimates consistent with the svd with the same rank threshold it can replace an svd where only subspace estimation is needed typical applications would e g be the detection of the number of signals in array signal processing and subspace estimation for source separation and interference mitigation such as the first step in music and esprit type algorithms simulation results demonstrate the numerical stability and confirm that this algorithm provides exact rank estimates and good principal subspace estimates as compared to the svd

theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes summary we suggest a technique related to the concept of detection boundary that was developed by ingster and by donoho and jin for comparing the theoretical performance of classifiers constructed from small training samples of very large vectors the resulting classification boundaries are obtained for a variety of distance based methods including the support vector machine distance weighted discrimination and kth nearest neighbour classifiers for thresholded forms of those methods and for techniques based on donoho and jin s higher criticism approach to signal detection assessed in these terms standard distance based methods are shown to be capable only of detecting differences between populations when those differences can be estimated consistently however the thresholded forms of distance based classifiers can do better and in particular can correctly classify data even when differences between distributions are only detectable not estimable other methods including higher criticism classifiers can on occasion perform better still but they tend to be more limited in scope requiring substantially more information about the marginal distributions moreover as tail weight becomes heavier the classification boundaries of methods designed for particular distribution types can converge to and achieve the boundary for thresholded nearest neighbour approaches for example although higher criticism has a lower classification boundary and in this sense performs better in the case of normal data the boundaries are identical for exponentially distributed data when both sample sizes equal 1

higher criticism for detecting sparse heterogeneous mixtures higher criticism or second level significance testing is a multiplecomparisons concept mentioned in passing by tukey it concerns a situation where there are many independent tests of significance and one is interested in rejecting the joint null hypothesis tukey suggested comparing the fraction of observed significances at a given level to the expected fraction under the joint null in fact he suggested standardizing the difference of the two quantities and forming a z score the resulting z score tests the significance of the body of significance tests we consider a generalization where we maximize this z score over a range of significance levels 0 0 we are able to show that the resulting higher criticism statistic is effective at resolving a very subtle testing problem testing whether n normal means are all zero versus the alternative that a small fraction is nonzero the subtlety of this sparse normal means testing problem can be seen from work of ingster and jin who studied such problems in great detail in their studies they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution of the p values either for the bulk of the tests or for the few most highly significant tests in this range when the amplitude of nonzero means is calibrated with the fraction of nonzero means the likelihood ratio test for a precisely specified alternative would still succeed in separating the two hypotheses we show that the higher criticism is successful throughout the same region of amplitude sparsity where the likelihood ratio test

approximation of haar distributed matrices and limiting distributions of eigenvalues of jacobi ensembles we develop a tool to approximate the entries of a large dimensional complex jacobi ensemble with independent complex gaussian random variables based on this and the author s earlier work in this direction we obtain the tracy widom law of the largest singular values of the jacobi emsemble moreover the circular law the marchenko pastur law the central limit theorem and the laws of large numbers for the spectral norms are also obtained

geometric representation of high dimension low sample size data summary high dimension low sample size data are emerging in various areas of science we find a common structure underlying many such data sets by using a non standard type of asymptotics the dimension tends to while the sample size is fixed our analysis shows a tendency for the data to lie deterministically at the vertices of a regular simplex essentially all the randomness in the data appears only as a random rotation of this simplex this geometric representation is used to obtain several new statistical insights

numerical evaluation of airy type integrals arising in uniform asymptotic analysis we describe a method to evaluate integrals that arise in the asymptotic analysis when two saddle points may be close together these integrals which appear in problems from optics acoustics or quantum mechanics as well as in a wide class of special functions can be transformed into airy type integrals and we use the trapezoidal rule to compute these integrals numerically the quadrature method which remains valid when two saddle points coalesce is illustrated with numerical examples

higher criticism thresholding optimal feature selection when useful features are rare and weak in important application fields today genomics and proteomics are examples selecting a small subset of useful features is crucial for success of linear classification analysis we study feature selection by thresholding of feature z scores and introduce a principle of threshold selection based on the notion of higher criticism hc for i 1 2 p let i denote the two sided p value associated with the ith feature z score and i denote the ith order statistic of the collection of p values the hc threshold is the absolute z score corresponding to the p value maximizing the hc objective i p i i p 1 i p we consider a rare weak rw feature model where the fraction of useful features is small and the useful features are each too weak to be of much use on their own hc thresholding hct has interesting behavior in this setting with an intimate link between maximizing the hc objective and minimizing the error rate of the designed classifier and very different behavior from popular threshold selection procedures such as false discovery rate thresholding fdrt in the most challenging rw settings hct uses an unconventionally low threshold this keeps the missed feature detection rate under better control than fdrt and yields a classifier with improved misclassification performance replacing cross validated threshold selection in the popular shrunken centroid classifier with the computationally less expensive and simpler hct reduces the variance of the selected threshold and the error rate of the constructed classifier results on standard real datasets and in asymptotic theory confirm the advantages of hct

optimal detection of sparse mixtures against a given null distribution detection of sparse signals arises in a wide range of modern scientific studies the focus so far has been mainly on gaussian mixture models in this paper we consider the detection problem under a general sparse mixture model and obtain explicit expressions for the detection boundary under mild regularity conditions in addition for gaussian null hypothesis we establish the adaptive optimality of the higher criticism procedure for all sparse mixtures satisfying the same conditions in particular the general results obtained in this paper recover and extend in a unified manner the previously known results on sparse detection far beyond the conventional gaussian model and other exponential families

sparse principal component analysis and iterative thresholding principal component analysis pca is a classical dimension reduction method which projects data onto the principal subspace spanned by the leading eigenvectors of the covariance matrix however it behaves poorly when the number of features p is comparable to or even much larger than the sample size n in this paper we propose a new iterative thresholding approach for estimating principal subspaces in the setting where the leading eigenvectors are sparse under a spiked covariance model we find that the new approach recovers the principal subspace and leading eigenvectors consistently and even optimally in a range of high dimensional sparse settings simulated examples also demonstrate its competitive performance

penalized discriminant analysis fisher s linear discriminant analysis lda is a popular data analytic tool for studying the relationship between a set of predictors and a categorical response in this paper we describe a penalized version of lda it is designed for situations in which there are many highly correlated predictors such as those obtained by discretizing a function or the grey scale values of the pixels in a series of images in cases such as these it is natural efficient and sometimes essential to impose a spatial smoothness constraint on the coefficients both for improved prediction performance and interpretability we cast the classification problem into a regression framework via optimal scoring using this our proposal facilitates the use of any penalized regression technique in the classification setting the technique is illustrated with examples in speech recognition and handwritten character recognition

a cure for variance inflation in high dimensional kernel principal component analysis small sample high dimensional principal component analysis pca suffers from variance inflation and lack of generalizability it has earlier been pointed out that a simple leave one out variance renormalization scheme can cure the problem in this paper we generalize the cure in two directions first we propose a computationally less intensive approximate leave one out estimator secondly we show that variance inflation is also present in kernel principal component analysis kpca and we provide a non parametric renormalization scheme which can quite efficiently restore generalizability in kpca as for pca our analysis also suggests a simplified approximate expression

deterministic parallel analysis an improved method for selecting factors and principal components factor analysis and principal component analysis are used in many application areas the first step choosing the number of components remains a serious challenge our work proposes improved methods for this important problem one of the most popular state of the art methods is parallel analysis pa which compares the observed factor strengths with simulated strengths under a noise only model the paper proposes improvements to pa we first derandomize it proposing deterministic pa which is faster and more reproducible than pa both pa and deterministic pa are prone to a shadowing phenomenon in which a strong factor makes it difficult to detect smaller but more interesting factors we propose deflation to counter shadowing we also propose to raise the decision threshold to improve estimation accuracy we prove several consistency results for our methods and test them in simulations we also illustrate our methods on data from the human genome diversity project where they significantly improve the accuracy

high dimensional influence measure influence diagnosis is important since presence of influential observations could lead to distorted analysis and misleading interpretations for high dimensional data it is particularly so as the increased dimensionality and complexity may amplify both the chance of an observation being influential and its potential impact on the analysis in this article we propose a novel high dimensional influence measure for regressions with the number of predictors far exceeding the sample size our proposal can be viewed as a high dimensional counterpart to the classical cook s distance however whereas the cook s distance quantifies the individual observation s influence on the least squares regression coefficient estimate our new diagnosis measure captures the influence on the marginal correlations which in turn exerts serious influence on downstream analysis including coefficient estimation variable selection and screening moreover we establish the asymptotic distribution of the proposed influence measure by letting the predictor dimension go to infinity availability of this asymptotic distribution leads to a principled rule to determine the critical value for influential observation detection both simulations and real data analysis demonstrate usefulness of the new influence diagnosis measure

hypothesis testing on linear structures of high dimensional covariance matrix this paper is concerned with test of significance on high dimensional covariance structures and aims to develop a unified framework for testing commonly used linear covariance structures we first construct a consistent estimator for parameters involved in the linear covariance structure and then develop two tests for the linear covariance structures based on entropy loss and quadratic loss used for covariance matrix estimation to study the asymptotic properties of the proposed tests we study related high dimensional random matrix theory and establish several highly useful asymptotic results with the aid of these asymptotic results we derive the limiting distributions of these two tests under the null and alternative hypotheses we further show that the quadratic loss based test is asymptotically unbiased we conduct monte carlo simulation study to examine the finite sample performance of the two tests our simulation results show that the limiting null distributions approximate their null distributions quite well and the corresponding asymptotic critical values keep type i error rate very well our numerical comparison implies that the proposed tests outperform existing ones in terms of controlling type i error rate and power our simulation indicates that the test based on quadratic loss seems to have better power than the test based on entropy loss

counterexamples to the maximal p norm multiplicativity conjecture for all p 1 for all p 1 we demonstrate the existence of quantum channels with non multiplicative maximal output p norms equivalently for all p 1 the minimum output renyi entropy of order p of a quantum channel is not additive the violations found are large in all cases the minimum output renyi entropy of order p for a product channel need not be significantly greater than the minimum output entropy of its individual factors since p 1 corresponds to the von neumann entropy these counterexamples demonstrate that if the additivity conjecture of quantum information theory is true it cannot be proved as a consequence of any channel independent guarantee of maximal p norm multiplicativity we also show that a class of channels previously studied in the context of approximate encryption lead to counterexamples for all p 2