high dimensional semiparametric scale invariant principal component analysis we propose a new high dimensional semiparametric principal component analysis pca method named copula component analysis coca the semiparametric model assumes that after unspecified marginally monotone transformations the distributions are multivariate gaussian coca improves upon pca and sparse pca in three aspects i it is robust to modeling assumptions ii it is robust to outliers and data contamination iii it is scale invariant and yields more interpretable results we prove that the coca estimators obtain fast estimation rates and are feature selection consistent when the dimension is nearly exponentially large relative to the sample size careful experiments confirm that coca outperforms sparse pca on both synthetic and real world data sets

a direct formulation for sparse pca using semidefinite programming given a covariance matrix we consider the problem of maximizing the variance explained by a particular linear combination of the input variables while constraining the number of nonzero coefficients in this combination this problem arises in the decomposition of a covariance matrix into sparse factors or sparse principal component analysis pca and has wide applications ranging from biology to finance we use a modification of the classical variational representation of the largest eigenvalue of a symmetric matrix where cardinality is constrained and derive a semidefinite programming based relaxation for our problem we also discuss nesterov s smooth minimization technique applied to the semidefinite program arising in the semidefinite relaxation of the sparse pca problem the method has complexity o n 4 sqrt log n epsilon where n is the size of the underlying covariance matrix and epsilon is the desired absolute accuracy on the optimal value of the problem

truncated power method for sparse eigenvalue problems this paper considers the sparse eigenvalue problem which is to extract dominant largest sparse eigenvectors with at most k non zero components we propose a simple yet effective solution called truncated power method that can approximately solve the underlying nonconvex optimization problem a strong sparse recovery result is proved for the truncated power method and this theory is our key motivation for developing the new algorithm the proposed method is tested on applications such as sparse principal component analysis and the densest k subgraph problem extensive experiments on several synthetic and real world data sets demonstrate the competitive empirical performance of our method

sparse principal component analysis via rotation and truncation sparse principal component analysis sparse pca aims at finding a sparse basis to improve the interpretability over the dense basis of pca while still covering the data subspace as much as possible in contrast to most existing work that addresses the problem by adding sparsity penalties on various objectives of pca we propose a new method sparse pca via rotation and truncation spcart which finds a rotation matrix and a sparse basis such that the sparse basis approximates the basis of pca after the rotation the algorithm of spcart consists of three alternating steps 1 rotating the pca basis 2 truncating small entries and 3 updating the rotation matrix its performance bounds are also given the spcart is efficient with each iteration scaling linearly with the data dimension parameter choice is simple due to explicit physical explanations we give a unified view to several existing sparse pca methods and discuss the connections with spcart some ideas from spcart are extended to gpower a popular sparse pca algorithm to address its limitations experimental results demonstrate that spcart achieves the state of the art performance along with a good tradeoff among various criteria including sparsity explained variance orthogonality balance of sparsity among loadings and computational speed

fast and guaranteed blind multichannel deconvolution under a bilinear system model we consider the multichannel blind deconvolution problem where we observe the output of multiple channels that are all excited with the same unknown input from these observations we wish to estimate the impulse responses of each of the channels we show that this problem is well posed if the channels follow a bilinear model where the ensemble of channel responses is modeled as lying in a low dimensional subspace but with each channel modulated by an independent gain under this model we show how the channel estimates can be found by minimizing a quadratic function over a non convex set we analyze two methods for solving this non convex program and provide performance guarantees for each the first is a method of alternating eigenvectors that breaks the program down into a series of eigenvalue problems the second is a truncated power iteration which can roughly be interpreted as a method for finding the largest eigenvector of a symmetric matrix with the additional constraint that it adheres to our bilinear model as with most non convex optimization algorithms the performance of both of these algorithms is highly dependent on having a good starting point we show how such a starting point can be constructed from the channel measurements our performance guarantees are non asymptotic and provide a sufficient condition on the number of samples observed per channel in order to guarantee channel estimates of certain accuracy our analysis uses a model with a generic subspace that is drawn at random and we show the performance bounds hold with high probability mathematically the key estimates are derived by quantifying how well the eigenvectors of certain random matrices approximate the eigenvectors of their mean we also present a series of numerical results demonstrating that the empirical performance is consistent with the presented theory

convergence of gradient descent for low rank matrix approximation this paper provides a proof of global convergence of gradient search for low rank matrix approximation such approximations have recently been of interest for large scale problems as well as for dictionary learning for sparse signal representations and matrix completion the proof is based on the interpretation of the problem as an optimization on the grassmann manifold and fubiny study distance on this space

near optimal stochastic approximation for online principal component estimation principal component analysis pca has been a prominent tool for high dimensional data analysis online algorithms that estimate the principal component by processing streaming data are of tremendous practical and theoretical interests despite its rich applications theoretical convergence analysis remains largely open in this paper we cast online pca into a stochastic nonconvex optimization problem and we analyze the online pca algorithm as a stochastic approximation iteration the stochastic approximation iteration processes data points incrementally and maintains a running estimate of the principal component we prove for the first time a nearly optimal finite sample error bound for the online pca algorithm under the subgaussian assumption we show that the finite sample error bound closely matches the minimax information lower bound

new results on the windy postman problem in this paper we study the windy postman problem wpp this is a well known arc routing problem that contains the mixed chinese postman problem mcpp as a special case we extend to arbitrary dimension some new inequalities that complete the description of the polyhedron associated with the windy postman problem over graphs with up to four vertices and ten edges we introduce two new families of facet inducing inequalities and prove that these inequalities along with the already known odd zigzag inequalities are chvatal gomory inequalities of rank at most 2 moreover a branch and cut algorithm that incorporates two new separation algorithms for all the previously mentioned inequalities and a new heuristic procedure to obtain upper bounds are presented finally the performance of a branch and cut algorithm over several sets of large wpp and mcpp instances with up to 3 000 nodes and 9 000 edges and arcs in the mcpp case shows that to our knowledge this is the best algorithm to date for the exact resolution of the wpp and the mcpp

on the distribution of the largest eigenvalue in principal components analysis let x 1 denote the square of the largest singular value of an n p matrix x all of whose entries are independent standard gaussian variates equivalently x 1 is the largest principal component variance of the covariance matrix x x or the largest eigenvalue of a p variate wishart distribution on n degrees of freedom with identity covariance consider the limit of large p and n with n p gamma ge 1 when centered by mu p sqrt n 1 sqrt p 2 and scaled by sigma p sqrt n 1 sqrt p 1 sqrt n 1 1 sqrt p 1 3 the distribution of x 1 approaches the tracey widom law of order 1 which is defined in terms of the painleve ii differential equation and can be numerically evaluated and tabulated in software simulations show the approximation to be informative for n and p as small as 5 the limit is derived via a corresponding result for complex wishart matrices using methods from random matrix theory the result suggests that some aspects of large p multivariate distribution theory may be easier to apply in practice than their fixed p counterparts

group factor analysis factor analysis fa provides linear factors that describe the relationships between individual variables of a data set we extend this classical formulation into linear factors that describe the relationships between groups of variables where each group represents either a set of related variables or a data set the model also naturally extends canonical correlation analysis to more than two sets in a way that is more flexible than previous extensions our solution is formulated as a variational inference of a latent variable model with structural sparsity and it consists of two hierarchical levels 1 the higher level models the relationships between the groups and 2 the lower models the observed variables given the higher level we show that the resulting solution solves the group factor analysis gfa problem accurately outperforming alternative fa based solutions as well as more straightforward implementations of gfa the method is demonstrated on two life science data sets one on brain activation and the other on systems biology illustrating its applicability to the analysis of different types of high dimensional data sources

asymptotically unbiased inference for a dynamic panel model with fixed effects when both n and t are large we consider a dynamic panel ar 1 model with fixed effects when both n and t are large under the t fixed n large asymptotic approximation the ordinary least squares ols or gaussian maximum likelihood estimator mle is known to be inconsistent due to the well known incidental parameter problem we consider an alternative asymptotic approximation where n and t grow at the same rate it is shown that although ols or the mle is asymptotically biased a relatively simple fix to ols or the mle results in an asymptotically unbiased estimator under the assumption of gaussian innovations the bias corrected mle is shown to be asymptotically efficient by a hajek type convolution theorem

orthogonal sparse pca and covariance estimation via procrustes reformulation the problem of estimating sparse eigenvectors of a symmetric matrix has attracted a lot of attention in many applications especially those with a high dimensional dataset while classical eigenvectors can be obtained as the solution of a maximization problem existing approaches formulate this problem by adding a penalty term into the objective function that encourages a sparse solution however the vast majority of the resulting methods achieve sparsity at the expense of sacrificing the orthogonality property in this paper we develop a new method to estimate dominant sparse eigenvectors without trading off their orthogonality the problem is highly nonconvex and hard to handle we apply the minorization maximization framework wherein we iteratively maximize a tight lower bound surrogate function of the objective function over the stiefel manifold the inner maximization problem turns out to be a rectangular procrustes problem which has a closed form solution in addition we propose a method to improve the covariance estimation problem when its underlying eigenvectors are known to be sparse we use the eigenvalue decomposition of the covariance matrix to formulate an optimization problem wherein we impose sparsity on the corresponding eigenvectors numerical experiments show that the proposed eigenvector extraction algorithm outperforms existing algorithms in terms of support recovery and explained variance whereas the covariance estimation algorithms improve the sample covariance estimator significantly

sparse pca optimal rates and adaptive estimation principal component analysis pca is one of the most commonly used statistical procedures with a wide range of applications this paper considers both minimax and adaptive estimation of the principal subspace in the high dimensional setting under mild technical conditions we first establish the optimal rates of convergence for estimating the principal subspace which are sharp with respect to all the parameters thus providing a complete characterization of the difficulty of the estimation problem in term of the convergence rate the lower bound is obtained by calculating the local metric entropy and an application of fano s lemma the rate optimal estimator is constructed using aggregation which however might not be computationally feasible we then introduce an adaptive procedure for estimating the principal subspace which is fully data driven and can be computed efficiently it is shown that the estimator attains the optimal rates of convergence simultaneously over a large collection of the parameter spaces a key idea in our construction is a reduction scheme which reduces the sparse pca problem to a high dimensional multivariate regression problem this method is potentially also useful for other related problems

another look at the instrumental variable estimation of error components models this article develops a framework for efficient iv estimators of random effects models with information in levels which can accommodate predetermined variables our formulation clarifies the relationship between the existing estimators and the role of transformations in panel data models we characterize the valid transformations for relevant models and show that optimal estimators are invariant to the transformation used to remove individual effects we present an alternative transformation for models with predetermined instruments which preserves the orthogonality among the errors finally we consider models with predetermined variables that have constant correlation with the effects and illustrate their importance with simulations

eca high dimensional elliptical component analysis in non gaussian distributions abstractwe present a robust alternative to principal component analysis pca called elliptical component analysis eca for analyzing high dimensional elliptically distributed data eca estimates the eigenspace of the covariance matrix of the elliptical data to cope with heavy tailed elliptical distributions a multivariate rank statistic is exploited at the model level we consider two settings either that the leading eigenvectors of the covariance matrix are nonsparse or that they are sparse methodologically we propose eca procedures for both nonsparse and sparse settings theoretically we provide both nonasymptotic and asymptotic analyses quantifying the theoretical performances of eca in the nonsparse setting we show that eca s performance is highly related to the effective rank of the covariance matrix in the sparse setting the results are twofold i we show that the sparse eca estimator based on a combinatoric program attains the optimal rate of convergence ii based on some recent d

multichannel sparse blind deconvolution on the sphere multichannel blind deconvolution is the problem of recovering an unknown signal f and multiple unknown channels x i from their circular convolution y i x i circledast f i 1 2 dots n we consider the case where the x i s are sparse and convolution with f is invertible our nonconvex optimization formulation solves for a filter h on the unit sphere that produces sparse output y i circledast h under some technical assumptions we show that all local minima of the objective function correspond to the inverse filter of f up to an inherent sign and shift ambiguity and all saddle points have strictly negative curvatures this geometric structure allows successful recovery of f and x i using a simple manifold gradient descent mgd algorithm the same approach is also applicable to blind gain and phase calibration with a fourier sensing matrix our algorithm and analysis require fewer assumptions than previous algorithms for the same problem our theoretical findings are complemented by numerical experiments which demonstrate superior performance of the proposed approach over the previous methods empirically our algorithm has low computation cost converging in a small number of iterations and low memory footprint solving only for the inverse filter of f

statistical and computational trade offs in estimation of sparse principal components in recent years sparse principal component analysis has emerged as an extremely popular dimension reduction technique for high dimensional data the theoretical challenge in the simplest case is to estimate the leading eigenvector of a population covariance matrix under the assumption that this eigenvector is sparse an impressive range of estimators have been proposed some of these are fast to compute while others are known to achieve the minimax optimal rate over certain gaussian or sub gaussian classes in this paper we show that under a widely believed assumption from computational complexity theory there is a fundamental trade off between statistical and computational performance in this problem more precisely working with new larger classes satisfying a restricted covariance concentration condition we show that there is an effective sample size regime in which no randomised polynomial time algorithm can achieve the minimax optimal rate we also study the theoretical performance of a polynomial time variant of the well known semidefinite relaxation estimator revealing a subtle interplay between statistical and computational efficiency

rate optimal denoising of simultaneously sparse and low rank matrices we study minimax rates for denoising simultaneously sparse and low rank matrices in high dimensions we show that an iterative thresholding algorithm achieves near optimal rates adaptively under mild conditions for a large class of loss functions numerical experiments on synthetic datasets also demonstrate the competitive performance of the proposed method

optimal rates of convergence for noisy sparse phase retrieval via thresholded wirtinger flow this paper considers the noisy sparse phase retrieval problem recovering a sparse signal mathbf x in mathbb r p from noisy quadratic measurements y j mathbf a j mathbf x 2 varepsilon j j 1 ldots m with independent sub exponential noise varepsilon j the goals are to understand the effect of the sparsity of mathbf x on the estimation precision and to construct a computationally feasible estimator to achieve the optimal rates adaptively inspired by the wirtinger flow ieee trans inform theory61 2015 1985 2007 proposed for non sparse and noiseless phase retrieval a novel thresholded gradient descent algorithm is proposed and it is shown to adaptively achieve the minimax optimal rates of convergence over a wide range of sparsity levels when the mathbf a j s are independent standard gaussian random vectors provided that the sample size is sufficiently large compared to the sparsity of mathbf x

non negative principal component analysis message passing algorithms and sharp asymptotics principal component analysis pca aims at estimating the direction of maximal variability of a high dimensional data set a natural question is does this task become easier and estimation more accurate when we exploit additional knowledge on the principal vector we study the case in which the principal vector is known to lie in the positive orthant similar constraints arise in a number of applications ranging from the analysis of gene expression data to spike sorting in neural signal processing in the unconstrained case the estimation performances of pca have been precisely characterized using the random matrix theory under a statistical model known as the spiked model it is known that the estimation error undergoes a phase transition as the signal to noise ratio crosses a certain threshold unfortunately tools from the random matrix theory have no bearing on the constrained problem despite this challenge we develop an analogous characterization in the constrained case within a one spike model in particular 1 we prove that the estimation error undergoes a similar phase transition albeit at a different thresholds in signal to noise ratio that we determine exactly 2 we prove that unlike in the unconstrained case the estimation error depends on the spike vector and characterize the least favorable vectors and 3 we show that a non negative principal component can be approximately computed under the spiked model in nearly linear time this despite the fact that the problem is non convex and in general np hard to solve exactly

sparse principal component analysis based on least trimmed squares sparse principal component analysis pca is used to obtain stable and interpretable principal components pcs from high dimensional data a robust sparse pca method is proposed to handle potentia

panel data and unobservable individual effects abstract an important purpose in pooling time series and cross section data is to control for individual specific unobservable effects which may be correlated with other explanatory variables e g latent ability in measuring returns to schooling in earnings equations or managerial ability in measuring returns to scale in firm cost functions using instrumental variables and the time invariant characteristics of the latent variable we derive 1 1 a test for the presence of this effect and for the over identifying restriction we use 2 2 necessary and sufficient conditions for identification of all the parameters in the model and 3 3 the asymptotically efficient instrumental variables estimator and conditions under which it differs from the within groups estimator we calculate efficient estimates of a wage equation from the michigan income dynamics data which indicate substantial differences from within groups and balestra nerlove estimates particularly a significantly higher estimate of the returns to schooling

majorization minimization algorithms in signal processing communications and machine learning this paper gives an overview of the majorization minimization mm algorithmic framework which can provide guidance in deriving problem driven algorithms with low computational cost a general introduction of mm is presented including a description of the basic principle and its convergence results the extensions acceleration schemes and connection to other algorithmic frameworks are also covered to bridge the gap between theory and practice upperbounds for a large number of basic functions derived based on the taylor expansion convexity and special inequalities are provided as ingredients for constructing surrogate functions with the pre requisites established the way of applying mm to solving specific problems is elaborated by a wide range of applications in signal processing communications and machine learning

quasi maximum likelihood estimators for spatial dynamic panel data with fixed effects when both n and t are large this paper investigates the asymptotic properties of quasi maximum likelihood estimators for spatial dynamic panel data with xed eects when both the number of individuals n and the number of time periods t are large we consider the case where t is asymptotically large relative to n the case where t is asymptotically proportional to n and the case where n is asymptotically large relative to t in the case where t is asymptotically large relative to n the estimators are p nt consistent and asymptotically normal with the limit distribution centered around 0 when n is asymptotically proportional to t the estimators are p nt consistent and asymptotically normal but the limit distribution is not centered around 0 and when n is large relative to t the estimators are consistent with rate t and have a degenerate limit distribution the estimators of the xed eects are p t consistent and asymptotically normal we also propose a bias correction for our estimators we show that when t grows faster than n 1 3 the correction will asymptotically eliminate the bias and yield a centered con dence interval

sparse principal component analysis and iterative thresholding principal component analysis pca is a classical dimension reduction method which projects data onto the principal subspace spanned by the leading eigenvectors of the covariance matrix however it behaves poorly when the number of features p is comparable to or even much larger than the sample size n in this paper we propose a new iterative thresholding approach for estimating principal subspaces in the setting where the leading eigenvectors are sparse under a spiked covariance model we find that the new approach recovers the principal subspace and leading eigenvectors consistently and even optimally in a range of high dimensional sparse settings simulated examples also demonstrate its competitive performance

sparse generalized eigenvalue problem optimal statistical rates via truncated rayleigh flow the sparse generalized eigenvalue problem gep plays a pivotal role in a large family of high dimensional statistical models including sparse fisher s discriminant analysis canonical correlation analysis and sufficient dimension reduction the sparse gep involves solving a non convex optimization problem most existing methods and theory in the context of specific statistical models that are special cases of the sparse gep require restrictive structural assumptions on the input matrices we propose a two stage computational framework to solve the sparse gep at the first stage we solve a convex relaxation of the sparse gep taking the solution as an initial value we then exploit a non convex optimization perspective and propose the truncated rayleigh flow method which we call rifle to estimate the leading generalized eigenvector we show that rifle converges linearly to a solution with the optimal statistical rate of convergence theoretically our method significantly improves on the existing literature by eliminating structural assumptions on the input matrices to achieve this our analysis involves two key ingredients a new analysis of the gradient based method on non convex objective functions and a fine grained characterization of the evolution of sparsity patterns along the solution path thorough numerical studies are provided to validate the theoretical results

heuristic and special case algorithms for dispersion problems the dispersion problem arises in selecting facilities to maximize some function of the distances between the facilities the problem also arises in selecting nondominated solutions for multiobjective decision making it is known to be np hard under two objectives maximizing the minimum distance max min between any pair of facilities and maximizing the average distance max avg we consider the question of obtaining near optimal solutions for max min we show that if the distances do not satisfy the triangle inequality there is no polynomial time relative approximation algorithm unless p np when the distances satisfy the triangle inequality we analyze an efficient heuristic and show that it provides a performance guarantee of two we also prove that obtaining a performance guarantee of less than two is np hard for max avg we analyze an efficient heuristic and show that it provides a performance guarantee of four when the distances satisfy the triangle inequality we also present a polynomial ti

optimal permutation recovery in permuted monotone matrix model motivated by recent research on quantifying bacterial growth dynamics based on genome assemblies we consider a permuted monotone matrix model y z where the rows represent different samples th

ruling out ptas for graph min bisection dense k subgraph and bipartite clique assuming that np not subseteq cap epsilon 0 bptime 2 n epsilon we show that graph min bisection dense k subgraph and bipartite clique have no polynomial time approximation scheme ptas we give a reduction from the minimum distance of code mdc problem starting with an instance of mdc we build a quasi random probabilistically checkable proof pcp that suffices to prove the desired inapproximability results in a quasi random pcp the query pattern of the verifier looks random in a certain precise sense among the several new techniques we introduce the most interesting one gives a way of certifying that a given polynomial belongs to a given linear subspace of polynomials as is important for our purpose the certificate itself happens to be another polynomial and it can be checked probabilistically by reading a constant number of its values

dense subgraph partition of positive hypergraphs in this paper we present a novel partition framework called dense subgraph partition dsp to automatically precisely and efficiently decompose a positive hypergraph into dense subgraphs a positive hypergraph is a graph or hypergraph whose edges except self loops have positive weights we first define the concepts of core subgraph conditional core subgraph and disjoint partition of a conditional core subgraph then define dsp based on them the result of dsp is an ordered list of dense subgraphs with decreasing densities which uncovers all underlying clusters as well as outliers a divide and conquer algorithm called min partition evolution is proposed to efficiently compute the partition dsp has many appealing properties first it is a nonparametric partition and it reveals all meaningful clusters in a bottom up way second it has an exact and efficient solution called min partition evolution algorithm the min partition evolution algorithm is a divide and conquer algorithm thus time efficient and memory friendly and suitable for parallel processing third it is a unified partition framework for a broad range of graphs and hypergraphs we also establish its relationship with the densest k subgraph problem d k s an np hard but fundamental problem in graph theory and prove that dsp gives precise solutions to d k s for all k in a graph dependent set called critical k set to our best knowledge this is a strong result which has not been reported before moreover as our experimental results show for sparse graphs especially web graphs the size of critical k set is close to the number of vertices in the graph we test the proposed partition framework on various tasks and the experimental results clearly illustrate its advantages

the sparse principal component of a constant rank matrix the computation of the sparse principal component of a matrix is equivalent to the identification of its principal submatrix with the largest maximum eigenvalue finding this optimal submatrix is what renders the problem np hard in this paper we prove that if the matrix is positive semidefinite and its rank is constant then its sparse principal component is polynomially computable our proof utilizes the auxiliary unit vector technique that has been recently developed to identify problems that are polynomially solvable in addition we use this technique to design an algorithm which for any sparsity value computes the sparse principal component with complexity o nd 1 where n and d are the matrix size and rank respectively our algorithm is fully parallelizable and memory efficient

on bias inconsistency and efficiency of various estimators in dynamic panel data models abstract when a model for panel data includes lagged dependent explanatory variables then the habitual estimation procedures are asymptotically valid only when the number of observations in the time dimension t gets large usually however such datasets have substantial sample size in the cross section dimension n whereas t is often a single digit number results on the asymptotic bias n in this situation have been published a decade ago but hence far analytic small sample assessments of the actual bias have not been presented here we derive a formula for the bias of the least squares dummy variable lsdv estimator which has a o n 1 t 3 2 approximation error in a simulation study this is found to be remarkably accurate due to the small variance of the lsdv estimator which is usually much smaller than the variance of consistent generalized method of moments estimators a very efficient procedure results when we remove the bias from the lsdv estimator the simulations contain results for a particular operational corrected lsdv estimation procedure which in many situations proves to be much more efficient than various instrumental variable type estimators