sparse linear discriminant analysis by thresholding for high dimensional data in many social economical biological and medical studies one objective is to classify a subject into one of several classes based on a set of variables observed from the subject because the probability distribution of the variables is usually unknown the rule of classification is constructed using a training sample the well known linear discriminant analysis lda works well for the situation where the number of variables used for classification is much smaller than the training sample size because of the advance in technologies modern statistical studies often face classification problems with the number of variables much larger than the sample size and the lda may perform poorly we explore when and why the lda has poor performance and propose a sparse lda that is asymptotically optimal under some sparsity conditions on the unknown parameters for illustration of application we discuss an example of classifying human cancer into two classes of leukemia based on a set of 7 129 genes and a training sample of size 72 a simulation is also conducted to check the performance of the proposed method

innovated higher criticism for detecting sparse signals in correlated noise higher criticism is a method for detecting signals that are both sparse and weak although first proposed in cases where the noise variables are independent higher criticism also has reasonable performance in settings where those variables are correlated in this paper we show that by exploiting the nature of the correlation performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer indeed it turns out that the case of independent noise is the most difficult of all from a statistical viewpoint and that more accurate signal detection for a given level of signal sparsity and strength can be obtained when correlation is present we characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary the boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is toeplitz

deblurring saturated night image with function form kernel deblurring saturated night images are a challenging problem because such images have low contrast combined with heavy noise and saturated regions unlike the deblurring schemes that discard saturated regions when estimating blur kernels this paper proposes a novel scheme to deduce blur kernels from saturated regions via a novel kernel representation and advanced algorithms our key technical contribution is the proposed function form representation of blur kernels which regularizes existing matrix form kernels using three functional components 1 trajectory 2 intensity and 3 expansion from automatically detected saturated regions their skeleton brightness and width are fitted into the corresponding three functional components of blur kernels such regularization significantly improves the quality of kernels deduced from saturated regions second we propose an energy minimizing algorithm to select and assign the deduced function form kernels to partitioned image regions as the initialization for non uniform deblurring finally we convert the assigned function form kernels into matrix form for more detailed estimation in a multi scale deconvolution experimental results show that our scheme outperforms existing schemes on challenging real examples

local strong homogeneity of a regularized estimator this paper deals with regularized pointwise estimation of discrete signals which contain large strongly homogeneous zones where typically they are constant or linear or more generally satisfy a linear equation the estimate is defined as the minimizer of an objective function combining a quadratic data fidelity term and a regularization prior term the latter term is the sum of the values obtained by applying a potential function pf to each component called a difference of a linear transform of the signal minimizers of functions of this form arise in various settings in statistics and optimization the features exhibited by such an estimate are closely related to the shape of the pf our goal is to determine estimators providing solutions which involve large strongly homogeneous zones where more precisely the differences are null in spite of the noise corrupting the data to this end we require that the strongly homogeneous zones recovered by the estimator be insensitive to any variation of th

non parametric blur map regression for depth of field extension real camera systems have a limited depth of field dof which may cause an image to be degraded due to visible misfocus or too shallow dof in this paper we present a blind deblurring pipeline able to restore such images by slightly extending their dof and recovering sharpness in regions slightly out of focus to address this severely ill posed problem our algorithm relies first on the estimation of the spatially varying defocus blur drawing on local frequency image features a machine learning approach based on the recently introduced regression tree fields is used to train a model able to regress a coherent defocus blur map of the image labeling each pixel by the scale of a defocus point spread function a non blind spatially varying deblurring algorithm is then used to properly extend the dof of the image the good performance of our algorithm is assessed both quantitatively using realistic ground truth data obtained with a novel approach based on a plenoptic camera and qualitatively with real images

regularized estimation of large covariance matrices this paper considers estimating a covariance matrix of p variables from n observations by either banding the sample covariance matrix or estimating a banded version of the inverse of the covariance we show that these estimates are consistent in the operator norm as long as logp 2 n 0 and obtain explicit rates the results are uniform over some fairly natural well conditioned families of covariance matrices we also introduce an analogue of the gaussian white noise model and show that if the population covariance is embeddable in that model and well conditioned then the banded approximations produce consistent estimates of the eigenvalues and associated eigenvectors of the covariance matrix the results can be extended to smooth versions of banding and to non gaussian distributions with su ciently short tails a resampling approach is proposed for choosing the banding parameter in practice this approach is illustrated numerically on both simulated and real data

richardson lucy deblurring for scenes under a projective motion path this paper addresses how to model and correct image blur that arises when a camera undergoes ego motion while observing a distant scene in particular we discuss how the blurred image can be modeled as an integration of the clear scene under a sequence of planar projective transformations i e homographies that describe the camera s path this projective motion path blur model is more effective at modeling the spatially varying motion blur exhibited by ego motion than conventional methods based on space invariant blur kernels to correct the blurred image we describe how to modify the richardson lucy rl algorithm to incorporate this new blur model in addition we show that our projective motion rl algorithm can incorporate state of the art regularization priors to improve the deblurred results the projective motion path blur model along with the modified rl algorithm is detailed together with experimental results demonstrating its overall effectiveness statistical analysis on the algorithm s convergence properties and robustness to noise is also provided

optimal classification in sparse gaussian graphic model consider a two class classification problem where the number of features is much larger than the sample size the features are masked by gaussian noise with mean zero and covariance matrix sigma where the precision matrix omega sigma 1 is unknown but is presumably sparse the useful features also unknown are sparse and each contributes weakly i e rare and weak to the classification decision by obtaining a reasonably good estimate of omega we formulate the setting as a linear regression model we propose a two stage classification method where we first select features by the method of innovated thresholding it and then use the retained features and fisher s lda for classification in this approach a crucial problem is how to set the threshold of it we approach this problem by adapting the recent innovation of higher criticism thresholding hct we find that when useful features are rare and weak the limiting behavior of hct is essentially just as good as the limiting behavior of ideal threshold the threshold one would choose if the underlying distribution of the signals is known if only somewhat surprisingly when omega is sufficiently sparse its off diagonal coordinates usually do not have a major influence over the classification decision compared to recent work in the case where omega is the identity matrix proc natl acad sci usa 105 2008 14790 14795 philos trans r soc lond ser a math phys eng sci 367 2009 4449 4470 the current setting is much more general which needs a new approach and much more sophisticated analysis one key component of the analysis is the intimate relationship between hct and fisher s separation another key component is the tight large deviation bounds for empirical processes for data with unconventional correlation structures where graph theory on vertex coloring plays an important role

partial deconvolution with inaccurate blur kernel most non blind deconvolution methods are developed under the error free kernel assumption and are not robust to inaccurate blur kernel unfortunately despite the great progress in blind deconvolution estimation error remains inevitable during blur kernel estimation consequently severe artifacts such as ringing effects and distortions are likely to be introduced in the non blind deconvolution stage in this paper we tackle this issue by suggesting 1 a partial map in the fourier domain for modeling kernel estimation error and 2 a partial deconvolution model for robust deblurring with inaccurate blur kernel the partial map is constructed by detecting the reliable fourier entries of estimated blur kernel and partial deconvolution is applied to wavelet based and learning based models to suppress the adverse effect of kernel estimation error furthermore an e m algorithm is developed for estimating the partial map and recovering the latent sharp image alternatively experimental results show that our partial deconvolution model is effective in relieving artifacts caused by inaccurate blur kernel and can achieve favorable deblurring quality on synthetic and real blurry images

high dimensional classification using features annealed independence rules classification using high dimensional features arises frequently in many contemporary statistical studies such as tumor classification using microarray or other high throughput data the impact of dimensionality on classifications is largely poorly understood in a seminal paper bickel and levina 2004 show that the fisher discriminant performs poorly due to diverging spectra and they propose to use the independence rule to overcome the problem we first demonstrate that even for the independence classification rule classification using all the features can be as bad as the random guessing due to noise accumulation in estimating population centroids in high dimensional feature space in fact we demonstrate further that almost all linear discriminants can perform as bad as the random guessing thus it is paramountly important to select a subset of important features for high dimensional classification resulting in features annealed independence rules fair the conditions under which all the important features can be selected by the two sample t statistic are established the choice of the optimal number of features or equivalently the threshold value of the test statistics are proposed based on an upper bound of the classification error simulation studies and real data analysis support our theoretical results and demonstrate convincingly the advantage of our new classification procedure

minimax risk over l p balls for l p error consider estimating the mean vector from datan n 2 i withl q norm loss q 1 when is known to lie in ann dimensionall p ball p 0 for largen the ratio of minimaxlinear risk to minimax risk can bearbitrarily large ifp q obvious exceptions aside the limiting ratio equals 1 only ifp q 2 our arguments are mostly indirect involving a reduction to a univariate bayes minimax problem whenp q simple non linear co ordinatewise threshold rules are asymptotically minimax at small signal to noise ratios and within a bounded factor of asymptotic minimaxity in general we also give asymptotic evaluations of the minimax linear risk our results are basic to a theory of estimation in besov spaces using wavelet bases to appear elsewhere

a constrained l1 minimization approach to sparse precision matrix estimation this article proposes a constrained l1 minimization method for estimating a sparse inverse covariance matrix based on a sample of n iid p variate random variables the resulting estimator is shown to have a number of desirable properties in particular the rate of convergence between the estimator and the true s sparse precision matrix under the spectral norm is when the population distribution has either exponential type tails or polynomial type tails we present convergence rates under the elementwise l norm and frobenius norm in addition we consider graphical model selection the procedure is easily implemented by linear programming numerical performance of the estimator is investigated using both simulated and real data in particular the procedure is applied to analyze a breast cancer dataset and is found to perform favorably compared with existing methods

image deblurring with an inaccurate blur kernel using a group based low rank image prior abstract we address the problem of restoring an original image from its blurry and noisy observation together with inaccurate information of the blurring process for this purpose we propose an enhanced regularized structured total least squares rstls model that can estimate the latent image and blur kernel simultaneously in the proposed model both the image and the blur kernel are characterized by a group based low rank prior which assumes that a group of vectorized similar data patches can be well approximated by a low rank matrix we develop an alternating minimization algorithm to solve the proposed model efficiently numerical experiments demonstrate the effectiveness of our method in terms of both quantitative measures and visual quality

estimation and confidence sets for sparse normal mixtures for high dimensional statistical models researchers have begun to focus on situations which can be described as having relatively few moderately large coefficients such situations lead to some very subtle statistical problems in particular ingster and donoho and jin have considered a sparse normal means testing problem in which they described the precise demarcation or detection boundary meinshausen and rice have shown that it is even possible to estimate consistently the fraction of nonzero coordinates on a subset of the detectable region but leave unanswered the question of exactly in which parts of the detectable region consistent estimation is possible in the present paper we develop a new approach for estimating the fraction of nonzero means for problems where the nonzero means are moderately large we show that the detection region described by ingster and donoho and jin turns out to be the region where it is possible to consistently estimate the expected fraction of nonzero coordinates this theory is developed further and minimax rates of convergence are derived a procedure is constructed which attains the optimal rate of convergence in this setting furthermore the procedure also provides an honest lower bound for confidence intervals while minimizing the expected length of such an interval simulations are used to enable comparison with the work of meinshausen and rice where a procedure is given but where rates of convergence have not been discussed extensions to more general gaussian mixture models are also given

theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes summary we suggest a technique related to the concept of detection boundary that was developed by ingster and by donoho and jin for comparing the theoretical performance of classifiers constructed from small training samples of very large vectors the resulting classification boundaries are obtained for a variety of distance based methods including the support vector machine distance weighted discrimination and kth nearest neighbour classifiers for thresholded forms of those methods and for techniques based on donoho and jin s higher criticism approach to signal detection assessed in these terms standard distance based methods are shown to be capable only of detecting differences between populations when those differences can be estimated consistently however the thresholded forms of distance based classifiers can do better and in particular can correctly classify data even when differences between distributions are only detectable not estimable other methods including higher criticism classifiers can on occasion perform better still but they tend to be more limited in scope requiring substantially more information about the marginal distributions moreover as tail weight becomes heavier the classification boundaries of methods designed for particular distribution types can converge to and achieve the boundary for thresholded nearest neighbour approaches for example although higher criticism has a lower classification boundary and in this sense performs better in the case of normal data the boundaries are identical for exponentially distributed data when both sample sizes equal 1

higher criticism for detecting sparse heterogeneous mixtures higher criticism or second level significance testing is a multiplecomparisons concept mentioned in passing by tukey it concerns a situation where there are many independent tests of significance and one is interested in rejecting the joint null hypothesis tukey suggested comparing the fraction of observed significances at a given level to the expected fraction under the joint null in fact he suggested standardizing the difference of the two quantities and forming a z score the resulting z score tests the significance of the body of significance tests we consider a generalization where we maximize this z score over a range of significance levels 0 0 we are able to show that the resulting higher criticism statistic is effective at resolving a very subtle testing problem testing whether n normal means are all zero versus the alternative that a small fraction is nonzero the subtlety of this sparse normal means testing problem can be seen from work of ingster and jin who studied such problems in great detail in their studies they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution of the p values either for the bulk of the tests or for the few most highly significant tests in this range when the amplitude of nonzero means is calibrated with the fraction of nonzero means the likelihood ratio test for a precisely specified alternative would still succeed in separating the two hypotheses we show that the higher criticism is successful throughout the same region of amplitude sparsity where the likelihood ratio test

goodness of fit tests via phi divergences a unified family of goodness of fit tests based on o divergences is introduced and studied the new family of test statistics s n s includes both the supremum version of the anderson darling statistic and the test statistic of berk and jones z wahrsch verw gebiete 47 1979 47 59 as special cases s 2 and s 1 resp we also introduce integral versions of the new statistics we show that the asymptotic null distribution theory of berk and jones z wahrsch verw gebiete 47 1979 47 59 and wellner and koltchinskii high dimensional probability iii 2003 321 332 birkhauser basel for the berk jones statistic applies to the whole family of statistics s n s with s 1 2 on the side of power behavior we study the test statistics under fixed alternatives and give extensions of the poisson boundary phenomena noted by berk and jones for their statistic we also extend the results of donoho and jin ann statist 32 2004 962 994 by showing that all our new tests for s 1 2 have the same optimal detection boundary for normal shift mixture alternatives as tukey s higher criticism statistic and the berk jones statistic

empirical bayes estimates for large scale prediction problems classical prediction methods such as fisher s linear discriminant function were designed for small scale problems in which the number of predictors n is much smaller than the number of observations n modern scientific devices often reverse this situation a microarray analysis for example might include n 100 subjects measured on n 10 000 genes each of which is a potential predictor this article proposes an empirical bayes approach to large scale prediction where the optimum bayes prediction rule is estimated employing the data from all of the predictors microarray examples are used to illustrate the method the results demonstrate a close connection with the shrunken centroids algorithm of tibshirani et al 2002 a frequentist regularization approach to large scale prediction and also with false discovery rate theory

robust image deblurring with an inaccurate blur kernel most existing nonblind image deblurring methods assume that the blur kernel is free of error however it is often unavoidable in practice that the input blur kernel is erroneous to some extent sometimes the error could be severe e g for images degraded by nonuniform motion blurring when an inaccurate blur kernel is used as the input significant distortions will appear in the image recovered by existing methods in this paper we present a novel convex minimization model that explicitly takes account of error in the blur kernel the resulting minimization problem can be efficiently solved by the so called accelerated proximal gradient method in addition a new boundary extension scheme is incorporated in the proposed model to further improve the results the experiments on both synthesized and real images showed the efficiency and robustness of our algorithm to both the image noise and the model error in the blur kernel

a direct estimation approach to sparse linear discriminant analysis this article considers sparse linear discriminant analysis of high dimensional data in contrast to the existing methods which are based on separate estimation of the precision matrix and the difference of the mean vectors we introduce a simple and effective classifier by estimating the product directly through constrained l1 minimization the estimator can be implemented efficiently using linear programming and the resulting classifier is called the linear programming discriminant lpd rule the lpd rule is shown to have desirable theoretical and numerical properties it exploits the approximate sparsity of and as a consequence allows cases where it can still perform well even when and or cannot be estimated consistently asymptotic properties of the lpd rule are investigated and consistency and rate of convergence results are given the lpd classifier has superior finite sample performance and significant computational advantages over the existing methods that require separate estimation

higher criticism thresholding optimal feature selection when useful features are rare and weak in important application fields today genomics and proteomics are examples selecting a small subset of useful features is crucial for success of linear classification analysis we study feature selection by thresholding of feature z scores and introduce a principle of threshold selection based on the notion of higher criticism hc for i 1 2 p let i denote the two sided p value associated with the ith feature z score and i denote the ith order statistic of the collection of p values the hc threshold is the absolute z score corresponding to the p value maximizing the hc objective i p i i p 1 i p we consider a rare weak rw feature model where the fraction of useful features is small and the useful features are each too weak to be of much use on their own hc thresholding hct has interesting behavior in this setting with an intimate link between maximizing the hc objective and minimizing the error rate of the designed classifier and very different behavior from popular threshold selection procedures such as false discovery rate thresholding fdrt in the most challenging rw settings hct uses an unconventionally low threshold this keeps the missed feature detection rate under better control than fdrt and yields a classifier with improved misclassification performance replacing cross validated threshold selection in the popular shrunken centroid classifier with the computationally less expensive and simpler hct reduces the variance of the selected threshold and the error rate of the constructed classifier results on standard real datasets and in asymptotic theory confirm the advantages of hct

a road to classification in high dimensional space the regularized optimal affine discriminant for high dimensional classification it is well known that naively performing the fisher discriminant rule leads to poor results due to diverging spectra and noise accumulation therefore researchers proposed independence rules to circumvent the diverging spectra and sparse independence rules to mitigate the issue of noise accumulation however in biological applications there are often a group of correlated genes responsible for clinical outcomes and the use of the covariance information can significantly reduce misclassification rates in theory the extent of such error rate reductions is unveiled by comparing the misclassification rates of the fisher discriminant rule and the independence rule to materialize the gain based on finite samples a regularized optimal affine discriminant road is proposed road selects an increasing number of features as the regularization relaxes further benefits can be achieved when a screening method is employed to narrow the feature pool before hitting the road an efficient constrained coordinate descent algorithm ccd is also developed to solve the associated optimization problems sampling properties of oracle type are established simulation studies and real data analysis support our theoretical results and demonstrate the advantages of the new classification procedure under a variety of correlation structures a delicate result on continuous piecewise linear solution path for the road optimization problem at the population level justifies the linear interpolation of the ccd algorithm

high order total variation based image restoration the total variation tv denoising method is a pde based technique that preserves edges well but has the sometimes undesirable staircase effect namely the transformation of smooth regions ramps into piecewise constant regions stairs in this paper we present an improved model constructed by adding a nonlinear fourth order diffusive term to the euler lagrange equations of the variational tv model our technique substantially reduces the staircase effect while preserving sharp jump discontinuities edges we show numerical evidence of the power of resolution of this novel model with respect to the tv model in some 1d and 2d numerical examples