a penalized likelihood method for structural equation modeling a penalized likelihood pl method for structural equation modeling sem was proposed as a methodology for exploring the underlying relations among both observed and latent variables compared to the usual likelihood method pl includes a penalty term to control the complexity of the hypothesized model when the penalty level is appropriately chosen the pl can yield an sem model that balances the model goodness of fit and model complexity in addition the pl results in a sparse estimate that enhances the interpretability of the final model the proposed method is especially useful when limited substantive knowledge is available for model specifications the pl method can be also understood as a methodology that links the traditional sem to the exploratory sem asparouhov muthen in struct equ model multidiscipl j 16 397 438 2009 an expectation conditional maximization algorithm was developed to maximize the pl criterion the asymptotic properties of the proposed pl were also derived the performance of pl was evaluated through a numerical experiment and two real data illustrations were presented to demonstrate its utility in psychological research

generalized thresholding and online sparsity aware learning in a union of subspaces this paper considers a sparse signal recovery task in time varying time adaptive environments the contribution of the paper to sparsity aware online learning is threefold first a generalized thresholding gt operator which relates to both convex and non convex penalty functions is introduced this operator embodies in a unified way the majority of well known thresholding rules which promote sparsity second a non convexly constrained sparsity promoting online learning scheme namely the adaptive projection based generalized thresholding apgt is developed that incorporates the gt operator with a computational complexity that scales linearly to the number of unknowns third the novel family of partially quasi nonexpansive mappings is introduced as a functional analytic tool for treating the gt operator by building upon the rich fixed point theory the previous class of mappings establishes also a link between the gt operator and a union of linear subspaces a non convex object which lies at the heart of any sparsity promoting technique batch or online based on this functional analytic framework a convergence analysis of the apgt is provided extensive experiments suggest that the apgt exhibits competitive performance when compared to computationally more demanding alternatives such as the sparsity promoting affine projection algorithm apa and recursive least squares rls based techniques

identification of spatially varying parameters in distributed parameter systems by discrete regularization abstract identification of spatially varying parameters in distributed parameter systems from noisy data is an ill posed problem the regularization identification approach developed by c kravaris and j h seinfeld siam j control optim 23 1985 217 241 provides stable approximate solutions to that problem in this work a discretized minimization of the smoothing functional is proposed by using finite dimensional convergent approximations in sobolev spaces a convergence theorem for the discretized minimization of the smoothing functional is established the performance of this discrete regularization approach is evaluated by numerical experiments on the identification of spatially varying diffusivity in the two dimensional diffusion equation

identification of parabolic distributed parameter systems by regularization with differential operators abstract the theory of identification of variable coefficients in parabolic distributed parameter systems by regularization is extended to the case in which the stabilizing functional is the norm of a differential operator stability and convergence of the method are proved a detailed computational evaluation of the approach is presented via the estimation of a spatially varying conductivity in the two dimensional heat equation

estimation of the mean of a multivariate normal distribution estimation of the means of independent normal random variables is considered using sum of squared errors as loss an unbiased estimate of risk is obtained for an arbitrary estimate and certain special classes of estimates are then discussed the results are applied to smoothing by use of moving averages and to trimmed analogs of the james stein estimate a suggestion is made for calculating approximate confidence sets for the mean vector centered at an arbitrary estimate

optimized wavelet denoising for self similar alpha stable processes we investigate the performance of wavelet shrinkage methods for the denoising of symmetric alpha stable text s alpha text s self similar stochastic processes corrupted by additive white gaussian noise awgn where alpha is tied to the sparsity of the process the wavelet transform is assumed to be orthonormal and the shrinkage function minimizes the mean square approximation error mmse estimator we derive the corresponding formula for the expected value of the averaged estimation error we show that the predicted mmse is a monotone function of a simple criterion that depends on the wavelet and the statistical parameters of the process using the calculus of variations we then optimize this criterion to find the best performing wavelet within the extended family of meyer wavelets which are bandlimited these are compared with the daubechies wavelets which are compactly supported in time we find that the wavelets that are shorter in time in particular the haar basis are better suited to denoise the sparser processes say alpha while the bandlimited ones including the held and shannon wavelets offer the best performance for alpha 1 6 the limit corresponding to the gaussian case fbm with alpha 2

a method for solving ill posed linear operator equations an algorithm is presented to compute least squares solutions of the linear equation tx y where t is a bounded pperator from one hilbert space into another the algorithm is a generalization of marti s method the convergence to a solution minimizing the seminorm lx is proved where the operator l is a closed densely defined linear operator with closed range the method is applied to integral equations and three examples are treated numerically

heuristics of instability and stabilization in model selection in model selection usually a best predictor is chosen from a collection hat mu cdot s of predictors where hat mu cdot s is the minimum least squares predictor in a collection mathsf u s of predictors here s is a complexity parameter that is the smaller s the lower dimensional smoother the models in mathsf u s if mathsf l is the data used to derive the sequence hat mu cdot s the procedure is called unstable if a small change in mathsf l can cause large changes in hat mu cdot s with a crystal ball one could pick the predictor in hat mu cdot s having minimum prediction error without prescience one uses test sets cross validation and so forth the difference in prediction error between the crystal ball selection and the statistician s choice we call predictive loss for an unstable procedure the predictive loss is large this is shown by some analytics in a simple case and by simulation results in a more complex comparison of four different linear regression methods unstable procedures can be stabilized by perturbing the data getting a new predictor sequence hat mu cdot s and then averaging over many such predictor sequences

random calibration with many measurements an application of stein estimation in the problem considered a vector of many imprecise measurements e g spectroscopic is used to linearly predict a quantity whose precise measurement is difficult or expensive the regression vector is estimated from a calibration experiment having both types of measurements for a random sample most previous approaches to this problem adjust for approximate multicollinearity which often results from the correlations among the imprecise measurements by inverting an approximation e g factor analytic to their covariance matrix in the approach here it is argued that the regression vector should lie in a lower dimensional suhspace determined by the principal components of the covariance matrix it is then estimated by applying a stein contraction of the least squares estimator to the principal components regression estimator examples using real data are presented in which the proposed estimator substantially improves on the ordinary least squares estimation

a quasi newton method for estimating the parameter in a nonlinear hyperbolic system a quasi newton method qnm in infinite dimensional spaces for estimating parameters involved in distributed parameter systems is presented in this paper next the global convergence of a sequence generated by the algorithm qnm is also proved we apply the algorithm qnm to an identification problem of a distributed parameter system governed by a nonlinear hyperbolic partial differential equation

approximated penalized maximum likelihood for exploratory factor analysis an orthogonal case the problem of penalized maximum likelihood pml for an exploratory factor analysis efa model is studied in this paper an efa model is typically estimated using maximum likelihood and then the estimated loading matrix is rotated to obtain a sparse representation penalized maximum likelihood simultaneously fits the efa model and produces a sparse loading matrix to overcome some of the computational drawbacks of pml an approximation to pml is proposed in this paper it is further applied to an empirical dataset for illustration a simulation study shows that the approximation naturally produces a sparse loading matrix and more accurately estimates the factor loadings and the covariance matrix in the sense of having a lower mean squared error than factor rotations under various conditions

identification of distributed parameter systems with pointwise constraints on the parameters abstract the problem of estimating k x y in the parabolic equation u t x k x y u x y k x y u y f x y t on the basis of noisy measurements of u at discrete locations is considered subject to direct measurements of k at other discrete locations necessary conditions for optimality are derived based on a regularization formulation and the computational implementation is carried out via a finite element representation of k

sparse signal approximation via nonseparable regularization the calculation of a sparse approximate solution to a linear system of equations is often performed using either l1 norm regularization and convex optimization or nonconvex regularization and nonconvex optimization combining these principles this paper describes a type of nonconvex regularization that maintains the convexity of the objective function thereby allowing the calculation of a sparse approximate solution via convex optimization the preservation of convexity is viable in the proposed approach because it uses a regularizer that is nonseparable the proposed method is motivated and demonstrated by the calculation of sparse signal approximation using tight frames examples of denoising demonstrate improvement relative to l1 norm regularization

spectral regularization algorithms for learning large incomplete matrices we use convex relaxation techniques to provide a sequence of regularized low rank solutions for large scale matrix completion problems using the nuclear norm as a regularizer we provide a simple and very efficient convex algorithm for minimizing the reconstruction error subject to a bound on the nuclear norm our algorithm soft impute iteratively replaces the missing elements with those obtained from a soft thresholded svd with warm starts this allows us to efficiently compute an entire regularization path of solutions on a grid of values of the regularization parameter the computationally intensive part of our algorithm is in computing a low rank svd of a dense matrix exploiting the problem structure we show that the task can be performed with a complexity of order linear in the matrix dimensions our semidefinite programming algorithm is readily scalable to large matrices for example soft impute takes a few hours to compute low rank approximations of a 106 x 106 incomplete matrix with 107 observed entries and fits a rank 95 approximation to the full netflix training set in 3 3 hours our methods achieve good training and test errors and exhibit superior timings when compared to other competitive state of the art techniques

cmenet a new method for bi level variable selection of conditional main effects this article introduces a novel method for selecting main effects and a set of reparameterized effects called conditional main effects cmes which capture the conditional effect of a factor at a

estimation of variable coefficients in parabolic distributed systems we present techniques based on cubic spline approximations for estimating coefficients e g diffusion convector velocity etc depending on time and the spatial variable in parabolic distributed systems such as those that arise in transport models convergence results and a summary of numerical performance of the resulting algorithms are given

decoupling shrinkage and selection in bayesian linear models a posterior summary perspective selecting a subset of variables for linear models remains an active area of research this article reviews many of the recent contributions to the bayesian model selection and shrinkage prior literature a posterior variable selection summary is proposed which distills a full posterior distribution over regression coefficients into a sequence of sparse linear predictors

a new criterion and a special class of k positive maps abstract we study k positive maps on operators we obtain a new criterion on k positivity in terms of the k numerical range and use it to improve and refine some earlier results on k positive maps related to the study of quantum information science we also consider a special class of positive maps extending the construction of choi on positive maps that are not completely positive some open questions on the decomposability of such maps are answered

generalized choi maps in three dimensional matrix algebra we consider a class of positive linear maps in the three dimensional matrix algebra which are generalizations of the positive linear map constructed by choi in the relation with positive semidefinite biquadratic forms we find conditions for which such maps are completely positive completely copositive decomposable and two positive

small confidence sets for the mean of a spherically symmetric distribution summary suppose that x has a k variate spherically symmetric distribution with mean vector 0 and identity covariance matrix wepresent two spherical confidence sets for 0 both centred at a positive part stein estimator t x in the first we obtain the radius by approximating the upper a point of the sampling disiribution of iits x 9112 by the first two non zero terms of its taylor series about the origin we can analyse some of the properties of this confidence set and see that it performs well in terms of coverage probability volume and conditional behaviour in the second method we find the radius by using a parametric bootstrap procedure here even greater improvement in terms of volume over the usual confidence set is possible at the expense of having a less explicit radius function a real data example is provided and extensions to the unknown covariance matrix and elliptically symmetric cases are discussed

local strong homogeneity of a regularized estimator this paper deals with regularized pointwise estimation of discrete signals which contain large strongly homogeneous zones where typically they are constant or linear or more generally satisfy a linear equation the estimate is defined as the minimizer of an objective function combining a quadratic data fidelity term and a regularization prior term the latter term is the sum of the values obtained by applying a potential function pf to each component called a difference of a linear transform of the signal minimizers of functions of this form arise in various settings in statistics and optimization the features exhibited by such an estimate are closely related to the shape of the pf our goal is to determine estimators providing solutions which involve large strongly homogeneous zones where more precisely the differences are null in spite of the noise corrupting the data to this end we require that the strongly homogeneous zones recovered by the estimator be insensitive to any variation of th

model selection via bayesian information criterion for quantile regression models bayesian information criterion bic is known to identify the true model consistently as long as the predictor dimension is finite recently its moderate modifications have been shown to be consistent in model selection even when the number of variables diverges those works have been done mostly in mean regression but rarely in quantile regression the best known results about bic for quantile regression are for linear models with a fixed number of variables in this article we investigate how bic can be adapted to high dimensional linear quantile regression and show that a modified bic is consistent in model selection when the number of variables diverges as the sample size increases we also discuss how it can be used for choosing the regularization parameters of penalized approaches that are designed to conduct variable selection and shrinkage estimation simultaneously moreover we extend the results to structured nonparametric quantile models with a diverging number of covariates we illustrate o

parameter estimation in flow through partially saturated porous materials a class of numerical simulators were developed and critically evaluated to be incorporated as the solver of a forward problem in the framework of an inverse modeling strategy the strategy couples a mass lumped galerkin linear finite element solution of the mixed form richards equation with an experimental time space series and the osborne more revised form of the levenberg marquardt algorithm to retrieve hydraulic parameters of a partially saturated porous medium the numerical simulator shows excellent agreement with a reference solution obtained on a dense grid and infinitesimal time step in terms of fluid pressure head fluid content and fluid volumetric flux density and perfectly conserves the global mass an adaptive algorithm was implemented to estimate sensitivity matrix in the inverse algorithm a multi criterion stopping rule was developed and successfully implemented to end the inverse code at the solution the result of the optimization was compared with a large scale in situ soil moisture space time series measured during the course of a drainage experiment and excellent agreements were found analysis of the parameter response surfaces and hyper space plots closeness of the gradient of the penalty function at minimum to zero and positive definiteness of the approximation for the hessian at the solution eigs h 0 indicate that the obtained solution is a strong local minimum a state of the art sensitivity analysis carried out to quantify sensitivity of the state variable with respect to uncertainty and changes in different model parameters

l_ 0 sparse inverse covariance estimation recently there has been focus on penalized log likelihood covariance estimation for sparse inverse covariance precision matrices the penalty is responsible for inducing sparsity and a very common choice is the convex l 1 norm however the best estimator performance is not always achieved with this penalty the most natural sparsity promoting norm is the nonconvex l 0 penalty but its lack of convexity has deterred its use in sparse maximum likelihood estimation in this paper we consider nonconvex l 0 penalized log likelihood inverse covariance estimation and present a novel cyclic descent algorithm for its optimization convergence to a local minimizer is proved which is highly nontrivial and we demonstrate via simulations the reduced bias and superior quality of the l 0 penalty as compared to the l 1 penalty

ep gig priors and applications in bayesian sparse learning in this paper we propose a novel framework for the construction of sparsity inducing priors in particular we define such priors as a mixture of exponential power distributions with a generalized inverse gaussian density ep gig ep gig is a variant of generalized hyperbolic distributions and the special cases include gaussian scale mixtures and laplace scale mixtures furthermore laplace scale mixtures can subserve a bayesian framework for sparse learning with nonconvex penalization the densities of ep gig can be explicitly expressed moreover the corresponding posterior distribution also follows a generalized inverse gaussian distribution we exploit these properties to develop em algorithms for sparse empirical bayesian learning we also show that these algorithms bear an interesting resemblance to iteratively reweighted l2 or l1 methods finally we present two extensions for grouped variable selection and logistic regression

penpc a two step approach to estimate the skeletons of high dimensional directed acyclic graphs estimation of the skeleton of a directed acyclic graph dag is of great importance for understanding the underlying dag and causal effects can be assessed from the skeleton when the dag is not identifiable we propose a novel method named penpc to estimate the skeleton of a high dimensional dag by a two step approach we first estimate the nonzero entries of a concentration matrix using penalized regression and then fix the difference between the concentration matrix and the skeleton by evaluating a set of conditional independence hypotheses for high dimensional problems where the number of vertices p is in polynomial or exponential scale of sample size n we study the asymptotic property of penpc on two types of graphs traditional random graphs where all the vertices have the same expected number of neighbors and scale free graphs where a few vertices may have a large number of neighbors as illustrated by extensive simulations and applications on gene expression data of cancer patients penpc has higher sensitivity and specificity than the state of the art method the pc stable algorithm

high dimensional sparse additive hazards regression high dimensional sparse modeling with censored survival data is of great practical importance as exemplified by modern applications in high throughput genomic data analysis and credit risk analysis in this article we propose a class of regularization methods for simultaneous variable selection and estimation in the additive hazards model by combining the nonconcave penalized likelihood approach and the pseudoscore method in a high dimensional setting where the dimensionality can grow fast polynomially or nonpolynomially with the sample size we establish the weak oracle property and oracle property under mild interpretable conditions thus providing strong performance guarantees for the proposed methodology moreover we show that the regularity conditions required by the l 1 method are substantially relaxed by a certain class of sparsity inducing concave penalties as a result concave penalties such as the smoothly clipped absolute deviation minimax concave penalty and smooth integration of co

the spike and slab lasso abstractdespite the wide adoption of spike and slab methodology for bayesian variable selection its potential for penalized likelihood estimation has largely been overlooked in this paper we bridge this gap by cross fertilizing these two paradigms with the spike and slab lasso procedure for variable selection and parameter estimation in linear regression we introduce a new class of self adaptive penalty functions that arise from a fully bayes spike and slab formulation ultimately moving beyond the separable penalty framework a virtue of these non separable penalties is their ability to borrow strength across coordinates adapt to ensemble sparsity information and exert multiplicity adjustment the spike and slab lasso procedure harvests efficient coordinate wise implementations with a path following scheme for dynamic posterior exploration we show on simulated data that the fully bayes penalty mimics oracle performance providing a viable alternative to cross validation we develop theory for the s

a statistical view of some chemometrics regression tools chemometrics is a field of chemistry that studies the application of statistical methods to chemical data analysis in addition to borrowing many techniques from the statistics and engineering literatures chemometrics itself has given rise to several new data analytical methods this article examines two methods commonly used in chemometrics for predictive modeling partial least squares and principal components regression from a statistical perspective the goal is to try to understand their apparent successes and in what situations they can be expected to work well and to compare them with other statistical methods intended for those situations these methods include ordinary least squares variable subset selection and ridge regression

confidence intervals for sparse penalized regression with random designs with the abundance of large data sparse penalized regression techniques are commonly used in data analysis due to the advantage of simultaneous variable selection and estimation a number of conve

extension of positive maps into b h soient 7b h et 7b k des espaces de hilbert complexes et soit a un sous espace lineaire auto adjoint norme clos d operateurs bornes b 7b k sur 7b k contenant l operateur identite si on note par b a 7b h les applications positives dans l ensemble b a 7b h des applications lineaires bornees de a dans b 7b h on etudie le probleme d extension pour des applications dans b a 7b h aux applications dans b b 7b k 7b h

sparsity aware data selective adaptive filters we propose two adaptive filtering algorithms that combine sparsity promoting schemes with data selection mechanisms sparsity is promoted via some well known nonconvex approximations to the l 0 norm in order to increase convergence speed of the algorithms when dealing with sparse compressible signals these approximations circumvent some difficulties of working with the l 0 norm thus allowing the development of online data selective algorithms data selection is implemented based on set membership filtering which yields robustness against noise and reduced computational burden the proposed algorithms are analyzed in order to set properly their parameters to guarantee stability in addition we characterize their updating processes from a geometrical viewpoint simulation results show that the proposed algorithms outperform the state of the art algorithms designed to exploit sparsity

sparsenet coordinate descent with nonconvex penalties we address the problem of sparse selection in linear models a number of nonconvex penalties have been proposed in the literature for this purpose along with a variety of convex relaxation algorithms for finding good solutions in this article we pursue a coordinate descent approach for optimization and study its convergence properties we characterize the properties of penalties suitable for this approach study their corresponding threshold functions and describe a df standardizing reparametrization that assists our pathwise algorithm the mc penalty is ideally suited to this task and we use it to demonstrate the performance of our algorithm certain technical derivations and experiments related to this article are included in the supplementary materials section

on l _ q optimization and sparse inverse covariance selection graphical models are well established in providing meaningful conditional probability descriptions of complex multivariable interactions in the gaussian case the conditional independencies between different variables correspond to zero entries in the precision inverse covariance matrix hence there has been much recent interest in sparse precision matrix estimation in areas such as statistics machine learning computer vision pattern recognition and signal processing a popular estimation method involves optimizing a penalized log likelihood problem the penalty is responsible for inducing sparsity and a common choice is the convex l1 norm even though the l0 penalty is the natural choice guaranteeing maximum sparsity it has been avoided due to lack of convexity as a result in this paper we bridge the gap between these two penalties and propose the non concave lq penalized log likelihood problem for sparse precision matrix estimation where 0 q 1 a novel algorithm is developed for the optimization and we provide some of its theoretic properties that are useful in sparse linear regression we illustrate on synthetic and real data showing reconstruction quality comparisons of sparsity inducing penalties l0 lq with 0 q 1 l1 and scad

does ell _ p minimization outperform ell _ 1 minimization in many application areas ranging from bioinformatics to imaging we are faced with the following question can we recover a sparse vector x o in mathbb r n from its undersampled set of noisy observations y in mathbb r n y a x o w the last decade has witnessed a surge of algorithms and theoretical results to address this question one of the most popular schemes is the ell p regularized least squares given by the following formulation hat x gamma p in arg min x 1 2 y ax 2 2 gamma x p p where p in 0 1 among these optimization problems the case p 1 also known as lasso is the best accepted in practice for the following two reasons first thanks to the extensive studies performed in the fields of high dimensional statistics and compressed sensing we have a clear picture of lasso s performance second it is convex and efficient algorithms exist for finding its global minima unfortunately neither of the above two properties hold for 0 leq p however they are still appealing because of the following folklores in the high dimensional statistics first hat x gamma p is closer to x o than hat x gamma 1 second if we employ iterative methods that aim to converge to a local minima of arg min x 1 2 y ax 2 2 gamma x p p then under good initialization these algorithms converge to a solution that is still closer to x o than hat x gamma 1 in spite of the existence of plenty of empirical results that support these folklore theorems the theoretical progress to establish them has been very limited this paper aims to study the above mentioned folklore theorems and establish their scope of validity starting with approximate message passing amp algorithm as a heuristic method for solving ell p regularized least squares we study the following questions first what is the impact of initialization on the performance of the algorithm second when does the algorithm recover the sparse signal x o under a good initialization third when does the algorithm converge to the sparse signal regardless of the initialization studying these questions will not only shed light on the second folklore theorem but also lead us to the answer the first one i e the performance of the global optima hat x gamma p for that purpose we employ the replica analysis 1 to show the connection between the solution of amp and hat x gamma p in the asymptotic settings this enables us to compare the accuracy of hat x gamma p and hat x gamma 1 in particular we will present an accurate characterization of the phase transition and noise sensitivity of ell p regularized least squares for every 0 leq p leq 1 our results in the noiseless setting confirm that ell p regularized least squares if gamma is tuned optimally exhibits the same phase transition for every 0 leq p and this phase transition is much better than that of lasso furthermore we show that in the noisy setting there is a major difference between the performance of ell p regularized least squares with different values of p for instance we will show that for very small and very large measurement noises p 0 and p 1 outperform the other values of p respectively 1 replica method is a widely accepted heuristic in statistical physics for analyzing large disordered systems

on the degrees of freedom of the lasso we study the effective degrees of freedom of the lasso in the framework of stein s unbiased risk estimation sure we show that the number of nonzero coefficients is an unbiased estimate forthe degrees of freedom of the lasso a conclusion that requires no special assumption on the predictors in addition the unbiased estimator is shown to be asymptotically consistent with these results on hand various model selection criteria cp aic and bic are available which along with the lars algorithm provide a principled and efficient approach to obtaining the optimal lasso fit with the computational effort of a single ordinary least squares fit

randomly spread cdma asymptotics via statistical physics this paper studies randomly spread code division multiple access cdma and multiuser detection in the large system limit using the replica method developed in statistical physics arbitrary input distributions and flat fading are considered a generic multiuser detector in the form of the posterior mean estimator is applied before single user decoding the generic detector can be particularized to the matched filter decorrelator linear minimum mean square error mmse detector the jointly or the individually optimal detector and others it is found that the detection output for each user although in general asymptotically non gaussian conditioned on the transmitted symbol converges as the number of users go to infinity to a deterministic function of a hidden gaussian statistic independent of the interferers thus the multiuser channel can be decoupled each user experiences an equivalent single user gaussian channel whose signal to noise ratio snr suffers a degradation due to the multiple access interference mai the uncoded error performance e g symbol error rate and the mutual information can then be fully characterized using the degradation factor also known as the multiuser efficiency which can be obtained by solving a pair of coupled fixed point equations identified in this paper based on a general linear vector channel model the results are also applicable to multiple input multiple output mimo channels such as in multiantenna systems

one step sparse estimates in nonconcave penalized likelihood models fan and li propose a family of variable selection methods via penalized likelihood using concave penalty functions the nonconcave penalized likelihood estimators enjoy the oracle properties but maximizing the penalized likelihood function is computationally challenging because the objective function is nondifferentiable and nonconcave in this article we propose a new unified algorithm based on the local linear approximation lla for maximizing the penalized likelihood for a broad class of concave penalty functions convergence and other theoretical properties of the lla algorithm are established a distinguished feature of the lla algorithm is that at each lla step the lla estimator can naturally adopt a sparse representation thus we suggest using the one step lla estimator from the lla algorithm as the final estimates statistically we show that if the regularization parameter is appropriately chosen the one step lla estimates enjoy the oracle properties with good initial estimators computationally the one step lla estimation methods dramatically reduce the computational cost in maximizing the nonconcave penalized likelihood we conduct some monte carlo simulation to assess the finite sample performance of the one step sparse estimation methods the results are very encouraging

orthogonalizing em a design based least squares algorithm we introduce an efficient iterative algorithm intended for various least squares problems based on a design of experiments perspective the algorithm called orthogonalizing em oem works for ordinary least squares ols and can be easily extended to penalized least squares the main idea of the procedure is to orthogonalize a design matrix by adding new rows and then solve the original problem by embedding the augmented design in a missing data framework we establish several attractive theoretical properties concerning oem for the ols with a singular regression matrix an oem sequence converges to the moore penrose generalized inverse based least squares estimator for ordinary and penalized least squares with various penalties it converges to a point having grouping coherence for fully aliased regression matrices convergence and the convergence rate of the algorithm are examined finally we demonstrate that oem is highly efficient for large scale least squares and penalized least squares proble

merging of positive maps a construction of various classes of positive maps on matrix algebras abstract for two positive maps i b k i b h i i 1 2 we construct a new linear map b h b k where k k 1 k 2 c h h 1 h 2 c by means of some additional ingredients such as operators and functionals we call it a merging of maps 1 and 2 the properties of this construction are discussed in particular conditions for positivity of as well as for 2 positivity complete positivity optimality and indecomposability are provided in particular we show that for a pair composed of 2 positive and 2 copositive maps there is an indecomposable merging of them one of our main results asserts that for a canonical merging of a pair composed of completely positive and completely copositive extremal maps their canonical merging is an exposed positive map this result provides a wide class of new examples of exposed positive maps as an application new examples of entangled ppt states are described

quantile regression for analyzing heterogeneity in ultra high dimension ultra high dimensional data often display heterogeneity due to either heteroscedastic variance or other forms of non location scale covariate effects to accommodate heterogeneity we advocate a more general interpretation of sparsity which assumes that only a small number of covariates influence the conditional distribution of the response variable given all candidate covariates however the sets of relevant covariates may differ when we consider different segments of the conditional distribution in this framework we investigate the methodology and theory of nonconvex penalized quantile regression in ultra high dimension the proposed approach has two distinctive features 1 it enables us to explore the entire conditional distribution of the response variable given the ultra high dimensional covariates and provides a more realistic picture of the sparsity pattern 2 it requires substantially weaker conditions compared with alternative methods in the literature thus it greatly alleviates the

multinomial inverse regression for text analysis text data including speeches stories and other document forms are often connected to sentiment variables that are of interest for research in marketing economics and elsewhere it is also very high dimensional and difficult to incorporate into statistical analyses this article introduces a straightforward framework of sentiment sufficient dimension reduction for text data multinomial inverse regression is introduced as a general tool for simplifying predictor sets that can be represented as draws from a multinomial distribution and we show that logistic regression of phrase counts onto document annotations can be used to obtain low dimensional document representations that are rich in sentiment information to facilitate this modeling a novel estimation technique is developed for multinomial logistic regression with very high dimensional response in particular independent laplace priors with unknown variance are assigned to each regression coefficient and we detail an efficient routine for ma

on a characterization of positive maps drawing on results of choi stormer and woronowicz we present a nearly complete characterization of certain important classes of positive maps in particular we construct a general class of positive linear maps acting between two matrix algebras b h and b k where h and k are finite dimensional hilbert spaces it turns out that elements of this class are characterized by operators from the dual cone of the set of all separable states on b h k subsequently the relation between entanglements and positive maps is described finally a new characterization of the cone b h b k is given

expandable factor analysis bayesian sparse factor models have proven useful for characterizing dependence in multivariate data but scaling computation to large numbers of samples and dimensions is problematic we propose expandable factor analysis for scalable inference in factor models when the number of factors is unknown the method relies on a continuous shrinkage prior for efficient maximum a posteriori estimation of a low rank and sparse loadings matrix the structure of the prior leads to an estimation algorithm that accommodates uncertainty in the number of factors we propose an information criterion to select the hyperparameters of the prior expandable factor analysis has better false discovery rates and true positive rates than its competitors across diverse simulation settings we apply the proposed approach to a gene expression study of ageing in mice demonstrating superior results relative to four competing methods

alternating linearization for structured regularization problems we adapt the alternating linearization method for proximal decomposition to structured regularization problems in particular to the generalized lasso problems the method is related to two well known operator splitting methods the douglas rachford and the peaceman rachford method but it has descent properties with respect to the objective function this is achieved by employing a special update test which decides whether it is beneficial to make a peaceman rachford step any of the two possible douglas rachford steps or none the convergence mechanism of the method is related to that of bundle methods of nonsmooth optimization we also discuss implementation for very large problems with the use of specialized algorithms and sparse data structures finally we present numerical results for several synthetic and real world examples including a three dimensional fused lasso problem which illustrate the scalability efficacy and accuracy of the method

bayesian robustness and the stein effect abstract in simultaneous estimation of normal means it is shown that through use of the stein effect surprisingly large gains of a bayesian nature can be achieved at little or no cost if the prior information is misspecified this provides a justification in terms of robustness with respect to mis specification of the prior for employing the stein effect even when combining a priori independent problems i e problems in which no empirical bayes effects are obtainable to study this issue a class of minimax estimators that closely mimic the conjugate prior bayes estimators is introduced

a statistical mechanics approach to large system analysis of cdma multiuser detectors we present a theory based on statistical mechanics to evaluate analytically the performance of uncoded fully synchronous randomly spread code division multiple access cdma multiuser detectors with additive white gaussian noise awgn channel under perfect power control and in the large system limit application of the replica method a tool developed in the literature of statistical mechanics allows us to derive analytical expressions for the bit error rate as well as the multiuser efficiency of the individually optimum io and jointly optimum jo multiuser detectors over the whole range of noise levels the information theoretic capacity of the randomly spread cdma channel and the performance of decorrelating and linear minimum mean square error mmse detectors are also derived in the same replica formulation thereby demonstrating validity of the statistical mechanical approach

optimal feature selection in high dimensional discriminant analysis we consider the high dimensional discriminant analysis problem for this problem different methods have been proposed and justified by establishing exact convergence rates for the classification risk as well as the ell 2 convergence results to the discriminative rule however sharp theoretical analysis for the variable selection performance of these procedures have not been established even though model interpretation is of fundamental importance in scientific data analysis this paper bridges the gap by providing sharp sufficient conditions for consistent variable selection using the sparse discriminant analysis through careful analysis we establish rates of convergence that are significantly faster than the best known results and admit an optimal scaling of the sample size n dimensionality p and sparsity level s in the high dimensional setting sufficient conditions are complemented by the necessary information theoretic limits on the variable selection problem in the context of high dimensional discriminant analysis exploiting a numerical equivalence result our method also establish the optimal results for the road estimator and the sparse optimal scoring estimator furthermore we analyze an exhaustive search procedure whose performance serves as a benchmark and show that it is variable selection consistent under weaker conditions extensive simulations demonstrating the sharpness of the bounds are also provided

mmse dimension if n is standard gaussian the minimum mean square error mmse of estimating a random variable x based on snr x n vanishes at least as fast as 1 snr as snr we define the mmse dimension of x as the limit as snr of the product of snr and the mmse mmse dimension is also shown to be the asymptotic ratio of nonlinear mmse to linear mmse for discrete absolutely continuous or mixed distribution we show that mmse dimension equals renyi s information dimension however for a class of self similar singular x e g cantor dis tribution we show that the product of snr and mmse oscillates around information dimension periodically in snr db we also show that these results extend considerably beyond gaussian noise under various technical conditions

differential geometric least angle regression a differential geometric approach to sparse generalized linear models sparsity is an essential feature of many contemporary data problems remote sensing various forms of automated screening and other high throughput measurement devices collect a large amount of information typically about few independent statistical subjects or units in certain cases it is reasonable to assume that the underlying process generating the data is itself sparse in the sense that only a few of the measured variables are involved in the process we propose an explicit method of monotonically decreasing sparsity for outcomes that can be modelled by an exponential family in our approach we generalize the equiangular condition in a generalized linear model although the geometry involves the fisher information in a way that is not obvious in the simple regression setting the equiangular condition turns out to be equivalent with an intuitive condition imposed on the rao score test statistics in certain special cases the method can be tweaked to obtain l1 penalized generalized linear model solution paths but the method itself defines sparsity more directly although the computation of the solution paths is not trivial the method compares favourably with other path following algorithms

estimation of coefficients and boundary parameters in hyperbolic systems we consider semi discrete galerkin approximation schemes in connection with inverse problems for the estimation of spatially varying coefficients and boundary condition parameters in second order hyperbolic systems typical of those arising in 1 d surface seismic problems spline based algorithms are proposed for which theoretical convergence results along with a representative sample of numerical findings are glven

exposed positive maps a sufficient condition exposed positive maps in matrix algebras define a dense subset of extremal maps we provide a sufficient condition for a positive map to be exposed this is an analogue of a spanning property which guarantees that a positive map is optimal we analyze a class of decomposable maps for which this condition is also necessary

distributed proximal gradient algorithm for partially asynchronous computer clusters with ever growing data volume and model size an error tolerant communication efficient yet versatile distributed algorithm has become vital for the success of many large scale machine learning applications in this work we propose m papg an implementation of the flexible proximal gradient algorithm in model parallel systems equipped with the partially asynchronous communication protocol the worker machines communicate asynchronously with a controlled staleness bound s and operate at different frequencies we characterize various convergence properties of m papg 1 under a general non smooth and nonconvex setting we prove that every limit point of the sequence generated by m papg is a critical point of the objective function 2 under an error bound condition of convex objective functions we prove that the optimality gap decays linearly for every s steps 3 under the kurdyka ojasiewicz inequality and a sufficient decrease assumption we prove that the sequences generated by m papg converge to the same critical point provided that a proximal lipschitz condition is satisfied

multi objective optimization in variably saturated fluid flow a general purpose multi objective inverse modeling strategy was developed and implemented to quantify fluid flow parameters in variably saturated porous materials the strategy combines a robust and mass conservative numerical simulator of the flow equation with an optimization algorithm and an experimental data set to estimate the parameters the numerical simulator of the direct problem shows excellent agreement with a reference solution and conserves global mass with near perfection an adaptive method was proposed in which the sensitivity matrix was calculated by one sided finite difference approximation at the early stages of the optimization and the more accurate two sided differentiation as the search approaches the minimum a combined termination criterium was developed to stop the inverse code at the solution the results of the multi objective optimization were compared with those of single objective minimization while single objective optimization generates reasonable results for either the fluid pressure head profile or the fluid content data the proposed multi objective optimization shows excellent agreement with both profiles simultaneously

hidden markov models with applications in cell adhesion experiments estimation of the number of hidden states is challenging in hidden markov models motivated by the analysis of a specific type of cell adhesion experiments a new framework based on a hidden markov model and double penalized order selection is proposed the order selection procedure is shown to be consistent in estimating the number of states a modified expectation maximization algorithm is introduced to efficiently estimate parameters in the model simulations show that the proposed framework outperforms existing methods applications of the proposed methodology to real data demonstrate the accuracy of estimating receptor ligand bond lifetimes and waiting times which are essential in kinetic parameter estimation supplementary materials for this article are available online

efficient shrinkage in parametric models this paper introduces shrinkage for general parametric models we show how to shrink maximum likelihood estimators towards parameter subspaces de ned by general nonlinear restrictions we derive the asymptotic distribution and risk of the generalized shrinkage estimator using a local asymptotic framework we show that if the shrinkage dimension exceeds two the asymptotic risk of the shrinkage estimator is strictly less than that of the mle this reduction holds globally in the parameter space we show that the reduction in asymptotic risk is substantial even for moderately large values of the parameters the formula simplify in a very convenient way in the context of high dimensional models we derive a simple bound for the asymptotic risk we also provide a new large sample minimax e ciency bound we use the concept of local asymptotic minimax bounds a generalization of the conventional asymptotic minimax bounds the

the sparse laplacian shrinkage estimator for high dimensional regression we propose a new penalized method for variable selection and estimation that explicitly incorporates the correlation patterns among predictors this method is based on a combination of the minimax concave penalty and laplacian quadratic associated with a graph as the penalty function we call it the sparse laplacian shrinkage sls method the sls uses the minimax concave penalty for encouraging sparsity and laplacian quadratic penalty for promoting smoothness among coefficients associated with the correlated predictors the sls has a generalized grouping property with respect to the graph represented by the laplacian quadratic we show that the sls possesses an oracle property in the sense that it is selection consistent and equal to the oracle laplacian shrinkage estimator with high probability this result holds in sparse high dimensional settings with p n under reasonable conditions we derive a coordinate descent algorithm for computing the sls estimates simulation studies are conducted to evaluate the performance of the sls method and a real data example is used to illustrate its application

on the structure of positive maps ii low dimensional matrix algebras we use a new idea that emerged in the examination of exposed positive maps between matrix algebras to investigate in more detail the differences and similarities between unital positive maps on m2 c and m3 c our main tool stems from classical grothendieck theorem on tensor product of banach spaces and is an older and more general version of choi jamiolkowski isomorphism between positive maps and block positive choi matrices it takes into account the correct topology on the latter set that is induced by the uniform topology on positive maps in this setting we show that in m2 c case a large class of nice positive maps can be generated from the small set of maps represented by self adjoint unitaries 2px with x maximally entangled vector and p 1 with p rank 1 projector we indicate problems with passing this result to m3 c case among similarities in both m2 c and m3 c cases any unital positive map represented by self adjoint unitary is unitarily equivalent to the transposition map consequently

restricted isometry constants where ell p sparse recovery can fail for 0 p leq 1 this paper investigates conditions under which the solution of an underdetermined linear system with minimal lscrp norm 0 p les 1 is guaranteed to be also the sparsest one matrices are constructed with restricted isometry constants ric delta2m arbitrarily close to 1 radic2 ap 0 707 where sparse recovery with p 1 fails for at least one m sparse vector as well as matrices with delta2m arbitrarily close to one where lscr1 minimization succeeds for any m sparse vector this highlights the pessimism of sparse recovery prediction based on the ric and indicates that there is limited room for improving over the best known positive results of foucart and lai which guarantee that lscr1 minimization recovers all m sparse vectors for any matrix with delta2m 2 3 radic2 7 ap 0 4531 these constructions are a by product of tight conditions for lscrp recovery 0 les p les 1 with matrices of unit spectral norm which are expressed in terms of the minimal singular values of 2m column submatrices compared to lscr1 minimization lscrp minimization recovery failure is shown to be only slightly delayed in terms of the ric values furthermore in this case the minimization is nonconvex and it is important to consider the specific minimization algorithm being used it is shown that when lscrp optimization is attempted using an iterative reweighted lscr1 scheme failure can still occur for delta2m arbitrarily close to 1 radic2

estimation of two phase petroleum reservoir properties by regularization abstract an algorithm is developed based on the theory of regularization and on spline approximation to estimate the absolute permeability in two phase petroleum reservoirs from noisy well pressure data the regularization feature of the algorithm converts the ill posed estimation problem to a well posed one the algorithm which employs the partial conjugate gradient method of nazareth as its core minimization technique automatically chooses the regularization parameter based on the non regularized estimation it is shown that regularized estimation is more stable and insensitive to the choice of initial guess as compared to non regularized conventional estimation

mixtures envelopes and hierarchical duality summary we develop a connection between mixture and envelope representations of objective functions that arise frequently in statistics we refer to this connection by using the term hierarchical duality our results suggest an interesting and previously underexploited relationship between marginalization and profiling or equivalently between the fenchel moreau theorem for convex functions and the bernstein widder theorem for laplace transforms we give several different sets of conditions under which such a duality result obtains we then extend existing work on envelope representations in several ways including novel generalizations to variance mean models and to multivariate gaussian location models this turns out to provide an elegant missing data interpretation of the proximal gradient method which is a widely used algorithm in machine learning we show several statistical applications in which the framework proposed leads to easily implemented algorithms including a robust version of the fused lasso non linear quantile regression via trend filtering and the binomial fused double pareto model code for the examples is available on github at https github com jgscott hierduals

identification of parameters in distributed parameter systems by regularization identification of spatially varying parameters in distributed parameter systems from noisy data is an ill posed problem the concept of regularization widely used in solving linear fredholm integral equations is developed for the identification of parameters in distributed parameter systems a general regularization identification theory is first presented and then applied to the identification of parabolic systems the performance of the regularization identification method is evaluated by numerical experiments on the identification of a spatially varying diffusivity in the diffusion equation

on minimax estimation of a sparse normal mean vector mallows has conjectured that among distributions which are gaussian but for occasional contamination by additive noise the one having least fisher information has two sided geometric contamination a very similar problem arises in estimation of a non negative vector parameter in gaussian white noise when it is known also that most i e 1i components are zero we provide a partial asymptotic expansion of the minimax risk as 0 while the conjecture seems unlikely to be exactly true for flnite we verify it asymptotically up to the accuracy of the expansion numerical work suggests the expansion is accurate for as large as 05 the best 1 estimation rule is flrst but not second order minimax the results bear on an earlier study of maximum entropy estimation and various questions in robustness and function estimation using wavelet bases

the bayesian bridge summary we propose the bayesian bridge estimator for regularized regression and classification two key mixture representations for the bayesian bridge model are developed a scale mixture of normal distributions with respect to an stable random variable a mixture of bartlett fejer kernels or triangle densities with respect to a two component mixture of gamma random variables both lead to markov chain monte carlo methods for posterior simulation and these methods turn out to have complementary domains of maximum efficiency the first representation is a well known result due to west and is the better choice for collinear design matrices the second representation is new and is more efficient for orthogonal problems largely because it avoids the need to deal with exponentially tilted stable random variables it also provides insight into the multimodality of the joint posterior distribution which is a feature of the bridge model that is notably absent under ridge or lasso type priors we prove a theorem that extends this representation to a wider class of densities representable as scale mixtures of beta distributions and we provide an explicit inversion formula for the mixing distribution the connections with slice sampling and scale mixtures of normal distributions are explored on the practical side we find that the bayesian bridge model outperforms its classical cousin in estimation and prediction across a variety of data sets both simulated and real we also show that the markov chain monte carlo algorithm for fitting the bridge model exhibits excellent mixing properties particularly for the global scale parameter this makes for a favourable contrast with analogous markov chain monte carlo algorithms for other sparse bayesian models all methods described in this paper are implemented in the r package bayesbridge an extensive set of simulation results is provided in two on line supplemental files

or forum an algorithmic approach to linear regression linear regression models are traditionally built through trial and error to balance many competing goals such as predictive power interpretability significance robustness to error in data and sparsity among others this problem lends itself naturally to a mixed integer quadratic optimization miqo approach but has not been modeled this way because of the belief in the statistics community that miqo is intractable for large scale problems however in the last 25 years 1991 2015 algorithmic advances in integer optimization combined with hardware improvements have resulted in an astonishing 450 billion factor speedup in solving mixed integer optimization problems we present an miqo based approach for designing high quality linear regression models that explicitly addresses various competing objectives and demonstrate the effectiveness of our approach on both real and synthetic data sets

a consistent information criterion for support vector machines in diverging model spaces information criteria have been popularly used in model selection and proved to possess nice theoretical properties for classification claeskens et al 2008 proposed support vector machine information criterion for feature selection and provided encouraging numerical evidence yet no theoretical justification was given there this work aims to fill the gap and to provide some theoretical justifications for support vector machine information criterion in both fixed and diverging model spaces we first derive a uniform convergence rate for the support vector machine solution and then show that a modification of the support vector machine information criterion achieves model selection consistency even when the number of features diverges at an exponential rate of the sample size this consistency result can be further applied to selecting the optimal tuning parameter for various penalized support vector machine methods finite sample performance of the proposed information criterion is investigated using monte carlo studies and one real world gene selection problem

l_ q sparsity penalized linear regression with cyclic descent recently there has been a lot of focus on penalized least squares problems for noisy sparse signal estimation the penalty induces sparsity and a very common choice has been the convex l1 norm however to improve sparsity and reduce the biases associated with the l1 norm one must move to non convex penalties such as the lq norm in this paper we present a novel cyclic descent algorithm for optimizing the resulting lq penalized least squares problem optimality conditions for this problem are derived and competing ones clarified coordinate wise convergence as well as convergence to a local minimizer of the algorithm which is highly non trivial is proved and we illustrate with simulations comparing the signal reconstruction quality with three penalty functions l0 l1 and lq with 0 q 1