learning low complexity autoregressive models via proximal alternating minimization abstract we consider the estimation of the state transition matrix in vector autoregressive models when time sequence data is limited but nonsequence steady state data is abundant to leverage both sources of data we formulate the least squares minimization problem regularized by a lyapunov penalty we impose cardinality or rank constraints to reduce the complexity of the autoregressive model the resulting nonconvex nonsmooth problem is solved by using the proximal alternating linearization method palm we prove that palm is globally convergent to a critical point and that the estimation error monotonically decreases explicit formulas are obtained for the proximal operators to facilitate the implementation of palm we demonstrate the effectiveness of the developed method by numerical experiments

biclustering via sparse singular value decomposition sparse singular value decomposition ssvd is proposed as a new exploratory analysis tool for biclustering or identifying interpretable row column associations within high dimensional data matrices ssvd seeks a low rank checkerboard structured matrix approximation to data matrices the desired checkerboard structure is achieved by forcing both the left and right singular vectors to be sparse that is having many zero entries by interpreting singular vectors as regression coefficient vectors for certain linear regressions sparsity inducing regularization penalties are imposed to the least squares regression to produce sparse singular vectors an efficient iterative algorithm is proposed for computing the sparse singular vectors along with some discussion of penalty parameter selection a lung cancer microarray dataset and a food nutrition dataset are used to illustrate ssvd as a biclustering method ssvd is also compared with some existing biclustering methods using simulated datasets

cooling schedules for optimal annealing a monte carlo optimization technique called simulated annealing is a descent algorithm modified by random ascent moves in order to escape local minima which are not global minima the level of randomization is determined by a control parameter t called temperature which tends to zero according to a deterministic cooling schedule we give a simple necessary and sufficient condition on the cooling schedule for the algorithm state to converge in probability to the set of globally minimum cost states in the special case that the cooling schedule has parametric form tt c log1 t the condition for convergence is that c be greater than or equal to the depth suitably defined of the deepest local minimum which is not a global minimum state

heuristics of instability and stabilization in model selection in model selection usually a best predictor is chosen from a collection hat mu cdot s of predictors where hat mu cdot s is the minimum least squares predictor in a collection mathsf u s of predictors here s is a complexity parameter that is the smaller s the lower dimensional smoother the models in mathsf u s if mathsf l is the data used to derive the sequence hat mu cdot s the procedure is called unstable if a small change in mathsf l can cause large changes in hat mu cdot s with a crystal ball one could pick the predictor in hat mu cdot s having minimum prediction error without prescience one uses test sets cross validation and so forth the difference in prediction error between the crystal ball selection and the statistician s choice we call predictive loss for an unstable procedure the predictive loss is large this is shown by some analytics in a simple case and by simulation results in a more complex comparison of four different linear regression methods unstable procedures can be stabilized by perturbing the data getting a new predictor sequence hat mu cdot s and then averaging over many such predictor sequences

a bayesian approach to sparse dynamic network identification modeling and identification of high dimensional systems involving signals with many components poses severe challenges to off the shelf techniques for system identification this is particularly so when relatively small data sets as compared to the number signal components have to be used it is often the case that each component of the measured signal can be described in terms of a few other measured variables and these dependences can be encoded in a graphical way via so called dynamic bayesian networks the problem of finding the interconnection structure as well as estimating the dynamic models can be posed as a system identification problem which involves variable selection while this variable selection could be performed via standard selection techniques computational complexity may however be a critical issue being combinatorial in the number of inputs and outputs in this paper we introduce two new nonparametric techniques which borrow ideas from a recently introduced kernel estimator called stable spline as well as from sparsity inducing priors which use 1 type penalties numerical experiments regarding estimation of large scale sparse armax models show that this technique provides a definite advantage over a group lar algorithm and state of the art parametric identification techniques based on prediction error minimization

shrinkage tuning parameter selection with a diverging number of parameters contemporary statistical research frequently deals with problems involving a diverging number of parameters for those problems various shrinkage methods e g the lasso and smoothly clipped absolute deviation are found to be particularly useful for variable selection nevertheless the desirable performances of those shrinkage methods heavily hinge on an appropriate selection of the tuning parameters with a fixed predictor dimension wang and co worker have demonstrated that the tuning parameters selected by a bayesian information criterion type criterion can identify the true model consistently in this work similar results are further extended to the situation with a diverging number of parameters for both unpenalized and penalized estimators consequently our theoretical results further enlarge not only the scope of applicability of the traditional bayesian information criterion type criteria but also that of those shrinkage estimation methods

quantum and semiclassical representations over lagrangian submanifolds in su 2 so 4 and su 1 1 irreducible representations of three simplest lie algebras are realized over lagrangian submanifolds over closed curves tori and spheres in coadjoint orbits the intertwining operator is constructed which enables to pass from an abstract representation to a representation the integral kernel of the intertwining operator involves special coherent states the simple integral formulas are obtained for exact and semiclassical eigenvectors and for solutions of the cauchy problem over lie algebras mentioned in the title the hamiltonians considered are nonanalytic in general

regularized estimation of linear functionals of precision matrices for high dimensional time series this paper studies a dantzig selector type regularized estimator for linear functionals of high dimensional linear processes explicit rates of convergence of the proposed estimator are obtained and they cover the broad regime from independent identically distributed samples to long range dependent time series and from sub gaussian innovations to those with mild polynomial moments it is shown that the convergence rates depend on the degree of temporal dependence and the moment conditions of the underlying linear processes the dantzig selector estimator is applied to the sparse markowitz portfolio allocation and the optimal linear prediction for time series in which the ratio consistency when compared with an oracle estimator is established the effect of dependence and innovation moment conditions is further illustrated in the simulation study finally the regularized estimator is applied to classify the cognitive states on a real functional magnetic resonance imaging dataset and to portfolio optimization on a financial dataset

hierarchically penalized cox regression with grouped variables summary in many biological and other scientific applications predictors are often naturally grouped for example in biological applications assayed genes or proteins are grouped by biological roles or biological pathways when studying the dependence of survival outcome on these grouped predictors it is desirable to select variables at both the group level and the within group level in this article we develop a new method to address the group variable selection problem in the cox proportional hazards model our method not only effectively removes unimportant groups but also maintains the flexibility of selecting variables within the identified groups we also show that the new method offers the potential for achieving the asymptotic oracle property

variable selection for cox s proportional hazards model and frailty model a class of variable selection procedures for parametric models via nonconcave penalized likelihood was proposed in fan and li 2001a it has been shown there that the resulting procedures perform as well as if the subset of significant variables were known in advance such a property is called an oracle property the proposed procedures were illustrated in the context of linear regression robust linear regression and generalized linear models in this paper the nonconcave penalized likelihood approach is extended further to the cox proportional hazards model and the cox proportional hazards frailty model two commonly used semi parametric models in survival analysis as a result new variable selection procedures for these two commonly used models are proposed it is demonstrated how the rates of convergence depend on the regularization parameter in the penalty function further with a proper choice of the regularization parameter and the penalty function the proposed estimators possess an oracle property standard error formulae are derived and their accuracies are empirically tested simulation studies show that the proposed procedures are more stable in prediction and more effective in computation than the best subset variable selection and they reduce model complexity as effectively as the best subset variable selection compared with the lasso which is the penalized likelihood method with the l 1 penalty proposed by tibshirani the newly proposed approaches have better theoretic properties and finite sample performance

nonconcave penalized likelihood with a diverging number of parameters a class of variable selection procedures for parametric models via nonconcave penalized likelihood was proposed by fan and li to simultaneously estimate parameters and select important variables they demonstrated that this class of procedures has an oracle property when the number of parameters is finite however in most model selection problems the number of parameters should be large and grow with the sample size in this paper some asymptotic properties of the nonconcave penalized likelihood are established for situations in which the number of parameters tends to as the sample size increases under regularity conditions we have established an oracle property and the asymptotic normality of the penalized likelihood estimators furthermore the consistency of the sandwich formula of the covariance matrix is demonstrated nonconcave penalized likelihood ratio statistics are discussed and their asymptotic distributions under the null hypothesis are obtained by imposing some mild conditions on the penalty functions the asymptotic results are augmented by a simulation study and the newly developed methodology is illustrated by an analysis of a court case on the sexual discrimination of salary

spatial shrinkage estimation of diffusion tensors on diffusion weighted imaging data diffusion tensor imaging dti based on the diffusion weighted imaging dwi data acquired from magnetic resonance experiments has been widely used to analyze the physical structure of white matter fibers in the human brain in vivo the raw dwi data however carry noise this contaminates the diffusion tensor dt estimates and introduces systematic bias into the induced eigenvalues these bias components affect the effectiveness of fiber tracking algorithms in this article we propose a two stage spatial shrinkage estimation spske procedure to accommodate the spatial information carried in dwi data in dt estimation and to reduce the bias components in the corresponding derived eigenvalues to this end in the framework of the heteroscedastic linear model spske incorporates l 1 type penalization and the locally weighted least square function the theoretical properties of spske are explored the effectiveness of spske is further illustrated by simulation and real data examples supplementary materi

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

high dimensional ordinary least squares projection for screening variables variable selection is a challenging issue in statistical applications when the number of predictors p far exceeds the number of observations n in this ultra high dimensional setting the sure independence screening sis procedure was introduced to significantly reduce the dimensionality by preserving the true model with overwhelming probability before a refined second stage analysis however the aforementioned sure screening property strongly relies on the assumption that the important variables in the model have large marginal correlations with the response which rarely holds in reality to overcome this we propose a novel and simple screening technique called the high dimensional ordinary least squares projection holp we show that holp possesses the sure screening property and gives consistent variable selection without the strong correlation assumption and has a low computational complexity a ridge type holp procedure is also discussed simulation study shows that holp performs competitively compared to many other marginal correlation based methods an application to a mammalian eye disease data illustrates the attractiveness of holp

structure detection of semiparametric structural equation models with bayesian adaptive group lasso structural equation models sems are widely recognized as the most important statistical tool for assessing the interrelationships among latent variables this study develops a bayesian adaptive group least absolute shrinkage and selection operator procedure to perform simultaneous model selection and estimation for semiparametric sems wherein the structural equation is formulated using the additive nonparametric functions of observed and latent variables we propose the use of basis expansions to approximate the unknown functions by introducing adaptive penalties to the groups of basis expansions the nonlinear linear or non existent effects of observed and latent variables in the structural equation can be automatically detected a simulation study demonstrates that the proposed method performs satisfactorily this paper presents an application of revealing the observed and latent risk factors of diabetic kidney disease copyright 2015 john wiley sons ltd

regression models with time series errors abstract the time series regression models in which the errors of regression equations follow stationary or nonstationary autoregressive moving average models are considered convergence properties of the sample autocorrelation function of observed series and the least squares estimates of the linear regression parameters are shown based upon these results a procedure for specifying the tentative order of the mixed arma errors is proposed two examples are given

on the statistical analysis of dirty pictures may 7th 1986 professor a f m smith in the chair summary a continuous two dimensional region is partitioned into a fine rectangular array of sites or pixels each pixel having a particular colour belonging to a prescribed finite set the true colouring of the region is unknown but associated with each pixel there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model the aim is to reconstruct the true scene with the additional knowledge that pixels close together tend to have the same or similar colours in this paper it is assumed that the local characteristics of the true scene can be represented by a nondegenerate markov random field such information can be combined with the records by bayes theorem and the true scene can be estimated according to standard criteria however the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field thus a simple iterative method of reconstruction is proposed which does not depend on these large scale characteristics the method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field some complications including parameter estimation are discussed potential applications are mentioned briefly

variable selection with the strong heredity constraint and its oracle property in this paper we extend the lasso method tibshirani 1996 for simultaneously fitting a regression model and identifying important interaction terms unlike most of the existing variable selection methods our method automatically enforces the heredity constraint that is an interaction term can be included in the model only if the corresponding main terms are also included in the model furthermore we extend our method to generalized linear models and show that it performs as well as if the true model were given in advance that is the oracle property as in fan and li 2001 and fan and peng 2004 the proof of the oracle property is given in online supplemental materials numerical results on both simulation data and real data indicate that our method tends to remove irrelevant variables more effectively and provide better prediction performance than previous work yuan joseph and lin 2007 and zhao rocha and yu 2009 as well as the classical lasso method

shrinkage estimation of the varying coefficient model the varying coefficient model is a useful extension of the linear regression model nevertheless how to conduct variable selection for the varying coefficient model in a computationally efficient manner is poorly understood to solve the problem we propose here a novel method which combines the ideas of the local polynomial smoothing and the least absolute shrinkage and selection operator lasso the new method can do nonparametric estimation and variable selection simultaneously with a local constant estimator and the adaptive lasso penalty the new method can identify the true model consistently and that the resulting estimator can be as efficient as the oracle estimator numerical studies clearly confirm our theories extension to other shrinkage methods e g the scad i e the smoothly clipped absolute deviation and other smoothing methods is straightforward

a theory of proximity based clustering structure detection by optimization abstract in this paper a systematic optimization approach for clustering proximity or similarity data is developed starting from fundamental invariance and robustness properties a set of axioms is proposed and discussed to distinguish different cluster compactness and separation criteria the approach covers the case of sparse proximity matrices and is extended to nested partitionings for hierarchical data clustering to solve the associated optimization problems a rigorous mathematical framework for deterministic annealing and mean field approximation is presented efficient optimization heuristics are derived in a canonical way which also clarifies the relation to stochastic optimization by gibbs sampling similarity based clustering techniques have a broad range of possible applications in computer vision pattern recognition and data analysis as a major practical application we present a novel approach to the problem of unsupervised texture segmentation which relies on statistical tests as a measure of homogeneity the quality of the algorithms is empirically evaluated on a large collection of brodatz like micro texture mondrians and on a set of real word images to demonstrate the broad usefulness of the theory of proximity based clustering the performances of different criteria and algorithms are compared on an information retrieval task for a document database the superiority of optimization algorithms for clustering is supported by extensive experiments

additive model building for spatial regression summary spatial regression is an important predictive tool in many scientific applications and an additive model provides a flexible regression relationship between predictors and a response variable we develop a regularized variable selection technique for building a spatial additive model we find that the methods developed for independent data do not work well for spatially dependent data this motivates us to propose a spatially weighted error norm with a group lasso type of penalty to select additive components in spatial additive models we establish the selection consistency of the approach proposed where the penalty parameter depends on several factors such as the order of approximation of additive components characteristics of the spatial weight and spatial dependence an extensive simulation study provides a vivid picture of the effects of dependent data structure and choice of a spatial weight on selection results as well as the asymptotic behaviour of the estimators as an illustrative example the method is applied to lung cancer mortality data over the period of 2000 2005 obtained from the surveillance epidemiology and end results programme national cancer institute usa

hierarchical unsupervised learning with growing via phase transitions we address unsupervised learning subject to structural constraints with particular emphasis placed on clustering with an imposed decision tree structure most known methods are greedy optimizing one node of the tree at a time to minimize a local cost by constrast we develop a joint optimization method derived based on information theoretic principles and closely related to known methods in statistical physics the approach is inspired by the deterministic annealing algorithm for unstructured data clustering which was based on maximum entropy inference the new approach is founded on the principle of minimum cross entropy using informative priors to approximate the unstructured clustering solution while imposing the structural constraint the resulting method incorporates supervised learning principles applied in an unsupervised problem setting in our approach the tree grows by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales thus estimates of the tree size and structure are naturally obtained at each temperature in the process examples demonstrate considerable improvement over known methods

on selection of spatial linear models for lattice data summary spatial linear models are popular for the analysis of data on a spatial lattice but statistical techniques for selection of covariates and a neighbourhood structure are limited here we develop new methodology for simultaneous model selection and parameter estimation via penalized maximum likelihood under a spatial adaptive lasso a computationally efficient algorithm is devised for obtaining approximate penalized maximum likelihood estimates asymptotic properties of penalized maximum likelihood estimates and their approximations are established a simulation study shows that the method proposed has sound finite sample properties and for illustration we analyse an ecological data set in western canada

regularization parameter selections via generalized information criterion we apply the nonconcave penalized likelihood approach to obtain variable selections as well as shrinkage estimators this approach relies heavily on the choice of regularization parameter which controls the model complexity in this paper we propose employing the generalized information criterion encompassing the commonly used akaike information criterion aic and bayesian information criterion bic for selecting the regularization parameter our proposal makes a connection between the classical variable selection criteria and the regularization parameter selections for the nonconcave penalized likelihood approaches we show that the bic type selector enables identification of the true model consistently and the resulting estimator possesses the oracle property in the terminology of fan and li 2001 in contrast however the aic type selector tends to overfit with positive probability we further show that the aic type selector is asymptotically loss efficient while the bic type selector is not

regression coefficient and autoregressive order shrinkage and selection via the lasso summary the least absolute shrinkage and selection operator lasso has been widely used in regression shrinkage and selection we extend its application to the regression model with autoregressive errors two types of lasso estimators are carefully studied the first is similar to the traditional lasso estimator with only two tuning parameters one for regression coefficients and the other for autoregression coefficients these tuning parameters can be easily calculated via a data driven method but the resulting lasso estimator may not be fully efficient to overcome this limitation we propose a second lasso estimator which uses different tuning parameters for each coefficient we show that this modified lasso can produce the estimator as efficiently as the oracle moreover we propose an algorithm for tuning parameter estimates to obtain the modified lasso estimator simulation studies demonstrate that the modified estimator is superior to the traditional estimator one empirical example is also presented to illustrate the usefulness of lasso estimators the extension of the lasso to the autoregression with exogenous variables model is briefly discussed