the em algorithm an old folk song sung to a fast new tune celebrating the 20th anniversary of the presentation of the paper by dempster laird and rubin which popularized the em algorithm we investigate after a brief historical account strategies that aim to make the em algorithm converge faster while maintaining its simplicity and stability e g automatic monotone convergence in likelihood first we introduce the idea of a working parameter to facilitate the search for efficient data augmentation schemes and thus fast em implementations second summarizing various recent extensions of the em algorithm we formulate a general alternating expectation conditional maximization algorithm aecm that couples flexible data augmentation schemes with model reduction schemes to achieve efficient computations we illustrate these methods using multivariate t models with known or unknown degrees of freedom and poisson models for image reconstruction we show through both empirical and theoretical evidence the potential for a dramatic reduction in computational time with little increase in human effort we also discuss the intrinsic connection between em type algorithms and the gibbs sampler and the possibility of using the techniques presented here to speed up the latter the main conclusion of the paper is that with the help of statistical considerations it is possible to construct algorithms that are simple stable and fast

auxiliary variable methods for markov chain monte carlo with applications abstract suppose that one wishes to sample from the density x using markov chain monte carlo mcmc an auxiliary variable u and its conditional distribution u x can be defined giving the joint distribution x u x u x a mcmc scheme that samples over this joint distribution can lead to substantial gains in efficiency compared to standard approaches the revolutionary algorithm of swendsen and wang is one such example besides reviewing the swendsen wang algorithm and its generalizations this article introduces a new auxiliary variable method called partial decoupling two applications in bayesian image analysis are considered a binary classification problem in which partial decoupling out performs swendsen wang and single site metropolis methods and a positron emission tomography pet reconstruction that uses the gray level prior of geman and mcclure a generalized swendsen wang algorithm is developed for this problem which reduces the computing time to the point where mcmc is a viab

bayesian sparse multiple regression for simultaneous rank reduction and variable selection we develop a bayesian methodology aimed at simultaneously estimating low rank and row sparse matrices in a high dimensional multiple response linear regression model we consider a carefully devised shrinkage prior on the matrix of regression coefficients which obviates the need to specify a prior on the rank and shrinks the regression matrix towards low rank and row sparse structures we provide theoretical support to the proposed methodology by proving minimax optimality of the posterior mean under the prediction risk in ultra high dimensional settings where the number of predictors can grow sub exponentially relative to the sample size a one step post processing scheme induced by group lasso penalties on the rows of the estimated coefficient matrix is proposed for variable selection with default choices of tuning parameters we additionally provide an estimate of the rank using a novel optimization function achieving dimension reduction in the covariate space we exhibit the performance of the proposed methodology in an extensive simulation study and a real data example

parameter expansion to accelerate em the px em algorithm summary the em algorithm and its extensions are popular tools for modal estimation but ar e often criticised for their slow convergence we propose a new method that can often make em much faster the intuitive idea is to use a covariance adjustment to correct the analysis of the m step capitalising on extra information captured in the imputed complete data the way we accomplish this is by parameter expansion we expand the complete data model while preserving the observed data model and use the expanded complete data model to generate em this parameter expanded ei m px em algorithm shares the simplicity and stability of ordinary em but has a faster rate of convergence since its m step performs a more efficient analysis the px em algorithm is illustrated for the multivariate t distribution a random effects model factor analysis probit regression and a poisson imaging model

exploring texture ensembles by efficient markov chain monte carlo toward a trichromacy theory of texture presents a mathematical definition of texture the julesz ensemble spl omega h which is the set of all images defined on z sup 2 that share identical statistics h then texture modeling is posed as an inverse problem given a set of images sampled from an unknown spl omega h sub we search for the statistics h sub which define the ensemble a spl omega h has an associated probability distribution q i h which is uniform over the images in spl omega h and has zero probability outside the authors previously 1999 showed q i h to be the limit distribution of the frame filter random field and minimax entropy model as the image lattice spl lambda spl rarr z sup 2 this conclusion establishes the intrinsic link between the scientific definition of texture on z sup 2 and the mathematical models of texture on finite lattices it brings two advantages the practice of texture image synthesis by matching statistics is put on a mathematical foundation and we need not learn the expensive frame model in feature pursuit model selection and texture synthesis an efficient markov chain monte carte algorithm is proposed for sampling julesz ensembles it generates random texture images by moving along the directions of filter coefficients and thus extends the traditional single site gibbs sampler we compare four popular statistical measures in the literature in terms of their descriptive abilities our experiments suggest that a small number of bins in marginal histograms are sufficient for capturing a variety of texture patterns

joint models for the association of longitudinal binary and continuous processes with application to a smoking cessation trial joint models for the association of a longitudinal binary and a longitudinal continuous process are proposed for situations in which their association is of direct interest the models are parameterized such that the dependence between the two processes is characterized by unconstrained regression coefficients bayesian variable selection techniques are used to parsimoniously model these coefficients a markov chain monte carlo mcmc sampling algorithm is developed for sampling from the posterior distribution using data augmentation steps to handle missing data several technical issues are addressed to implement the mcmc algorithm efficiently the models are motivated by and are used for the analysis of a smoking cessation clinical trial in which an important question of interest was the effect of the exercise treatment on the relationship between smoking cessation and weight gain

convergence rates and asymptotic standard errors for markov chain monte carlo algorithms for bayesian probit regression consider a probit regression problem in which y 1 y n are independent bernoulli random variables such that pr y i 1 x i t where x i is a p dimensional vector of known covariates that are associated with y i is a p dimensional vector of unknown regression coefficients and denotes the standard normal distribution function we study markov chain monte carlo algorithms for exploring the intractable posterior density that results when the probit regression likelihood is combined with a flat prior on we prove that albert and chib s data augmentation algorithm and liu and wu s px da algorithm both converge at a geometric rate which ensures the existence of central limit theorems for ergodic averages under a second moment condition although these two algorithms are essentially equivalent in terms of computational complexity results of hobert and marchev imply that the px da algorithm is theoretically more efficient in the sense that the asymptotic variance in the central limit theorem under the px da algorithm is no larger than that under albert and chib s algorithm we also construct minorization conditions that allow us to exploit regenerative simulation techniques for the consistent estimation of asymptotic variances as an illustration we apply our results to van dyk and meng s lupus data this example demonstrates that huge gains in efficiency are possible by using the px da algorithm instead of albert and chib s algorithm

seeking efficient data augmentation schemes via conditional and marginal augmentation data augmentation sometimes known as the method of auxiliary variables is a powerful tool for constructing optimisation and simulation algorithms in the context of optimisation meng van dyk 1997 1998 reported several successes of the working parameter approach for constructing efficient data augmentation schemes for fast and simple em type algorithms this paper investigates the use of working parameters in the context of markov chain monte carlo in particular in the context of tanner wong s 1987 data augmentation algorithm via a theoretical study of two working parameter approaches the conditional augmentation approach and the marginal augmentation approach posterior sampling under the univariate t model is used as a running example which particularly illustrates how the marginal augmentation approach obtains a fast mixing positive recurrent markov chain by first constructing a nonpositive recurrent markov chain in a larger space

sampling based approaches to calculating marginal densities abstract stochastic substitution the gibbs sampler and the sampling importance resampling algorithm can be viewed as three alternative sampling or monte carlo based approaches to the calculation of numerical estimates of marginal probability distributions the three approaches will be reviewed compared and contrasted in relation to various joint probability structures frequently encountered in applications in particular the relevance of the approaches to calculating bayesian posterior densities for a variety of structured models will be discussed and illustrated

marginal bayesian semiparametric modeling of mismeasured multivariate interval censored data motivated by data gathered in an oral health study we propose a bayesian nonparametric approach for population averaged modeling of correlated time to event data when the responses can only be de

an expanded theoretical treatment of iteration dependent majorize minimize algorithms the majorize minimize mm optimization technique has received considerable attention in signal and image processing applications as well as in statistics literature at each iteration of an mm algorithm one constructs a tangent majorant function that majorizes the given cost function and is equal to it at the current iterate the next iterate is obtained by minimizing this tangent majorant function resulting in a sequence of iterates that reduces the cost function monotonically a well known special case of mm methods are expectation maximization algorithms in this paper we expand on previous analyses of mm due to fessler and hero that allowed the tangent majorants to be constructed in iteration dependent ways also this paper overcomes an error in one of those earlier analyses there are three main aspects in which our analysis builds upon previous work first our treatment relaxes many assumptions related to the structure of the cost function feasible set and tangent majorants for example the cost function can be nonconvex and the feasible set for the problem can be any convex set second we propose convergence conditions based on upper curvature bounds that can be easier to verify than more standard continuity conditions furthermore these conditions allow for considerable design freedom in the iteration dependent behavior of the algorithm finally we give an original characterization of the local region of convergence of mm algorithms based on connected e g convex tangent majorants for such algorithms cost function minimizers will locally attract the iterates over larger neighborhoods than typically is guaranteed with other methods this expanded treatment widens the scope of the mm algorithm designs that can be considered for signal and image processing applications allows us to verify the convergent behavior of previously published algorithms and gives a fuller understanding overall of how these algorithms behave

reducing the state space dimension in a large tvp var abstract this paper proposes a new approach to estimating high dimensional time varying parameter structural vector autoregressive models tvp svars by taking advantage of an empirical feature of tvp s vars tvp s var models are rarely used with more than 4 5 variables however recent work has shown the advantages of modelling vars with large numbers of variables and interest has naturally increased in modelling large dimensional tvp vars a feature that has not yet been utilized is that the covariance matrix for the state equation when estimated freely is often near singular we propose a specification that uses this singularity to develop a factor like structure to estimate a tvp svar for many variables using a generalization of the recentering approach a rank reduced state covariance matrix and judicious parameter expansions we obtain efficient and simple computation of a high dimensional tvp svar an advantage of our approach is that we retain a formal inferential framework such that we can propose formal inference on impulse responses variance decompositions and important for our model the rank of the state equation covariance matrix in a system with 15 variables we show clear empirical evidence in favour of our model and improvements in estimates of impulse responses

fuzzy control systems design and analysis a linear matrix inequality approach k tanaka h o wang eds john wiley sons inc 2001 320pp isbn 0 471 32324 1

parameter expansion for data augmentation abstract viewing the observed data of a statistical model as incomplete and augmenting its missing parts are useful for clarifying concepts and central to the invention of two well known statistical algorithms expectation maximization em and data augmentation recently liu rubin and wu demonstrated that expanding the parameter space along with augmenting the missing data is useful for accelerating iterative computation in an em algorithm the main purpose of this article is to rigorously define a parameter expanded data augmentation px da algorithm and to study its theoretical properties the px da is a special way of using auxiliary variables to accelerate gibbs sampling algorithms and is closely related to reparameterization techniques we obtain theoretical results concerning the convergence rate of the px da algorithm and the choice of prior for the expansion parameter to understand the role of the expansion parameter we establish a new theory for iterative conditional sampling under the tra

batch means and spectral variance estimators in markov chain monte carlo calculating a monte carlo standard error mcse is an important step in the statistical analysis of the simulation output obtained from a markov chain monte carlo experiment an mcse is usually based on an estimate of the variance of the asymptotic normal distribution we consider spectral and batch means methods for estimating this variance in particular we establish conditions which guarantee that these estimators are strongly consistent as the simulation effort increases in addition for the batch means and overlapping batch means methods we establish conditions ensuring consistency in the mean square sense which in turn allows us to calculate the optimal batch size up to a constant of proportionality finally we examine the empirical finite sample properties of spectral variance and batch means estimators and provide recommendations for practitioners

the collapsed gibbs sampler in bayesian computations with applications to a gene regulation problem abstract this article describes a method of grouping and collapsing in using the gibbs sampler and proves from an operator theory viewpoint that the method is in general beneficial the norms of the forward operators associated with the corresponding nonreversible markov chains are used to discriminate among different simulation schemes when applied to bayesian missing data problems the idea of collapsing suggests skipping the steps of sampling parameter s values in standard data augmentation by doing this we obtain a predictive update version of the gibbs sampler a procedure of calculating the posterior odds ratio via the collapsed gibbs sampler when incomplete observations are involved is presented as an illustration of possible applications three examples along with a bayesian treatment for identifying common protein binding sites in unaligned dna sequences are provided

analysis of variance why it is more important than ever analysis of variance anova is an extremely important method in exploratory and confirmatory data analysis unfortunately in complex problems e g split plot designs it is not always easy to set up an appropriate anova we propose a hierarchical analysis that automatically gives the correct anova comparisons even in complex scenarios the inferences for all means and variances are performed under a model with a separate batch of effects for each row of the anova table we connect to classical anova by working with finite sample variance components fixed and random effects models are characterized by inferences about existing levels of a factor and new levels respectively we also introduce a new graphical display showing inferences about the standard deviations of each batch of effects we illustrate with two examples from our applied data analysis first illustrating the usefulness of our hierarchical computations and displays and second showing how the ideas of anova are helpful in understanding a previously fit hierarchical model

improving and evaluating topic models and other models of text abstractan ongoing challenge in the analysis of document collections is how to summarize content in terms of a set of inferred themes that can be interpreted substantively in terms of topics the current practice of parameterizing the themes in terms of most frequent words limits interpretability by ignoring the differential use of words across topics here we show that words that are both frequent and exclusive to a theme are more effective at characterizing topical content and we propose a regularization scheme that leads to better estimates of these quantities we consider a supervised setting where professional editors have annotated documents to topic categories organized into a tree in which leaf nodes correspond to more specific topics each document is annotated to multiple categories at different levels of the tree we introduce a hierarchical poisson convolution model to analyze these annotated documents a parallelized hamiltonian monte carlo sampler allows the inference to scale to millio

generalised gibbs sampler and multigrid monte carlo for bayesian computation summary although monte carlo methods have frequently been applied with success indiscriminate use of markov chain monte carlo leads to unsatisfactory performances in numerous applications we present a generalised version of the gibbs sampler that is based on conditional moves along the traces of groups of transformations in the sample space we explore its connection with the multigrid monte carlo method and its use in designing more efficient samplers the generalised gibbs sampler provides a framework encompassing a class of recently proposed tricks such as parameter expansion and reparameterisation to illustrate we apply this new method to bayesian inference problems for nonlinear state space models ordinal data and stochastic differential equations with discrete observations

sparse bayesian infinite factor models we focus on sparse modelling of high dimensional covariance matrices using bayesian latent factor models we propose a multiplicative gamma process shrinkage prior on the factor loadings which allows introduction of infinitely many factors with the loadings increasingly shrunk towards zero as the column index increases we use our prior on a parameter expanded loading matrix to avoid the order dependence typical in factor analysis models and develop an efficient gibbs sampler that scales well as data dimensionality increases the gain in efficiency is achieved by the joint conjugacy property of the proposed prior which allows block updating of the loadings matrix we propose an adaptive gibbs sampler for automatically truncating the infinite loading matrix through selection of the number of important factors theoretical results are provided on the support of the prior and truncation approximation bounds a fast algorithm is proposed to produce approximate bayes estimates latent factor regression methods are developed for prediction and variable selection in applications with high dimensional correlated predictors operating characteristics are assessed through simulation studies and the approach is applied to predict survival times from gene expression data

a second order accurate material order independent interface reconstruction technique for multi material flow simulations a new second order accurate volume conservative material order independent interface reconstruction method for multi material flow simulations is presented first materials are located in multi material computational cells using a piecewise linear reconstruction of the volume fraction function these material locator points are then used as generators to reconstruct the interface with a weighted voronoi diagram that matches the volume fractions the interfaces are then improved by minimizing an objective function that smoothes interface normals while enforcing convexity and volume constraints for the pure material subcells convergence tests are shown demonstrating second order accuracy static and dynamic examples are shown illustrating the superior performance of the method over existing material order dependent methods

functional gradient ascent for probit regression this paper proposes two gradient based methods to fit a probit regression model by maximizing the sample log likelihood function using the property of the hessian of the objective function the first method performs weighted least square regression in each iteration of the newton raphson framework resulting in probitboost a boosting like algorithm motivated by the gradient boosting algorithm 10 the second proposed approach maximizes the sample log likelihood function by updating the fitted function a small step in the gradient direction performing gradient ascent in functional space resulting in gradient probitboost we also generalize the algorithms to multi class problems by two strategies one of which is to use the gradient ascent to maximize the multi class sample log likelihood function for fitting all the classifiers simultaneously and the second approach uses the one versus all scheme to reduce the multi class problem to a series of binary classification problems the proposed algorithms are tested on typical classification problems including face detection cancer classification and handwritten digit recognition the results show that compared to the alternative methods the proposed algorithms perform similar or better in terms of testing error rates

covariance structure of the gibbs sampler with applications to the comparisons of estimators and augmentation schemes summary we study the covariance structure of a markov chain generated by the gibbs sampler with emphasis on data augmentation when applied to a bayesian missing data problem the gibbs sampler produces two natural approximations for the posterior distribution of the parameter vector the empirical distribution based on the sampled values of the parameter vector and a mixture of complete data posteriors we prove that rao blackwellization causes a one lag delay for the autocovariances among dependent samples obtained from data augmentation and consequently the mixture approximation produces estimates with smaller variances than the empirical approximation the covariance structure results are used to compare different augmentation schemes it is shown that collapsing and grouping random components in a gibbs sampler with two or three components usually result in more efficient sampling schemes

learning to recognize objects with little supervision this paper shows i improvements over state of the art local feature recognition systems ii how to formulate principled models for automatic local feature selection in object class recognition when there is little supervised data and iii how to formulate sensible spatial image context models using a conditional random field for integrating local features and segmentation cues superpixels by adopting sparse kernel methods bayesian learning techniques and data association with constraints the proposed model identifies the most relevant sets of local features for recognizing object classes achieves performance comparable to the fully supervised setting and obtains excellent results for image classification

bayesian semiparametric structural equation models with latent variables structural equation models sems with latent variables are widely useful for sparse covariance structure modeling and for inferring relationships among latent variables bayesian sems are appealing in allowing for the incorporation of prior information and in providing exact posterior distributions of unknowns including the latent variables in this article we propose a broad class of semiparametric bayesian sems which allow mixed categorical and continuous manifest variables while also allowing the latent variables to have unknown distributions in order to include typical identifiability restrictions on the latent variable distributions we rely on centered dirichlet process cdp and cdp mixture cdpm models the cdp will induce a latent class model with an unknown number of classes while the cdpm will induce a latent trait model with unknown densities for the latent traits a simple and efficient markov chain monte carlo algorithm is developed for posterior computation and the methods are illustrated using simulated examples and several applications

efficient parametrisations for normal linear mixed models summary the generality and easy programmability of modern sampling based methods for maximisation of likelihoods and summarisation of posterior distributions have led to a tremendous increase in the complexity and dimensionality of the statistical models used in practice however these methods can often be extremely slow to converge due to high correlations between or weak identifiability of certain model parameters we present simple hierarchical centring reparametrisations that often give improved convergence for a broad class of normal linear mixed models in particular we study the two stage hierarchical normal linear model the laird ware model for longitudinal data and a general structure for hierarchically nested linear models using analytical arguments simulation studies and an example involving clinical markers of acquired immune deficiency syndrome aids we indicate when reparametrisation is likely to provide substantial gains in efficiency

fast bayesian factor analysis via automatic rotations to sparsity abstractrotational post hoc transformations have traditionally played a key role in enhancing the interpretability of factor analysis regularization methods also serve to achieve this goal by prioritizing sparse loading matrices in this work we bridge these two paradigms with a unifying bayesian framework our approach deploys intermediate factor rotations throughout the learning process greatly enhancing the effectiveness of sparsity inducing priors these automatic rotations to sparsity are embedded within a pxl em algorithm a bayesian variant of parameter expanded em for posterior mode detection by iterating between soft thresholding of small factor loadings and transformations of the factor basis we obtain a dramatic accelerations b robustness against poor initializations and c better oriented sparse solutions to avoid the prespecification of the factor cardinality we extend the loading matrix to have infinitely many columns with the indian buffet process ibp prior the factor dimen

partially collapsed gibbs samplers theory and methods ever increasing computational power along with ever more sophisticated statistical computing techniques is making it possible to fit ever more complex statistical models among the more computationally intensive methods the gibbs sampler is popular because of its simplicity and power to effectively generate samples from a high dimensional probability distribution despite its simple implementation and description however the gibbs sampler is criticized for its sometimes slow convergence especially when it is used to fit highly structured complex models here we present partially collapsed gibbs sampling strategies that improve the convergence by capitalizing on a set of functionally incompatible conditional distributions such incompatibility generally is avoided in the construction of a gibbs sampler because the resulting convergence properties are not well understood we introduce three basic tools marginalization permutation and trimming that allow us to transform a gibbs sampler into a part

parameterization and bayesian modeling progress in statistical computation often leads to advances in statistical modeling for example it is surprisingly common that an existing model is reparameterized solely for computational purposes but then this new configuration motivates a new family of models that is useful in applied statistics one reason why this phenomenon may not have been noticed in statistics is that reparameterizations do not change the likelihood in a bayesian framework however a transformation of parameters typically suggests a new family of prior distributions we discuss examples in censored and truncated data mixture modeling multivariate imputation stochastic processes and multilevel models

the calculation of posterior distributions by data augmentation abstract the idea of data augmentation arises naturally in missing value problems as exemplified by the standard ways of filling in missing cells in balanced two way tables thus data augmentation refers to a scheme of augmenting the observed data so as to make it more easy to analyze this device is used to great advantage by the em algorithm dempster laird and rubin 1977 in solving maximum likelihood problems in situations when the likelihood cannot be approximated closely by the normal likelihood maximum likelihood estimates and the associated standard errors cannot be relied upon to make valid inferential statements from the bayesian point of view one must now calculate the posterior distribution of parameters of interest if data augmentation can be used in the calculation of the maximum likelihood estimate then in the same cases one ought to be able to use it in the computation of the posterior distribution it is the purpose of this article to explain how this can be done the basic idea