the em algorithm an old folk song sung to a fast new tune celebrating the 20th anniversary of the presentation of the paper by dempster laird and rubin which popularized the em algorithm we investigate after a brief historical account strategies that aim to make the em algorithm converge faster while maintaining its simplicity and stability e g automatic monotone convergence in likelihood first we introduce the idea of a working parameter to facilitate the search for efficient data augmentation schemes and thus fast em implementations second summarizing various recent extensions of the em algorithm we formulate a general alternating expectation conditional maximization algorithm aecm that couples flexible data augmentation schemes with model reduction schemes to achieve efficient computations we illustrate these methods using multivariate t models with known or unknown degrees of freedom and poisson models for image reconstruction we show through both empirical and theoretical evidence the potential for a dramatic reduction in computational time with little increase in human effort we also discuss the intrinsic connection between em type algorithms and the gibbs sampler and the possibility of using the techniques presented here to speed up the latter the main conclusion of the paper is that with the help of statistical considerations it is possible to construct algorithms that are simple stable and fast

inference for imputation estimators we derive an estimator of the asymptotic variance of both single and multiple imputation estimators we assume a parametric imputation model but allow for non and semiparametric analysis models our variance estimator in contrast to the estimator proposed by rubin 1987 is consistent even when the imputation and analysis models are misspecified and incompatible with one another

global dynamics of a system governing an algorithm for regression with censored and non censored data under general errors we present an investigation into the dynamics of a system which underlies a new estimating algorithm for regression with grouped and nongrouped data the algorithm springs from a simplification of the well known em algorithm in which the expectation step of the em is substituted by a modal step this avoids awkward integrations when the error distribution is assumed to be general the sequences generated by the estimating procedure proposed here define our objective system which is piecewise linear the study tackles the system s asymptotic stability as well as its speed of convergence to the equilibrium point in this sense to reduce the speed of convergence we propose an alternative estimating procedure numerical examples illustrate the theoretical results compare the proposed procedures and analyze the precision of the estimate

a probabilistic construction of model validation we describe a procedure to assess the predictive accuracy of process models subject to approximation error and uncertainty the proposed approach is a functional analysis based probabilistic approach for which we represent random quantities using polynomial chaos expansions pces the approach permits the formulation of the uncertainty assessment in validation a significant component of the process as a problem of approximation theory it has two essential parts first a statistical procedure is implemented to calibrate uncertain parameters of the candidate model from experimental or model based measurements such a calibration technique employs pces to represent the inherent uncertainty of the model parameters based on the asymptotic behavior of the statistical parameter estimator the associated pce coefficients are then characterized as independent random quantities to represent epistemic uncertainty due to lack of information second a simple hypothesis test is implemented to explore the validation of the computational model assumed for the physics of the problem the above validation path is implemented for the case of dynamical system validation challenge exercise

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

inference of nested variance components in a longitudinal myopia intervention trial this paper was motivated by a double blind randomized clinical trial of myopia intervention in addition to the primary goal of comparing treatment effects we are concerned with the modelling of correlation that may come from two possible sources one among the longitudinal observations and the other between measurements taken from both eyes per subject the data are nested repeated measurements we suggest three models for analysis each one expresses the correlation differently in various covariance structures we articulate their differences and describe the implementations in estimation using commercial statistical software the computer output can be further utilized to perform model selection with schwarz criterion simulation studies are conducted to evaluate the performance under each model data of the myopia intervention trial are reanalysed with these models for illustration the results indicate that atropine is more effective in reducing the progression rate the rates are homogeneous across subjects and among the suggested models the one with independent random effects of two eyes fits best we conclude that model selection is a crucial step before making inference with estimates otherwise the correlation may be attributed incorrectly to a different mechanism the same conclusion applies to other variance components as well copyright 2005 john wiley sons ltd

analysis of contingency tables by correspondence models subject to order constraints abstract inferential correspondence analysis which has gained much attention in recent years is applied here to contingency tables with ordered categories to reflect such order the parameters of the underlying correspondence models are constrained to follow the order induced by the categories of the analyzed table a reparameterization of the correspondence model in terms of a latent variable model is presented this allows a simple and straightforward use of the em algorithm to obtain efficient order restricted estimates a goodness of fit test is also discussed and an example is analyzed a small monte carlo example is presented

incompressible smoothed particle hydrodynamics for free surface flows a generalised diffusion based algorithm for stability and validations for impulsive flows and propagating waves the incompressible smoothed particle hydrodynamics isph method with projection based pressure correction has been shown to be highly accurate and stable for internal flows and importantly for many problems the pressure field is virtually noise free in contrast to the weakly compressible sph approach xu et al 2009 31 however for almost inviscid fluids instabilities at the free surface occur due to errors associated with the truncated kernels a new algorithm is presented which remedies this issue giving stable and accurate solutions to both internal and free surface flows generalising the particle shifting approach of xu et al 2009 31 the algorithm is based upon fick s law of diffusion and shifts particles in a manner that prevents highly anisotropic distributions and the onset of numerical instability the algorithm is validated against analytical solutions for an internal flow at higher reynolds numbers than previously the flow due to an impulsively started plate and highly accurate solutions for wet bed dam break problems at zero and small times the method is then validated for progressive regular waves with paddle motion defined by linear theory the accurate predictions demonstrate the effectiveness of the algorithm in stabilising solutions and minimising the surface instabilities generated by the inevitable errors associated with truncated kernels the test cases are thought to provide a more thorough quantitative validation than previously undertaken

proportional hazards regression with missing covariates abstract nonparametric maximum likelihood npml is used to estimate regression parameters in a proportional hazards regression model with missing covariates the npml estimator is shown to be consistent and asymptotically normally distributed under some conditions em type algorithms are applied to solve the maximization problem variance estimates of the regression parameters are obtained by a profile likelihood approach that uses em aided numerical differentiation simulation results indicate that the npml estimates of the regression parameters are more efficient than the approximate partial likelihood estimates and estimates from complete case analysis when missing covariates are missing completely at random and that the proposed method corrects for bias when the missing covariates are missing at random

accuracy and stability in incompressible sph isph based on the projection method and a new approach the stability and accuracy of three methods which enforce either a divergence free velocity field density invariance or their combination are tested here through the standard taylor green and spin down vortex problems while various approaches to incompressible sph isph have been proposed in the past decade the present paper is restricted to the projection method for the pressure and velocity coupling it is shown that the divergence free isph method cannot maintain stability in certain situations although it is accurate before instability sets in the density invariant isph method is stable but inaccurate with random noise like disturbances the combined isph combining advantages in divergence free isph and density invariant isph can maintain accuracy and stability although at a higher computational cost redistribution of particles on a fixed uniform mesh is also shown to be effective but the attraction of a mesh free method is lost a new divergence free isph approach is proposed here which maintains accuracy and stability while remaining mesh free without increasing computational cost by slightly shifting particles away from streamlines although the necessary interpolation of hydrodynamic characteristics means the formulation ceases to be strictly conservative this avoids the highly anisotropic particle spacing which eventually triggers instability importantly pressure fields are free from spurious oscillations up to the highest reynolds numbers tested

convergence of a stochastic approximation version of the em algorithm the expectation maximization em algorithm is a powerful computational technique for locating maxima of functions it is widely used in statistics for maximum likelihood or maximum a posteriori estimation in incomplete data models in certain situations however this method is not applicable because the expectation step cannot be performed in closed form to deal with these problems a novel method is introduced the stochastic approximation em saem which replaces the expectation step of the em algorithm by one iteration of a stochastic approximation procedure the convergence of the saem algorithm is established under conditions that are applicable to many practical situations moreover it is proved that under mild additional conditions the attractive stationary points of the saem algorithm correspond to the local maxima of the function presented to support our findings

adaptive iterative detectors for phase uncertain channels via variational bounding the problem of iterative detection decoding of data symbols transmitted over an additive white gaussian noise awgn channel in the presence of phase uncertainty is addressed in this paper by modelling the phase uncertainty either as an unknown deterministic variable process or random variable process with a known a priori probability density function a number of non bayesian and bayesian detection algorithms with various amount of suboptimality have been proposed in the literature to solve the problem in this paper a new set of suboptimal iterative detection algorithms is obtained by utilizing the variational bounding technique especially applying the generic variational bayesian vb framework efficient iterative joint estimation and detection decoding schemes are derived for the constant phase model as well as for the dynamic phase model in addition the relation of the vb based approach to the optimal noncoherent receiver as well as to the classical approach via the expectation maximization em algorithm is provided performance of the proposed detectors in the presence of a strong dynamic phase noise is compared to the performance of the existing detectors furthermore an incremental scheduling of the vb or em algorithm is shown to reduce the overall complexity of the receiver

an mps based particle method for simulation of multiphase flows characterized by high density ratios by incorporation of space potential particle concept abstract simulation of multiphase flows characterized by high density ratios is targeted by using a refined particle method the proposed method is founded on moving particle semi implicit mps method which is a projection based particle method the proposed method comprises of a decoupled two step computational algorithm through application of a consistent scheme for coupling the light heavy phases by incorporating the concept of space potential particles spp the proposed coupling scheme guarantees the continuity of pressure and space volume conservation at the phase interface without any need for commonly applied density smoothing averaging schemes or application of numerical stabilizing terms verification of the proposed multiphase particle method is conducted through the simulations of four benchmark tests and the method is shown to possess acceptable accuracy and convergence properties

state space models from the em algorithm to a gradient approach slow convergence is observed in the em algorithm for linear state space models we propose to circumvent the problem by applying any off the shelf quasi newton type optimizer which operates on the gradient of the log likelihood function such an algorithm is a practical alternative due to the fact that the exact gradient of the log likelihood function can be computed by recycling components of the expectation maximization em algorithm we demonstrate the efficiency of the proposed method in three relevant instances of the linear state space model in high signal to noise ratios where em is particularly prone to converge slowly we show that gradient based learning results in a sizable reduction of computation time

a monte carlo implementation of the em algorithm and the poor man s data augmentation algorithms abstract the first part of this article presents the monte carlo implementation of the e step of the em algorithm given the current guess to the maximizer of the posterior distribution latent data patterns are generated from the conditional predictive distribution the expected value of the augmented log posterior is then updated as a mixture of augmented log posteriors mixed over the generated latent data patterns multiple imputations in the m step of the algorithm this mixture is maximized to obtain the update to the maximizer of the observed posterior the gradient and hessian of the observed log posterior are also expressed as mixtures mixed over the multiple imputations the relation between the monte carlo em mcem algorithm and the data augmentation algorithm is noted two modifications to the mcem algorithm the poor man s data augmentation algorithms which allow for the calculation of the entire posterior are then presented these approximations serve as diagnostics for the validity o

estimation in covariance components models abstract estimation techniques for linear covariance components models are developed and illustrated with special emphasis on explaining computational processes the estimation of fixed and random effects when the variances and covariances are known is presented in bayesian terms point estimates of the unknown variances and covariances are computed using the em algorithm for maximum likelihood estimation from incomplete data the techniques are illustrated with data on law schools field mice and professional football teams

maximum a posteriori estimation of time varying arma processes from noisy observations the estimation of the parameters of discrete time autoregressive moving average arms processes observed in white noise is considered a class of time varying arma processes in which the parameter process is the output of a known linear system driven by white gaussian noise is examined the maximum a posteriori map estimator is defined for the trajectory of the parameter s random process a two step estimate and maximize em based e step and m step iterative algorithm is derived the posterior probability of the parameters is increased in each iteration and convergence to stationary points of the posterior probability is guaranteed each iteration involves two linear systems and is easily implemented it is shown that similar results can be obtained for a wide class of parameter estimation problems

sequential algorithms for parameter estimation based on the kullback leibler information measure methods of stochastic approximation are used to convert iterative algorithms for maximizing the kullback leibler information measure into sequential algorithms special attention is given to the case of incomplete data and several algorithms are presented to deal with situations of this kind the application of these algorithms to the identification of finite impulse response systems is considered

a new open boundary formulation for incompressible sph abstract in this work a new formulation for inflow outflow boundary conditions in an incompressible smoothed particles hydrodynamics isph model is proposed it relies on the technique of unified semi analytical boundary conditions that was first proposed for wall boundary conditions in 2013 then extended to open boundaries in the framework of weakly compressible sph wcsph an isph model relying on that formulation for solid boundaries was then proposed which is the one considered here it includes a buoyancy model for temperature effects and a k turbulence closure there are two main requirements for the imposition of open boundaries in isph an algorithm to let particles enter and leave the domain and the correct imposition of open boundary conditions on the fields regarding the algorithm for particles creation destruction it relies on the variation of mass of the particles located at the open boundaries when the mass of such a particle reaches a threshold a new particle is released on the other hand the imposition of open boundary conditions on the fields is done by prescribing the value of the boundary terms appearing in the semi analytical formulation the formulation was first validated in 2 d on a cut dam break a case of propagation of a solitary wave and a creager weir it was then extended to 3 d and tested on a 3 d circular pipe a preliminary application case consisting of two connected pipes at different temperatures was then simulated the results are promising since in all cases the fluid enters and leaves the domain as prescribed and generating none or very few reflected waves

variances for maximum penalized likelihood estimates obtained via the em algorithm we address the problem of providing variances for parameter estimates obtained under a penalized likelihood formulation through use of the em algorithm the solution proposed represents a synthesis of two existent techniques firstly we exploit the supplemented em algorithm developed by meng and rubin that provides variance estimates for maximum likelihood estimates obtained via the em algorithm their procedure relies on evaluating the jacobian of the mapping induced by the em algorithm secondly we utilize results from green that provide expressions for jacobians of mappings induced by em algorithms applied to a penalized likelihood the resultant procedure requires no more code than that needed for the penalized em algorithm itself

a new method for evaluating the log likelihood gradient the hessian and the fisher information matrix for linear dynamic systems a method is presented for evaluating the log likelihood gradient score the hessian and the fisher information matrix of the parameters of linear dynamic stochastic systems the method incorporates the optimal kalman smoothing equations and is therefore ideal for simultaneous state estimation and parameter identification the result can be used for efficient implementation of gradient based algorithms for maximum likelihood identification of the unknown system parameters and for assessing the mean square estimation accuracy

information matrix computation from conditional information via normal approximation summary this paper provides a method for computing the asymptotic covariance matrix from a likelihood function with known maximum likelihood estimate of the parameters philosophically the basic idea is to assume that the likelihood function should be well approximated by a normal density when asymptotic results about the maximum likelihood estimate are applied for statistical inference technically the method makes use of two facts the information for a one dimensional parameter can be well computed when the loglikelihood is approximately quadratic over the range corresponding to a small positive confidence interval and the covariance matrix of a normal distribution can be obtained from its one dimensional conditional distributions whose sample spaces span the sample space of the joint distribution we illustrate the method with its application to a linear mixed effects model

unified semi analytical wall boundary conditions applied to 2 d incompressible sph this work aims at improving the 2 d incompressible sph model isph by adapting it to the unified semi analytical wall boundary conditions proposed by ferrand et al 10 the isph algorithm considered is as proposed by lind et al 25 based on the projection method with a divergence free velocity field and using a stabilising procedure based on particle shifting however we consider an extension of this model to reynolds averaged navier stokes equations based on the k e turbulent closure model as done in 10 the discrete sph operators are modified by the new description of the wall boundary conditions in particular a boundary term appears in the laplacian operator which makes it possible to accurately impose a von neumann pressure wall boundary condition that corresponds to impermeability the shifting and free surface detection algorithms have also been adapted to the new boundary conditions moreover a new way to compute the wall renormalisation factor in the frame of the unified semi analytical boundary conditions is proposed in order to decrease the computational time we present several verifications to the present approach including a lid driven cavity a water column collapsing on a wedge and a periodic schematic fish pass our results are compared to finite volumes methods using volume of fluids in the case of free surface flows we briefly investigate the convergence of the method and prove its ability to model complex free surface and turbulent flows the results are generally improved when compared to a weakly compressible sph model with the same boundary conditions especially in terms of pressure prediction

open boundary conditions for isph and their application to micro flow open boundary conditions for incompressible smoothed particle hydrodynamics isph are rare for stable simulations with open boundary conditions one needs to specify all boundary conditions correctly in the pressure force as well as in the linear equation system for pressure calculation especially for homogeneous or non homogeneous dirichlet boundary conditions for pressure there exist several possibilities but only a few lead to stable results however this isn t trivial for open boundary conditions we introduce a new approach for open boundary conditions for isph to enable stable simulations in contrast to existing models for weakly compressible sph we can specify open pressure boundary conditions because in isph pressure can be calculated independently of the density the presented approach is based on the mirror particle approach already introduced for solid wall boundary conditions here we divide the mirror axis in several segments with time dependent positions we validate the presented approach for the example of poiseuille flow and flow around a cylinder at different reynolds numbers and show that we get good agreement with references then we demonstrate that the approach can be applied to free surface flows finally we apply the new approach to micro flow through a random porous medium with a different number of in and outlets and demonstrate its benefits

maximum likelihood analysis of logistic regression models with incomplete covariate data and auxiliary information summary this article presents a new method for maximum likelihood estimation of logistic regression models with incomplete covariate data where auxiliary information is available this auxiliary information is extraneous to the regression model of interest but predictive of the covariate with missing data ibrahim 1990 journal of the american statistical association85 765 769 provides a general method for estimating generalized linear regression models with missing covariates using the em algorithm that is easily implemented when there is no auxiliary data vach 1997 statistics in medicine16 57 72 describes how the method can be extended when the outcome and auxiliary data are conditionally independent given the covariates in the model the method allows the incorporation of auxiliary data without making the conditional independence assumption we suggest tests of conditional independence and compare the performance of several estimators in an example concerning mental health service utilization in children using an artificial dataset we compare the performance of several estimators when auxiliary data are available

acceleration by aggregation of successive approximation methods abstract methods of successive approximation for solving linear systems or minimization problems are accelerated by aggregation disaggregation processes these processes which modify the iterates being produced are characterized by a two directional flow of information between the original higher dimensional problem and a lower dimensional aggregated version this technique is characterized by means of galerkin approximations and this in turn permits analysis of the method a deterministic as well as probabilistic analysis is given of a number of specific aggregation disaggregation examples numerical experiments have been performed and these confirm the analysis and demonstrate the acceleration

empirical bayes estimates of cancer mortality rates using spatial models we give results of empirical bayes eb estimation of mortality rates designed to smooth observed smr when random fluctuation of the observed deaths is important we have specially studied the case where the prior distributions of the eb method have a spatial structure the need for spatial modelling of cancer mortality rates in france is first shown with testing autocorrelation and fitting autoregressive spatial models conditional car or simultaneous sar a positive autocorrelation of the rates is shown for most cancer sites studied as expected eb estimates of mortality rates for common tumours are similar to smrs for rare tumours the eb method identifies the extreme rates more clearly than smrs by smoothing the smrs with large variances car or sar models are adequate prior distributions for autocorrelated rates and produce quite similar rate estimates

interval mapping of quantitative trait loci for time to event data with the proportional hazards mixture cure model summary interval mapping using normal mixture models has been an important tool for analyzing quantitative traits in experimental organisms when the primary phenotype is time to event it is natural to use survival models such as cox s proportional hazards model instead of normal mixtures to model the phenotype distribution an extra challenge for modeling time to event data is that the underlying population may consist of susceptible and nonsusceptible subjects in this article we propose a semiparametric proportional hazards mixture cure model which allows missing covariates we discuss applications to quantitative trait loci qtl mapping when the primary trait is time to event from a population of mixed susceptibility this model can be used to characterize qtl effects on both susceptibility and time to event distribution and to estimate qtl location the model can naturally incorporate covariate effects of other risk factors maximum likelihood estimates for the parameters in the model as well as their corresponding variance estimates can be obtained numerically using an em type algorithm the proposed methods are assessed by simulations under practical settings and illustrated using a real data set containing survival times of mice after infection with listeria monocytogenes an extension to multiple intervals is also discussed

parameter estimation of superimposed signals using the em algorithm a computationally efficient algorithm for parameter estimation of superimposed signals based on the two step iterative em estimate and maximize with an e step and an m step algorithm is developed the idea is to decompose the observed data into their signal components and then to estimate the parameters of each signal component separately the algorithm iterates back and forth using the current parameter estimates to decompose the observed data better and thus increase the likelihood of the next parameter estimates the application of the algorithm to the multipath time delay and multiple source location estimation problems is considered

estimate maximize algorithms for multichannel time delay and signal estimation computationally efficient iterative algorithms are developed for the joint maximum likelihood estimation of the time delays and the spectral parameters of stationary gaussian signals radiated from a stationary source and observed in the presence of uncorrelated additive noise at two or more spatially distributed receivers the first set of algorithms is based on the estimate maximize approach while the second set achieves superlinear performance by using a modified approach these algorithms decouple the estimation of all the unknowns separately estimating the signal the signal spectral parameters and each channel s delay attenuation and noise power this decoupling considerably simplifies the estimator s structure and computation convergence is guaranteed to the set of stationary points of the likelihood function and each iteration increases the likelihood the convergence rates are analyzed theoretically and the algorithm s performance demonstrated via simulation

dynamic refinement and boundary contact forces in sph with applications in fluid flow problems this paper presents a general method for dynamic particle refinement in smoothed particle hydrodynamics sph candidate particles are split into several daughter particles according to a given refinement pattern centred about the original particle through the solution of a non linear minimization problem the optimal mass distribution of the daughter particles is obtained so as to reduce the errors introduced to the underlying density field this procedure necessarily conserves the mass of the system conservation of energy and momentum results are also discussed copyright 2007 john wiley sons ltd

standard errors for em estimates in generalized linear models with random effects a procedure is derived for computing standard errors of em estimates in generalized linear models with random effects quadrature formulas are used to approximate the integrals in the em algorithm where two different approaches are pursued i e gauss hermite quadrature in the case of gaussian random effects and nonparametric maximum likelihood estimation for an unspecified random effect distribution an approximation of the expected fisher information matrix is derived from an expansion of the em estimating equations this allows for inferential arguments based on em estimates as demonstrated by an example and simulations

conjugate gradient acceleration of the em algorithm the em algorithm is a very popular and widely applicable algorithm for the computation of maximum likelihood estimates although its implementation is generally simple the em algorithm often exhibits slow convergence and is costly in some areas of application past attempts to accelerate the em algorithm have most commonly been based on some form of aitken acceleration here we propose an alternative method based on conjugate gradients the key as we show is that the em step can be viewed approximately at least as a generalized gradient making it natural to apply generalized conjugate gradient methods in an attempt to accelerate the em algorithm the proposed method is relatively simple to implement and can handle problems with a large number of parameters an important feature of most em algorithms to demonstrate the effectiveness of the proposed acceleration method we consider its application to several problems in each of the following areas estimation of a covariance matrix from incomplete mu

on the rate of convergence of the ecme algorithm for multiple regression models with t distributed errors although much work has been done on comparing and contrasting the em and ecme algorithms in terms of their rates of convergence it is not clear what mechanism underlies each and furthermore what factors may determine and influence their rates of convergence in this paper we examine the convergence rates and properties of these two popular optimisation algorithms as used in computing the maximum likelihood estimates from regression models with t distributed errors by approaching this computing problem through the use of two data augmentation schemes as well as variations of these well known algorithms we offer a more composite view on the performance of each

acceleration of convergence of vector sequences a general approach to the construction of convergence acceleration methods for vector sequences is proposed using this approach one can generate some known methods such as the minimal polynomial extrapolation the reduced rank extrapolation and the topological epsilon algorithm and also some new ones some of the new methods are easier to implement than the known methods and are observed to have similar numerical properties the convergence analysis of these new methods is carried out and it is shown that they are especially suitable for accelerating the convergence of vector sequences that are obtained when one solves linear systems of equations iteratively a stability analysis is also given and numerical examples are provided the convergence and stability properties of the topological epsilon algorithm are likewise given

on the construction and analysis of stochastic models characterization and propagation of the errors associated with limited data this paper investigates the predictive accuracy of stochastic models in particular a formulation is presented for the impact of data limitations associated with the calibration of parameters for these models on their overall predictive accuracy in the course of this development a new method for the characterization of stochastic processes from corresponding experimental observations is obtained specifically polynomial chaos representations of these processes are estimated that are consistent in some useful sense with the data the estimated polynomial chaos coefficients are themselves characterized as random variables with known probability density function thus permitting the analysis of the dependence of their values on further experimental evidence moreover the error in these coefficients associated with limited data is propagated through a physical system characterized by a stochastic partial differential equation spde this formalism permits the rational allocation of resources in view of studying the possibility of validating a particular predictive model a bayesian inference scheme is relied upon as the logic for parameter estimation with its computational engine provided by a metropolis hastings markov chain monte carlo procedure

space alternating generalized expectation maximization algorithm the expectation maximization em method can facilitate maximizing likelihood functions that arise in statistical estimation problems in the classical em paradigm one iteratively maximizes the conditional log likelihood of a single unobservable complete data space rather than maximizing the intractable likelihood function for the measured or incomplete data em algorithms update all parameters simultaneously which has two drawbacks 1 slow convergence and 2 difficult maximization steps due to coupling when smoothness penalties are used the paper describes the space alternating generalized em sage method which updates the parameters sequentially by alternating between several small hidden data spaces defined by the algorithm designer the authors prove that the sequence of estimates monotonically increases the penalized likelihood objective derive asymptotic convergence rates and provide sufficient conditions for monotone convergence in norm two signal processing applications illustrate the method estimation of superimposed signals in gaussian noise and image reconstruction from poisson measurements in both applications the sage algorithms easily accommodate smoothness penalties and converge faster than the em algorithms

acceleration of the em algorithm by using quasi newton methods the em algorithm is a popular method for maximum likelihood estimation its simplicity in many applications and desirable convergence properties make it very attractive its sometimes slow convergence however has prompted researchers to propose methods to accelerate it we review these methods classifying them into three groups pure hybrid and em type accelerators we propose a new pure and a new hybrid accelerator both based on quasi newton methods and numerically compare these and two other quasi newton accelerators for this we use examples in each of three areas poisson mixtures the estimation of covariance from incomplete data and multivariate normal mixtures in these comparisons the new hybrid accelerator was fastest on most of the examples and often dramatically so in some cases it accelerated the em algorithm by factors of over 100 the new pure accelerator is very simple to implement and competed well with the other accelerators it accelerated the em algorithm in some cases by factors of over 50 to obtain standard errors we propose to approximate the inverse of the observed information matrix by using auxiliary output from the new hybrid accelerator a numerical evaluation of these approximations indicates that they may be useful at least for exploratory purposes

using em to obtain asymptotic variance covariance matrices the sem algorithm abstract the expectation maximization em algorithm is a popular and often remarkably simple method for maximum likelihood estimation in incomplete data problems one criticism of em in practice is that asymptotic variance covariance matrices for parameters e g standard errors are not automatic byproducts as they are when using some other methods such as newton raphson in this article we define and illustrate a procedure that obtains numerically stable asymptotic variance covariance matrices using only the code for computing the complete data variance covariance matrix the code for em itself and code for standard matrix operations the basic idea is to use the fact that the rate of convergence of em is governed by the fractions of missing information to find the increased variability due to missing information to add to the complete data variance covariance matrix we call this supplemented em algorithm the sem algorithm theory and particular examples reinforce the conclusion that the sem alg

sph with the multiple boundary tangent method in this article we present an improved solid boundary treatment formulation for the smoothed particle hydrodynamics sph method benchmark simulations using previously reported boundary treatments can suffer from particle penetration and may produce results that numerically blow up near solid boundaries as well current sph boundary approaches do not properly treat curved boundaries in complicated flow domains these drawbacks have been remedied in a new boundary treatment method presented in this article called the multiple boundary tangent mbt approach in this article we present two important benchmark problems to validate the developed algorithm and show that the multiple boundary tangent treatment produces results that agree with known numerical and experimental solutions the two benchmark problems chosen are the lid driven cavity problem and flow over a cylinder the sph solutions using the mbt approach and the results from literature are in very good agreement these solutions involved solid boundaries but the approach presented herein should be extendable to time evolving free surface boundaries copyright 2008 john wiley sons ltd

a hidden seasonal switching model for high resolution breakpoint rainfall data 1 a nonhomogeneous hidden semi markov model nhsmm for breakpoint rainfall data is proposed which extends the homogeneous hidden semi markov model hsmm of sansom and thomson 2001 to incorporate stochastic seasonality the nhsmm model is able to switch seasons at times that are earlier or later than expected and in this way is able to explain additional seasonal variability due to varying length seasons the model s hidden rainfall states align with precipitation mechanisms that are seasonally invariant but the state dynamics are assumed to vary with season recursions for constructing the likelihood are developed and the em algorithm used to fit the parameters of the model an application of the model to breakpoint rainfall measurements from invercargill new zealand is discussed and the results of fitting a number of different nhsmms are compared to those from fitting the non seasonal homogeneous hsmm

likelihood analysis of multivariate probit models using a parameter expanded mcem algorithm multivariate binary data arise in a variety of settings in this article we propose a practical and efficient computational framework for maximum likelihood estimation of multivariate probit regression models this approach uses the monte carlo expectation maximization mcem algorithm with parameter expansion to complete the m step to avoid the direct evaluation of the intractable multivariate normal orthant probabilities the parameter expansion not only enables a closed form solution in the m step but also improves efficiency using the simulation studies we compare the performance of our approach with the mcem algorithms developed by chib and greenberg 1998 and song and lee 2005 as well as the iterative approach proposed by li and schafer 2008 our approach is further illustrated using a real world example

extensions of estimation methods using the em algorithm abstract the em algorithm described by dempster laid and rubin 1977 is reviewed with the purpose of clarifying several misconceptions in the statistical and econometric literature the clarifications lead to several applications of the algorithm to models that have appeared to be less tractable the relationship between the em algorithm and the method of scoring is also explained providing estimators of the score and the information from the em algorithm the em algorithm is extended to missing data problems and an estimation method based on simulations

one step bootstrapping for smooth iterative procedures summary resampling techniques have the potential to provide useful information about the sampling distribution of estimators of many population characteristics ambitious schemes such as the bootstrap and iterated bootstrap imply a substantial increase in computational effort for some iterative procedures such as generalized least squares or the em algorithm it is possible to avoid fully iterating each bootstrap replication to convergence by analysing expansions of the defining equation we can extract asymptotically correct bootstrap estimates from a single step for each replication in this paper we demonstrate the large sample validity of this computationally efficient approach and illustrate its small sample applicability whether or not the adjustment represents an adequate replacement for full iteration depends on the nature of the problem and the desired accuracy for the bootstrap quantiles if subsequent iterations are adjusted then greater enhancement of the rate is achieved and the practical increase in accuracy is significant

statistical guarantees for the em algorithm from population to sample based analysis we develop a general framework for proving rigorous guarantees on the performance of the em algorithm and a variant known as gradient em our analysis is divided into two parts a treatment of these algorithms at the population level in the limit of infinite data followed by results that apply to updates based on a finite set of samples first we characterize the domain of attraction of any global maximizer of the population likelihood this characterization is based on a novel view of the em updates as a perturbed form of likelihood ascent or in parallel of the gradient em updates as a perturbed form of standard gradient ascent leveraging this characterization we then provide non asymptotic guarantees on the em and gradient em algorithms when applied to a finite set of samples we develop consequences of our general theory for three canonical examples of incompletedata problems mixture of gaussians mixture of regressions and linear regression with covariates missing completely at random in each case our theory guarantees that with a suitable initialization a relatively small number of em or gradient em steps will yield with high probability an estimate that is within statistical error of the mle we provide simulations to confirm this theoretically predicted behavior

comparisons of weakly compressible and truly incompressible algorithms for the sph mesh free particle method in the smoothed particle hydrodynamics sph discretisation method for the navier stokes equations the most widespread method to solve for pressure and mass conservation is the weakly compressible assumption wcsph this includes hydraulics applications and leads to some drawbacks such as severe artificial pressure fluctuations and a limitation to very small time steps related to the wcsph mach number and explicit method this paper presents comparisons of a semi implicit and truly incompressible sph isph algorithm with the classical wcsph method showing how some of the problems encountered in wcsph have been resolved by using isph to simulate incompressible flows mathematical models are presented before describing sph formalism several standard boundary conditions are introduced and special attention is given to tracking the surface particles the lid driven cavity flow re 400 and 1000 is performed as a benchmarking test a bluff body test case a square cylinder in a closed channel re d 20 and 100 based on the cylinder diameter shows that pressure fields extracted from wcsph are very unreliable whereas isph predict pressures and forces in closer agreement with classical finite volume cfd methods dam breaking cases with dry or wet beds downstream are then presented to highlight free surface flow and rapid dynamics effects the wcsph and isph results are generally verified with reference data from experiment and or another numerical method all the comparisons show improvement with isph and good agreement in general

majorization minimization algorithms in signal processing communications and machine learning this paper gives an overview of the majorization minimization mm algorithmic framework which can provide guidance in deriving problem driven algorithms with low computational cost a general introduction of mm is presented including a description of the basic principle and its convergence results the extensions acceleration schemes and connection to other algorithmic frameworks are also covered to bridge the gap between theory and practice upperbounds for a large number of basic functions derived based on the taylor expansion convexity and special inequalities are provided as ingredients for constructing surrogate functions with the pre requisites established the way of applying mm to solving specific problems is elaborated by a wide range of applications in signal processing communications and machine learning

iterative and sequential algorithms for multisensor signal enhancement in problems of enhancing a desired signal in the presence of noise multiple sensor measurements will typically have components from both the signal and the noise sources when the systems that couple the signal and the noise to the sensors are unknown the problem becomes one of joint signal estimation and system identification the authors specifically consider the two sensor signal enhancement problem in which the desired signal is modeled as a gaussian autoregressive ar process the noise is modeled as a white gaussian process and the coupling systems are modeled as linear time invariant finite impulse response fir filters the main approach consists of modeling the observed signals as outputs of a stochastic dynamic linear system and the authors apply the estimate maximize em algorithm for jointly estimating the desired signal the coupling systems and the unknown signal and noise spectral parameters the resulting algorithm can be viewed as the time domain version of the frequency domain approach of feder et al 1989 where instead of the noncausal frequency domain wiener filter the kalman smoother is used this approach leads naturally to a sequential adaptive algorithm by replacing the kalman smoother with the kalman filter and in place of successive iterations on each data block the algorithm proceeds sequentially through the data with exponential weighting applied to allow adaption to nonstationary changes in the structure of the data a computationally efficient implementation of the algorithm is developed an expression for the log likelihood gradient based on the kalman smoother filter output is also developed and used to incorporate efficient gradient based algorithms in the estimation process

direct calculation of the information matrix via the em algorithm summary a simple explicit formula is provided for the matrix of second derivatives of the observed data log likelihood in terms of derivatives of the criterion function the conditional expectation of the complete data log likelihood given the observed data invoked by the em algorithm

vector extrapolated fast maximum likelihood estimation algorithms for emission tomography a new class of fast maximum likelihood estimation mle algorithms for emission computed tomography ect is developed in these cyclic iterative algorithms vector extrapolation techniques are integrated with the iterations in gradient based mle algorithms with the objective of accelerating the convergence of the base iterations this results in a substantial reduction in the effective number of base iterations required for obtaining an emission density estimate of specified quality the mathematical theory behind the minimal polynomial and reduced rank vector extrapolation techniques in the context of emission tomography is presented these extrapolation techniques are implemented in a positron emission tomography system the new algorithms are evaluated using computer experiments with measurements taken from simulated phantoms it is shown that with minimal additional computations the proposed approach results in substantial improvement in reconstruction

a fast improvement to the em algorithm on its own terms the em algorithm is a numerical technique for the evaluation of maximum likelihood estimates for parameters describing incomplete data models it is easy to apply in many problems and is stable but slow the algorithm fails to provide a consistent estimator of the standard errors of the maximum likelihood estimates unless the additional analysis required by the louis method is performed newton type or other gradient methods are faster and provide error estimates but tend to be unstable and require the analytical evaluation of likelihoods to derive expressions for the score function and at least approximations to the fisher information matrix the purpose of this paper is to expand on a result by fisher that permits a unification of em methodology and newton methods the evaluation of the individual observation by observation score functions of the incomplete data is a byproduct of the application of the e step of the em algorithm once these become available the fisher information matrix may be consistently estimated and the m step may be replaced by a fast newton type step

a statistical model for positron emission tomography abstract positron emission tomography pet still in its research stages is a technique that promises to open new medical frontiers by enabling physicians to study the metabolic activity of the body in a pictorial manner much as in x ray transmission tomography and other modes of computerized tomography the quality of the reconstructed image in pet is very sensitive to the mathematical algorithm to be used for reconstruction in this article we tailor a mathematical model to the physics of positron emissions and we use the model to describe the basic image reconstruction problem of pet as a standard problem in statistical estimation from incomplete data we describe various estimation procedures such as the maximum likelihood ml method using the em algorithm the method of moments and the least squares method a computer simulation of a pet experiment is then used to demonstrate the ml and the least squares reconstructions the main purposes of this article are to report on what we believe is an

tensor deflation for candecomp parafac part ii initialization and error analysis in part i of the study of the tensor deflation for candecomp parafac we have shown that the rank 1 tensor deflation is applicable under some conditions part ii of the study presents several initialization algorithms suitable for the algorithm proposed in part i in addition part ii contains an algorithm for the case when one or more factor matrices in the estimated model is constrained to be orthogonal finally part ii provides an error analysis of the tensor deflation algorithm which shows that there is a marginal loss of accuracy of the deflation algorithm compared to the ordinary cp decomposition

meshfree numerical approach based on the finite pointset method for static linear elasticity problems abstract in this work a promising meshfree numerical approach is extended and implemented for the first time in the field of linear static elasticity a real meshfree method the so called finite pointset method fpm is applied and implemented in order to solve the strong classical form of the governing partial differential equations pdes the well known navier cauchy equations fpm numerical performance is demonstrated for several classical and well known studied benchmark problems and comparison with other approaches is also reported numerical results suggest that fpm method is stable with good convergence behavior for 2d linear static elasticity problems

finding the observed information matrix when using the em algorithm a procedure is derived for extracting the observed information matrix when the em algorithm is used to find maximum likelihood estimates in incomplete data problems the technique requires computation of a complete data gradient vector or second derivative matrix but not those associated with the incomplete data likelihood in addition a method useful in speeding up the convergence of the em algorithm is developed two examples are presented

empirical bayes analysis for systems of mixed models with linked autocorrelated random effects abstract empirical bayes concepts are implemented in a simultaneous analysis of a system of mixed linear models having linked and serially correlated random effects emphasis is placed on the estimation of the random effects and exploration of the relationships between them application is made to the investigation of several series of laboratory assay data that were observed during overlapping time intervals and were therefore subjected to common systematic errors or daily effects the motivation for this work was the need to investigate methods of adjustment for such daily effects and to estimate the degree to which concurrently run series are impacted in common attention is given to the construction of confidence intervals for daily effects tractable methods are proposed that yield approximately correct coverage for large samples although derived within a bayes empirical bayes framework these intervals are somewhat similar to intervals constructed by the method of kackar and harville implement

convergence theorems for generalized alternating minimization procedures the em algorithm is widely used to develop iterative parameter estimation procedures for statistical models in cases where these procedures strictly follow the em formulation the convergence properties of the estimation procedures are well understood in some instances there are practical reasons to develop procedures that do not strictly fall within the em framework we study em variants in which the e step is not performed exactly either to obtain improved rates of convergence or due to approximations needed to compute statistics under a model family over which e steps cannot be realized since these variants are not em procedures the standard g em convergence results do not apply to them we present an information geometric framework for describing such algorithms and analyzing their convergence properties we apply this framework to analyze the convergence properties of incremental em and variational em for incremental em we discuss conditions under these algorithms converge in likelihood for variational em we show how the e step approximation prevents convergence to local maxima in likelihood

kullback proximal algorithms for maximum likelihood estimation accelerated algorithms for maximum likelihood image reconstruction are essential for emerging applications such as three dimensional 3 d tomography dynamic tomographic imaging and other high dimensional inverse problems in this paper we introduce and analyze a class of fast and stable sequential optimization methods for computing maximum likelihood estimates and study its convergence properties these methods are based on a proximal point algorithm implemented with the kullback liebler kl divergence between posterior densities of the complete data as a proximal penalty function when the proximal relaxation parameter is set to unity one obtains the classical expectation maximization em algorithm for a decreasing sequence of relaxation parameters relaxed versions of em are obtained which can have much faster asymptotic convergence without sacrifice of monotonicity we present an implementation of the algorithm using more s 1983 trust region update strategy for illustration the method is applied to a nonquadratic inverse problem with poisson distributed data