the em algorithm an old folk song sung to a fast new tune celebrating the 20th anniversary of the presentation of the paper by dempster laird and rubin which popularized the em algorithm we investigate after a brief historical account strategies that aim to make the em algorithm converge faster while maintaining its simplicity and stability e g automatic monotone convergence in likelihood first we introduce the idea of a working parameter to facilitate the search for efficient data augmentation schemes and thus fast em implementations second summarizing various recent extensions of the em algorithm we formulate a general alternating expectation conditional maximization algorithm aecm that couples flexible data augmentation schemes with model reduction schemes to achieve efficient computations we illustrate these methods using multivariate t models with known or unknown degrees of freedom and poisson models for image reconstruction we show through both empirical and theoretical evidence the potential for a dramatic reduction in computational time with little increase in human effort we also discuss the intrinsic connection between em type algorithms and the gibbs sampler and the possibility of using the techniques presented here to speed up the latter the main conclusion of the paper is that with the help of statistical considerations it is possible to construct algorithms that are simple stable and fast

accelerated iterative reconstruction for positron emission tomography based on the em algorithm for maximum likelihood estimation the em method that was originally developed for maximum likelihood estimation in the context of mathematical statistics may be applied to a stochastic model of positron emission tomography pet the result is an iterative algorithm for image reconstruction that is finding increasing use in pet due to its attractive theoretical and practical properties its major disadvantage is the large amount of computation that is often required due to the algorithm s slow rate of convergence this paper presents an accelerated form of the em algorithm for pet in which the changes to the image as calculated by the standard algorithm are multiplied at each iteration by an overrelaxation parameter the accelerated algorithm retains two of the important practical properties of the standard algorithm namely the selfnormalization and nonnegativity of the reconstructed images experimental results are presented using measured data obtained from a hexagonal detector system for pet the likelihood function and the norm of the data residual were monitored during the iterative process according to both of these measures the images reconstructed at iterations 7 and 11 of the accelerated algorithm are similar to those at iterations 15 and 30 of the standard algorithm for two different sets of data important theoretical properties remain to be investigated namely the convergence of the accelerated algorithm and its performance as a maximum likelihood estimator

on estimation and prediction for spatial generalized linear mixed models we use spatial generalized linear mixed models glmm to model non gaussian spatial variables that are observed at sampling locations in a continuous area in many applications prediction of random effects in a spatial glmm is of great practical interest we show that the minimum mean squared error mmse prediction can be done in a linear fashion in spatial glmms analogous to linear kriging we develop a monte carlo version of the em gradient algorithm for maximum likelihood estimation of model parameters a by product of this approach is that it also produces the mmse estimates for the realized random effects at the sampled sites this method is illustrated through a simulation study and is also applied to a real data set on plant root diseases to obtain a map of disease severity that can facilitate the practice of precision agriculture

parameter expansion to accelerate em the px em algorithm summary the em algorithm and its extensions are popular tools for modal estimation but ar e often criticised for their slow convergence we propose a new method that can often make em much faster the intuitive idea is to use a covariance adjustment to correct the analysis of the m step capitalising on extra information captured in the imputed complete data the way we accomplish this is by parameter expansion we expand the complete data model while preserving the observed data model and use the expanded complete data model to generate em this parameter expanded ei m px em algorithm shares the simplicity and stability of ordinary em but has a faster rate of convergence since its m step performs a more efficient analysis the px em algorithm is illustrated for the multivariate t distribution a random effects model factor analysis probit regression and a poisson imaging model

reliable h_ infty control of linear systems with adaptive mechanism this technical note studies the problem of designing reliable h controllers with adaptive mechanism for linear systems a new method for designing indirect adaptive reliable controller via state feedback is presented for actuator fault compensations based on the on line estimation of eventual faults the proposed reliable controller parameters are updated automatically to compensate the fault effects on systems a notion of adaptive h performance index is proposed to describe the disturbance attenuation performances of closed loop systems the design conditions are given in terms of solutions to a set of linear matrix inequalities lmis the resultant designs can guarantee the asymptotic stability and adaptive h performances of closed loop systems even in the cases of actuator failures the effectiveness of the proposed design method is illustrated via a numerical example

computability structure of the sobolev spaces and its applications in this paper we present a definition of s computability for functions in sobolev spaces hs rd s r two applications of s computability are then discussed one is to address several open problems regarding classical computable functions the other is to conduct computable analysis of several important classes of partial differential equations

interpolation function and computability computability of banach spaces is discussed a compatible relation is shown to hold between the complex interpolation spaces of calderon studia math 24 1964 113 and the computability structures introduced by pour el and richards computability in analysis and physics springer berlin 1989 namely it is verified that calderon s original construction of the complex interpolation functor is valid in the context of computability if a mild effective separability condition is fulfilled by a compatible couple of banach spaces

joint state and parameter estimation for a target directed nonlinear dynamic system model we present a new approach to joint state and parameter estimation for a target directed nonlinear dynamic system model with switching states the model recently proposed for representing speech dynamics is called the hidden dynamic model hdm the model parameters subject to statistical estimation consist of the target vector and the system matrix also called time constants as well as parameters characterizing the nonlinear mapping from the hidden state to the observation we implement these parameters as the weights of a three layer feedforward multilayer perceptron mlp network the new estimation approach is based on the extended kalman filter ekf and its performance is compared with the traditional expectation maximization em based approach extensive simulation results are presented using both approaches and under typical hdm speech modeling conditions the ekf based algorithm demonstrates superior convergence performance compared with the em algorithm but the former suffers from excessive computational loads when adopted for training the mlp weights in all cases the simulated model output converges to the given observation sequence however only in the case where the mlp weights or the target vector are assumed known do the time constant parameters converge to their true values we also show that the mlp weights never converge to their true values thus demonstrating the many to one mapping property of the feedforward mlp we conclude that for the system to be identifiable restrictions on the parameter space are needed

maximum likelihood estimation via the ecm algorithm a general framework two major reasons for the popularity of the em algorithm are that its maximum step involves only complete data maximum likelihood estimation which is often computationally simple and that its convergence is stable with each iteration increasing the likelihood when the associated complete data maximum likelihood estimation itself is complicated em is less attractive because the m step is computationally unattractive in many cases however complete data maximum likelihood estimation is relatively simple when conditional on some function of the parameters being estimated

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

recursive parameter estimation using incomplete data abstract stochastic approximation procedures are considered for the estimation of parameters using incomplete data one procedure is stated and illustrated which often leads to asymptotically efficient estimators others are developed which although possibly not optimal in the above sense will be very much easier to apply this will be particularly advantageous when quick recursive estimates are required examples are given and a link is made between one of the sub optimal methods and the em algorithm author

a computational model for metric spaces for every metric space x we define a continuous poset bx such that x is homeomorphic to the set of maximal elements of bx with the relative scott topology the poset bx is a dcpo iff x is complete and continuous iff x is separable the computational model bx is used to give domain theoretic proofs of banach s fixed point theorem and of two classical results of hutchinson on a complete metric space every hyperbolic iterated function system has a unique non empty compact attractor and every iterated function system with probabilities has a unique invariant measure with bounded support we also show that the probabilistic power domain of bx provides an continuous computational model for measure theory on a separable complete metric space x

a disaggregate negative binomial regression procedure for count data analysis various research areas face the methodological problems presented by nonnegative integer count data drawn from heterogeneous populations we present a disaggregate negative binomial regression procedure for analysis of count data observed for a heterogeneous sample of cross sections possibly over some fixed time periods this procedure simultaneously pools or groups cross sections while estimating a separate negative binomial regression model for each group an e m algorithm is described within a maximum likelihood framework to estimate the group proportions the group specific regression coefficients and the degree of overdispersion in event rates within each derived group the proposed procedure is illustrated with count data entailing nonnegative integer counts of purchases events for a frequently bought consumer good

generalized linear latent variable models for repeated measures of spatially correlated multivariate data summary observations of multiple response variables across space and over time occur often in environmental and ecological studies compared to purely spatial models for a single response variable in the exponential family of distributions fewer statistical tools are available for multiple response variables that are not necessarily gaussian an exception is a common factor model developed for multivariate spatial data by wang and wall 2003 biostatistics 4 569 582 the purpose of this article is to extend this multivariate space only model and develop a flexible class of generalized linear latent variable models for multivariate spatial temporal data for statistical inference maximum likelihood estimates and their standard deviations are obtained using a monte carlo em algorithm we also use a novel way to automatically adjust the monte carlo sample size which facilitates the convergence of the monte carlo em algorithm the methodology is illustrated by an ecological study of red pine trees in response to bark beetle challenges in a forest stand of wisconsin

alternative em methods for nonparametric finite mixture models summary this research focuses on a general class of maximum likelihood problems in which it is desired to maximise a nonparametric mixture likelihood with finitely many known component densities over the set of unknown weight parameters convergence of the conventional em algorithm for this problem is extremely slow when the component densities are poorly separated and when the maximum likelihood estimator requires some of the weights to be zero as the algorithm can never reach such a boundary point alternative methods based on the principles of em are developed using a two stage approach first a new data augmentation scheme provides improved convergence rates in certain parameter directions secondly two cyclic versions of this data augmentation are created by changing the missing data formulation between the em steps these extend the acceleration directions to the whole parameter space giving another order of magnitude increase in convergence rate examples indicate that the new cyclic versions of the data augmentation schemes can converge up to 500 times faster than the conventional em algorithm for fitting nonparametric finite mixture models

large sample joint posterior approximations when full conditionals are approximately normal modern bayesian statistical methods such as gibbs and metropolis hastings sampling were developed to liberate statisticians from the necessity of making large sample assumptions and to facilitate the numerical approximation of problems that had previously been analytically intractable counter to this trend we develop a method for constructing asymptotic joint posterior approximations based on models with k blocks of parameters and where the corresponding properly normalized full conditionals are themselves asymptotically normal we illustrate these techniques by applying them to particular linear and generalized linear mixed models glmms we also consider the relevance of different parameterizations with regard to our asymptotics recent work has indicated that gibbs samplers based on so called centering parameterizations result in better convergence properties for the resulting markov chains our results for the one way random effects model shed some light on this issue for this example we also

convergence of a stochastic approximation version of the em algorithm the expectation maximization em algorithm is a powerful computational technique for locating maxima of functions it is widely used in statistics for maximum likelihood or maximum a posteriori estimation in incomplete data models in certain situations however this method is not applicable because the expectation step cannot be performed in closed form to deal with these problems a novel method is introduced the stochastic approximation em saem which replaces the expectation step of the em algorithm by one iteration of a stochastic approximation procedure the convergence of the saem algorithm is established under conditions that are applicable to many practical situations moreover it is proved that under mild additional conditions the attractive stationary points of the saem algorithm correspond to the local maxima of the function presented to support our findings

maximum pseudo likelihood estimation in network tomography network monitoring and diagnosis are key to improving network performance the difficulties of performance monitoring lie in today s fast growing internet accompanied by increasingly heterogeneous and unregulated structures moreover these tasks become even harder since one cannot rely on the collaboration of individual routers and servers to measure network traffic directly even though the aggregative nature of possible network measurements gives rise to inverse problems existing methods for solving inverse problems are usually computationally intractable or statistically inefficient a pseudo likelihood approach is proposed to solve a group of network tomography problems the basic idea of pseudo likelihood is to form simple subproblems and ignore the dependences among the subproblems to form a product likelihood of the subproblems as a result this approach keeps a good balance between the computational complexity and the statistical efficiency of the parameter estimation some statistical properties of the pseudo likelihood estimator such as consistency and asymptotic normality are established a pseudo expectation maximization em algorithm is developed to maximize the pseudo log likelihood function two examples with simulated or real data are used to illustrate the pseudo likelihood proposal 1 inference of the internal link delay distributions through multicast end to end measurements 2 origin destination matrix estimation through link traffic counts

a monte carlo implementation of the em algorithm and the poor man s data augmentation algorithms abstract the first part of this article presents the monte carlo implementation of the e step of the em algorithm given the current guess to the maximizer of the posterior distribution latent data patterns are generated from the conditional predictive distribution the expected value of the augmented log posterior is then updated as a mixture of augmented log posteriors mixed over the generated latent data patterns multiple imputations in the m step of the algorithm this mixture is maximized to obtain the update to the maximizer of the observed posterior the gradient and hessian of the observed log posterior are also expressed as mixtures mixed over the multiple imputations the relation between the monte carlo em mcem algorithm and the data augmentation algorithm is noted two modifications to the mcem algorithm the poor man s data augmentation algorithms which allow for the calculation of the entire posterior are then presented these approximations serve as diagnostics for the validity o

the wave equation with computable initial data such that its unique solution is not computable we consider the three dimensional wave equation it is well known that the solution u x y z t is uniquely determined by two initial conditions the values of u and au at at time t 0 our question is can computable initial data give rise to noncomputable solutions the answer is yes and two quite different types of noncomputability can occur theorem 1 below gives an example in which the solution u x y z t takes a noncomputable real value at a computable point in space time by contrast theorem 2 provides an example in which the solution maps each computable sequence of points in space time into a computable sequence nevertheless u x y z t is not a computable function we note in passing that computability as used in this paper is a technical term familiar to mathematical logicians the precise definitions are spelled out in the next section the proof of theorem 1 is quite short that for theorem 2 is considerably more intricate it should be mentioned that these noncomputable solutions of the wave equation are of the type commonly referred to as weak solutions i e although continuous they are not twice differentiable at all points weak solutions describe creases cusps and other non differentiable patterns which frequently appear in models of wave phenomena the use of weak solutions is inevitable we will prove that no cz solutions of the desired kind exist for the convenience of readers unfamiliar with weak solutions we have put them into a coherent framework in an addendum basing our presentation on the energy integral associated with the wave just as the initial conditions are computable the energy integral in our examples is also a computable real finally the proof that all noncomputable solutions must be of weak type is sketched at the end of this addendum we note however that the main part of the paper may be read without reference to the addendum the results in this paper are related to comments of kreisel in 4 kreisel asks whether existing physical theories e g classical mechanics or quantum mechanics can predict theoretically the existence of a physical constant which is not a recursive real previous work of the authors in this area 9 was concerned with ordinary differential equations it was proved that there exists a computable and hence continuous function f such that 215 oool 8708 81 030215 25905 00 o

an algorithm based on discrete response regression models suitable to correct the bias of non response in surveys with several capture tries the use of survey plans which contemplate several tries or call backs when endeavouring to capture individual data may supply unarguable information in certain sampling situations with non ignorable non response this paper presents an algorithm whose final aim is the estimation of the individual non response probabilities from a general perspective of discrete response regression models which includes the well known probit and logit models it will be assumed that the respondents supply all the variables of interest when they are captured nevertheless the call backs continue even after previous captures for a small number of tries r which has been fixed beforehand only for estimating purposes the different retries or call backs are supposed to be carried out with different capture intensities as mentioned above the response probabilities which may vary from one individual to another are sought by discrete response regression models whose parameters are estimated from conditioned likelihoods evaluated on the respondents only the algorithm quick and easy to implement may be used even when the capture indicator matrix has been partially recorded finally the practical performance of the proposed procedure is tested and evaluated from empirical simulations whose results are undoubtedly encouraging

brief paper non fragile h filter design for linear continuous time systems this paper studies the problem of non fragile h filter design for linear continuous time systems the filter to be designed is assumed to include additive gain variations which result from filter implementations a notion of structured vertex separator is proposed to approach the problem and exploited to develop sufficient conditions for the non fragile h filter design in terms of solutions to a set of linear matrix inequalities lmis the designs guarantee the asymptotic stability of the estimation errors and the h performance of the system from the exogenous signals to the estimation errors below a prescribed level a numerical example is given to illustrate the effect of the proposed method

latent variable models for mixed discrete and continuous outcomes we propose a latent variable model for mixed discrete and continuous outcomes the model accommodates any mixture of outcomes from an exponential family and allows for arbitrary covariate effects as well as direct modelling of covariates on the latent variable an em algorithm is proposed for parameter estimation and estimates of the latent variables are produced as a by product of the analysis a generalized likelihood ratio test can be used to test the significance of covariates affecting the latent outcomes this method is applied to birth defects data where the outcomes of interest are continuous measures of size and binary indicators of minor physical anomalies infants who were exposed in utero to anticonvulsant medications are compared with controls

the dynamic expectation conditional maximization either algorithm summary the expectation conditional maximization either ecme algorithm has proven to be an effective way of accelerating the expectation maximization algorithm for many problems recognizing the limitation of using prefixed acceleration subspaces in the ecme algorithm we propose a dynamic ecme decme algorithm which allows the acceleration subspaces to be chosen dynamically the simplest decme implementation is what we call decme 1 which uses the line that is determined by the two most recent estimates as the acceleration subspace the investigation of decme 1 leads to an efficient simple stable and widely applicable decme implementation which uses two dimensional acceleration subspaces and is referred to as decme 2 the fast convergence of decme 2 is established by the theoretical result that in a small neighbourhood of the maximum likelihood estimate it is equivalent to a conjugate direction method the remarkable accelerating effect of decme 2 and its variant is also demonstrated with several numerical examples

on use of the em algorithm for penalized likelihood estimation summary the em algorithm is a popular approach to maximum likelihood estimation but has not been much used for penalized likelihood or maximum a posteriori estimation this paper discusses properties of the em algorithm in such contexts concentrating on rates of conver gence and presents an alternative that is usually more practical and converges at least as quickly the em algorithm is a general approach to maximum likelihood estimation rather than a specific algorithm dempster et al 1977 discussed the method and derived basic properties demonstrating that a variety of procedures previously developed rather informally could be unified the common strand to problems where the approach is applicable is a notion of incomplete data this includes the conventional sense of missing data but is much broader than that the em algorithm demon strates its strength in situations where some hypothetical experiment yields data from which estimation is particularly convenient and economical the incomplete data actually at hand are regarded as observable functions of these complete data the resulting algorithms while usually slow to converge are often extremely simple and remain practical in large problems where no other approaches may be feasible dempster et al 1977 briefly refer to the use of the same approach to the problem of finding the posterior mode maximum a posteriori estimate in a bayesian estima

low rank structure learning via nonconvex heuristic recovery in this paper we propose a nonconvex framework to learn the essential low rank structure from corrupted data different from traditional approaches which directly utilizes convex norms to measure the sparseness our method introduces more reasonable nonconvex measurements to enhance the sparsity in both the intrinsic low rank structure and the sparse corruptions we will respectively introduce how to combine the widely used lp norm 0 p 1 and log sum term into the framework of low rank structure learning although the proposed optimization is no longer convex it still can be effectively solved by a majorization minimization mm type algorithm with which the nonconvex objective function is iteratively replaced by its convex surrogate and the nonconvex problem finally falls into the general framework of reweighed approaches we prove that the mm type algorithm can converge to a stationary point after successive iterations the proposed model is applied to solve two typical problems robust principal component analysis and low rank representation experimental results on low rank structure learning demonstrate that our nonconvex heuristic methods especially the log sum heuristic recovery algorithm generally perform much better than the convex norm based method 0 p 1 for both data with higher rank and with denser corruptions

robust statistical modeling using the t distribution abstract the t distribution provides a useful extension of the normal for statistical modeling of data sets involving errors with longer than normal tails an analytical strategy based on maximum likelihood for a general model with multivariate t errors is suggested and applied to a variety of problems including linear and nonlinear regression robust estimation of the mean and covariance matrix with missing data unbalanced multivariate repeated measures data multivariate modeling of pedigree data and multivariate nonlinear regression the degrees of freedom parameter of the t distribution provides a convenient dimension for achieving robust statistical inference with moderate increases in computational complexity for many models estimation of precision from asymptotic theory and the bootstrap is discussed and graphical methods for checking the appropriateness of the t distribution are presented

time varying network tomography router link data abstract the origin destination od traffic matrix of a computer network is useful for solving problems in design routing configuration debugging monitoring and pricing directly measuring this matrix is not usually feasible but less informative link measurements are easy to obtain this work studies the inference of od byte counts from link byte counts measured at router interfaces under a fixed routing scheme a basic model of the od counts assumes that they are independent normal over od pairs and iid over successive measurement periods the normal means and variances are functionally related through a power law we deal with the time varying nature of the counts by fitting the basic iid model locally using a moving data window identifiability of the model is proved for router link data and maximum likelihood is used for parameter estimation the od counts are estimated by their conditional expectations given the link counts and estimated parameters thus od estimates are forced to be positive

an expanded theoretical treatment of iteration dependent majorize minimize algorithms the majorize minimize mm optimization technique has received considerable attention in signal and image processing applications as well as in statistics literature at each iteration of an mm algorithm one constructs a tangent majorant function that majorizes the given cost function and is equal to it at the current iterate the next iterate is obtained by minimizing this tangent majorant function resulting in a sequence of iterates that reduces the cost function monotonically a well known special case of mm methods are expectation maximization algorithms in this paper we expand on previous analyses of mm due to fessler and hero that allowed the tangent majorants to be constructed in iteration dependent ways also this paper overcomes an error in one of those earlier analyses there are three main aspects in which our analysis builds upon previous work first our treatment relaxes many assumptions related to the structure of the cost function feasible set and tangent majorants for example the cost function can be nonconvex and the feasible set for the problem can be any convex set second we propose convergence conditions based on upper curvature bounds that can be easier to verify than more standard continuity conditions furthermore these conditions allow for considerable design freedom in the iteration dependent behavior of the algorithm finally we give an original characterization of the local region of convergence of mm algorithms based on connected e g convex tangent majorants for such algorithms cost function minimizers will locally attract the iterates over larger neighborhoods than typically is guaranteed with other methods this expanded treatment widens the scope of the mm algorithm designs that can be considered for signal and image processing applications allows us to verify the convergent behavior of previously published algorithms and gives a fuller understanding overall of how these algorithms behave

constrained maximum likelihood estimation of two level covariance structure model via em type algorithms in this paper the constrained maximum likelihood estimation of a two level covariance structure model with unbalanced designs is considered the two level model is reformulated as a single level model by treating the group level latent random vectors as hypothetical missing data then the popular em algorithm is extended to obtain the constrained maximum likelihood estimates for general nonlinear constraints the multiplier method is used at them step to find the constrained minimum of the conditional expectation an accelerated em gradient procedure is derived to handle linear constraints the empirical performance of the proposed em type algorithms is illustrated by some artifical and real examples

from axiomatics to intrinsic characterization some open problems in computable analysis abstract five open problems are presented the framework for these problems is the notion of a computability structure on a banach space which is discussed below the study of banach spaces in particular the study of hilbert space is an important topic in mathematics and its applications problems a and b are concerned with transfer principles for obtaining effective versions of classical theorems of mathematics problems c and d deal with separability and effective separability problem e is concerned with computability for classes of mathematical structures other than banach spaces

concrete models of computation for topological algebras a concrete model of computation for a topological algebra is based on a representation of the algebra made from functions on the natural numbers the functions computable in a concrete model are computable in the representation in the classical sense of the chruch turing thesis moreover the functions turn out to be continuous in the topology of the algebra in this paper we consider different concrete models for computing in topological algebras and prove their mutual equivalence in certain commonly occurring circumstances for topological algebras the concrete models we use are effective representation by algebraic domains stoltenberg hansen and tucker effective representation by continuous domains edelat effective representation by type two recursion on baire space weihrauch and for metric and normed algebras we use effective metric spaces moschovakis and computability structures pour el and richards the result are evidence that computability theory for topological algebras is a stable theory independent of the specific models of computation just as classical computability theory for discrete algebras is stable

uniform computational complexity of the derivatives of c functions we discuss the uniform computational complexity of the derivatives of c functions in the model of ko and friedman ko complexity theory of real functions birkhauser basel 1991 ko friedman theor comput sci 20 1982 323 352 we construct a polynomial time computable real function g c 1 1 such that the sequence g n 0 n n is not bounded by any recursive function on the other hand we show that if f c 1 1 is polynomial time computable and the sequence of the derivatives of f is uniformly polynomially bounded i e f n x is bounded by 2p n for all x 1 1 for some polynomial p then the sequence f n n n is uniformly polynomial time computable

vector l_0 sparse variable pca principal component analysis pca achieves dimension reduction by replacing the original measured variables with a smaller set of derived variables called the principal components sparse pca improves this with sparsity there are two kinds of sparse pca sparse loading pca slpca which keeps all the measured variables but zeroes out some of their loadings and sparse variable pca svpca which removes some measured variables completely by simultaneously zeroing out all their loadings because it zeroes out some measured variables completely svpca is capable of huge additional dimension reduction beyond pca while slpca keeps all measured variables and does not have this capability here we consider a vector l0 penalized likelihood approach to svpca and develop a penalized expectation maximization pem algorithm which remarkably in an l0 setting leads to a closed form m step and we provide a convergence analysis

maximum likelihood estimation of nonlinear structural equation models the existing maximum likelihood theory and its computer software in structural equation modeling are established based on linear relationships among manifest variables and latent variables however models with nonlinear relationships are often encountered in social and behavioral sciences in this article an em type algorithm is developed for maximum likelihood estimation of a general nonlinear structural equation model to avoid computation of the complicated multiple integrals involved the e step is completed by a metropolis hastings algorithm it is shown that the m step can be completed efficiently by simple conditional maximization standard errors of the maximum likelihood estimates are obtained via louis s formula the methodology is illustrated with results from a simulation study and two real examples

domain representations of partial functions with applications to spatial objects and constructive volume geometry a partial spatial object is a partial map from space to data data types of partial spatial objects are modelled by topological algebras of partial maps and are the foundation for a high level approach to volume graphics called constructive volume geometry cvg where space and data are subspaces of n dimensional euclidean space we investigate the computability of partial spatial object data types in general and in volume graphics using the theory of effective domain representations for topological algebras the basic mathematical problem considered is to classify which partial functions between topological spaces can be represented by total continuous functions between given domain representations of the spaces we prove theorems about partial functions on regular hausdorff spaces and their domain representations and apply the results to partial spatial objects and cvg algebras

sparse variable pca using geodesic steepest descent principal component analysis pca is a dimensionality reduction technique used in most fields of science and engineering it aims to find linear combinations of the input variables that maximize variance a problem with pca is that it typically assigns nonzero loadings to all the variables which in high dimensional problems can require a very large number of coefficients but in many applications the aim is to obtain a massive reduction in the number of coefficients there are two very different types of sparse pca problems sparse loadings pca slpca which zeros out loadings while generally keeping all of the variables and sparse variable pca which zeros out whole variables typically leaving less than half of them in this paper we propose a new svpca which we call sparse variable noisy pca svnpca it is based on a statistical model and this gives access to a range of modeling and inferential tools estimation is based on optimizing a novel penalized log likelihood able to zero out whole variables rather than just some loadings the estimation algorithm is based on the geodesic steepest descent algorithm finally we develop a novel form of bayesian information criterion bic for tuning parameter selection the svnpca algorithm is applied to both simulated data and real functional magnetic resonance imaging fmri data

weighted breslow type and maximum likelihood estimation in semiparametric transformation models a semiparametric transformation model comprises a parametric component for covariate effects and a nonparametric component for the baseline hazard intensity the breslow type estimator has been proposed for estimating the nonparametric component in some inefficient estimation procedures we show that introducing weights into this estimator leads to nonparametric maximum likelihood estimation with the weights depending on the martingale residuals the weighted breslow type estimator suggests an iterative reweighting algorithm for nonparametric maximum likelihood estimation which can be implemented by a weighted variant of the existing algorithms for inefficient estimation and can be computationally more efficient than an em type algorithm the weighting idea is further extended to semiparametric transformation models with mismeasured covariates

one step sparse estimates in nonconcave penalized likelihood models fan and li propose a family of variable selection methods via penalized likelihood using concave penalty functions the nonconcave penalized likelihood estimators enjoy the oracle properties but maximizing the penalized likelihood function is computationally challenging because the objective function is nondifferentiable and nonconcave in this article we propose a new unified algorithm based on the local linear approximation lla for maximizing the penalized likelihood for a broad class of concave penalty functions convergence and other theoretical properties of the lla algorithm are established a distinguished feature of the lla algorithm is that at each lla step the lla estimator can naturally adopt a sparse representation thus we suggest using the one step lla estimator from the lla algorithm as the final estimates statistically we show that if the regularization parameter is appropriately chosen the one step lla estimates enjoy the oracle properties with good initial estimators computationally the one step lla estimation methods dramatically reduce the computational cost in maximizing the nonconcave penalized likelihood we conduct some monte carlo simulation to assess the finite sample performance of the one step sparse estimation methods the results are very encouraging

robust h_2 control of linear systems with mismatched quantization this paper is concerned with the h 2 control of linear systems with multiple quantization channels the quantization parameters of each channel are not required to be identical the resultant mismatches are represented by polytopic uncertainties a composite controller composed of linear and nonlinear parts is designed to meet the required h 2 performance and offset the quantization error resorting to a vertex separation technique and finsler lemma instead of matrix inverse operations new synthesis conditions for the desired linear part are derived in terms of linear matrix inequalities which are further extended to treat systems with norm bounded uncertainties a comparison of conservativeness between the proposed methods and the existing ones is demonstrated by two numerical examples

variable selection using mm algorithms variable selection is fundamental to high dimensional statistical modeling many variable selection techniques may be implemented by maximum penalized likelihood using various penalty functions optimizing the penalized likelihood function is often challenging because it may be nondifferentiable and or nonconcave this article proposes a new class of algorithms for finding a maximizer of the penalized likelihood for a broad class of penalty functions these algorithms operate by perturbing the penalty function slightly to render it differentiable then optimizing this differentiable function using a minorize maximize mm algorithm mm algorithms are useful extensions of the well known class of em algorithms a fact that allows us to analyze the local and global convergence of the proposed algorithm using some of the techniques employed for em algorithms in particular we prove that when our mm algorithms converge they must converge to a desirable point we also discuss conditions under which this convergence may be guaranteed we exploit the newton raphson like aspect of these algorithms to propose a sandwich estimator for the standard errors of the estimators our method performs well in numerical tests

theory of representations abstract an approach for a simple general and unified theory of effectivity on sets with cardinality not greater than that of the continuum is presented a standard theory of effectivity on f f n n has been developed in a previous paper by representations f m this theory is extended to other sets m topological and recursion theoretical properties of representations are studied where the final topology of a representation plays an essential role it is shown that for any separable t0 space an up to equivalence unique admissible representation can be defined which reflects the topological properties correctly

system identification of nonlinear state space models this paper is concerned with the parameter estimation of a general class of nonlinear dynamic systems in state space form more specifically a maximum likelihood ml framework is employed and an expectation maximisation em algorithm is derived to compute these ml estimates the expectation e step involves solving a nonlinear state estimation problem where the smoothed estimates of the states are required this problem lends itself perfectly to the particle smoother which provides arbitrarily good estimates the maximisation m step is solved using standard techniques from numerical optimisation theory simulation examples demonstrate the efficacy of our proposed solution

globally convergent algorithms for maximum a posteriori transmission tomography this paper reviews and compares three maximum likelihood algorithms for transmission tomography one of these algorithms is the em algorithm one is based on a convexity argument devised by de pierro see ieee trans med imaging vol 12 p 328 333 1993 in the context of emission tomography and one is an ad hoc gradient algorithm the algorithms enjoy desirable local and global convergence properties and combine gracefully with bayesian smoothing priors preliminary numerical testing of the algorithms on simulated data suggest that the convex algorithm and the ad hoc gradient algorithm are computationally superior to the em algorithm this superiority stems from the larger number of exponentiations required by the em algorithm the convex and gradient algorithms are well adapted to parallel computing

estimating survival curves with left truncated and interval censored data under monotone hazards we show that the nonparametric maximum likelihood estimator npmle of a survival function may severely underestimate the survival probabilities at very early times for left truncated and interval censored data as an alternative we propose to compute the nonparametric mle under a nondecreasing hazard assumption the monotone mle by a gradient projection algorithm when the assumption holds the projection step is accomplished via an isotonic regression algorithm the pool adjacent violators algorithm this gradient projection algorithm is computationally efficient and converges globally monte carlo simulations show superior performance of the monotone mle over that of the npmle in terms of either bias or variance even for large samples the methodology is illustrated with the application to the wisconsin epidemiological study of diabetic retinopathy data to estimate the probability of incidence of retinopathy

computable banach spaces via domain theory this paper extends the order theoretic approach to computable analysis via continuous domains to complete metric spaces and banach spaces we employ the domain of formal balls to define a computability theory for complete metric spaces for banach spaces the domain specialises to the domain of closed balls ordered by reversed inclusion we characterise computable linear operators as those which map computable sequences to computable sequences and are effectively bounded we show that the domain theoretic computability theory is equivalent to the well established approach by pour el and richards

observer based control for piecewise affine systems with both input and output quantization this technical note is concerned with the problem of simultaneous design of observers and controllers for a class of piecewise affine systems against signal quantization occurring in both measurement output and control input channels the general scenario is considered that system state and estimated state may not be in the same operating region by a novel quantization error dependent lyapunov function the stability and mathcal h infty performance criteria are first established for the augmented system composed of a closed loop control system and an estimation error system with the aid of s procedure involving the region partition information of the original system then by the cone complementary linearization algorithm the desired observer and controller gains are solved simultaneously such that the resulting closed loop system is asymptotically stable with a prescribed mathcal h infty performance index finally a networked single link robot arm is utilized to demonstrate the effectiveness of the proposed control strategy

acceleration of the em algorithm by using quasi newton methods the em algorithm is a popular method for maximum likelihood estimation its simplicity in many applications and desirable convergence properties make it very attractive its sometimes slow convergence however has prompted researchers to propose methods to accelerate it we review these methods classifying them into three groups pure hybrid and em type accelerators we propose a new pure and a new hybrid accelerator both based on quasi newton methods and numerically compare these and two other quasi newton accelerators for this we use examples in each of three areas poisson mixtures the estimation of covariance from incomplete data and multivariate normal mixtures in these comparisons the new hybrid accelerator was fastest on most of the examples and often dramatically so in some cases it accelerated the em algorithm by factors of over 100 the new pure accelerator is very simple to implement and competed well with the other accelerators it accelerated the em algorithm in some cases by factors of over 50 to obtain standard errors we propose to approximate the inverse of the observed information matrix by using auxiliary output from the new hybrid accelerator a numerical evaluation of these approximations indicates that they may be useful at least for exploratory purposes

comment feature screening and variable selection via iterative ridge regression ridge regression was originally introduced by hoerl and kennard 1970 to deal with collinearity issue in linear regression in the presence of highly correlated covariates it solves l2 penalized l

adaptive learning vector quantization for online parametric estimation this paper addresses the problem of parameter estimation in a quantized and online setting a sensing unit collects random vector valued samples from the environment these samples are quantized and transmitted to a central processor which generates an online estimate of the unknown parameter this paper provides a closed form expression of the excess mean square error mse caused by quantization in the high rate regime i e when the number of quantization levels is supposed to be large next we determine the quantizers which mitigate the excess mse the optimal quantization rule unfortunately depends on the unknown parameter to circumvent this issue we introduce a novel adaptive learning vector quantization scheme which allows to simultaneously estimate the parameter of interest and select an efficient quantizer

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

an em algorithm for fitting two level structural equation models maximum likelihood is an important approach to analysis of two level structural equation models different algorithms for this purpose have been available in the literature in this paper we present a new formulation of two level structural equation models and develop an em algorithm for fitting this formulation this new formulation covers a variety of two level structural equation models as a result the proposed em algorithm is widely applicable in practice a practical example illustrates the performance of the em algorithm and the maximum likelihood statistic

effective metric spaces and representations of the reals based on standard notions of classical recursion theory a natural model of approximate computability for partial functions between effective metric spaces is presented it generalizes the ko friedman approach to computability of real functions by means of oracle turing machines follows the main ideas of weihrauch s type 2 theory of effectivity but it avoids the explicit use of representations the topological arithmetical hierarchy is introduced and shown to be strict if the underlying space contains an effectively discrete sequence the domains of computable functions are exactly the 2 sets of this hierarchy if the space admits a finitary stratification finally this framework is used to investigate and characterize the standard representations of the real numbers they are just those functions from the name space onto the reals which have both computable extensions and inversions that are computable as relations

estimation for general birth death processes birth death processes bdps are continuous time markov chains that track the number of particles in a system over time while widely used in population biology genetics and ecology statistical inference of the instantaneous particle birth and death rates remains largely limited to restrictive linear bdps in which per particle birth and death rates are constant researchers often observe the number of particles at discrete times necessitating data augmentation procedures such as expectation maximization em to find maximum likelihood estimates mles for bdps on finite state spaces there are powerful matrix methods for computing the conditional expectations needed for the e step of the em algorithm for bdps on infinite state spaces closed form solutions for the e step are available for some linear models but most previous work has resorted to time consuming simulation remarkably we show that the e step conditional expectations can be expressed as convolutions of computable transition probabi

on line expectation maximization algorithm for latent data models in this contribution we propose a generic online also sometimes called adaptive or recursive version of the expectation maximisation em algorithm applicable to latent variable models of independent observations compared to the algorithm of titterington 1984 this approach is more directly connected to the usual em algorithm and does not rely on integration with respect to the complete data distribution the resulting algorithm is usually simpler and is shown to achieve convergence to the stationary points of the kullback leibler divergence between the marginal distribution of the observation and the model distribution at the optimal rate i e that of the maximum likelihood estimator in addition the proposed approach is also suitable for conditional or regression models as illustrated in the case of the mixture of linear regressions model

a gradient algorithm locally equivalent to the em algorithm in many problems of maximum likelihood estimation it is impossible to carry out either the e step or the m step of the em algorithm the present paper introduces a gradient algorithm that is closely related to the em algorithm this em gradient algorithm approximately solves the m step of the em algorithm by one iteration of newton s method since newton s method converges quickly the local properties of the em gradient algorithm are almost identical with those of the em algorithm any strict local maximum point of the observed likelihood locally attracts the em and em gradient algorithm at the same rate of convergence and near the maximum point the em gradient algorithm always produces an increase in the likelihood with proper modification the em gradient algorithm also exhibits global convergence properties that are similar to those of the em algorithm our proof of global convergence applies and improves existing theory for the em algorithm these theoretical points are reinforced by a discussion of three realistic examples illustrating how the em gradient algorithm can succeed where the em algorithm is intractable

majorization minimization algorithms in signal processing communications and machine learning this paper gives an overview of the majorization minimization mm algorithmic framework which can provide guidance in deriving problem driven algorithms with low computational cost a general introduction of mm is presented including a description of the basic principle and its convergence results the extensions acceleration schemes and connection to other algorithmic frameworks are also covered to bridge the gap between theory and practice upperbounds for a large number of basic functions derived based on the taylor expansion convexity and special inequalities are provided as ingredients for constructing surrogate functions with the pre requisites established the way of applying mm to solving specific problems is elaborated by a wide range of applications in signal processing communications and machine learning

mm algorithms for generalized bradley terry models the bradley terry model for paired comparisons is a simple and muchstudied means to describe the probabilities of the possible outcomes when individuals are judged against one another in pairs among the many studies of the model in the past 75 years numerous authors have generalized it in several directions sometimes providing iterative algorithms for obtaining maximum likelihood estimates for the generalizations building on a theory of algorithms known by the initials mm for minorization maximization this paper presents a powerful technique for producing iterative maximum likelihood estimation algorithms for a wide class of generalizations of the bradley terry model while algorithms for problems of this type have tended to be custom built in the literature the techniques in this paper enable their mass production simple conditions are stated that guarantee that each algorithm described will produce a sequence that converges to the unique maximum likelihood estimator several of the algorithms and convergence results herein are new

pcf extended with real numbers we extend the programming language pcf with a type for total and partial real numbers by a partial real number we mean an element of a cpo of intervals whose subspace of maximal elements single point intervals is homeomorphic to the euclidean real line we show that partial real numbers can be considered as continuous words concatenation of continuous words corresponds to refinement of partial information the usual basic operations cons head and tail used to explicitly or recursively define functions on words generalize to partial real numbers we use this fact to give an operational semantics to the above referred extension of pcf we prove that the operational semantics is sound and complete with respect to the denotational semantics a program of real number type evaluates to a head normal form iff its value is different from b if its value is different from b then it successively evaluates to head normal forms giving better and better partial results converging to its value author s abstract

the restricted em algorithm for maximum likelihood estimation under linear restrictions on the parameters abstract the em algorithm is one of the most powerful algorithms for obtaining maximum likelihood estimates for many incomplete data problems but when the parameters must satisfy a set of linear restrictions the em algorithm may be too complicated to apply directly in this article we propose maximum likelihood estimation procedures under a set of linear restrictions for situations in which the em algorithm could be used if there were no such restrictions on the parameters we develop a modification to the em algorithm which we call the restricted em algorithm incorporating the linear restrictions on the parameters this algorithm is easily updated by using the code for the complete data information matrix and the code for the usual em algorithm major applications of the restricted em algorithm are to construct likelihood ratio tests and profile likelihood confidence intervals we illustrate the procedure with two models a variance component model and a bivariate normal model

computability on computable metric spaces in previous papers e g weihrauch 1987 type 2 theory of effectivity tte has been shown to be a very general and powerful computer oriented theory for studying effectivity in mathematics in this contribution computability on certain computable separable metric spaces is studied in detail by applying the framework of tte computationally admissible representations of metric spaces are introduced after showing that four different effective representations are computationally equivalent for extending computability to the set of continuous functions several effective namings which are related to definitions of continuity are introduced and compared the definitions are used to prove effective g extension theorems for continuous functions and an effective g characterization of the domains of strongly continous functions which generalize the known properties of real functions kreitz 1984

asymptotic convergence rate of the em algorithm for gaussian mixtures it is well known that the convergence rate of the expectation maximization em algorithm can be faster than those of convention first order iterative algorithms when the overlap in the given mixture is small but this argument has not been mathematically proved yet this article studies this problem asymptotically in the setting of gaussian mixtures under the theoretical framework of xu and jordan 1996 it has been proved that the asymptotic convergence rate of the em algorithm for gaussian mixtures locally around the true solution is o e0 5 e where e 0 is an arbitrarily small number o x means that it is a higher order infinitesimal as x 0 and e is a measure of the average overlap of gaussians in the mixture in other words the large sample local convergence rate for the em algorithm tends to be asymptotically superlinear when e tends to zero

classes of matrices for the traveling salesman problem abstract some of the foundational structure for the traveling salesman problem is addressed we investigate square matrices a aij where the aij are from the computable real field and the aii may be assigned values as needed we may transform a into a nonnegative matrix and apply the hungarian algorithm to obtain an equivalent matrix which we call a hungarian form of a we denote by h a the class of all hungarian forms associated with a we determine some of the combinatorial structure of the classes h a which yields new insights about traveling salesman problems for each matrix in the class h a we show how to obtain its triangular block form the transformation from the original matrix a to its triangular block form can be accomplished in polynomial time the original matrix a and its triangular block form have the same set of optimal tours up to a renumbering of the cities

unsupervised frequency tracking beyond the nyquist frequency using markov chains this paper deals with the estimation of a sequence of frequencies from a corresponding sequence of signals this problem arises in fields such as doppler imaging where its specificity is twofold first only short noisy data records are available typically four sample long and experimental constraints may cause spectral aliasing so that measurements provide unreliable ambiguous information second the frequency sequence is smooth here this information is accounted for by a markov model and application of the bayes rule yields the a posteriori density the maximum a posteriori is computed by a combination of viterbi and descent procedures one of the major features of the method is that it is entirely unsupervised adjusting the hyperparameters that balance data based and prior based information is done automatically by maximum likelihood ml using an expectation maximization em based gradient algorithm we compared the proposed estimate to a reference one and found that it performed better variance was greatly reduced and tracking was correct even beyond the nyquist frequency

high dimensional exploratory item factor analysis by a metropolis hastings robbins monro algorithm a metropolis hastings robbins monro mh rm algorithm for high dimensional maximum marginal likelihood exploratory item factor analysis is proposed the sequence of estimates from the mh rm algorithm converges with probability one to the maximum likelihood solution details on the computer implementation of this algorithm are provided the accuracy of the proposed algorithm is demonstrated with simulations as an illustration the proposed algorithm is applied to explore the factor structure underlying a new quality of life scale for children it is shown that when the dimensionality is high mh rm has advantages over existing methods such as numerical quadrature based em algorithm extensions of the algorithm to other modeling frameworks are discussed

the eigenvalues of an effectively determined self adjoint operator are computable but the sequence of eigenvalues is not on etudie la structure de calculabilite des valeurs propres et du spectre d operateurs lineaires bornes et non bornes sur un espace de hilbert

experimental computation of real numbers by newtonian machines following a methodology we have proposed for analysing the nature of experimental computation we prove that there is a 3 dimensional newtonian machine which given any point x 2 0 1 can generate an infinite sequence pn qn for n 1 2 of rational number interval approximations that converges to x as n 1 the machine is a system for scattering and collecting particles the theorem implies that every point x 2 0 1 is computable by a simple newtonian kinematic system that is bounded in space and mass and for which the calculation of the n th approximation of x takes place in o n time with o n energy we describe a set of variants of the scatter machine which explain why our machine is non deterministic

building an identifiable latent class model with covariate effects on underlying and measured variables in recent years latent class models have proven useful for analyzing relationships between measured multiple indicators and covariates of interest such models summarize shared features of the multiple indicators as an underlying categorical variable and the indicators substantive associations with predictors are built directly and indirectly in unique model parameters in this paper we provide a detailed study on the theory and application of building models that allow mediated relationships between primary predictors and latent class membership but that also allow direct effects of secondary covariates on the indicators themselves theory for model identification is developed we detail an expectation maximization algorithm for parameter estimation standard error calculation and convergent properties comparison of the proposed model with models underlying existing latent class modeling software is provided a detailed analysis of how visual impairments affect older persons functioning requiring distance vision is used for illustration

computability and noncomputability in classical analysis this paper treats in a systematic way the following question which basic constructions in real and complex analysis lead from the computable to the noncomputable and which do not the topics treated include computability for c c real analytic functions fourier series and fourier transforms a final section presents a more general approach via translation invariant operators particular attention is paid to those processes which occur in physical applications the approach to computability is via the standard notion of recursive function from mathematical logic this paper represents an excursion through some of the main topics in classical analysis with a view to determining their computability we shall move in order through three basic domains c functions analytic functions and fourier series and transforms in a final section a general approach to several of our results is given via translation invariant operators particular attention will be paid to those operations which are useful in physical applications our viewpoint will be that of recursion theory thus as is standard in recursion theory we make free use of nonconstructive methods to delineate the set of computable functions within the larger set of all functions in this way our work differs from that of the intuitionists such as brouwer and heyting the leningrad school of sanin zaslavskii and others and the work of bishop all of whom are concerned with constructive methods of proof cf 1 2 3 6 16 20 the purpose of this paper is to show which of the basic operations of analysis lead from the computable to the noncomputable and which do not sometimes this is easy to guess however there are surprises particularly curious in our view are certain theorems which give computability in cases where one s first classical intuition would be to look for the obvious counterexample for instance we would expect that differentiation is not a computable process because d dx is not a bounded operator this is true but the mere existence of a continuous second derivative of a computable function forces the computability of the first by extension of course if a function is c but only itself is assumed to be computable then all of its derivatives are computable one then asks whether the set of derivatives is computable as a sequence uniformly in n here our intuition is correct this is too much to ask