a hybrid method combining continuous tabu search and nelder mead simplex algorithms for the global optimization of multiminima functions abstract tabu search ts is a metaheuristic which proved efficient to solve various combinatorial optimization problems however few works deal with its application to the global minimization of functions depending on continuous variables to perform this task we propose an hybrid method combining tabu search and simplex search ss ts allows to cover widely the solution space to stimulate the search towards solutions far from the current solution and to avoid the risk of trapping into a local minimum ss is used to accelerate the convergence towards a minimum the nelder mead simplex algorithm is a classical very powerful local descent algorithm making no use of the objective function derivatives a simplex is a geometrical figure consisting in n dimensions of n 1 points if any point of a simplex is taken as the origin the n other points define vector directions that span the n dimension vector space through a sequence of elementary geometric transformations reflection contraction and extension the initial simplex moves expands or contracts to select the appropriate transformation the method only uses the values of the function to be optimized at the vertices of the simplex considered after each transformation the current worst vertex is replaced by a better one our algorithm called continuous tabu simplex search ctss implemented in two different forms ctsssingle ctssmultiple is made up of two steps first an adaptation of ts to continuous optimization problems allowing to localize a promising area then intensification within this promising area involving ss the efficiency of ctss is extensively tested by using analytical test functions of which global and local minima are known a comparison is proposed with several variants of tabu search genetic algorithms and simulated annealing ctss is applied to the design of a eddy current sensor aimed at non destructive control

hybrid optimization methods for geophysical inversion local and global optimization algorithms are used commonly in geophysical data inversion each type of algorithm has unique advantages and disadvantages here we propose several methods of combining the two algorithms such that we can overcome their drawbacks and make use of the salient features of the two methods in particular we combined a local conjugate gradient cg method with a global very fast simulated annealing vfsa approach to solve problems of geophysical interests we conducted a systematic study to find an efficient strategy to combine cg and vfsa optimization schemes and recommend a couple of ways for future implementations seven different hybrid algorithms were first tested on a set of field 1 d schlumberger resistivity sounding data and their performances were compared with stand alone genetic algorithm ga simulated annealing and local search algorithms almost all of the proposed hybrid algorithms were found to be computationally more efficient than the conventional global optim

convergence of a stochastic approximation version of the em algorithm the expectation maximization em algorithm is a powerful computational technique for locating maxima of functions it is widely used in statistics for maximum likelihood or maximum a posteriori estimation in incomplete data models in certain situations however this method is not applicable because the expectation step cannot be performed in closed form to deal with these problems a novel method is introduced the stochastic approximation em saem which replaces the expectation step of the em algorithm by one iteration of a stochastic approximation procedure the convergence of the saem algorithm is established under conditions that are applicable to many practical situations moreover it is proved that under mild additional conditions the attractive stationary points of the saem algorithm correspond to the local maxima of the function presented to support our findings

a monte carlo implementation of the em algorithm and the poor man s data augmentation algorithms abstract the first part of this article presents the monte carlo implementation of the e step of the em algorithm given the current guess to the maximizer of the posterior distribution latent data patterns are generated from the conditional predictive distribution the expected value of the augmented log posterior is then updated as a mixture of augmented log posteriors mixed over the generated latent data patterns multiple imputations in the m step of the algorithm this mixture is maximized to obtain the update to the maximizer of the observed posterior the gradient and hessian of the observed log posterior are also expressed as mixtures mixed over the multiple imputations the relation between the monte carlo em mcem algorithm and the data augmentation algorithm is noted two modifications to the mcem algorithm the poor man s data augmentation algorithms which allow for the calculation of the entire posterior are then presented these approximations serve as diagnostics for the validity o

genetic and nelder mead algorithms hybridized for a more accurate global optimization of continuous multiminima functions abstract a hybrid method combining two algorithms is proposed for the global optimization of multiminima functions to localize a promising area likely to contain a global minimum it is necessary to well explore the whole search domain when a promising area is detected the appropriate tools must be used to exploit this area and obtain the optimum as accurately and quickly as possible both tasks are hardly performed through only one method we propose an algorithm using two processes each one devoted to one task global metaheuristics such as simulated annealing tabu search and genetic algorithms gas are efficient to localize the best areas on the other hand local search methods are classically available in particular the hill climbing e g the quasi newton method and the nelder mead simplex search ss therefore we worked out an hybrid method called continuous hybrid algorithm cha performing the exploration with a ga and the exploitation with a nelder mead ss to evaluate the efficiency of cha we implemented a set of benchmark functions and compared our results to the ones supplied by other competitive methods

inversion of acoustic data using a combination of genetic algorithms and the gauss newton approach an inversion procedure for obtaining speeds attenuation densities and thicknesses for a layered medium is described the inversion is carried out using the least squares technique and the forward modeling is based on safari the optimization is a hybrid method combining the global genetic algorithms and the local gauss newton method this is done by taking several gradient steps between each update of the object function for each individual in the population the gradients for the gauss newton method are computed analytically this makes the computation faster and more stable than computing the gradients by numerical differentiation the combination of a global and a local method makes the hybrid method faster and it gets closer to the global minimum than a pure global method examples based on both real and synthetic data in wave number frequency and range frequency domains show that the method works well

a class of spectral two level preconditioners it is well known that the convergence of krylov methods for solving the linear system often depends to a large extent on the eigenvalue distribution in many cases it is observed that removing the smallest eigenvalues can greatly improve the convergence several techniques have been proposed in the past few years that attempt to tackle this problem the proposed approaches can be split into two main families depending on whether the scheme enlarges the generated krylov space or adaptively updates the preconditioner in this paper we follow the second approach and propose a class of preconditioners both for unsymmetric and for symmetric linear systems that can also be adapted for symmetric positive definite problems we effectively solve the preconditioned system exactly in the low dimensional space associated with the smallest eigenvalues and use this to update the preconditioned residual this update results in shifting eigenvalues from close to the origin to near to one for the new preconditioner this is ideal when there are only a few eigenvalues near the origin while all the others are close to one because the updated preconditioned system becomes close to the identity we illustrate the performance of our method through extensive numerical experiments on a set of general linear systems finally we show the advantages of the preconditioners for solving dense linear systems arising in electromagnetism applications which were the main motivation for this work

asymptotic properties of the maximum likelihood estimator in autoregressive models with markov regime an autoregressive process with markov regime is an autoregressive process for which the regression function at each time point is given by a nonobservable markov chain in this paper we consider the asymptotic properties of the maximum likelihood estimator in a possibly nonstationary process of this kind for which the hidden state space is compact but not necessarily finite consistency and asymptotic normality are shown to follow from uniform exponential forgetting of the initial distribution for the hidden markov chain conditional on the observations

simulated annealing searching for an optimal temperature schedule a sizable part of the theoretical literature on simulated annealing deals with a property called convergence which asserts that the simulated annealing chain is in the set of global minimum states of the objective function with probability tending to 1 however in practice the convergent algorithms are considered too slow whereas a number of nonconvergent ones are usually preferred we attempt a detailed analysis of various temperature schedules examples will be given of when it is both practically and theoretically justified to use boiling fixed temperature or even fast cooling schedules which have a small probability of reaching global minima applications to traveling salesman problems of various sizes are also given

analysis of computer experiments with functional response this article is motivated by a computer experiment conducted for optimizing residual stresses in the machining of metals although kriging is widely used in the analysis of computer experiments it cannot be easily applied to model the residual stresses because they are obtained as a profile the high dimensionality caused by this functional response introduces severe computational challenges in kriging it is well known that if the functional data are observed on a regular grid the computations can be simplified using an application of kronecker products however the case of irregular grid is quite complex in this article we develop a gibbs sampling based expectation maximization algorithm which converts the irregularly spaced data into a regular grid so that the kronecker product based approach can be employed for efficiently fitting a kriging model to the functional data supplementary materials are available online

predictive coarse graining abstract we propose a data driven coarse graining formulation in the context of equilibrium statistical mechanics in contrast to existing techniques which are based on a fine to coarse map we adopt the opposite strategy by prescribing a probabilistic coarse to fine map this corresponds to a directed probabilistic model where the coarse variables play the role of latent generators of the fine scale all atom data from an information theoretic perspective the framework proposed provides an improvement upon the relative entropy method 1 and is capable of quantifying the uncertainty due to the information loss that unavoidably takes place during the coarse graining process furthermore it can be readily extended to a fully bayesian model where various sources of uncertainties are reflected in the posterior of the model parameters the latter can be used to produce not only point estimates of fine scale reconstructions or macroscopic observables but more importantly predictive posterior distributions on these quantities predictive posterior distributions reflect the confidence of the model as a function of the amount of data and the level of coarse graining the issues of model complexity and model selection are seamlessly addressed by employing a hierarchical prior that favors the discovery of sparse solutions revealing the most prominent features in the coarse grained model a flexible and parallelizable monte carlo expectation maximization mc em scheme is proposed for carrying out inference and learning tasks a comparative assessment of the proposed methodology is presented for a lattice spin system and the spc e water model

fitting full information item factor models and an empirical investigation of bridge sampling abstract based on item response theory bock and aitken introduced a method of item factor analysis termed full information item factor fiif analysis by bartholomew because it uses all distinct item response vectors as data but a limitation of their fitting algorithm is its reliance on fixed point gauss hermite quadrature which can produce appreciable numerical errors especially in high dimension problems the first purpose of this article is to offer more reliable methods by using recent advances in statistical computation specifically we illustrate two ways of implementing monte carlo expectation maximization em algorithm to fit a fiif model using the gibbs sampler to carry out the computation for the e steps we also show how to use bridge sampling to simulate the likelihood ratios for monitoring the convergence of a monte carlo em a strategy that is useful in general simulations demonstrate substantial improvement over bock and aitken s algorithm in recovering known factor loadings in hig

maximizing generalized linear mixed model likelihoods with an automated monte carlo em algorithm summary two new implementations of the em algorithm are proposed for maximum likelihood fitting of generalized linear mixed models both methods use random independent and identically distributed sampling to construct monte carlo approximations at the e step one approach involves generating random samples from the exact conditional distribution of the random effects given the data by rejection sampling using the marginal distribution as a candidate the second method uses a multivariate t importance sampling approximation in many applications the two methods are complementary rejection sampling is more efficient when sample sizes are small whereas importance sampling is better with larger sample sizes monte carlo approximation using random samples allows the monte carlo error at each iteration to be assessed by using standard central limit theory combined with taylor series methods specifically we construct a sandwich variance estimate for the maximizer at each approximate e step this suggests a rule for automatically increasing the monte carlo sample size after iterations in which the true em step is swamped by monte carlo error in contrast techniques for assessing monte carlo error have not been developed for use with alternative implementations of monte carlo em algorithms utilizing markov chain monte carlo e step approximations three different data sets including the infamous salamander data of mccullagh and nelder are used to illustrate the techniques and to compare them with the alternatives the results show that the methods proposed can be considerably more efficient than those based on markov chain monte carlo algorithms however the methods proposed may break down when the intractable integrals in the likelihood function are of high dimension

convergence analysis of stochastic algorithms this paper investigates asymptotic convergence to stationary points of gradient descent algorithms where the functions involved are not available in closed form but are approximated by sequences of random functions the algorithms take large stepsizes and use progressively finer precision at successive iterations two kinds of optimization algorithms are considered one where the limiting function to be minimized is differentiable but the random approximating functions can be nondifferentiable and the other where both the limiting and approximating functions are nondifferentiable and convex such functions often arise in discrete event dynamic system models in various application areas the analysis brings together ideas and techniques from the disciplines of nonlinear programming and nondifferentiable optimization on one hand and stochastic processes and especially uniform laws of large numbers on the other hand a general algorithmic frame work is first developed and analyzed and then applied to the specific algorithms considered the analysis extends the results derived to date for similar algorithms and has a potential for further extensions in proving convergence of other algorithms as well

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

modeling the random effects covariance matrix for longitudinal data with covariates measurement error longitudinal data occur frequently in practice such as medical studies and life sciences generalized linear mixed models glmms are commonly used to analyze such data it is typically assumed that the random effects covariance matrix is constant among subjects in these models in many situations however the correlation structure may differ among subjects and ignoring this heterogeneity can lead to biases in model parameters estimate recently lee et al developed a heterogeneous random effects covariance matrix for glmms for error free covariates covariates measured with error also happen frequently in the longitudinal data set up eg blood pressure and cholesterol level ignoring this issue in the data may produce bias in model parameters estimate and lead to wrong conclusions in this paper we propose an approach to properly model the random effects covariance matrix based on covariates in the class of glmms where we also have covariates measured with error the resulting parameters from the decomposition of random effects covariance matrix have a sensible interpretation and can be easily modeled without the concern of positive definiteness of the resulting estimator the performance of the proposed approach is evaluated through simulation studies which show that the proposed method performs very well in terms of bias mean squared error and coverage rate an application of the proposed method is also provided using a longitudinal data from manitoba follow up study

on perturbed proximal gradient algorithms we study a version of the proximal gradient algorithm for which the gradient is intractable and is approximated by monte carlo methods and in particular markov chain monte carlo we derive conditions on the step size and the monte carlo batch size under which convergence is guaranteed both increasing batch size and constant batch size are considered we also derive non asymptotic bounds for an averaged version our results cover both the cases of biased and unbiased monte carlo approximation to support our findings we discuss the inference of a sparse generalized linear model with random effect and the problem of learning the edge structure and parameters of sparse undirected graphical models

maximum likelihood estimation for probit linear mixed models with correlated random effects the probit normal model for binary data mcculloch 1994 journal of the american statistical association 89 330 335 is extended to allow correlated random effects to obtain maximum likelihood estimates we use the em algorithm with its m step greatly simplified under the assumption of a probit link and its e step made feasible by gibbs sampling standard errors are calculated by inverting a monte carlo approximation of the information matrix rather than via the sem algorithm a method is also suggested that accounts for the monte carlo variation explicitly as an illustration we present a new analysis of the famous salamander mating data unlike previous analyses we find it necessary to introduce different variance components for different species of animals finally we consider models with correlated errors as well as correlated random effects

network inference from co occurrences the discovery of networks is a fundamental problem arising in numerous fields of science and technology including communication systems biology sociology and neuroscience unfortunately it is often difficult or impossible to obtain data that directly reveal network structure and so one must infer a network from incomplete data this paper considers inferring network structure from co occurrence data observations that identify which network components e g switches routers genes carry each transmission but do not indicate the order in which they handle the transmission without order information the number of networks that are consistent with the data grows exponentially with the size of the network i e the number of nodes yet the basic engineering evolutionary principles underlying most networks strongly suggest that not all data consistent networks are equally likely in particular nodes that co occur in many observations are probably closely connected with this in mind we model the co occurrence observations as independent realizations of a random walk on the network subjected to a random permutation to account for the lack of order information treating permutations as missing data we derive an expectation maximization em algorithm for estimating the random walk parameters the model and em algorithm significantly simplify the problem but the computational complexity of the reconstruction process does grow exponentially in the length of each transmission path for networks with long paths the exact e step may be computationally intractable we propose a polynomial time monte carlo em algorithm based on importance sampling and derive conditions that ensure convergence of the algorithm with high probability simulations and experiments with internet measurements demonstrate the promise of this approach

monte carlo em estimation for time series models involving counts abstract the observations in parameter driven models for time series of counts are generated from latent unobservable processes that characterize the correlation structure these models result in very complex likelihoods and even the em algorithm which is usually well suited for problems of this type involves high dimensional integration in this article we discuss a monte carlo em mcem algorithm that uses a markov chain sampling technique in the calculation of the expectation in the e step of the em algorithm we propose a stopping criterion for the algorithm and provide rules for selecting the appropriate monte carlo sample size we show that under suitable regularity conditions an mcem algorithm will with high probability get close to a maximizer of the likelihood of the observed data we also discuss the asymptotic efficiency of the procedure we illustrate our monte carlo estimation method on a time series involving small counts the polio incidence time series previously analyzed by zeger

on a class of limited memory preconditioners for large scale linear systems with multiple right hand sides this work studies a class of limited memory preconditioners lmps for solving linear positive definite systems of equations with multiple right hand sides we propose a class of lmps whose construction requires a small number of linearly independent vectors after exploring the theoretical properties of the preconditioners we focus on three particular members spectral lmp quasi newton lmp and ritz lmp we show that the first two are well known while the third is new numerical tests indicate that the ritz lmp is efficient on a real life nonlinear optimization problem arising in a data assimilation system for oceanography

a novel mutation operator based on the immunity operation this paper presents a novel mutation operator based on the immunity operation also the selection is integrated into both crossover and mutation operation experiments with our approach on many test functions resulted in near optimal solutions in all cases in comparison with several other algorithms our approach achieved improved accuracy

convergence of the monte carlo expectation maximization for curved exponential families the monte carlo expectation maximization mcem algorithm is a versatile tool for inference in incomplete data models especially when used in combination with markov chain monte carlo simulation methods in this contribution the almost sure convergence of the mcem algorithm is established it is shown using uniform versions of ergodic theorems for markov chains that mcem converges under weak conditions on the simulation kernel practical illustrations are presented using a hybrid random walk metropolis hastings sampler and an independence sampler the rate of convergence is studied showing the impact of the simulation schedule on the fluctuation of the parameter estimate at the convergence a novel averaging procedure is then proposed to reduce the simulation variance and increase the rate of convergence

combining monte carlo and mean field like methods for inference in hidden markov random fields issues involving missing data are typical settings where exact inference is not tractable as soon as nontrivial interactions occur between the missing variables approximations are required and most of them are based either on simulation methods or on deterministic variational methods while variational methods provide fast and reasonable approximate estimates in many scenarios simulation methods offer more consideration of important theoretical issues such as accuracy of the approximation and convergence of the algorithms but at a much higher computational cost in this work we propose a new class of algorithms that combine the main features and advantages of both simulation and deterministic methods and consider applications to inference in hidden markov random fields hmrfs these algorithms can be viewed as stochastic perturbations of variational expectation maximization vem algorithms which are not tractable for hmrf we focus more specifically on one of these perturbations and we prove their almost sure convergence to the same limit set as the limit set of vem in addition experiments on synthetic and real world images show that the algorithm performance is very close and sometimes better than that of other existing simulation based and variational em like algorithms

a regression model for time series of counts summary this paper discusses a model for regression analysis with a time series of counts correlation is assumed to arise from an unobservable process added to the linear predictor in a log linear model an estimating equation approach used for parameter estimation leads to an iterative weighted and filtered least squares algorithm asymptotic properties for the regression coefficients are presented we illustrate the technique with an analysis of trends in u s polio incidence since 1970

an intelligent genetic algorithm designed for global optimization of multi minima functions many practical problems often lead to large non convex non linear programming problems that have multi minima the global optimization algorithms of these problems have received much attention over the last few years generally stochastic algorithms are suitable for these problems but not efficient when there are too many minima genetic algorithms are stochastic search approaches based on randomized operators such as selection crossover and mutation inspired by the natural reproduction and evolution of the living creatures however the existing genetic algorithms cannot solve global optimization of multi minima functions effectively a new algorithm called intelligent genetic algorithm iga is proposed for the global optimization of multi minima functions iga integrates many cross operator mutation operator and reattempt operation it can select the appropriate cross operator mutation operator or reattempt operation according to the current optimization result it converges to the global optimization solution without the influence of random searching process at first this paper introduces the foundation for designing intelligent genetic algorithm secondly this paper reports on the development of an intelligent genetic algorithm approach for global optimization problems thirdly the proposed method is illustrated by means of some numerical examples finally the conclusions of this study are drawn with possible directions for subsequent studies the feasibility the efficiency and the effectiveness of iga are tested in detail through a set of benchmark multi modal functions of which global and local minima are known the experimental results suggest that results from iga are better than results from other methods in conclusion the performance of iga is better than that of other methods iga results are satisfactory for all the functions

a monte carlo approach to filtering for a class of marked doubly stochastic poisson processes marked doubly stochastic poisson processes are a particular type of marked point processes that are characterized by the number of events in any time interval as being conditionally poisson distributed given another positive stochastic process called intensity here we consider a subclass of these processes in which the intensity is assumed to be a deterministic function of another nonexplosive marked point process in particular we will investigate an intensity jump process with an exponential decay having an analytic form for the distribution of the times and sizes of the jumps which can be seen as a generalization of the classical shot noise process assuming that the intensity is unobservable interest here is in its filtering that is in the computation of its conditional distribution over a whole time interval given an observed trajectory of realized events because in general this computation cannot be performed analytically we propose a simulation method that provides an approximate solut