discriminant analysis by gaussian mixtures fisher rao linear discriminant analysis lda is a valuable tool for multigroup classification lda is equivalent to maximum likelihood classification assuming gaussian distributions for each class in this paper we fit gaussian mixtures to each class to facilitate effective classification in non normal settings especially when the classes are clustered low dimensional views are an important by product of lda our new techniques inherit this feature we can control the within class spread of the subclass centres relative to the between class spread our technique for fitting these models permits a natural blend with nonparametric versions of lda

bayesian approaches to gaussian mixture modeling a bayesian based methodology is presented which automatically penalizes overcomplex models being fitted to unknown data we show that with a gaussian mixture model the approach is able to select an optimal number of components in the model and so partition data sets the performance of the bayesian method is compared to other methods of optimal model selection and found to give good results the methods are tested on synthetic and real data sets

model based gaussian and non gaussian clustering abstract the classification maximum likelihood approach is sufficiently general to encompass many current clustering algorithms including those based on the sum of squares criterion and on the criterion of friedman and rubin 1967 however as currently implemented it does not allow the specification of which features orientation size and shape are to be common to all clusters and which may differ between clusters also it is restricted to gaussian distributions and it does not allow for noise we propose ways of overcoming these limitations a reparameterization of the covariance matrix allows us to specify that some features but not all be the same for all clusters a practical framework for non gaussian clustering is outlined and a means of incorporating noise in the form of a poisson process is described an approximate bayesian method for choosing the number of clusters is given the performance of the proposed methods is studied by simulation with encouraging results the methods are applied to the analysis of a data set arising in the study of diabetes and the results seem better than those of previous analyses rh

a robust em clustering algorithm for gaussian mixture models clustering is a useful tool for finding structure in a data set the mixture likelihood approach to clustering is a popular clustering method in which the em algorithm is the most used method however the em algorithm for gaussian mixture models is quite sensitive to initial values and the number of its components needs to be given a priori to resolve these drawbacks of the em we develop a robust em clustering algorithm for gaussian mixture models first creating a new way to solve these initialization problems we then construct a schema to automatically obtain an optimal number of clusters therefore the proposed robust em algorithm is robust to initialization and also different cluster volumes with automatically obtaining an optimal number of clusters some experimental examples are used to compare our robust em algorithm with existing clustering methods the results demonstrate the superiority and usefulness of our proposed method

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

genetic based em algorithm for learning gaussian mixture models we propose a genetic based expectation maximization ga em algorithm for learning gaussian mixture models from multivariate data this algorithm is capable of selecting the number of components of the model using the minimum description length mdl criterion our approach benefits from the properties of genetic algorithms ga and the em algorithm by combination of both into a single procedure the population based stochastic search of the ga explores the search space more thoroughly than the em method therefore our algorithm enables escaping from local optimal solutions since the algorithm becomes less sensitive to its initialization the ga em algorithm is elitist which maintains the monotonic convergence property of the em algorithm the experiments on simulated and real data show that the ga em outperforms the em method since 1 we have obtained a better mdl score while using exactly the same termination condition for both algorithms 2 our approach identifies the number of components which were used to generate the underlying data more often than the em algorithm

global optimization of wavelet domain hidden markov tree for image segmentation this work presents a global energy minimization method for multiscale image segmentation using convex optimization theory the construction of energy function is motivated by the intuition that the larger the entropy the less a priori information one has on the value of the random variables first we represent the wavelet domain hidden markov tree whmt model of the original image as a structured energy function which is proved convex in marginal distributions next we derive the maximum lower bound of the energy function through lagrange dual transform for the purpose of incorporating marginal constraints into optimization finally a modified belief propagation optimization algorithm is used to perform global energy minimization of the dual convex energy function experiments on real image segmentation problems demonstrate the superior performance of this new algorithm when compared with nonconvex ones

competitive em algorithm for finite mixture models abstract in this paper we present a novel competitive em cem algorithm for finite mixture models to overcome the two main drawbacks of the em algorithm often getting trapped at local maxima and sometimes converging to the boundary of the parameter space the proposed algorithm is capable of automatically choosing the clustering number and selecting the split or merge operations efficiently based on the new competitive mechanism we propose it is insensitive to the initial configuration of the mixture component number and model parameters experiments on synthetic data show that our algorithm has very promising performance for the parameter estimation of mixture models the algorithm is also applied to the structure analysis of complicated chinese characters the results show that the proposed algorithm performs much better than previous methods with slightly heavier computation burden

unsupervised learning of finite mixture models this paper proposes an unsupervised algorithm for learning a finite mixture model from multivariate data the adjective unsupervised is justified by two properties of the algorithm 1 it is capable of selecting the number of components and 2 unlike the standard expectation maximization em algorithm it does not require careful initialization the proposed method also avoids another drawback of em for mixture fitting the possibility of convergence toward a singular estimate at the boundary of the parameter space the novelty of our approach is that we do not use a model selection criterion to choose one among a set of preestimated candidate models instead we seamlessly integrate estimation and model selection in a single algorithm our technique can be applied to any type of parametric mixture model for which it is possible to write an em algorithm in this paper we illustrate it with experiments involving gaussian mixtures these experiments testify for the good performance of our approach

stepwise possibilistic c regressions in 1993 hathaway and bezdek combined switching regressions with fuzzy c means fcm to create fuzzy c regressions fcr the fcr algorithm had been widely studied and applied in various areas however membership of the fcr does not always correspond to the degree of belonging and it can be inaccurate in a noisy environment krishnapuram and keller 1993 proposed a possibilistic c means pcm whereby membership gives a much better explanation of the degree of belonging and it is more robust to noise and outliers than the fcm therefore we incorporate possibilistic clustering into switching regression models and term it possibilistic c regressions pcr although a pcr ameliorates the problem of outliers and noisy points a pcr still depends heavily on initial values in this paper we propose a schema for a nested stepwise procedure for a pcr called a stepwise pcr spcr method which repeats the pcr on a series of nested subsets using the clustering results of the previous subset as good initial values for the pcr for the succeeding subset when the smallest subset d1 is determined at the location where the c regression models separate sufficiently the number of clusters and the cluster centers in d1 are determined using a modified mountain method so data in d1 is properly partitioned which provides good initial values for the following pcr the proposed spcr is unsupervised does not require initialization and is unaffected by noise and outliers several experiments and real examples demonstrate the superiority and effectiveness of the proposed spcr method

blobworld image segmentation using expectation maximization and its application to image querying retrieving images from large and varied collections using image content as a key is a challenging and important problem we present a new image representation that provides a transformation from the raw pixel data to a small set of image regions that are coherent in color and texture this blobworld representation is created by clustering pixels in a joint color texture position feature space the segmentation algorithm is fully automatic and has been run on a collection of 10 000 natural images we describe a system that uses the blobworld representation to retrieve images from this collection an important aspect of the system is that the user is allowed to view the internal representation of the submitted image and the query results similar systems do not offer the user this view into the workings of the system consequently query results from these systems can be inexplicable despite the availability of knobs for adjusting the similarity metrics by finding image regions that roughly correspond to objects we allow querying at the level of objects rather than global image properties we present results indicating that querying for images using blobworld produces higher precision than does querying using color and texture histograms of the entire image in cases where the image contains distinctive objects

a dynamical trajectory based methodology for systematically computing multiple optimal solutions of general nonlinear programming problems in this paper a novel dynamical trajectory based methodology is developed for systematically computing multiple local optimal solutions of general nonlinear programming problems with disconnected feasible components satisfying nonlinear equality inequality constraints the proposed methodology deterministic in nature exploits trajectories of two different nonlinear dynamical systems to find multiple local optimal solutions the methodology consists of two phases phase i starts from an arbitrary infeasible initial point and finds systematically multiple or all the disconnected feasible components phase ii finds an adjacent local optimal solution from a local optimum via a decomposition point thereby systematically locating multiple local optimal solutions which lie within each feasible component found in phase i by alternating between these two phases the methodology locates multiple or all the local optimal solutions which lie in all the disconnected feasible components a theoretical foundation for the proposed methodology is also developed the methodology is illustrated with a numerical example with promising results

the extended symplectic pencil and the finite horizon lq problem with two sided boundary conditions this technical note introduces a new approach to the solution of a very general class of finite horizon optimal control problems for discrete time systems this approach provides a parametric expression for the optimal control sequences as well as the corresponding optimal state trajectories by exploiting a new decomposition of the so called extended symplectic pencil this decomposition provides an original strategy for a more direct solution of the problem with no need of the system theoretic hypotheses including regularity of the symplectic pencil that have always been assumed in the literature so far

on convergence properties of the em algorithm for gaussian mixtures we build up the mathematical connection between the expectation maximization em algorithm and gradient based approaches for maximum likelihood learning of finite gaussian mixtures we show that the em step in parameter space is obtained from the gradient via a projection matrix p and we provide an explicit expression for the matrix we then analyze the convergence of em in terms of special properties of p and provide new results analyzing the effect that p has on the likelihood surface based on these mathematical results we present a comparative discussion of the advantages and disadvantages of em and other algorithms for the learning of gaussian mixture models

minimum entropy data partitioning using reversible jump markov chain monte carlo problems in data analysis often require the unsupervised partitioning of a data set into classes several methods exist for such partitioning but many have the weakness of being formulated via strict parametric models e g each class is modeled by a single gaussian or being computationally intensive in high dimensional data spaces we reconsider the notion of such cluster analysis in information theoretic terms and show that an efficient partitioning may be given via a minimization of partition entropy a reversible jump sampling is introduced to explore the variable dimension space of partition models

trust tech based expectation maximization for learning finite mixture models the expectation maximization em algorithm is widely used for learning finite mixture models despite its greedy nature most popular model based clustering techniques might yield poor clusters if the parameters are not initialized properly to reduce the sensitivity of initial points a novel algorithm for learning mixture models from multivariate data is introduced in this paper the proposed algorithm takes advantage of trust tech transformation under stability retaining equilibria characterization to compute neighborhood local maxima on the likelihood surface using stability regions basically our method coalesces the advantages of the traditional em with that of the dynamic and geometric characteristics of the stability regions of the corresponding nonlinear dynamical system of the log likelihood function two phases namely the em phase and the stability region phase are repeated alternatively in the parameter space to achieve local maxima with improved likelihood values the em phase obtains the local maximum of the likelihood function and the stability region phase helps to escape out of the local maximum by moving toward the neighboring stability regions though applied to gaussian mixtures in this paper our technique can be easily generalized to any other parametric finite mixture model the algorithm has been tested on both synthetic and real data sets and the improvements in the performance compared to other approaches are demonstrated the robustness with respect to initialization is also illustrated experimentally

elite ensemble of optimal input pruned neural networks using trust tech the ensemble of optimal input pruned neural networks using trust tech elite method for constructing high quality ensemble through an optimal linear combination of accurate and diverse neural networks is developed the optimization problems in the proposed methodology are solved by a global optimization a global optimization method called transformation under stability retraining equilibrium characterization trust tech whose main features include its capability in identifying multiple local optimal solutions in a deterministic systematic and tier by tier manner elite creates a diverse population via a feature selection procedure of different local optimal neural networks obtained using tier 1 trust tech search in addition the capability of each input pruned network is fully exploited through a trust tech based optimal training finally finding the optimal linear combination weights for an ensemble is modeled as a nonlinear programming problem and solved using trust tech and the interior point method where the issue of non convexity can be effectively handled extensive numerical experiments have been carried out for pattern classification on the synthetic and benchmark datasets numerical results show that elite consistently outperforms existing methods on the benchmark datasets the results show that elite can be very promising for constructing high quality neural network ensembles

reversible jump markov chain monte carlo computation and bayesian model determination markov chain monte carlo methods for bayesian computation have until recently been restricted to problems where the joint distribution of all variables has a density with respect to some fixed standard underlying measure they have therefore not been available for application to bayesian model determination where the dimensionality of the parameter vector is typically not fixed this paper proposes a new framework for the construction of reversible markov chain samplers that jump between parameter subspaces of differing dimensionality which is flexible and entirely constructive it should therefore have wide applicability in model determination problems the methodology is illustrated with applications to multiple change point analysis in one and two dimensions and to a bayesian comparison of binomial experiments

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

on bayesian analysis of mixtures with an unknown number of components with discussion new methodology for fully bayesian mixture analysis is developed making use of reversible jump markov chain monte carlo methods that are capable of jumping between the parameter subspaces corresponding to different numbers of components in the mixture a sample from the full joint distribution of all unknown variables is thereby generated and this can be used as a basis for a thorough presentation of many aspects of the posterior distribution the methodology is applied here to the analysis of univariate normal mixtures using a hierarchical prior model that offers an approach to dealing with weak prior information while avoiding the mathematical pitfalls of using improper priors in the mixture context

lower bounds for quantum communication complexity we prove lower bounds on the bounded error quantum communication complexity our methods are based on the fourier transform of the considered functions first we generalize a method for proving classical communication complexity lower bounds developed by raz comput complexity 5 1995 pp 205 221 to the quantum case applying this method we give an exponential separation between bounded error quantum communication complexity and nondeterministic quantum communication complexity we develop several other lower bound methods based on the fourier transform notably showing that sqrt bar s f log n for the average sensitivity bar s f of a function f yields a lower bound on the bounded error quantum communication complexity of f x wedge y oplus z where x is a boolean word held by alice and y z are boolean words held by bob we then prove the first large lower bounds on the bounded error quantum communication complexity of functions for which a polynomial quantum speedup is possible for all the functions we investigate the only previously applied general lower bound method based on discrepancy yields bounds that are o log n

efficient greedy learning of gaussian mixture models this article concerns the greedy learning of gaussian mixtures in the greedy approach mixture components are inserted into the mixture one after the other we propose a heuristic for searching for the optimal component to insert in a randomized manner a set of candidate new components is generated for each of these candidates we find the locally optimal new component and insert it into the existing mixture the resulting algorithm resolves the sensitivity to initialization of state of the art methods like expectation maximization and has running time linear in the number of data points and quadratic in the final number of mixture components due to its greedy nature the algorithm can be particularly useful when the optimal number of mixture components is unknown experimental results comparing the proposed algorithm to other methods on density estimation and texture segmentation are provided