a natural approach to the numerical integration of riccati differential equations this paper introduces a new class of methods which we call mobius schemes for the numerical solution of matrix riccati differential equations the approach is based on viewing the riccati equation in its natural geometric setting as a flow on the grassmannian of m dimensional subspaces of an n m dimensional vector space since the grassmannians are compact differentiable manifolds and the coefficients of the equation are assumed continuous there are no singularities or intrinsic instabilities in the associated flow the presence of singularities and numerical instabilities is an artifact of the coordinate system but since mobius schemes are based on the natural geometry they are able to deal with numerical instability and pass accurately through the singularities a number of examples are given to demonstrate these properties

inferring the eigenvalues of covariance matrices from limited noisy data the eigenvalue spectrum of covariance matrices is of central importance to a number of data analysis techniques usually the sample covariance matrix is constructed from a limited number of noisy samples we describe a method of inferring the true eigenvalue spectrum from the sample spectrum results of silverstein 1986 which characterize the eigenvalue spectrum of the noise covariance matrix and inequalities between the eigenvalues of hermitian matrices are used to infer probability densities for the eigenvalues of the noise free covariance matrix using bayesian inference posterior densities for each eigenvalue are obtained which yield error estimates the evidence framework gives estimates of the noise variance and permits model order selection by estimating the rank of the covariance matrix the method is illustrated with numerical examples

gtm the generative topographic mapping latent variable models represent the probability density of data in a space of several dimensions in terms of a smaller number of latent or hidden variables a familiar example is factor analysis which is based on a linear transformations between the latent space and the data space in this paper we introduce a form of non linear latent variable model called the generative topographic mapping for which the parameters of the model can be determined using the em algorithm gtm provides a principled alternative to the widely used self organizing map som of kohonen 1982 and overcomes most of the significant limitations of the som we demonstrate the performance of the gtm algorithm on a toy problem and on simulated data from flow diagnostics for a multi phase oil pipeline

online model selection based on the variational bayes the bayesian framework provides a principled way of model selection this framework estimates a probability distribution over an ensemble of models and the prediction is done by averaging over the ensemble of models accordingly the uncertainty of the models is taken into account and complex models with more degrees of freedom are penalized however integration over model parameters is often intractable and some approximation scheme is needed recently a powerful approximation scheme called the variational bayes vb method has been proposed this approach defines the free energy for a trial probability distribution which approximates a joint posterior probability distribution over model parameters and hidden variables the exact maximization of the free energy gives the true posterior distribution the vb method uses factorized trial distributions the integration over model parameters can be done analytically and an iterative expectation maximization like algorithm whose convergence is guaranteed is derived in this article we derive an online version of the vb algorithm and prove its convergence by showing that it is a stochastic approximation for finding the maximum of the free energy by combining sequential model selection procedures the online vb method provides a fully online learning method with a model selection mechanism in preliminary experiments using synthetic data the online vb method was able to adapt the model structure to dynamic environments

data assimilation for linear parabolic equations minimax projection method in this paper we propose a state estimation method for linear parabolic partial differential equations pde that accounts for errors in the model truncation and observations it is based on an extension of the galerkin projection method the extended method models projection coefficients representing the state of the pde in some basis by means of a differential algebraic equation dae the original estimation problem for the pde is then recast as a state estimation problem for the constructed dae using a linear continuous minimax filter we construct a numerical time integrator that preserves the monotonic decay of a nonstationary lyapunov function along the solution to conclude we demonstrate the efficacy of the proposed method by applying it to the tracking of a discharged pollutant slick in a two dimensional fluid

invariant finsler metrics on homogeneous manifolds in this paper we study invariant finsler metrics on homogeneous manifolds we first give an algebraic description of these metrics and obtain a necessary and sufficient condition for a homogeneous manifold to have invariant finsler metrics as a special case we study bi invariant finsler metrics on lie groups and obtain a necessary and sufficient condition for a lie group to have bi invariant finsler metrics finally we provide some conditions for a homogeneous manifold to admit invariant non riemannian finsler metrics and present some interesting examples

least squares quantization in pcm it has long been realized that in pulse code modulation pcm with a given ensemble of signals to handle the quantum values should be spaced more closely in the voltage regions where the signal amplitude is more likely to fall it has been shown by panter and dite that in the limit as the number of quanta becomes infinite the asymptotic fractional density of quanta per unit voltage should vary as the one third power of the probability density per unit voltage of signal amplitudes in this paper the corresponding result for any finite number of quanta is derived that is necessary conditions are found that the quanta and associated quantization intervals of an optimum finite quantization scheme must satisfy the optimization criterion used is that the average quantization noise power be a minimum it is shown that the result obtained here goes over into the panter and dite result as the number of quanta become large the optimum quautization schemes for 2 b quanta b 1 2 cdots 7 are given numerically for gaussian and for laplacian distribution of signal amplitudes

exponentiated gradient versus gradient descent for linear predictors we consider two algorithm for on line prediction based on a linear model the algorithms are the well known gradient descent gd algorithm and a new algorithm which we call eg they both maintain a weight vector using simple updates for the gd algorithm the update is based on subtracting the gradient of the squared error made on a prediction the eg algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively we present worst case loss bounds for eg and compare them to previously known bounds for the gd algorithm the bounds suggest that the losses of the algorithms are in general incomparable but eg has a much smaller loss if only a few components of the input are relevant for the predictions we have performed experiments which show that our worst case upper bounds are quite tight already on simple artificial data

hierarchical mixtures of experts and the em algorithm we present a tree structured architecture for supervised learning the statistical model underlying the architecture is a hierarchical mixture model in which both the mixture coefficients and the mixture components are generalized linear models glim s learning is treated as a maximum likelihood problem in particular we present an expectation maximization em algorithm for adjusting the parameters of the architecture we also develop an on line learning algorithm in which the parameters are updated incrementally comparative simulation results are presented in the robot dynamics domain

variational properties of value functions regularization plays a key role in a variety of optimization formulations of inverse problems a recurring theme in regularization approaches is the selection of regularization parameters and their effect on the solution and on the optimal value of the optimization problem the sensitivity of the value function to the regularization parameter can be linked directly to the lagrange multipliers this paper characterizes the variational properties of the value functions for a broad class of convex formulations which are not all covered by standard lagrange multiplier theory an inverse function theorem is given that links the value functions of different regularization formulations not necessarily convex these results have implications for the selection of regularization parameters and the development of specialized algorithms numerical examples illustrate the theoretical results

resolution based complexity control for gaussian mixture models in the domain of unsupervised learning mixtures of gaussians have become a popular tool for statistical modeling for this class of generative models we present a complexity control scheme which provides an effective means for avoiding the problem of overfitting usually encountered with unconstrained mixtures of gaussians in high dimensions according to some prespecified level of resolution as implied by a fixed variance noise model the scheme provides an automatic selection of the dimensionalities of some local signal subspaces by maximum likelihood estimation together with a resolution based control scheme for adjusting the number of mixture components we arrive at an incremental model refinement procedure within a common deterministic annealing framework which enables an efficient exploration of the model space the advantages of the resolution based framework are illustrated by experimental results on synthetic and high dimensional real world data

a unifying review of linear gaussian models factor analysis principal component analysis mixtures of gaussian clusters vector quantization kalman filter models and hidden markov models can all be unified as variations of unsupervised learning under a single basic generative model this is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity through the use of other nonlinearities we show how independent component analysis is also a variation of the same basic generative model we show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term we introduce a new model for static data known as sensible principal component analysis as well as a novel concept of spatially adaptive observation noise we also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

variational learning for switching state space models we introduce a new statistical model for time series that iteratively segments data into regimes with approximately linear dynamics and learns the parameters of each of these linear regimes this model combines and generalizes two of the most widely used stochastic time series models hidden markov models and linear dynamical systems and is closely related to models that are widely used in the control and econometrics literatures it can also be derived by extending the mixture of experts neural network jacobs jordan nowlan hinton 1991 to its fully dynamical version in which both expert and gating networks are recurrent inferring the posterior probabilities of the hidden states of this model is computationally intractable and therefore the exact expectation maximization em algorithm cannot be applied however we present a variational approximation that maximizes a lower bound on the log likelihood and makes use of both the forward and backward recursions for hidden markov models and the kalman filter recursions for linear dynamical systems we tested the algorithm on artificial data sets and a natural data set of respiration force from a patient with sleep apnea the results suggest that variational approximations are a viable method for inference and learning in switching state space models

on locally and globally symmetric berwald spaces in this paper we generalize cartan s work on riemannian locally and globally symmetric spaces to locally and globally symmetric berwald spaces we prove that a berwald space is locally symmetric if and only if the flag curvature is invariant under parallel displacements and a locally symmetric berwald space is locally isometric to a globally symmetric berwald space

linear state space models for blind source separation we apply a type of generative modelling to the problem of blind source separation in which prior knowledge about the latent source signals such as time varying auto correlation and quasi periodicity are incorporated into a linear state space model in simulations we show that in terms of signal to error ratio the sources are inferred more accurately as a result of the inclusion of strong prior knowledge we explore different schemes of maximum likelihood optimization for the purpose of learning the model parameters the expectation maximization algorithm which is often considered the standard optimization method in this context results in slow convergence when the noise variance is small in such scenarios quasi newton optimization yields substantial improvements in a range of signal to noise ratios we analyze the performance of the methods on convolutive mixtures of speech signals

efficient blind system identification of non gaussian autoregressive models with hmm modeling of the excitation we have previously proposed a blind system identification method that exploits the underlying dynamics of non gaussian signals in li and andersen blind identification of non gaussian autoregressive models for efficient analysis of speech signals proceedings of the international conference on acoustics speech and signal processing icassp may 2006 vol 1 pp i 1205 i 1208 the signal model being identified is an autoregressive ar model driven by a discrete state hidden markov process an exact expectation maximization em algorithm was derived for the joint estimation of the ar parameters and the hidden markov model hmm parameters in this paper we extend the system model by introducing an additive measurement noise the identification of the extended system model becomes much more complicated since the system output is now hidden we propose an exact em algorithm that incorporates a novel switching kalman smoother which obtains nonlinear minimum mean square error mmse estimates of the system output based on the state information given by the hmm filter the exact em algorithms for both models are obtainable only by appropriate constraints in the model design and have better convergence properties than algorithms employing generalized em algorithm or empirical iterative schemes the proposed methods also enjoy good data efficiency since only second order statistics are involved in the computation the signal models are general and suitable to numerous signals such as speech and baseband communication signals this paper describes the two system identification algorithms in an integrated form and provides supplementary results to the noise free model and new results to the extended model with applications in speech analysis and channel equalization

mixtures of probabilistic principal component analyzers principal component analysis pca is one of the most popular techniques for processing compressing and visualizing data although its effectiveness is limited by its global linearity while nonlinear variants of pca have been proposed an alternative paradigm is to capture data complexity by a combination of local linear pca projections however conventional pca does not correspond to a probability density and so there is no unique way to combine pca models therefore previous attempts to formulate mixture models for pca have been ad hoc to some extent in this article pca is formulated within a maximum likelihood framework based on a specific form of gaussian latent variable model this leads to a well defined mixture model for probabilistic principal component analyzers whose parameters can be determined using an expectationmaximization algorithm we discuss the advantages of this model in the context of clustering density modeling and local dimensionality reduction and we demonstrate its application to image compression and handwritten digit recognition

a least squares framework for component analysis over the last century component analysis ca methods such as principal component analysis pca linear discriminant analysis lda canonical correlation analysis cca locality preserving projections lpp and spectral clustering sc have been extensively used as a feature extraction step for modeling classification visualization and clustering ca techniques are appealing because many can be formulated as eigen problems offering great potential for learning linear and nonlinear representations of data in closed form however the eigen formulation often conceals important analytic and computational drawbacks of ca techniques such as solving generalized eigen problems with rank deficient matrices e g small sample size problem lacking intuitive interpretation of normalization factors and understanding commonalities and differences between ca methods this paper proposes a unified least squares framework to formulate many ca methods we show how pca lda cca lpp sc and its kernel and regularized extensions correspond to a particular instance of least squares weighted kernel reduced rank regression ls wkrrr the ls wkrrr formulation of ca methods has several benefits 1 provides a clean connection between many ca techniques and an intuitive framework to understand normalization factors 2 yields efficient numerical schemes to solve ca techniques 3 overcomes the small sample size problem 4 provides a framework to easily extend ca methods we derive weighted generalizations of pca lda sc and cca and several new ca techniques

invariant randers metrics on homogeneous riemannian manifolds this paper studies randers metrics on homogeneous riemannian manifolds it turns out that we can give a complete description of the invariant randers metrics on a homogeneous riemannian manifold as well as the geodesics the flag curvatures this result provides a convenient method to construct globally defined berwald space which is neither riemannian nor locally minkowskian and gives another explanation of the example of bao et al 1999 an introduction to riemannian finsler geometry berlin springer

dynamic linear models with switching abstract the problem of modeling change in a vector time series is studied using a dynamic linear model with measurement matrices that switch according to a time varying independent random process we derive filtered estimators for the usual state vectors and also for the state occupancy probabilities of the underlying nonstationary measurement process a maximum likelihood estimation procedure is given that uses a pseudo expectation maximization algorithm in the initial stages and nonlinear optimization we relate the models to those considered previously in the literature and give an application involving the tracking of multiple targets

localised sequential state estimation for advection dominated flows with non gaussian uncertainty description abstract this paper presents a new iterative state estimation algorithm for advection dominated flows with non gaussian uncertainty description of l type uncertain initial condition and model error are assumed to be pointwise bounded in space and time and the observation noise has uncertain but bounded second moments the algorithm approximates this l type bounding set by a union of possibly overlapping ellipsoids which are localised in space on a number of sub domains on each sub domain the state of the original system is estimated by the standard l 2 type filter e g kalman minimax filter which uses gaussian ellipsoidal uncertainty description and observations if any which correspond to this sub domain the resulting local state estimates are stitched together by the iterative d adn schwarz method to reconstruct the state of the original system the efficacy of the proposed method is demonstrated with a set of numerical examples

em algorithms for ml factor analysis the details of em algorithms for maximum likelihood factor analysis are presented for both the exploratory and confirmatory models the algorithm is essentially the same for both cases and involves only simple least squares regression operations the largest matrix inversion required is for aq q symmetric matrix whereq is the matrix of factors the example that is used demonstrates that the likelihood for the factor analysis model may have multiple modes that are not simply rotations of each other such behavior should concern users of maximum likelihood factor analysis and certainly should cast doubt on the general utility of second derivatives of the log likelihood as measures of precision of estimation

the indian buffet process an introduction and review the indian buffet process is a stochastic process defining a probability distribution over equivalence classes of sparse binary matrices with a finite number of rows and an unbounded number of columns this distribution is suitable for use as a prior in probabilistic models that represent objects using a potentially infinite array of features or that involve bipartite graphs in which the size of at least one class of nodes is unknown we give a detailed derivation of this distribution and illustrate its use as a prior in an infinite latent feature model we then review recent applications of the indian buffet process in machine learning discuss its extensions and summarize its connections to other stochastic processes

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

learning and inferring motion patterns using parametric segmental switching linear dynamic systems switching linear dynamic system slds models are a popular technique for modeling complex nonlinear dynamic systems an slds can describe complex temporal patterns more concisely and accurately than an hmm by using continuous hidden states however the use of slds models in practical applications is challenging for three reasons first exact inference in slds models is computationally intractable second the geometric duration model induced in standard sldss limits their representational power third standard sldss do not provide a principled way to interpret systematic variations governed by higher order parameters the contributions in this paper address all of these three challenges first we present a data driven mcmc dd mcmc sampling method for approximate inference in sldss we show dd mcmc provides an efficient method for estimation and learning in slds models second we present segmental sldss s slds where the geometric distributions of the switching state durations are replaced with arbitrary duration models third we extend the standard slds model with additional global parameters that can capture systematic temporal and spatial variations the resulting parametric slds model p slds uses em to robustly interpret parametrized motions by incorporating additional global parameters that underly systematic variations of the overall motion the overall development of the extensions for sldss provide a principled framework to interpret complex motions the framework is applied to the honey bee dance interpretation task in the context of the on going biotracking project at the georgia institute of technology the experimental results suggest that the enhanced models provide an effective framework for a wide range of motion analysis applications

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

modeling the manifolds of images of handwritten digits this paper describes two new methods for modeling the manifolds of digitized images of handwritten digits the models allow a priori information about the structure of the manifolds to be combined with empirical data accurate modeling of the manifolds allows digits to be discriminated using the relative probability densities under the alternative models one of the methods is grounded in principal components analysis the other in factor analysis both methods are based on locally linear low dimensional approximations to the underlying data manifold links with other methods that model the manifold are discussed

regularized principal manifolds many settings of unsupervised learning can be viewed as quantization problems the minimization of the expected quantization error subject to some restrictions this allows the use of tools such as regularization from the theory of supervised risk minimization for unsupervised learning this setting turns out to be closely related to principal curves the generative topographic map and robust coding we explore this connection in two ways 1 we propose an algorithm for finding principal manifolds that can be regularized in a variety of ways and 2 we derive uniform convergence bounds and hence bounds on the learning rates of the algorithm in particular we give bounds on the covering numbers which allows us to obtain nearly optimal learning rates for certain types of regularization operators experimental results demonstrate the feasibility of the approach