on some convergence properties of the subspace constrained mean shift abstract subspace constrained mean shift scms is a non parametric iterative algorithm that has recently been proposed to find principal curves and surfaces based on a new definition involving the gradient and hessian of a kernel probability density estimate although simulation results using synthetic and real data have demonstrated the usefulness of the scms algorithm a rigorous study of its convergence is still missing this paper aims to take initial steps in this direction by showing that the scms algorithm inherits some important convergence properties of the mean shift ms algorithm in particular the monotonicity and convergence of the density estimate values along the sequence of output values of the algorithm is shown also it is shown that the distance between consecutive points of the output sequence converges to zero as does the projection of the gradient vector onto the subspace spanned by the d d eigenvectors corresponding to the d d largest eigenvalues of the local inverse covariance matrix these last two properties provide theoretical guarantees for stopping criteria by modifying the projection step three variations of the scms algorithm are proposed and the running times and performance of the resulting algorithms are compared

diffusion approximation of frequency sensitive competitive learning the focus of this paper is a convergence study of the frequency sensitive competitive learning fscl algorithm we approximate the final phase of fscl learning by a diffusion process described by the fokker plank equation sufficient and necessary conditions are presented for the convergence of the diffusion process to a local equilibrium the analysis parallels that by ritter schulten 1988 for kohonen s self organizing map we show that the convergence conditions involve only the learning rate and that they are the same as the conditions for weak convergence described previously our analysis thus broadens the class of algorithms that have been shown to have these types of convergence characteristics

learning nonlinear image manifolds by global alignment of local linear models appearance based methods based on statistical models of the pixel values in an image region rather than geometrical object models are increasingly popular in computer vision in many applications the number of degrees of freedom dof in the image generating process is much lower than the number of pixels in the image if there is a smooth function that maps the dof to the pixel values then the images are confined to a low dimensional manifold embedded in the image space we propose a method based on probabilistic mixtures of factor analyzers to 1 model the density of images sampled from such manifolds and 2 recover global parameterizations of the manifold a globally nonlinear probabilistic two way mapping between coordinates on the manifold and images is obtained by combining several locally valid linear mappings we propose a parameter estimation scheme that improves upon an existing scheme and experimentally compare the presented approach to self organizing maps generative topographic mapping and mixtures of factor analyzers in addition we show that the approach also applies to finding mappings between different embeddings of the same manifold

an integrable sis model we provide a demonstration of the integrability of a classical model of an infectious disease which neither kills nor induces autoimmunity by means of the painleve analysis and use the lie theory of transformation groups to present an explicit solution

gtm the generative topographic mapping latent variable models represent the probability density of data in a space of several dimensions in terms of a smaller number of latent or hidden variables a familiar example is factor analysis which is based on a linear transformations between the latent space and the data space in this paper we introduce a form of non linear latent variable model called the generative topographic mapping for which the parameters of the model can be determined using the em algorithm gtm provides a principled alternative to the widely used self organizing map som of kohonen 1982 and overcomes most of the significant limitations of the som we demonstrate the performance of the gtm algorithm on a toy problem and on simulated data from flow diagnostics for a multi phase oil pipeline

relative perturbation bound for invariant subspaces of graded indefinite hermitian matrices abstract we give a bound for the perturbations of invariant subspaces of graded indefinite hermitian matrix h d ad which is perturbed into h h d a a d such relative perturbations include an important case where h is given with an element wise relative error application of our bounds requires only the knowledge of the size of relative perturbation a and not the perturbation a itself this typically occurs when data are given with relative uncertainties when the matrix is being stored into computer memory and when analyzing some numerical algorithms subspace perturbations are measured in terms of perturbations of angles between subspaces and our bound is therefore a relative variant of the well known davis kahan sin theorem our bounds generalize some of the recent relative perturbation results

principal curve algorithms for partitioning high dimensional data spaces most partitioning algorithms iteratively partition a space into cells that contain underlying linear or nonlinear structures using linear partitioning strategies the compactness of each cell depends on how well the locally linear partitioning strategy approximates the intrinsic structure to partition a compact structure for complex data in a nonlinear context this paper proposes a nonlinear partition strategy this is a principal curve tree pc tree which is implemented iteratively given that a pc passes through the middle of the data distribution it allows for partitioning based on the arc length of the pc to enhance the partitioning of a given space a residual version of the pc tree algorithm is developed denoted here as the principal component analysis tree pcr tree algorithm because of its residual property the pcr tree can yield the intrinsic dimension of high dimensional data comparisons presented in this paper confirm that the proposed pc tree and pcr tree approaches show a better performance than several other competing partitioning algorithms in terms of vector quantization error and nearest neighbor search the comparison also shows that the proposed algorithms outperform competing linear methods in total average coverage which measures the nonlinear compactness of partitioning algorithms

an implicit jacobi like method for computing generalized hyperbolic svd abstract in this paper we introduce a joint hyperbolic orthogonal decomposition of two matrices which we call a generalized hyperbolic singular value decomposition or ghsvd this decomposition can be used for finding the eigenvalues and eigenvectors of a symmetric definite pencil x t x y t y where diag 1 we also present an implicit jacobi like method for computing this ghsvd

relative perturbation theory for hyperbolic eigenvalue problem we give relative perturbation bounds for eigenvalues and perturbation bounds for eigenspaces of a hyperbolic eigenvalue problem hx jx where h is a positive definite matrix and j is a diagonal matrix of signs we consider two types of perturbations when a graded matrix h d ad is perturbed in a graded sense to h h d a a d and the multiplicative perturbations of the form h h i e h i e our bounds are simple to compute compare well to the classical results and can be used when analyzing numerical algorithms

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

constraint local principal curve existing principal curve algorithms have some drawbacks such as time consuming and narrow application scope in practice since these algorithms are mainly based on global optimization in this paper we present the concept of constraint local principal curve clpc which uses local optimization methods and restricts the principal curve with two fixed endpoints to reduce the computational complexity in addition we propose three clpc algorithms by local optimization and adaptive radius to expand the range of applications and increase the solution quality the first algorithm i e clpcg is based on greedy thinking the second algorithm i e clpcs uses one dimensional search and the last algorithm clpcc combines the greedy thinking and one dimensional search then we define six performance indices to evaluate the performance of the clpc algorithms finally we present some numerical experiments with three simulation data sets and two gps measured data sets in both highway and railway the results indicate that all of the three clpc algorithms can obtain high accuracy data from multiple low accuracy data efficiently the clpc algorithms can improve the accuracy and computational speed compared with the existing k segment principal curve kpc algorithm in addition clpcc outperforms clpcg and clpcs according to the comprehensive experiments while clpcg runs much faster than other ones we present the concept of constraint local principal curve clpc to reduce the computational complexity we propose three clpc algorithms that combine local optimization and adaptive radius to expand the range of applications and increase the solution quality we present some numerical experiments with three simulation data sets and two measured gps data sets in highway and railway the clpc algorithms can improve the accuracy and computational speed compared with the existing kpc algorithms the features of the each clpc algorithm are analyzed according to the comprehensive experiments

the minmax k means clustering algorithm applying k means to minimize the sum of the intra cluster variances is the most popular clustering approach however after a bad initialization poor local optima can be easily obtained to tackle the initialization problem of k means we propose the minmax k means algorithm a method that assigns weights to the clusters relative to their variance and optimizes a weighted version of the k means objective weights are learned together with the cluster assignments through an iterative procedure the proposed weighting scheme limits the emergence of large variance clusters and allows high quality solutions to be systematically uncovered irrespective of the initialization experiments verify the effectiveness of our approach and its robustness over bad initializations as it compares favorably to both k means and other methods from the literature that consider the k means initialization problem 2014 elsevier ltd all rights reserved

clustering based on conditional distributions in an auxiliary space we study the problem of learning groups or categories that are local in the continuous primary space but homogeneous by the distributions of an associated auxiliary random variable over a discrete auxiliary space assuming that variation in the auxiliary space is meaningful categories will emphasize similarly meaningful aspects of the primary space from a data set consisting of pairs of primary and auxiliary items the categories are learned by minimizing a kullback leibler divergence based distortion between implicitly estimated distributions of the auxiliary data conditioned on the primary data still the categories are defined in terms of the primary space an online algorithm resembling the traditional hebb type competitive learning is introduced for learning the categories minimizing the distortion criterion turns out to be equivalent to maximizing the mutual information between the categories and the auxiliary data in addition connections to density estimation and to the distributional clustering paradigm are outlined the method is demonstrated by clustering yeast gene expression data from dna chips with biological knowledge about the functional classes of the genes as the auxiliary data

similarity preserving principal curve an optimal 1 d feature extractor for data representation this paper discusses the problem of what kind of learning model is suitable for the tasks of feature extraction for data representation and suggests two evaluation criteria for nonlinear feature extractors reconstruction error minimization and similarity preservation based on the suggested evaluation criteria a new type of principal curve similarity preserving principal curve sppc is proposed sppcs minimize the reconstruction error under the condition that the similarity between similar samples are preserved in the extracted features thus giving researchers effective and reliable cognition of the inner structure of data sets the existence and properties of sppcs are analyzed a practical learning algorithm is proposed and high dimensional extensions of sppcs are also discussed experimental results show the virtues of sppcs in preserving inner structures of data sets and discovering manifolds with high nonlinearity

highly accurate symmetric eigenvalue decomposition and hyperbolic svd let g be a m n real matrix with full column rank and let j be a n n diagonal matrix of signs jii 1 1 the hyperbolic singular value decomposition hsvd of the pair g j is defined as g u v 1 where u is orthogonal is positive definite diagonal and v is j orthogonal matrix vtjv j we analyze when it is possible to compute the hsvd with high relative accuracy this essentially means that each computed hyperbolic singular value is guaranteed to have some correct digits even if they have widely varying magnitudes we show that one sided j orthogonal jacobi method method computes the hsvd with high relative accuracy more precisely let b gd 1 where d is diagonal such that the columns of b have unit norms essentially we show that the computed hyperbolic singular values of the pair g j will have log10 e min b correct decimal digits where e is machine precision we give the necessary relative perturbation bounds and error analysis of the algorithm our numerical tests confirmed all theoretical results for the symmetric non singular eigenvalue problem hx x we analyze the two step algorithm which consists of factorization h gjgt followed by the computation of the hsvd of the pair g j here g is square and non singular let b dg where d is diagonal such that the rows of b have unit norms and let b be defined as above essentially we show that the computed eigenvalues of h will have log10 e 2min b e min b correct decimal digits this accuracy can be much higher then the one obtained by the classical qr and jacobi methods applied to h where the accuracy depends on the spectral condition number of h particularly if the matrices b and b are well conditioned and we are interested in the accurate computation of tiny eigenvalues again we give the perturbation and error bounds and our theoretical predictions are confirmed by a series of numerical experiments we also give the corresponding results for eigenvectors and hyperbolic singular vectors

probabilistic self organizing maps for continuous data the original self organizing feature map did not define any probability distribution on the input space however the advantages of introducing probabilistic methodologies into self organizing map models were soon evident this has led to a wide range of proposals which reflect the current emergence of probabilistic approaches to computational intelligence the underlying estimation theories behind them derive from two main lines of thought the expectation maximization methodology and stochastic approximation methods here we present a comprehensive view of the state of the art with a unifying perspective of the involved theoretical frameworks in particular we examine the most commonly used continuous probability distributions self organization mechanisms and learning schemes special emphasis is given to the connections among them and their relative advantages depending on the characteristics of the problem at hand furthermore we evaluate their performance in two typical applications of self organizing maps classification and visualization

learning and design of principal curves principal curves have been defined as self consistent smooth curves which pass through the middle of a d dimensional probability distribution or data cloud they give a summary of the data and also serve as an efficient feature extraction tool we take a new approach by defining principal curves as continuous curves of a given length which minimize the expected squared distance between the curve and points of the space randomly chosen according to a given distribution the new definition makes it possible to theoretically analyze principal curve learning from training data and it also leads to a new practical construction our theoretical learning scheme chooses a curve from a class of polygonal lines with k segments and with a given total length to minimize the average squared distance over n training points drawn independently convergence properties of this learning scheme are analyzed and a practical version of this theoretical algorithm is implemented in each iteration of the algorithm a new vertex is added to the polygonal line and the positions of the vertices are updated so that they minimize a penalized squared distance criterion simulation results demonstrate that the new algorithm compares favorably with previous methods both in terms of performance and computational complexity and is more robust to varying data models

relative residual bounds for indefinite hermitian matrices abstract we prove several residual bounds for relative perturbations of the eigenvalues of indefinite hermitian matrix the bounds fall into two categories the weyl type bounds and the hofmann wielandt type bounds the bounds are expressed in terms of sines of acute principal angles between certain subspaces associated with the indefinite decomposition of the given matrix the bounds are never worse than the classical residual bounds and can be much sharper in some cases the bounds generalize the existing relative residual bounds for positive definite matrices to indefinite case

frequency sensitive competitive learning for scalable balanced clustering on high dimensional hyperspheres competitive learning mechanisms for clustering in general suffer from poor performance for very high dimensional 1000 data because of curse of dimensionality effects in applications such as document clustering it is customary to normalize the high dimensional input vectors to unit length and it is sometimes also desirable to obtain balanced clusters i e clusters of comparable sizes the spherical kmeans spkmeans algorithm which normalizes the cluster centers as well as the inputs has been successfully used to cluster normalized text documents in 2000 dimensional space unfortunately like regular kmeans and its soft expectation maximization based version spkmeans tends to generate extremely imbalanced clusters in high dimensional spaces when the desired number of clusters is large tens or more this paper first shows that the spkmeans algorithm can be derived from a certain maximum likelihood formulation using a mixture of von mises fisher distributions as the generative model and in fact it can be considered as a batch mode version of normalized competitive learning the proposed generative model is then adapted in a principled way to yield three frequency sensitive competitive learning variants that are applicable to static data and produced high quality and well balanced clusters for high dimensional data like kmeans each iteration is linear in the number of data points and in the number of clusters for all the three algorithms a frequency sensitive algorithm to cluster streaming data is also proposed experimental results on clustering of high dimensional text data sets are provided to show the effectiveness and applicability of the proposed techniques

self organization as an iterative kernel smoothing process kohonen s self organizing map when described in a batch processing mode can be interpreted as a statistical kernel smoothing problem the batch som algorithm consists of two steps first the training data are partitioned according to the voronoi regions of the map unit locations second the units are updated by taking weighted centroids of the data falling into the voronoi regions with the weighing function given by the neighborhood then the neighborhood width is decreased and steps 1 2 are repeated the second step can be interpreted as a statistical kernel smoothing problem where the neighborhood function corresponds to the kernel and neighborhood width corresponds to kernel span to determine the new unit locations kernel smoothing is applied to the centroids of the voronoi regions in the topological space this interpretation leads to some new insights concerning the role of the neighborhood and dimensionality reduction it also strengthens the algorithm s connection with the principal curve algorithm a generalized self organizing algorithm is proposed where the kernel smoothing step is replaced with an arbitrary nonparametric regression method

principal surfaces from unsupervised kernel regression we propose a nonparametric approach to learning of principal surfaces based on an unsupervised formulation of the nadaraya watson kernel regression estimator as compared with previous approaches to principal curves and surfaces the new method offers several advantages first it provides a practical solution to the model selection problem because all parameters can be estimated by leave one out cross validation without additional computational cost in addition our approach allows for a convenient incorporation of nonlinear spectral methods for parameter initialization beyond classical initializations based on linear pca furthermore it shows a simple way to fit principal surfaces in general feature spaces beyond the usual data space setup the experimental results illustrate these convenient features on simulated and real data

codeword distribution for frequency sensitive competitive learning with one dimensional input data we study the codeword distribution for a conscience type competitive learning algorithm frequency sensitive competitive learning fscl using one dimensional input data we prove that the asymptotic codeword density in the limit of large number of codewords is given by a power law of the form q x c spl middot p x sup spl alpha where p x is the input data density and spl alpha depends on the algorithm and the form of the distortion measure to be minimized we further show that the algorithm can be adjusted to minimize any l sub p distortion measure with p ranging in 0 2

clustering on the unit hypersphere using von mises fisher distributions several large scale data mining applications such as text categorization and gene expression analysis involve high dimensional data that is also inherently directional in nature often such data is l2 normalized so that it lies on the surface of a unit hypersphere popular models such as mixtures of multi variate gaussians are inadequate for characterizing such data this paper proposes a generative mixture model approach to clustering directional data based on the von mises fisher vmf distribution which arises naturally for data distributed on the unit hypersphere in particular we derive and analyze two variants of the expectation maximization em framework for estimating the mean and concentration parameters of this mixture numerical estimation of the concentration parameters is non trivial in high dimensions since it involves functional inversion of ratios of bessel functions we also formulate two clustering algorithms corresponding to the variants of em that we derive our approach provides a theoretical basis for the use of cosine similarity that has been widely employed by the information retrieval community and obtains the spherical kmeans algorithm kmeans with cosine similarity as a special case of both variants empirical results on clustering of high dimensional text and gene expression data based on a mixture of vmf distributions show that the ability to estimate the concentration parameter for each vmf component which is not present in existing approaches yields superior results especially for difficult clustering tasks in high dimensional spaces

a note on the existence of the hyperbolic singular value decomposition abstract in this note we present a new matrix decomposition for a matrix pair a b with a hermitian the hyperbolic svd comes as a special case of the decomposition with a set to be the signature matrix

lie point symmetries and first integrals the kowalevski top we show how the lie group analysis method can be used in order to obtain first integrals of any system of ordinary differential equations the method of reduction increase of order developed by nucci j math phys 37 1772 1775 1996 is essential noether s theorem is neither necessary nor considered the most striking example we present is the relationship between lie group analysis and the famous first integral of the kowalevski top

relative perturbation theory for hyperbolic singular value problem we give relative perturbation bounds for singular values and perturbation bounds for singular subspaces of a hyperbolic singular value problem for the pair g j where g is a full rank matrix and j is a diagonal matrix of signs we consider two types of relative perturbations g g b b d and g g d b b depending whether g has full column or full row rank respectively in both cases we also consider relative element wise perturbations of g which typically occur in numerical computations

a bound for the condition of a hyperbolic eigenvector matrix the hyperbolic eigenvector matrix is a matrix x which simultaneously diagonalizes the pair h j where h is hermitian positive definite and j diag 1 such that x hx and x jx j we prove that the spectral condition of x x is bounded byk x mink d hd where the minimum is taken over all non singular matrices d which commute with j this bound is attainable and it can be simply computed similar results hold for other signature matrices j like in the discretized klein gordon equation

a sin2θ theorem for graded indefinite hermitian matrices this paper gives double angle theorems that bound the change in an invariant subspace of an indefinite hermitian matrix in the graded form h d ad subject to a perturbation h h d a a d these theorems extend recent results on a definite hermitian

hidden linearity in systems for competition with evolution in ecology and finance abstract in j c sprott competition with evolution in ecology and finance phys lett a 325 2004 329 333 a variant of the multispecies lotka volterra model was proposed in which species competing for fixed finite resources are replaced by a new randomly chosen species whenever they become extinct we applied lie group analysis to this system and found that it can be reduced to a linear equation if certain values are given to the parameters involved which rules out any chaotic behavior in these instances also different routes to linearization are shown

floating point perturbations of hermitian matrices abstract we consider the perturbation properties of the eigensolution of hermitian matrices for the matrix entries and the eigenvalues we use the realistic floating point error measure a a recently demmel and veselic considered the same problem for a positive definite matrix h showing that the floating point perturbation theory holds with constants depending on the condition number of the matrix a dhd where aii 1 and d is a diagonal scaling we study the general hermitian case along the same lines thus obtaining new classes of well behaved matrices and matrix pairs our theory is applicable to the already known class of scaled diagonally dominant matrices as well as to matrices given by factors like those in symmetric indefinite decompositions we also obtain norm estimates for the perturbations of the eigenprojections and show that some of our techniques extend to non hermitian matrices however unlike in the positive definite case we are still unable to describe simply the set of all well behaved hermitian matrices

lorenz integrable system moves a la poinsot a transformation is derived which takes the lorenz integrable system into the well known euler equations of a torque free rigid body about a fixed point i e the famous motion a la poinsot the proof is based on lie group analysis applied to two third order ordinary differential equations admitting the same two dimensional lie symmetry algebra lie s classification of two dimensional symmetry algebras in the plane is used if the same transformation is applied to the lorenz system with any values of the parameters then one obtains euler equations of a rigid body about a fixed point subjected to a torsion depending on time and angular velocity the numerical solution of this system yields a three dimensional picture which looks like a tornado the cross section of which has a butterfly shape thus lorenz s butterfly has been transformed into a tornado

a note on unifying absolute and relative perturbation bounds abstract perturbation bounds for invariant subspaces and eigenvalues of complex matrices are presented that lead to absolute as well as a large class of relative bounds in particular it is shown that absolute bounds such as those by davis and kahan bauer and fike and hoffman and wielandt and some relative bounds are special cases of universal bounds as a consequence we obtain a new relative bound for subspaces of normal matrices which contains a deviation of the matrix from positive definiteness we also investigate how row scaling affects eigenvalues and their sensitivity to perturbations and we illustrate how the departure from normality can affect the condition number with respect to inversion of the scaled eigenvectors

exploratory projection pursuit abstract a new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods a number of practical issues concerning its application are addressed a connection to multivariate density estimation is established and its properties are investigated through simulation studies and application to real data the goal of exploratory projection pursuit is to use the data to find low one two or three dimensional projections that provide the most revealing views of the full dimensional data with these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance since linear effects are directly captured by the covariance structure of the variable pairs which are straightforward to estimate the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables although arbitrary

quadratic convergence estimate of scaled iterates by j symmetric jacobi method abstract this paper estimates the quadratic convergence reduction of scaled iterates by j symmetric jacobi method numer math 64 1993 241 although the method is well defined for a general definite pair h j h ht j diag im in m the paper considers the most important case when h is positive definite in that case the method is an accurate floating point eigensolver for the pair h j as such it is used in a compound algorithm for accurate floating point computation of eigenvalues and eigenvectors of a non singular indefinite symmetric matrix the new result is proved for scaled diagonally dominant matrices in the general case of multiple eigenvalues it uses frobenius norm of the off diagonal part of symmetrically scaled iteration matrix and a relative gap in the spectrum of h j it can be effectively used in connection with stopping criterion of the method especially with its one sided version

the complete kepler group can be derived by lie group analysis it is shown that the complete symmetry group for the kepler problem as introduced by krause can be derived by lie group analysis the same result is true for any autonomous system

relative perturbation theory for a class of diagonalizable hermitian matrix pairs abstract veselic and slapnicar gave a general perturbation result for the eigenvalues of the hermitian matrix pair h k where k is positive definite in this paper their result is generalized to a wider class of hermitian matrix pairs especially estimates for the relative perturbation of eigenvalues of definite pairs are also obtained

extremal properties of principal curves in the plane principal curves were introduced to formalize the notion of a curve passing through the middle of a dataset vaguely speaking a curve is said to pass through the middle of a dataset if every point on the curve is the average of the observations projecting onto it this idea can be made precise by defining principal curves for probability densities in this paper we study principal curves in the plane like linear principal components principal curves are critical points of the expected squared distance from the data however the largest and smallest principal components are extrema of the distance whereas all principal curves are saddle points this explains why cross validation does not appear to be a viable method for choosing the complexity of principal curve estimates

mixtures of probabilistic principal component analyzers principal component analysis pca is one of the most popular techniques for processing compressing and visualizing data although its effectiveness is limited by its global linearity while nonlinear variants of pca have been proposed an alternative paradigm is to capture data complexity by a combination of local linear pca projections however conventional pca does not correspond to a probability density and so there is no unique way to combine pca models therefore previous attempts to formulate mixture models for pca have been ad hoc to some extent in this article pca is formulated within a maximum likelihood framework based on a specific form of gaussian latent variable model this leads to a well defined mixture model for probabilistic principal component analyzers whose parameters can be determined using an expectationmaximization algorithm we discuss the advantages of this model in the context of clustering density modeling and local dimensionality reduction and we demonstrate its application to image compression and handwritten digit recognition

relative perturbation theory iii more bounds on eigenvalue variation abstract classically the relative error in ga 1 as an approximation to is measured by relative error in ga a the quantity log10 is usually used for the number of correct decimal digits in numerical results although this measure is clearly not a metric since it lacks symmetry between and ga in part i of this series two other kinds of relative distances which have much better mathematical properties have been introduced and employed to establish theories it is shown that these different measurements are topologically equivalent however the measure is more convenient to use in practice in this part we established relative perturbation bounds directly using the classical measure the new bounds for diagonalizable matrices are cleaner than the corresponding ones in part i and yield nice bounds for hermitian matrices too but when applied to nonnegative definite hermitian matrices the new bounds are weaker than those in part i

relative perturbation techniques for singular value problems a technique is presented for deriving bounds on the relative change in the singular values of a real matrix or the eigenvalues of a real symmetric matrix due to a perturbation as well as bounds on the angles between the unperturbed and perturbed singular vectors or eigenvectors the class of perturbations considered consists of all delta b for which b delta b d l bd r for some nonsingular matrices d l and d r this class includes componentwise relative perturbations of a bidiagonal or biacyclic matrix and perturbations that annihilate the off diagonal block in a block triangular matrix many existing relative perturbation and deflation bounds are derived from results for this general class of perturbations also some new relative perturbation and deflation results for the singular values and vectors of biacyclic triangular and shifted triangular matrices are presented

the hyberbolic singular value decomposition and applications a new generalization of the singular value decomposition svd the hyperbolic svd is advanced and its existence is established under mild restrictions the hyperbolic svd accurately and efficiently finds the eigenstructure of any matrix that is expressed as the difference of two matrix outer products signal processing applications where this task arises include the covariance differencing algorithm for bearing estimation in sensor arrays sliding rectangular windowing and array calibration two algorithms for effecting this decomposition are detailed one is sequential and follows a similar pattern to the sequential bidiagonal based svd algorithm the other is for parallel implementation and mimics hestenes svd algorithm 1958 numerical examples demonstrate that like its conventional counterpart the hyperbolic svd exhibits superior numerical behavior relative to explicit formation and solution of the normal equations furthermore the hyperbolic svd applies in problems where the conventional svd cannot be employed

calogero s goldfish is indeed a school of free particles a many body system of n nonlinear ordinary differential equations of second order which is amenable to exact treatments a goldfish calogero 2001 the neatest many body problem amenable to exact treatments a goldfish physica d 152 153 78 84 is shown to be equivalent through an exact transformation to the equations of one dimensional motion of n 1 free particles a school of free particles indeed the transformation is obtained by applying the reduction method and lie group analysis as introduced in nucci 1996 the complete kepler group can be derived by lie group analysis j math phys 37 1772 5

locally weighted regression an approach to regression analysis by local fitting abstract locally weighted regression or loess is a way of estimating a regression surface through a multivariate smoothing procedure fitting a function of the independent variables locally and in a moving fashion analogous to how a moving average is computed for a time series with local fitting we can estimate a much wider class of regression surfaces than with the usual classes of parametric functions such as polynomials the goal of this article is to show through applications how loess can be used for three purposes data exploration diagnostic checking of parametric models and providing a nonparametric regression surface along the way the following methodology is introduced a a multivariate smoothing procedure that is an extension of univariate locally weighted regression b statistical procedures that are analogous to those used in the least squares fitting of parametric functions c several graphical methods that are useful tools for understanding loess estimates and checking the a

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

locally defined principal curves and surfaces principal curves are defined as self consistent smooth curves passing through the middle of the data and they have been used in many applications of machine learning as a generalization dimensionality reduction and a feature extraction tool we redefine principal curves and surfaces in terms of the gradient and the hessian of the probability density estimate this provides a geometric understanding of the principal curves and surfaces as well as a unifying view for clustering principal curve fitting and manifold learning by regarding those as principal manifolds of different intrinsic dimensionalities the theory does not impose any particular density estimation method can be used with any density estimator that gives continuous first and second derivatives therefore we first present our principal curve surface definition without assuming any particular density estimation method afterwards we develop practical algorithms for the commonly used kernel density estimation kde and gaussian mixture models gmm results of these algorithms are presented in notional data sets as well as real applications with comparisons to other approaches in the principal curve literature all in all we present a novel theoretical understanding of principal curves and surfaces practical algorithms as general purpose machine learning tools and applications of these algorithms to several practical problems

absolute and relative perturbation bounds for invariant subspaces of matrices absolute and relative perturbation bounds are derived for angles between invariant subspaces of complex square matrices in the two norm and in the frobenius norm the absolute bounds can be considered extensions of davis and kahan s sin theorem to general matrices and invariant subspaces of any dimension the relative bounds are the most general relative bounds for invariant subspaces because they place no restrictions on the matrix or the perturbation when the perturbed subspace has dimension one the relative bound is implied by the absolute bound

a unified model for probabilistic principal surfaces principal curves and surfaces are nonlinear generalizations of principal components and subspaces respectively they can provide insightful summary of high dimensional data not typically attainable by classical linear methods solutions to several problems such as proof of existence and convergence faced by the original principal curve formulation have been proposed in the past few years nevertheless these solutions are not generally extensible to principal surfaces the mere computation of which presents a formidable obstacle consequently relatively few studies of principal surfaces are available we previously 2000 proposed the probabilistic principal surface pps to address a number of issues associated with current principal surface algorithms pps uses a manifold oriented covariance noise model based on the generative topographical mapping gtm which can be viewed as a parametric formulation of kohonen s self organizing map building on the pps we introduce a unified covariance model that implements pps 0 1 by varying the clamping parameter spl alpha then we comprehensively evaluate the empirical performance of pps gtm and the manifold aligned gtm on three popular benchmark data sets it is shown in two different comparisons that the pps outperforms the gtm under identical parameter settings convergence of the pps is found to be identical to that of the gtm and the computational overhead incurred by the pps decreases to 40 percent or less for more complex manifolds these results show that the generalized pps provides a flexible and effective way of obtaining principal surfaces

regularization free principal curve estimation principal curves and manifolds provide a framework to formulate manifold learning within a statistical context principal curves define the notion of a curve passing through the middle of a distribution while the intuition is clear the formal definition leads to some technical and practical difficulties in particular principal curves are saddle points of the mean squared projection distance which poses severe challenges for estimation and model selection this paper demonstrates that the difficulties in model selection associated with the saddle point property of principal curves are intrinsically tied to the minimization of the mean squared projection distance we introduce a new objective function facilitated through a modification of the principal curve estimation approach for which all critical points are principal curves and minima thus the new formulation removes the fundamental issue for model selection in principal curve estimation a gradient descent based estimator demonstrates the effectiveness of the new formulation for controlling model complexity on numerical experiments with synthetic and real data

lie symmetries of the lorenz model abstract we study the generalized symmetries of the lorenz model to find the parameter values at which one or more time dependent integrals of motion exist in these cases the integrals are found trivially from the symmetries themselves a complete study of the one completely algebraically integrable case shows a the dynamics can be integrated exactly by reducing it first to a lower dimensional system b the symmetry vector field is hamiltonian these facts hold for other dissipative completely integrable dynamical systems as well the analytic study of a natural two form reveals that it is an entire function of time the foliation of phase space induced by the two form for the partially integrable cases has a simple description in terms of the coefficients occurring in the laurent series expansions of the dependent variables

limitations of nonlinear pca as performed with generic neural networks kramer s 1991 nonlinear principal components analysis nlpca neural networks are feedforward autoassociative networks with five layers the third layer has fewer nodes than the input or output layers this paper proposes a geometric interpretation for kramer s method by showing that nlpca fits a lower dimensional curve or surface through the training data the first three layers project observations onto the curve or surface giving scores the last three layers define the curve or surface the first three layers are a continuous function which we show has several implications nlpca projections are suboptimal producing larger approximation error nlpca is unable to model curves and surfaces that intersect themselves and nlpca cannot parameterize curves with parameterizations having discontinuous jumps we establish results on the identification of score values and discuss their implications on interpreting score values we discuss the relationship between nlpca and principal curves and surfaces another nonlinear feature extraction method

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

reversible jump markov chain monte carlo computation and bayesian model determination markov chain monte carlo methods for bayesian computation have until recently been restricted to problems where the joint distribution of all variables has a density with respect to some fixed standard underlying measure they have therefore not been available for application to bayesian model determination where the dimensionality of the parameter vector is typically not fixed this paper proposes a new framework for the construction of reversible markov chain samplers that jump between parameter subspaces of differing dimensionality which is flexible and entirely constructive it should therefore have wide applicability in model determination problems the methodology is illustrated with applications to multiple change point analysis in one and two dimensions and to a bayesian comparison of binomial experiments

a jacobi eigenreduction algorithm for definite matrix pairs we propose a jacobi eigenreduction algorithm for symmetric definite matrix pairsa j of small to medium size real symmetric matrices withj 2 i j diagonal neitherj nora itself need be definite our jacobi reduction works only on one matrix and usesj orthogonal elementary congruences which include both trigonometric and hyperbolic rotations and preserve the symmetry throughout the process for the rotation parameters only the pivotal elements of the current matrix are needed which facilitates parallelization we prove the global convergence of the method the quadratic convergence was observed in all experiments we apply our method in two situations i eigenreducing a single real symmetric matrix and ii eigenreducing an overdamped quadratic matrix pencil in both cases our method is preceded by a symmetric indefinite decomposition and performed in its one sided variant on the thus obtained factors our method outdoes the standard methods like standard jacobi orqr ql in accuracy in spite of the use of hyperbolic transformations which are not orthogonal a theoretical justification of this behaviour is made elsewhere the accuracy advantage of our method can be particularly drastic if the eigenvalues are of different order in addition in working with quadratic pencils our method is shown to either converge or to detect non overdampedness

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

adaptive nonlinear manifolds and their applications to pattern recognition dimensionality reduction has long been associated with retinotopic mapping for understanding cortical maps multisensory information is processed fused and mapped to an essentially 2 d cortex in an information preserving manner data processing and projection techniques inspired by this biological mechanism are playing an increasingly important role in pattern recognition computational intelligence data mining information retrieval and image recognition dimensionality reduction involves reduction of features or volume of data and has become an essential step of information processing in many fields the topic of manifold learning has recently attracted a great deal of attention and a number of advanced techniques for extracting nonlinear manifolds and reducing data dimensions have been proposed from statistics geometry theory and adaptive neural networks this paper provides an overview of this challenging and emerging topic and discusses various recent methods such as self organizing map som kernel pca principal manifold isomap local linear embedding and laplacian eigenmap many of them can be considered in a learning manifold framework the paper further elaborates on the biologically inspired som model and its metric preserving variant visom under the framework of adaptive manifold and their applications in dimensionality reduction with face recognition are investigated the experiments demonstrate that adaptive visom based methods produce markedly improved performance over the others due to their metric scaling and preserving properties along the nonlinear manifold

perturbations of the eigenprojections of a factorized hermitian matrix abstract we give the perturbation bounds for the eigenprojections of a hermitian matrix h gjg where g has full column rank and j is nonsingular under the perturbations of the factor g our bounds hold for example when g is given with elementwise relative error our bounds contain relative gaps between the eigenvalues and may thus be much less pessimistic than the standard norm estimates

singular values doubly stochastic matrices and applications abstract the hadamard square of any square matrix a is bounded above and below by some doubly stochastic matrices times the square of the largest and the smallest singular values of a applications to graphs permanents and eigenvalue perturbations are discussed

modeling the manifolds of images of handwritten digits this paper describes two new methods for modeling the manifolds of digitized images of handwritten digits the models allow a priori information about the structure of the manifolds to be combined with empirical data accurate modeling of the manifolds allows digits to be discriminated using the relative probability densities under the alternative models one of the methods is grounded in principal components analysis the other in factor analysis both methods are based on locally linear low dimensional approximations to the underlying data manifold links with other methods that model the manifold are discussed

adaptive principal surfaces abstract we develop a nonlinear generalization of principal components analysis a principal surface of the data is constructed adaptively using some ideas from the mars procedure of friedman we explore applications to curve and surface reconstruction and to data summarization