on some convergence properties of the subspace constrained mean shift abstract subspace constrained mean shift scms is a non parametric iterative algorithm that has recently been proposed to find principal curves and surfaces based on a new definition involving the gradient and hessian of a kernel probability density estimate although simulation results using synthetic and real data have demonstrated the usefulness of the scms algorithm a rigorous study of its convergence is still missing this paper aims to take initial steps in this direction by showing that the scms algorithm inherits some important convergence properties of the mean shift ms algorithm in particular the monotonicity and convergence of the density estimate values along the sequence of output values of the algorithm is shown also it is shown that the distance between consecutive points of the output sequence converges to zero as does the projection of the gradient vector onto the subspace spanned by the d d eigenvectors corresponding to the d d largest eigenvalues of the local inverse covariance matrix these last two properties provide theoretical guarantees for stopping criteria by modifying the projection step three variations of the scms algorithm are proposed and the running times and performance of the resulting algorithms are compared

learning nonlinear image manifolds by global alignment of local linear models appearance based methods based on statistical models of the pixel values in an image region rather than geometrical object models are increasingly popular in computer vision in many applications the number of degrees of freedom dof in the image generating process is much lower than the number of pixels in the image if there is a smooth function that maps the dof to the pixel values then the images are confined to a low dimensional manifold embedded in the image space we propose a method based on probabilistic mixtures of factor analyzers to 1 model the density of images sampled from such manifolds and 2 recover global parameterizations of the manifold a globally nonlinear probabilistic two way mapping between coordinates on the manifold and images is obtained by combining several locally valid linear mappings we propose a parameter estimation scheme that improves upon an existing scheme and experimentally compare the presented approach to self organizing maps generative topographic mapping and mixtures of factor analyzers in addition we show that the approach also applies to finding mappings between different embeddings of the same manifold

gtm the generative topographic mapping latent variable models represent the probability density of data in a space of several dimensions in terms of a smaller number of latent or hidden variables a familiar example is factor analysis which is based on a linear transformations between the latent space and the data space in this paper we introduce a form of non linear latent variable model called the generative topographic mapping for which the parameters of the model can be determined using the em algorithm gtm provides a principled alternative to the widely used self organizing map som of kohonen 1982 and overcomes most of the significant limitations of the som we demonstrate the performance of the gtm algorithm on a toy problem and on simulated data from flow diagnostics for a multi phase oil pipeline

power law distribution in high frequency financial data an econometric analysis abstract power law distributions are very common in natural sciences we analyze high frequency financial data from xetra and the nyse using maximum likelihood estimation and the kolmogorov smirnov statistic to test whether the power law hypothesis holds also for these data we find that the universality and scale invariance properties of the power law are violated furthermore the returns of daimler chrysler and sap traded simultaneously on both exchanges follow a power law at one exchange but not at the other these results raise some questions about the no arbitrage condition finally we find that an exponential function provides a better fit for the tails of the sample distributions than a power law function

principal curve algorithms for partitioning high dimensional data spaces most partitioning algorithms iteratively partition a space into cells that contain underlying linear or nonlinear structures using linear partitioning strategies the compactness of each cell depends on how well the locally linear partitioning strategy approximates the intrinsic structure to partition a compact structure for complex data in a nonlinear context this paper proposes a nonlinear partition strategy this is a principal curve tree pc tree which is implemented iteratively given that a pc passes through the middle of the data distribution it allows for partitioning based on the arc length of the pc to enhance the partitioning of a given space a residual version of the pc tree algorithm is developed denoted here as the principal component analysis tree pcr tree algorithm because of its residual property the pcr tree can yield the intrinsic dimension of high dimensional data comparisons presented in this paper confirm that the proposed pc tree and pcr tree approaches show a better performance than several other competing partitioning algorithms in terms of vector quantization error and nearest neighbor search the comparison also shows that the proposed algorithms outperform competing linear methods in total average coverage which measures the nonlinear compactness of partitioning algorithms

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

constraint local principal curve existing principal curve algorithms have some drawbacks such as time consuming and narrow application scope in practice since these algorithms are mainly based on global optimization in this paper we present the concept of constraint local principal curve clpc which uses local optimization methods and restricts the principal curve with two fixed endpoints to reduce the computational complexity in addition we propose three clpc algorithms by local optimization and adaptive radius to expand the range of applications and increase the solution quality the first algorithm i e clpcg is based on greedy thinking the second algorithm i e clpcs uses one dimensional search and the last algorithm clpcc combines the greedy thinking and one dimensional search then we define six performance indices to evaluate the performance of the clpc algorithms finally we present some numerical experiments with three simulation data sets and two gps measured data sets in both highway and railway the results indicate that all of the three clpc algorithms can obtain high accuracy data from multiple low accuracy data efficiently the clpc algorithms can improve the accuracy and computational speed compared with the existing k segment principal curve kpc algorithm in addition clpcc outperforms clpcg and clpcs according to the comprehensive experiments while clpcg runs much faster than other ones we present the concept of constraint local principal curve clpc to reduce the computational complexity we propose three clpc algorithms that combine local optimization and adaptive radius to expand the range of applications and increase the solution quality we present some numerical experiments with three simulation data sets and two measured gps data sets in highway and railway the clpc algorithms can improve the accuracy and computational speed compared with the existing kpc algorithms the features of the each clpc algorithm are analyzed according to the comprehensive experiments

similarity preserving principal curve an optimal 1 d feature extractor for data representation this paper discusses the problem of what kind of learning model is suitable for the tasks of feature extraction for data representation and suggests two evaluation criteria for nonlinear feature extractors reconstruction error minimization and similarity preservation based on the suggested evaluation criteria a new type of principal curve similarity preserving principal curve sppc is proposed sppcs minimize the reconstruction error under the condition that the similarity between similar samples are preserved in the extracted features thus giving researchers effective and reliable cognition of the inner structure of data sets the existence and properties of sppcs are analyzed a practical learning algorithm is proposed and high dimensional extensions of sppcs are also discussed experimental results show the virtues of sppcs in preserving inner structures of data sets and discovering manifolds with high nonlinearity

probabilistic self organizing maps for continuous data the original self organizing feature map did not define any probability distribution on the input space however the advantages of introducing probabilistic methodologies into self organizing map models were soon evident this has led to a wide range of proposals which reflect the current emergence of probabilistic approaches to computational intelligence the underlying estimation theories behind them derive from two main lines of thought the expectation maximization methodology and stochastic approximation methods here we present a comprehensive view of the state of the art with a unifying perspective of the involved theoretical frameworks in particular we examine the most commonly used continuous probability distributions self organization mechanisms and learning schemes special emphasis is given to the connections among them and their relative advantages depending on the characteristics of the problem at hand furthermore we evaluate their performance in two typical applications of self organizing maps classification and visualization

learning and design of principal curves principal curves have been defined as self consistent smooth curves which pass through the middle of a d dimensional probability distribution or data cloud they give a summary of the data and also serve as an efficient feature extraction tool we take a new approach by defining principal curves as continuous curves of a given length which minimize the expected squared distance between the curve and points of the space randomly chosen according to a given distribution the new definition makes it possible to theoretically analyze principal curve learning from training data and it also leads to a new practical construction our theoretical learning scheme chooses a curve from a class of polygonal lines with k segments and with a given total length to minimize the average squared distance over n training points drawn independently convergence properties of this learning scheme are analyzed and a practical version of this theoretical algorithm is implemented in each iteration of the algorithm a new vertex is added to the polygonal line and the positions of the vertices are updated so that they minimize a penalized squared distance criterion simulation results demonstrate that the new algorithm compares favorably with previous methods both in terms of performance and computational complexity and is more robust to varying data models

power law distributions in empirical data power law distributions occur in many situations of scientific interest and have significant consequences for our understanding of natural and man made phenomena unfortunately the detection and characterization of power laws is complicated by the large fluctuations that occur in the tail of the distribution the part of the distribution representing large but rare events and by the difficulty of identifying the range over which power law behavior holds commonly used methods for analyzing power law data such as least squares fitting can produce substantially inaccurate estimates of parameters for power law distributions and even in cases where such methods return accurate answers they are still unsatisfactory because they give no indication of whether the data obey a power law at all here we present a principled statistical framework for discerning and quantifying power law behavior in empirical data our approach combines maximum likelihood fitting methods with goodness of fit tests based on the kolmogorov smirnov ks statistic and likelihood ratios we evaluate the effectiveness of the approach with tests on synthetic data and give critical comparisons to previous approaches we also apply the proposed methods to twenty four real world data sets from a range of different disciplines each of which has been conjectured to follow a power law distribution in some cases we find these conjectures to be consistent with the data while in others the power law is ruled out

self organization as an iterative kernel smoothing process kohonen s self organizing map when described in a batch processing mode can be interpreted as a statistical kernel smoothing problem the batch som algorithm consists of two steps first the training data are partitioned according to the voronoi regions of the map unit locations second the units are updated by taking weighted centroids of the data falling into the voronoi regions with the weighing function given by the neighborhood then the neighborhood width is decreased and steps 1 2 are repeated the second step can be interpreted as a statistical kernel smoothing problem where the neighborhood function corresponds to the kernel and neighborhood width corresponds to kernel span to determine the new unit locations kernel smoothing is applied to the centroids of the voronoi regions in the topological space this interpretation leads to some new insights concerning the role of the neighborhood and dimensionality reduction it also strengthens the algorithm s connection with the principal curve algorithm a generalized self organizing algorithm is proposed where the kernel smoothing step is replaced with an arbitrary nonparametric regression method

frequency sensitive competitive learning for scalable balanced clustering on high dimensional hyperspheres competitive learning mechanisms for clustering in general suffer from poor performance for very high dimensional 1000 data because of curse of dimensionality effects in applications such as document clustering it is customary to normalize the high dimensional input vectors to unit length and it is sometimes also desirable to obtain balanced clusters i e clusters of comparable sizes the spherical kmeans spkmeans algorithm which normalizes the cluster centers as well as the inputs has been successfully used to cluster normalized text documents in 2000 dimensional space unfortunately like regular kmeans and its soft expectation maximization based version spkmeans tends to generate extremely imbalanced clusters in high dimensional spaces when the desired number of clusters is large tens or more this paper first shows that the spkmeans algorithm can be derived from a certain maximum likelihood formulation using a mixture of von mises fisher distributions as the generative model and in fact it can be considered as a batch mode version of normalized competitive learning the proposed generative model is then adapted in a principled way to yield three frequency sensitive competitive learning variants that are applicable to static data and produced high quality and well balanced clusters for high dimensional data like kmeans each iteration is linear in the number of data points and in the number of clusters for all the three algorithms a frequency sensitive algorithm to cluster streaming data is also proposed experimental results on clustering of high dimensional text data sets are provided to show the effectiveness and applicability of the proposed techniques

projection pursuit density estimation abstract the projection pursuit methodology is applied to the multivariate density estimation problem the resulting nonparametric procedure is often less biased than the kernel and near neighbor methods in addition graphical information is produced that can be used to help gain geometric insight into the multivariate data distribution

principal surfaces from unsupervised kernel regression we propose a nonparametric approach to learning of principal surfaces based on an unsupervised formulation of the nadaraya watson kernel regression estimator as compared with previous approaches to principal curves and surfaces the new method offers several advantages first it provides a practical solution to the model selection problem because all parameters can be estimated by leave one out cross validation without additional computational cost in addition our approach allows for a convenient incorporation of nonlinear spectral methods for parameter initialization beyond classical initializations based on linear pca furthermore it shows a simple way to fit principal surfaces in general feature spaces beyond the usual data space setup the experimental results illustrate these convenient features on simulated and real data

quasiconformal kernel common locality discriminant analysis with application to breast cancer diagnosis dimensionality reduction dr is a popular method in recognition and classification in many areas such as facial and medical imaging in this paper we propose a novel supervised dr method namely quasiconformal kernel common locality discriminant analysis qkclda qkclda preserves the local and discriminative relationships of the data moreover it adjusts the kernel structure according to the distribution of the input data and thus possesses a classification advantage over traditional kernel based methods in qkclda the parameter of the quasiconformal kernel is automatically calculated through optimizing an objective function of maximizing the class discriminative ability qkclda is employed in breast cancer diagnoses and some experiments using wisconsin diagnostic breast cancer wdbc and mini mias databases have tested its feasibility and performance in assigning these diagnoses

confidence regions for evolutionary trajectories we derive confidence regions for the evolutionary trajectories derived from a model of via and lande 1985 evolution 39 505 522 we utilize a nested parametric bootstrap to calculate the confidence regions and show that a likelihood based approach provides an unsatisfactory solution we calculate the predicted evolutionary trajectories and confidence regions using data from an experiment on pupal weight of tribolium beetles in two different flour types with realistic sample sizes for experiments that estimate genetic parameters in multiple environments the confidence regions are likely to be large potentially limiting the ability to test hypotheses about evolutionary trajectories the problem may be less acute for trajectories of characters expressed in only a single environment because sample sizes can generally be larger than in multienvironmental studies

on a notion of data depth based on random simplices for a distribution f on mathbb r p and a point x in mathbb r p the simplical depth d x is introduced which is the probability that the point x is contained inside a random simplex whose vertices are p 1 independent observations from f mathematically and heuristically it is argued that d x indeed can be viewed as a measure of depth of the point x with respect to f an empirical version of d cdot gives rise to a natural ordering of the data points from the center outward the ordering thus obtained leads to the introduction of multivariate generalizations of the univariate sample median and l statistics this generalized sample median and l statistics are affine equivariant

exploratory projection pursuit abstract a new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods a number of practical issues concerning its application are addressed a connection to multivariate density estimation is established and its properties are investigated through simulation studies and application to real data the goal of exploratory projection pursuit is to use the data to find low one two or three dimensional projections that provide the most revealing views of the full dimensional data with these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance since linear effects are directly captured by the covariance structure of the variable pairs which are straightforward to estimate the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables although arbitrary

mixtures of probabilistic principal component analyzers principal component analysis pca is one of the most popular techniques for processing compressing and visualizing data although its effectiveness is limited by its global linearity while nonlinear variants of pca have been proposed an alternative paradigm is to capture data complexity by a combination of local linear pca projections however conventional pca does not correspond to a probability density and so there is no unique way to combine pca models therefore previous attempts to formulate mixture models for pca have been ad hoc to some extent in this article pca is formulated within a maximum likelihood framework based on a specific form of gaussian latent variable model this leads to a well defined mixture model for probabilistic principal component analyzers whose parameters can be determined using an expectationmaximization algorithm we discuss the advantages of this model in the context of clustering density modeling and local dimensionality reduction and we demonstrate its application to image compression and handwritten digit recognition

extremal properties of principal curves in the plane principal curves were introduced to formalize the notion of a curve passing through the middle of a dataset vaguely speaking a curve is said to pass through the middle of a dataset if every point on the curve is the average of the observations projecting onto it this idea can be made precise by defining principal curves for probability densities in this paper we study principal curves in the plane like linear principal components principal curves are critical points of the expected squared distance from the data however the largest and smallest principal components are extrema of the distance whereas all principal curves are saddle points this explains why cross validation does not appear to be a viable method for choosing the complexity of principal curve estimates

active curve axis gaussian mixture models gaussian mixture models gmm have been broadly applied for the fitting of probability density function however due to the intrinsic linearity of gmm usually many components are needed to appropriately fit the data distribution when there are curve manifolds in the data cloud in order to solve this problem and represent data with curve manifolds better in this paper we propose a new nonlinear probability model called active curve axis gaussian model intuitively this model can be imagined as gaussian model being bent at the first principal axis for estimating parameters of mixtures of this model the em algorithm is employed experiments on synthetic data and chinese characters show that the proposed nonlinear mixture models can approximate distributions of data clouds with curve manifolds in a more concise and compact way than gmm does the performance of the proposed nonlinear mixture models is promising

nonextensive statistical features of the polish stock market fluctuations the statistics of return distributions on various time scales constitutes one of the most informative characteristics of the financial dynamics here we present a systematic study of such characteristics for the polish stock market index wig20 over the period 04 01 1999 31 10 2005 for the time lags ranging from 1min1min up to 1 h this market is commonly classified as emerging still on the shortest time scales studied we find that the tails of the return distributions are consistent with the inverse cubic power law as identified previously for majority of the mature markets within the time scales studied a quick and considerable departure from this law towards a gaussian can however be traced interestingly all the forms of the distributions observed can be comprised by the single q gaussians which provide a satisfactory and at the same time compact representation of the distribution of return fluctuations over all magnitudes of their variation the corresponding nonextensivity parameter q was found to systematically decrease when increasing the time scales the temporal correlations quantified here in terms of multifractality provide further arguments in favor of nonextensivity

prepivoting to reduce level error of confidence sets summary approximate confidence sets for a parameter 0 may be obtained by referring a function of 0 and of the sample to an estimated quantile of that function s sampling distribution we call this function the root of the confidence set either asymptotic theory or bootstrap methods can be used to estimate the desired quantile when the root is not a pivot in the sense of classical statistics the actual level of the approximate confidence set may differ substantially from the intended level prepivoting is the transformation of a confidence set root by its estimated bootstrap cumulative distribution function prepivoting can be iterated bootstrap confidence sets generated from a root prepivoted one or more times have smaller error in level than do confidence sets based on the original root the first prepivoting is nearly equivalent to studentizing when that operation is appropriate further iterations of prepivoting make higher order corrections automatically

locally defined principal curves and surfaces principal curves are defined as self consistent smooth curves passing through the middle of the data and they have been used in many applications of machine learning as a generalization dimensionality reduction and a feature extraction tool we redefine principal curves and surfaces in terms of the gradient and the hessian of the probability density estimate this provides a geometric understanding of the principal curves and surfaces as well as a unifying view for clustering principal curve fitting and manifold learning by regarding those as principal manifolds of different intrinsic dimensionalities the theory does not impose any particular density estimation method can be used with any density estimator that gives continuous first and second derivatives therefore we first present our principal curve surface definition without assuming any particular density estimation method afterwards we develop practical algorithms for the commonly used kernel density estimation kde and gaussian mixture models gmm results of these algorithms are presented in notional data sets as well as real applications with comparisons to other approaches in the principal curve literature all in all we present a novel theoretical understanding of principal curves and surfaces practical algorithms as general purpose machine learning tools and applications of these algorithms to several practical problems

locally weighted regression an approach to regression analysis by local fitting abstract locally weighted regression or loess is a way of estimating a regression surface through a multivariate smoothing procedure fitting a function of the independent variables locally and in a moving fashion analogous to how a moving average is computed for a time series with local fitting we can estimate a much wider class of regression surfaces than with the usual classes of parametric functions such as polynomials the goal of this article is to show through applications how loess can be used for three purposes data exploration diagnostic checking of parametric models and providing a nonparametric regression surface along the way the following methodology is introduced a a multivariate smoothing procedure that is an extension of univariate locally weighted regression b statistical procedures that are analogous to those used in the least squares fitting of parametric functions c several graphical methods that are useful tools for understanding loess estimates and checking the a

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

a unified model for probabilistic principal surfaces principal curves and surfaces are nonlinear generalizations of principal components and subspaces respectively they can provide insightful summary of high dimensional data not typically attainable by classical linear methods solutions to several problems such as proof of existence and convergence faced by the original principal curve formulation have been proposed in the past few years nevertheless these solutions are not generally extensible to principal surfaces the mere computation of which presents a formidable obstacle consequently relatively few studies of principal surfaces are available we previously 2000 proposed the probabilistic principal surface pps to address a number of issues associated with current principal surface algorithms pps uses a manifold oriented covariance noise model based on the generative topographical mapping gtm which can be viewed as a parametric formulation of kohonen s self organizing map building on the pps we introduce a unified covariance model that implements pps 0 1 by varying the clamping parameter spl alpha then we comprehensively evaluate the empirical performance of pps gtm and the manifold aligned gtm on three popular benchmark data sets it is shown in two different comparisons that the pps outperforms the gtm under identical parameter settings convergence of the pps is found to be identical to that of the gtm and the computational overhead incurred by the pps decreases to 40 percent or less for more complex manifolds these results show that the generalized pps provides a flexible and effective way of obtaining principal surfaces

regularization free principal curve estimation principal curves and manifolds provide a framework to formulate manifold learning within a statistical context principal curves define the notion of a curve passing through the middle of a distribution while the intuition is clear the formal definition leads to some technical and practical difficulties in particular principal curves are saddle points of the mean squared projection distance which poses severe challenges for estimation and model selection this paper demonstrates that the difficulties in model selection associated with the saddle point property of principal curves are intrinsically tied to the minimization of the mean squared projection distance we introduce a new objective function facilitated through a modification of the principal curve estimation approach for which all critical points are principal curves and minima thus the new formulation removes the fundamental issue for model selection in principal curve estimation a gradient descent based estimator demonstrates the effectiveness of the new formulation for controlling model complexity on numerical experiments with synthetic and real data

scaling of the distribution of fluctuations of financial market indices up to more than one month we find that the distributions for dt 4d 1560 min are consistent with a power law asymptotic behavior characterized by an exponent a 3 well outside the stable levy regime 0 a 2 to test the robustness of the s p result we perform a parallel analysis on two other financial market indices database iv contains 3560 daily records of the nikkei index for the 14 year period 1984 1997 and database v contains 4649 daily records of the hang seng index for the 18 year period 1980 1997 we find estimates ofa consistent with those describing the distribution of s p 500 daily returns one possible reason for the scaling of these distributions is the long persistence of the autocorrelation function of the volatility for time scales longer than dt 3 4 d our results are consistent with a slow convergence to gaussian behavior s1063 651x 99 11211 x pacs number s 05 40 fb 05 45 tp 89 90 1n

blind source separation by sparse decomposition in a signal dictionary the blind source separation problem is to extract the underlying source signals from a set of linear mixtures where the mixing matrix is unknown this situation is common in acoustics radio medical signal and image processing hyperspectral imaging and other areas we suggest a two stage separation process a priori selection of a possibly overcomplete signal dictionary for instance a wavelet frame or a learned dictionary in which the sources are assumed to be sparsely representable followed by unmixing the sources by exploiting the their sparse representability we consider the general case of more sources than mixtures but also derive a more efficient algorithm in the case of a nonovercomplete dictionary and an equal numbers of sources and mixtures experiments with artificial signals and musical sounds demonstrate significantly better separation than other known techniques

limitations of nonlinear pca as performed with generic neural networks kramer s 1991 nonlinear principal components analysis nlpca neural networks are feedforward autoassociative networks with five layers the third layer has fewer nodes than the input or output layers this paper proposes a geometric interpretation for kramer s method by showing that nlpca fits a lower dimensional curve or surface through the training data the first three layers project observations onto the curve or surface giving scores the last three layers define the curve or surface the first three layers are a continuous function which we show has several implications nlpca projections are suboptimal producing larger approximation error nlpca is unable to model curves and surfaces that intersect themselves and nlpca cannot parameterize curves with parameterizations having discontinuous jumps we establish results on the identification of score values and discuss their implications on interpreting score values we discuss the relationship between nlpca and principal curves and surfaces another nonlinear feature extraction method

independent factor analysis we introduce the independent factor analysis ifa method for recovering independent hidden sources from their observed mixtures ifa generalizes and unifies ordinary factor analysis fa principal component analysis pca and independent component analysis ica and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy ifa is a two step procedure in the first step the source densities mixing matrix and noise covariance are estimated from the observed data by maximum likelihood for this purpose we present an expectation maximization em algorithm which performs unsupervised learning of an associated probabilistic model of the mixing situation each source in our model is described by a mixture of gaussians thus all the probabilistic calculations can be performed analytically in the second step the sources are reconstructed from the observed data by an optimal nonlinear estimator a variational approximation of this algorithm is derived for cases with a large number of sources where the exact algorithm becomes intractable our ifa algorithm reduces to the one for ordinary fa when the sources become gaussian and to an em algorithm for pca in the zero noise limit we derive an additional em algorithm specifically for noiseless ifa this algorithm is shown to be superior to ica since it can learn arbitrary source densities from the data beyond blind separation ifa can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding

robust estimation of dispersion matrices and principal components abstract this paper uses monte carlo methods to compare the performances of several robust procedures for estimating a correlation matrix and its principal components the estimators are formed either from separate bivariate analyses or by simultaneous manipulation of all variables by using techniques such as multivariate trimming and m estimation the m estimators stand up exceptionally well they and the multivariate trimming procedure are especially effective at estimating the principal components including a near singularity however the m estimators can break down relatively easily when the dimensionality is large and the outliers are asymmetric with missing data the element wise approach becomes more attractive

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

adaptive nonlinear manifolds and their applications to pattern recognition dimensionality reduction has long been associated with retinotopic mapping for understanding cortical maps multisensory information is processed fused and mapped to an essentially 2 d cortex in an information preserving manner data processing and projection techniques inspired by this biological mechanism are playing an increasingly important role in pattern recognition computational intelligence data mining information retrieval and image recognition dimensionality reduction involves reduction of features or volume of data and has become an essential step of information processing in many fields the topic of manifold learning has recently attracted a great deal of attention and a number of advanced techniques for extracting nonlinear manifolds and reducing data dimensions have been proposed from statistics geometry theory and adaptive neural networks this paper provides an overview of this challenging and emerging topic and discusses various recent methods such as self organizing map som kernel pca principal manifold isomap local linear embedding and laplacian eigenmap many of them can be considered in a learning manifold framework the paper further elaborates on the biologically inspired som model and its metric preserving variant visom under the framework of adaptive manifold and their applications in dimensionality reduction with face recognition are investigated the experiments demonstrate that adaptive visom based methods produce markedly improved performance over the others due to their metric scaling and preserving properties along the nonlinear manifold

kernel class wise locality preserving projection in the recent years the pattern recognition community paid more attention to a new kind of feature extraction method the manifold learning methods which attempt to project the original data into a lower dimensional feature space by preserving the local neighborhood structure among them locality preserving projection lpp is one of the most promising feature extraction techniques however when lpp is applied to the classification tasks it shows some limitations such as the ignorance of the label information in this paper we propose a novel local structure based feature extraction method called class wise locality preserving projection clpp clpp utilizes class information to guide the procedure of feature extraction in clpp the local structure of the original data is constructed according to a certain kind of similarity between data points which takes special consideration of both the local information and the class information the kernelized nonlinear counterpart of this linear feature extractor is also established in the paper moreover a kernel version of clpp namely kernel clpp kclpp is developed through applying the kernel trick to clpp to increase its performance on nonlinear feature extraction experiments on orl face database and yale face database are performed to test and evaluate the proposed algorithm

power law classification scheme of time series correlations on the example of g20 group abstract a power law classification scheme plcs of time series correlations is proposed it is shown that plcs provides the ability to classify nonlinear correlations and measure their stability plcs has been applied to gross domestic product gdp per capita of g20 members and their correlations analysed it has been shown that the method does not only recognise linear correlations properly but also allows to point out converging time series as well as to distinguish nonlinear correlations plcs is capable of crash recognition as it is shown in the argentina example finally the strength of correlations and the stability of correlation matrices have been used to construct a minimum spanning tree mst the results were compared with those based on the ultrametric distance ud comparing the structures of mst ud and plcs indicates that the latter one is more complicated but better fits the expected economic relations within the g20

prepivoting test statistics a bootstrap view of asymptotic refinements abstract approximate tests for a composite null hypothesis about a parameter may be obtained by referring a test statistic to an estimated critical value either asymptotic theory or bootstrap methods can be used to estimate the desired quantile the simple asymptotic test a refers the test statistic to a quantile of its asymptotic null distribution after unknown parameters have been estimated the bootstrap approach used here is based on the concept of prepivoting prepivoting is the transformation of a test statistic by the cdf of its bootstrap null distribution the simple bootstrap test b refers the prepivoted test statistic to a quantile of the uniform 0 1 distribution under regularity conditions the bootstrap test b has a smaller asymptotic order of error in level than does the asymptotic test a provided that the asymptotic null distribution of the test statistic does not depend on unknown parameters in the contrary case both a and b have the same order of level error certain class

on the bootstrap and likelihood based confidence regions summary we describe a method for constructing likelihood based confidence regions for a vector parameter using the bootstrap and nonparametric density estimation the technique is illustrated by application to a numerical example and its theoretical properties are elucidated it is argued that likelihood based regions should not be approximated by ellipses if we are to have any hope of capturing first order departures from normality bootstrap algorithms for constructing simultaneous confidence intervals for the components of a vector parameter are also presented advantages of the percentile t method over the ordinary percentile method are demonstrated in a multivariate setting

adaptive principal surfaces abstract we develop a nonlinear generalization of principal components analysis a principal surface of the data is constructed adaptively using some ideas from the mars procedure of friedman we explore applications to curve and surface reconstruction and to data summarization