the helmholtz machine discovering the structure inherent in a set of patterns is a fundamental aim of statistical inference or learning one fruitful approach is to build a parameterized stochastic generative model independent draws from which are likely to produce the patterns for all but the simplest generative models each pattern can be generated in exponentially many ways it is thus intractable to adjust the parameters to maximize the probability of the observed patterns we describe a way of finessing this combinatorial explosion by maximizing an easily computed lower bound on the probability of the observations our method can be viewed as a form of hierarchical self supervised learning that may relate to the function of bottom up and top down cortical processing pathways

learning gaussian mixture models with entropy based criteria in this paper we address the problem of estimating the parameters of gaussian mixture models although the expectation maximization em algorithm yields the maximum likelihood ml solution its sensitivity to the selection of the starting parameters is well known and it may converge to the boundary of the parameter space furthermore the resulting mixture depends on the number of selected components but the optimal number of kernels may be unknown beforehand we introduce the use of the entropy of the probability density function pdf associated to each kernel to measure the quality of a given mixture model with a fixed number of kernels we propose two methods to approximate the entropy of each kernel and a modification of the classical em algorithm in order to find the optimum number of components of the mixture moreover we use two stopping criteria a novel global mixture entropy based criterion called gaussianity deficiency gd and a minimum description length mdl principle based one our algorithm called entropy based em ebem starts with a unique kernel and performs only splitting by selecting the worst kernel attending to gd we have successfully tested it in probability density estimation pattern classification and color image segmentation experimental results improve the ones of other state of the art model order selection methods

learning nonlinear image manifolds by global alignment of local linear models appearance based methods based on statistical models of the pixel values in an image region rather than geometrical object models are increasingly popular in computer vision in many applications the number of degrees of freedom dof in the image generating process is much lower than the number of pixels in the image if there is a smooth function that maps the dof to the pixel values then the images are confined to a low dimensional manifold embedded in the image space we propose a method based on probabilistic mixtures of factor analyzers to 1 model the density of images sampled from such manifolds and 2 recover global parameterizations of the manifold a globally nonlinear probabilistic two way mapping between coordinates on the manifold and images is obtained by combining several locally valid linear mappings we propose a parameter estimation scheme that improves upon an existing scheme and experimentally compare the presented approach to self organizing maps generative topographic mapping and mixtures of factor analyzers in addition we show that the approach also applies to finding mappings between different embeddings of the same manifold

efficient recursive algorithms for detection of abrupt changes in signals and control systems this paper addresses a number of open problems concerning the generalized likelihood ratio glr rules for online detection of faults and parameter changes in control systems it is shown that with an appropriate choice of the threshold and window size these glr rules are asymptotically optimal the rules are also extended to non likelihood statistics that are widely used in monitoring adaptive algorithms for system identification and control by establishing gaussian approximations to these statistics when the window size is chosen suitably recursive algorithms are developed for practical implementation of the procedure and importance sampling techniques are introduced for determining the threshold of the rule to satisfy prescribed bounds on the false alarm rate

information theoretic matching of two point sets this paper describes the theoretic roadmap of least relative entropy matching of two point sets the novel feature is to align two point sets without needing to establish explicit point correspondences the recovery of transformational geometry is achieved using a mixture of principal axes registrations whose parameters are estimated by minimizing the relative entropy between the two point distributions and using the expectation maximization algorithm we give evidence of the optimality of the method and we then evaluate the algorithm s performance in both rigid and nonrigid image registration cases

tail behaviour for suprema of empirical processes abstract this document considers multi variate empirical processes with an empirical distribution function based on i i d variables with certain distribution functions

linear fuzzy clustering with selection of variables using graded possibilistic approach linear fuzzy clustering is a useful tool for knowledge discovery in databases kdd and several modifications have been proposed in order to analyze real world data this paper proposes a new approach for estimating local linear models in which linear fuzzy clustering is performed by selecting variables that are useful for extracting correlation structure in each cluster the new clustering model uses two types of memberships one is the conventional membership that represents the degree of membership of each sample in each cluster the other is the additional parameter that represents the relative responsibility of each variable for estimation of local linear models the additional membership takes large values when the variable has close relationship with local principal components and is calculated by using the graded possibilistic approach numerical experiments demonstrate that the proposed method is useful for identifying local linear model taking typicality of each variable into account

compressive sensing on manifolds using a nonparametric mixture of factor analyzers algorithm and performance bounds nonparametric bayesian methods are employed to constitute a mixture of low rank gaussians for data x rn that are of high dimension n but are constrained to reside in a low dimensional subregion of rn the number of mixture components and their rank are inferred automatically from the data the resulting algorithm can be used for learning manifolds and for reconstructing signals from manifolds based on compressive sensing cs projection measurements the statistical cs inversion is performed analytically we derive the required number of cs random measurements needed for successful reconstruction based on easily computed quantities drawing on block sparsity properties the proposed methodology is validated on several synthetic and real datasets

class specific feature polynomial classifier for pattern classification and its application to handwritten numeral recognition the polynomial classifier pc that takes the binomial terms of reduced subspace features as inputs has shown superior performance to multilayer neural networks in pattern classification in this paper we propose a class specific feature polynomial classifier cfpc that extracts class specific features from class specific subspaces unlike the ordinary pc that uses a class independent subspace the cfpc can be viewed as a hybrid of ordinary pc and projection distance method the class specific features better separate one class from the others and the incorporation of class specific projection distance further improves the separability the connecting weights of cfpc are efficiently learned class by class to minimize the mean square error on training samples to justify the promise of cfpc we have conducted experiments of handwritten digit recognition and numeral string recognition on the nist special database 19 sd19 the digit recognition task was also benchmarked on two standard databases usps and mnist the results show that the performance of cfpc is superior to that of ordinary pc and is competitive with support vector classifiers svcs

a nonlinear neural network model of mixture of local principal component analysis application to handwritten digits recognition abstract principal component analysis pca is a popular tool in multivariate statistics and pattern recognition recently some mixture models of local principal component analysis have attracted attention due to a number of benefits over global pca in this paper we propose a mixture model by concurrently performing global data partition and local linear pca the partition is optimal or near optimal which is realized by a soft competition algorithm called neural gas the local pca type representation is approximated by a neural learning algorithm in a nonlinear autoencoder network which is set up on the generalization of the least squares reconstruction problem leading to the standard pca such a local pca type representation has a number of numerical advantages for example faster convergence and insensitive to local minima based on this mixture model we describe a modular classification scheme to solve the problem of handwritten digits recognition we use 10 networks modules to capture different features in the 10 classes of handwritten digits with each network being a mixture model of local pca type representations when a test digit is presented to all the modules each module provides a reconstructed pattern by a prescribed principle and the system outputs the class label by comparing the reconstruction errors from the 10 networks compared with some traditional neural network based classifiers our scheme converges faster and recognizes with higher accuracy for a relatively small size of each module the classification accuracy reaches 98 6 on the training set and 97 8 on the testing set

an unbiased parametric imaging algorithm for nonuniformly sampled biomedical system parameter estimation an unbiased algorithm of generalized linear least squares glls for parameter estimation of nonuniformly sampled biomedical systems is proposed the basic theory and detailed derivation of the algorithm are given this algorithm removes the initial values required and computational burden of nonlinear least regression and achieves a comparable estimation quality in terms of the estimates bias and standard deviation therefore this algorithm is particular useful in image wide pixel by pixel based parameter estimation e g to generate parametric images from tracer dynamic studies with positron emission tomography an example is presented to demonstrate the performance of this new technique this algorithm is also generally applicable to other continuous system parameter estimation

maxima of asymptotically gaussian random fields and moderate deviation approximations to boundary crossing probabilities of sums of random variables with multidimensional indices several classical results on boundary crossing probabilities of brownian motion and random walks are extended to asymptotically gaussian random fields which include sums of i i d random variables with multidimensional indices multivariate empirical processes and scan statistics in change point and signal detection as special cases some key ingredients in these extensions are moderate deviation approximations to marginal tail probabilities and weak convergence of the conditional distributions of certain clumps around high level crossings we also discuss how these results are related to the poisson clumping heuristic and tube formulas of gaussian random fields and describe their applications to laws of the iterated logarithm in the form of the kolmogorov erdos feller integral tests

competitive em algorithm for finite mixture models abstract in this paper we present a novel competitive em cem algorithm for finite mixture models to overcome the two main drawbacks of the em algorithm often getting trapped at local maxima and sometimes converging to the boundary of the parameter space the proposed algorithm is capable of automatically choosing the clustering number and selecting the split or merge operations efficiently based on the new competitive mechanism we propose it is insensitive to the initial configuration of the mixture component number and model parameters experiments on synthetic data show that our algorithm has very promising performance for the parameter estimation of mixture models the algorithm is also applied to the structure analysis of complicated chinese characters the results show that the proposed algorithm performs much better than previous methods with slightly heavier computation burden

unsupervised learning of finite mixture models this paper proposes an unsupervised algorithm for learning a finite mixture model from multivariate data the adjective unsupervised is justified by two properties of the algorithm 1 it is capable of selecting the number of components and 2 unlike the standard expectation maximization em algorithm it does not require careful initialization the proposed method also avoids another drawback of em for mixture fitting the possibility of convergence toward a singular estimate at the boundary of the parameter space the novelty of our approach is that we do not use a model selection criterion to choose one among a set of preestimated candidate models instead we seamlessly integrate estimation and model selection in a single algorithm our technique can be applied to any type of parametric mixture model for which it is possible to write an em algorithm in this paper we illustrate it with experiments involving gaussian mixtures these experiments testify for the good performance of our approach

projection pursuit mixture density estimation in this paper we seek a gaussian mixture model gmm of an n variate probability density function usually the parameters of gmms are determined in the original n dimensional space by optimizing a maximum likelihood ml criterion a practical deficiency of this method of fitting gmms is its poor performance when dealing with high dimensional data since a large sample size is needed to match the accuracy that is possible in low dimensions we propose a method for fitting the gmm based on the projection pursuit strategy this gmm is highly constrained and hence its ability to model structure in subspaces is enhanced compared to a direct ml fitting of a gmm in high dimensions our method is closely related to recently developed independent factor analysis ifa mixture models the comparisons with ml fitting of gmm in n dimensions and ifa mixtures show that the proposed method is an attractive choice for fitting gmms using small sizes of training sets

invariance principles for sums of banach space valued random elements and empirical processes almost sure and probability invariance principles are established for sums of independent not necessarily measurable random elements with values in a not necessarily separable banach space it is then shown that empirical processes readily fit into this general framework thus we bypass the problems of measurability and topology characteristic for the previous theory of weak convergence of empirical processes

discriminant adaptive nearest neighbor classification nearest neighbour classification expects the class conditional probabilities to be locally constant and suffers from bias in high dimensions we propose a locally adaptive form of nearest neighbour classification to try to ameliorate this curse of dimensionality we use a local linear discriminant analysis to estimate an effective metric for computing neighbourhoods we determine the local decision boundaries from centroid information and then shrink neighbourhoods in directions orthogonal to these local decision boundaries and elongate them parallel to the boundaries thereafter any neighbourhood based classifier can be employed using the modified neighbourhoods the posterior probabilities tend to be more homogeneous in the modified neighbourhoods we also propose a method for global dimension reduction that combines local dimension information in a number of examples the methods demonstrate the potential for substantial improvements over nearest neighbour classification

quantifying curvelike structures of measures by using l2 jones quantities we study the curvelike structure of special measures on n in a multiscale fashion more precisely we consider the existence and construction of a sufficiently short curve with a sufficiently large measure our main tool is an l2 variant of jones numbers which measure the scaled deviations of the given measure from a best approximating line at different scales and locations the jones function is formed by adding the squares of the l2 jones numbers at different scales and the same location using a special l2 jones function we construct a sufficiently short curve with a sufficiently large measure the length and measure estimates of the underlying curve are expressed in terms of the size of this jones function 2003 wiley periodicals inc

mode finding for mixtures of gaussian distributions gradient quadratic and fixed point iteration algorithms and appropriate values for their control parameters are derived for finding all modes of a gaussian mixture a problem with applications in clustering and regression the significance of the modes found is quantified locally by hessian based error bars and globally by the entropy as sparseness measure

a unifying review of linear gaussian models factor analysis principal component analysis mixtures of gaussian clusters vector quantization kalman filter models and hidden markov models can all be unified as variations of unsupervised learning under a single basic generative model this is achieved by collecting together disparate observations and derivations made by many previous authors and introducing a new way of linking discrete and continuous state models using a simple nonlinearity through the use of other nonlinearities we show how independent component analysis is also a variation of the same basic generative model we show that factor analysis and mixtures of gaussians can be implemented in autoencoder neural networks and learned using squared error plus the same regularization term we introduce a new model for static data known as sensible principal component analysis as well as a novel concept of spatially adaptive observation noise we also review some of the literature involving global and local mixtures of the basic models and provide pseudocode for inference and learning for all the basic models

bayesian analysis of mixtures of factor analyzers for bayesian inference on the mixture of factor analyzers natural conjugate priors on the parameters are introduced and then a gibbs sampler that generates parameter samples following the posterior is constructed in addition a deterministic estimation algorithm is derived by taking modes instead of samples from the conditional posteriors used in the gibbs sampler this is regarded as a maximum a posteriori estimation algorithm with hyperparameter search the behaviors of the gibbs sampler and the deterministic algorithm are compared on a simulation experiment

fast transformation invariant component analysis dimensionality reduction techniques such as principal component analysis and factor analysis are used to discover a linear mapping between high dimensional data samples and points in a lower dimensional subspace previously frey and jojic introduced transformation invariant component analysis tca to learn a linear mapping invariant to a set of known form of global transformations however parameter estimation in that model using the previously proposed expectation maximization em algorithm required scalar operations in the order of n2 where n is the dimensionality of each training example this is prohibitive for many applications of interest such as modeling mid to large size images where for instance i n may be as high as 786432 512 512 rgb image in this paper we present an efficient algorithm that reduces the computational requirements to order of nlogn with this speedup we show the effectiveness of transformation invariant component analysis in various applications including tracking learning video textures clustering object recognition and object detection in images software for tca can be downloaded from http www psi toronto edu fasttca htm

mixtures of factor analyzers with common factor loadings applications to the clustering and visualization of high dimensional data mixtures of factor analyzers enable model based density estimation to be undertaken for high dimensional data where the number of observations n is not very large relative to their dimension p in practice there is often the need to further reduce the number of parameters in the specification of the component covariance matrices to this end we propose the use of common component factor loadings which considerably reduces further the number of parameters moreover it allows the data to be displayed in low dimensional plots

fuzzy pca guided robust k means clustering this paper proposes a new approach to robust clustering in which a robust k means partition is derived by using a noise rejection mechanism based on the noise clustering approach the responsibility weight of each sample for the k means process is estimated by considering the noise degree of the sample and cluster indicators are calculated in a fuzzy principal component analysis pca guided manner where fuzzy pca guided robust k means is performed by considering responsibility weights of samples then the proposed method achieves cluster core estimation in a deterministic way the validity of the derived cluster cores is visually assessed through distance sensitive ordering which considers responsibility weights of samples numerical experiments demonstrate that the proposed method is useful for capturing cluster cores by rejecting noise samples and we can easily assess cluster validity by using cluster crossing curves

a model based approach to simultaneous clustering and dimensional reduction of ordinal data the literature on clustering for continuous data is rich and wide differently that one developed for categorical data is still limited in some cases the clustering problem is made more difficult by the presence of noise variables dimensions that do not contain information about the clustering structure and could mask it the aim of this paper is to propose a model for simultaneous clustering and dimensionality reduction of ordered categorical data able to detect the discriminative dimensions discarding the noise ones following the underlying response variable approach the observed variables are considered as a discretization of underlying first order latent continuous variables distributed as a gaussian mixture to recognize discriminative and noise dimensions these variables are considered to be linear combinations of two independent sets of second order latent variables where only one contains the information about the cluster structure while the other one contains noise dimensions the model specification involves multidimensional integrals that make the maximum likelihood estimation cumbersome and in some cases infeasible to overcome this issue the parameter estimation is carried out through an em like algorithm maximizing a composite log likelihood based on low dimensional margins examples of application of the proposal on real and simulated data are performed to show the effectiveness of the proposal

images frames and connectionist hierarchies the representation of hierarchically structured knowledge in systems using distributed patterns of activity is an abiding concern for the connectionist solution of cognitively rich problems here we use statistical unsupervised learning to consider semantic aspects of structured knowledge representation we meld unsupervised learning notions formulated for multilinear models with tensor product ideas for representing rich information we apply the model to images of faces

mixtures of probabilistic principal component analyzers principal component analysis pca is one of the most popular techniques for processing compressing and visualizing data although its effectiveness is limited by its global linearity while nonlinear variants of pca have been proposed an alternative paradigm is to capture data complexity by a combination of local linear pca projections however conventional pca does not correspond to a probability density and so there is no unique way to combine pca models therefore previous attempts to formulate mixture models for pca have been ad hoc to some extent in this article pca is formulated within a maximum likelihood framework based on a specific form of gaussian latent variable model this leads to a well defined mixture model for probabilistic principal component analyzers whose parameters can be determined using an expectationmaximization algorithm we discuss the advantages of this model in the context of clustering density modeling and local dimensionality reduction and we demonstrate its application to image compression and handwritten digit recognition

merging and splitting eigenspace models we present new deterministic methods that given two eigenspace models each representing a set of n dimensional observations will 1 merge the models to yield a representation of the union of the sets and 2 split one model from another to represent the difference between the sets as this is done we accurately keep track of the mean here we give a theoretical derivation of the methods empirical results relating to the efficiency and accuracy of the techniques and three general applications including the construction of gaussian mixture models that are dynamically updateable

incremental nonlinear dimensionality reduction by manifold learning understanding the structure of multidimensional patterns especially in unsupervised cases is of fundamental importance in data mining pattern recognition and machine learning several algorithms have been proposed to analyze the structure of high dimensional data based on the notion of manifold learning these algorithms have been used to extract the intrinsic characteristics of different types of high dimensional data by performing nonlinear dimensionality reduction most of these algorithms operate in a batch mode and cannot be efficiently applied when data are collected sequentially in this paper we describe an incremental version of isomap one of the key manifold learning algorithms our experiments on synthetic data as well as real world images demonstrate that our modified algorithm can maintain an accurate low dimensional representation of the data in an efficient manner

intrinsic dimension estimation via nearest constrained subspace classifier we consider the problems of classification and intrinsic dimension estimation on image data a new subspace based classifier is proposed for supervised classification or intrinsic dimension estimation the distribution of the data in each class is modeled by a union of a finite number of affine subspaces of the feature space the affine subspaces have a common dimension which is assumed to be much less than the dimension of the feature space the subspaces are found using regression based on the 0 norm the proposed method is a generalisation of classical nn nearest neighbor nfl nearest feature line classifiers and has a close relationship to ns nearest subspace classifier the proposed classifier with an accurately estimated dimension parameter generally outperforms its competitors in terms of classification accuracy we also propose a fast version of the classifier using a neighborhood representation to reduce its computational complexity experiments on publicly available datasets corroborate these claims highlightsa new classifier called ncsc and its fast version called ncsc ii are proposed classifiers of nn nfl and ns are the special cases of ncsc ncsc ncsc ii can be used as intrinsic dimension estimator using well tuned intrinsic parameter ncsc ncsc ii outperforms nn nfl and ns

a re weighting strategy for improving margins we present a simple general scheme for improving margins that is inspired on well known margin theory principles the scheme is based on a sample re weighting strategy the very basic idea is in fact to add to the training set new replicas of samples which are not classified with a sufficient margin as a study case we present a new algorithm namely tvq which is an instance of the proposed scheme and involves a tangent distance based 1 nn classifier implementing a sort of quantization of the tangent distance prototypes the tangent distance models created in this way have shown a significant improvement in generalization power with respect to standard tangent models moreover the obtained models were able to outperform other state of the art algorithms such as svm in an ocr task

learning data manifolds with a cutting plane method we consider the problem of classifying data manifolds where each manifold represents invariances that are parameterized by continuous degrees of freedom conventional data augmentation methods rely

em algorithms for ml factor analysis the details of em algorithms for maximum likelihood factor analysis are presented for both the exploratory and confirmatory models the algorithm is essentially the same for both cases and involves only simple least squares regression operations the largest matrix inversion required is for aq q symmetric matrix whereq is the matrix of factors the example that is used demonstrates that the likelihood for the factor analysis model may have multiple modes that are not simply rotations of each other such behavior should concern users of maximum likelihood factor analysis and certainly should cast doubt on the general utility of second derivatives of the log likelihood as measures of precision of estimation

laplacian embedded regression for scalable manifold regularization semi supervised learning ssl as a powerful tool to learn from a limited number of labeled data and a large number of unlabeled data has been attracting increasing attention in the machine learning community in particular the manifold regularization framework has laid solid theoretical foundations for a large family of ssl algorithms such as laplacian support vector machine lapsvm and laplacian regularized least squares laprls however most of these algorithms are limited to small scale problems due to the high computational cost of the matrix inversion operation involved in the optimization problem in this paper we propose a novel framework called laplacian embedded regression by introducing an intermediate decision variable into the manifold regularization framework by using insensitive loss we obtain the laplacian embedded support vector regression lapesvr algorithm which inherits the sparse solution from svr also we derive laplacian embedded rls laperls corresponding to rls under the proposed framework both lapesvr and laperls posses a simpler form of a transformed kernel which is the summation of the original kernel and a graph kernel that captures the manifold structure the benefits of the transformed kernel are two fold 1 we can deal with the original kernel matrix and the graph laplacian matrix in the graph kernel separately and 2 if the graph laplacian matrix is sparse we only need to perform the inverse operation for a sparse matrix which is much more efficient when compared with that for a dense one inspired by kernel principal component analysis we further propose to project the introduced decision variable into a subspace spanned by a few eigenvectors of the graph laplacian matrix in order to better reflect the data manifold as well as accelerate the calculation of the graph kernel allowing our methods to efficiently and effectively cope with large scale ssl problems extensive experiments on both toy and real world data sets show the effectiveness and scalability of the proposed framework

the diabolo classifier we present a new classification architecture based on autoassociative neural networks that are used to learn discriminant models of each class the proposed architecture has several interesting properties with respect to other model based classifiers like nearest neighbors or radial basis functions it has a low computational complexity and uses a compact distributed representation of the models the classifier is also well suited for the incorporation of a priori knowledge by means of a problemspecific distance measure in particular we will show that tangent distance simard le cun denker 1993 can be used to achieve transformation invariance during learning and recognition we demonstrate the application of this classifier to optical character recognition where it has achieved state of the art results on several reference databases relations to other models in particular those based on principal component analysis are also discussed

on extremal theory for stationary processes let x t t 0 be a stationary stochastic process with one dimensional distribution function g we develop a method to determine an asymptotic expression for pr sup 0 t h x t u when u sup v g v 1 applicable when g belongs to a domain of attraction of extremes and we show that if g belongs to such a domain then so does the distribution function of sup 0 t h x t applications are given to hitting probabilities for small sets for r m valued gaussian processes and to extrema of rayleigh processes we establish the asymptotic poisson character of e upcrossings and local e maxima

first exit densities of brownian motion through one sided moving boundaries let a a e be a sequence of curved boundaries which tend to infinity as a increases let t a inf t 0 w t geqq psi a t where w t denotes the standard brownian motion under regularity conditions on the boundaries uniform approximations for the first exit densities of ta are derived the consequences for upper and lower class functions are discussed the approximations for the first exit densities of brownian motion with drift which are also derived lead to uniform approximations for the power functions of sequential tests the quality of the approximations is demonstrated by some figures

discriminant pattern recognition using transformation invariant neurons to overcome the problem of invariant pattern recognition simard lecun and denker 1993 proposed a successful nearest neighbor approach based on tangent distance attaining state of the art accuracy since this approach needs great computational and memory effort hastie simard and sackinger 1995 proposed an algorithm hss based on singular value decomposition svd for the generation of nondiscriminant tangent models in this article we propose a different approach based on a gradientdescent constructive algorithm called td neuron that develops discriminant models we present as well comparative results of our constructive algorithm versus hss and learning vector quantization lvq algorithms specifically we tested the hss algorithm using both the original version based on the two sided tangent distance and a new version based on the one sided tangent distance empirical results over the nist 3 database show that the td neuron is superior to both svd and lvq based algorithms since it reaches a better trade off between error and rejection

limit theorems for functionals of moving averages let x n i 1 a i e n 1 where the e i are i i d with mean 0 and finite second moment and the a i are either summable or regularly varying with index 1 1 2 the sequence x n has short memory in the former case and long memory in the latter for a large class of functions k a new approach is proposed to develop both central n rate and noncentral non n rate limit theorems for s n n 1 n k x n ek x n specifically we show that in the short memory case the central limit theorem holds for s n and in the long memory case s n can be decomposed into two asymptotically uncorrelated parts that follow a central limit and a non central limit theorem respectively further we write the noncentral part as an expansion of uncorrelated components that follow noncentral limit theorems connections with the usual hermite expansion in the gaussian setting are also explored

face recognition using the second order mixture of eigenfaces method the well known eigenface method uses an eigenface set obtained from principal component analysis however the single eigenface set is not enough to represent the complicated face images with large variations of poses and or illuminations to overcome this weakness we propose a second order mixture of eigenfaces method that combines the second order eigenface method iso mpg m5750 noordwijkerhout march 2000 and the mixture of eigenfaces method a k a gaussian mixture model proceedings ijcnn2001 2001 in this method we use a couple of mixtures of multiple eigenface sets one is a mixture of multiple approximate eigenface sets for face images and another is a mixture of multiple residual eigenface sets for residual face images each mixture of multiple eigenface sets has been obtained from expectation maximization learning consecutively based on two mixture of multiple eigenface sets each face image is represented by a couple of feature vectors obtained by projecting the face image onto a selected approximate eigenface set and then by projecting the residual face image onto a selected residual eigenface set recognition is performed by the distance in the feature space between the input image and the template image stored in the face database simulation results show that the proposed second order mixture of eigenfaces method is best for face images with illumination variations and the mixture of eigenfaces method is best for the face images with pose variations in terms of average of the normalized modified retrieval rank and false identification rate

support vector machine with manifold regularization and partially labeling privacy protection a novel support vector machine with manifold regularization and partially labeling privacy protection termed as svm mr plpp is proposed for semi supervised learning ssl scenarios where only few labeled data and the class proportion of unlabeled data due to privacy protection concerns are available it integrates manifold regularization and privacy protection regularization into the laplacian support vector machine lapsvm to improve the classification accuracy privacy protection here refers to use only the class proportion of data in order to circumvent the high computational burden of the matrix inversion operation involved in svm mr plpp its scalable version called ssvm mr plpp is further developed by introducing intermediate decision variables into the original regularization framework so that the computational burden of the corresponding transformed kernel in ssvm mr plpp can be greatly reduced making it highly scalable to large datasets the experimental results on numerous datasets show the effectiveness of the proposed classifiers

transformation invariant clustering using the em algorithm clustering is a simple effective way to derive useful representations of data such as images and videos clustering explains the input as one of several prototypes plus noise in situations where each input has been randomly transformed e g by translation rotation and shearing in images and videos clustering techniques tend to extract cluster centers that account for variations in the input due to transformations instead of more interesting and potentially useful structure for example if images from a video sequence of a person walking across a cluttered background are clustered it would be more useful for the different clusters to represent different poses and expressions instead of different positions of the person and different configurations of the background clutter we describe a way to add transformation invariance to mixture models by approximating the nonlinear transformation manifold by a discrete set of points we show how the expectation maximization algorithm can be used to jointly learn clusters while at the same time inferring the transformation associated with each input we compare this technique with other methods for filtering noisy images obtained from a scanning electron microscope clustering images from videos of faces into different categories of identification and pose and removing foreground obstructions from video we also demonstrate that the new technique is quite insensitive to initial conditions and works better than standard techniques even when the standard techniques are provided with extra data

modeling the manifolds of images of handwritten digits this paper describes two new methods for modeling the manifolds of digitized images of handwritten digits the models allow a priori information about the structure of the manifolds to be combined with empirical data accurate modeling of the manifolds allows digits to be discriminated using the relative probability densities under the alternative models one of the methods is grounded in principal components analysis the other in factor analysis both methods are based on locally linear low dimensional approximations to the underlying data manifold links with other methods that model the manifold are discussed

hyperdisk based large margin classifier we introduce a large margin linear binary classification framework that approximates each class with a hyperdisk the intersection of the affine support and the bounding hypersphere of its training samples in feature space and then finds the linear classifier that maximizes the margin separating the two hyperdisks we contrast this with support vector machines svms which find the maximum margin separator of the pointwise convex hulls of the training samples arguing that replacing convex hulls with looser convex class models such as hyperdisks provides safer margin estimates that improve the accuracy on some problems both the hyperdisks and their separators are found by solving simple quadratic programs the method is extended to nonlinear feature spaces using the kernel trick and multi class problems are dealt with by combining binary classifiers in the same ways as for svms experiments on a range of data sets show that the method compares favourably with other popular large margin classifiers