outlier detection with the kernelized spatial depth function statistical depth functions provide from the deepest point a center outward ordering of multidimensional data in this sense depth functions can measure the extremeness or outlyingness of a data point with respect to a given data set hence they can detect outliers observations that appear extreme relative to the rest of the observations of the various statistical depths the spatial depth is especially appealing because of its computational efficiency and mathematical tractability in this article we propose a novel statistical depth the kernelized spatial depth ksd which generalizes the spatial depth via positive definite kernels by choosing a proper kernel the ksd can capture the local structure of a data set while the spatial depth fails we demonstrate this by the half moon data and the ring shaped data based on the ksd we propose a novel outlier detection algorithm by which an observation with a depth value less than a threshold is declared as an outlier the proposed algorithm is simple in structure the threshold is the only one parameter for a given kernel it applies to a one class learning setting in which normal observations are given as the training data as well as to a missing label scenario where the training set consists of a mixture of normal observations and outliers with unknown labels we give upper bounds on the false alarm probability of a depth based detector these upper bounds can be used to determine the threshold we perform extensive experiments on synthetic data and data sets from real applications the proposed outlier detector is compared with existing methods the ksd outlier detector demonstrates a competitive performance

arbitrary norm support vector machines support vector machines svm are state of the art classifiers typically l2 norm or l1 norm is adopted as a regularization term in svms while other norm based svms for example the l0 norm svm or even the l norm svm are rarely seen in the literature the major reason is that l0 norm describes a discontinuous and nonconvex term leading to a combinatorially np hard optimization problem in this letter motivated by bayesian learning we propose a novel framework that can implement arbitrary norm based svms in polynomial time one significant feature of this framework is that only a sequence of sequential minimal optimization problems needs to be solved thus making it practical in many real applications the proposed framework is important in the sense that bayesian priors can be efficiently plugged into most learning methods without knowing the explicit form hence this builds a connection between bayesian learning and the kernel machines we derive the theoretical framework demonstrate how our approach works on the l0 norm svm as a typical example and perform a series of experiments to validate its advantages experimental results on nine benchmark data sets are very encouraging the implemented l0 norm is competitive with or even better than the standard l2 norm svm in terms of accuracy but with a reduced number of support vectors 9 46 of the number on average when compared with another sparse model the relevance vector machine our proposed algorithm also demonstrates better sparse properties with a training speed over seven times faster

a hybrid probabilistic framework for content based image retrieval with feature weighting in this paper we propose a probabilistic framework for efficient retrieval and indexing of image collections this framework uncovers the hierarchical structure underlying the collection from image features based on a hybrid model that combines both generative and discriminative learning we adopt the generalized dirichlet mixture and maximum likelihood for the generative learning in order to estimate accurately the statistical model of the data then the resulting model is refined by a new discriminative likelihood that enhances the power of relevant features consequently this new model is suitable for modeling high dimensional data described by both semantic and low level visual features the semantic features are defined according to a known ontology while visual features represent the visual appearance such as color shape and texture for validation purposes we propose a new visual feature which has nice invariance properties to image transformations experiments on the microsoft s collection msrcid show clearly the merits of our approach in both retrieval and indexing

variational learning of a dirichlet process of generalized dirichlet distributions for simultaneous clustering and feature selection this paper introduces a novel enhancement for unsupervised feature selection based on generalized dirichlet gd mixture models our proposal is based on the extension of the finite mixture model previously developed in 1 to the infinite case via the consideration of dirichlet process mixtures which can be viewed actually as a purely nonparametric model since the number of mixture components can increase as data are introduced the infinite assumption is used to avoid problems related to model selection i e determination of the number of clusters and allows simultaneous separation of data in to similar clusters and selection of relevant features our resulting model is learned within a principled variational bayesian framework that we have developed the experimental results reported for both synthetic data and real world challenging applications involving image categorization automatic semantic annotation and retrieval show the ability of our approach to provide accurate models by distinguishing between relevant and irrelevant features without over or under fitting the data

model based gaussian and non gaussian clustering abstract the classification maximum likelihood approach is sufficiently general to encompass many current clustering algorithms including those based on the sum of squares criterion and on the criterion of friedman and rubin 1967 however as currently implemented it does not allow the specification of which features orientation size and shape are to be common to all clusters and which may differ between clusters also it is restricted to gaussian distributions and it does not allow for noise we propose ways of overcoming these limitations a reparameterization of the covariance matrix allows us to specify that some features but not all be the same for all clusters a practical framework for non gaussian clustering is outlined and a means of incorporating noise in the form of a poisson process is described an approximate bayesian method for choosing the number of clusters is given the performance of the proposed methods is studied by simulation with encouraging results the methods are applied to the analysis of a data set arising in the study of diabetes and the results seem better than those of previous analyses rh

algebraic feature extraction for image recognition based on an optimal discriminant criterion abstract a novel algebraic feature extraction method for image recognition is presented the main difference between the present method and any other existing algebraic feature extraction method is that the present method uses an optimal discriminant criterion to extract algebraic features for the given training image samples a set of optimal discriminant projection vectors is calculated according to a generalized fisher criterion function on the basis of the optimal discriminant projection vector the projective feature vectors of an image i e algebraic features are extracted by projecting the image onto all optimal discriminant projection vectors the method of calculating the set of optimal discriminant projection vectors is discussed a minimum distance classifier is also proposed for classifying projective feature vectors experimental results showed that the present method has good recognition performance an important conclusion about the present method is that the foley sammon optimal set of discriminant vectors is a special case of the set of optimal discriminant projection vectors

a multi manifold discriminant analysis method for image feature extraction in this paper we propose a multi manifold discriminant analysis mmda method for an image feature extraction and pattern recognition based on graph embedded learning and under the fisher discriminant analysis framework in an mmda the within class graph and between class graph are respectively designed to characterize the within class compactness and the between class separability seeking for the discriminant matrix to simultaneously maximize the between class scatter and minimize the within class scatter in addition in an mmda the within class graph can represent the sub manifold information while the between class graph can represent the multi manifold information the proposed mmda is extensively examined by using the feret ar and orl face databases and the polyu finger knuckle print databases the experimental results demonstrate that an mmda is effective in feature extraction leading to promising image recognition performance

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

selecting critical patterns based on local geometrical and statistical information pattern selection methods have been traditionally developed with a dependency on a specific classifier in contrast this paper presents a method that selects critical patterns deemed to carry essential information applicable to train those types of classifiers which require spatial information of the training data set critical patterns include those edge patterns that define the boundary and those border patterns that separate classes the proposed method selects patterns from a new perspective primarily based on their location in input space it determines class edge patterns with the assistance of the approximated tangent hyperplane of a class surface it also identifies border patterns between classes using local probability the proposed method is evaluated on benchmark problems using popular classifiers including multilayer perceptrons radial basis functions support vector machines and nearest neighbors the proposed approach is also compared with four state of the art approaches and it is shown to provide similar but more consistent accuracy from a reduced data set experimental results demonstrate that it selects patterns sufficient to represent class boundary and to preserve the decision surface

perturbation lda learning the difference between the class empirical mean and its expectation fisher s linear discriminant analysis lda is popular for dimension reduction and extraction of discriminant features in many pattern recognition applications especially biometric learning in deriving the fisher s lda formulation there is an assumption that the class empirical mean is equal to its expectation however this assumption may not be valid in practice in this paper from the perturbation perspective we develop a new algorithm called perturbation lda p lda in which perturbation random vectors are introduced to learn the effect of the difference between the class empirical mean and its expectation in fisher criterion this perturbation learning in fisher criterion would yield new forms of within class and between class covariance matrices integrated with some perturbation factors moreover a method is proposed for estimation of the covariance matrices of perturbation random vectors for practical implementation the proposed p lda is evaluated on both synthetic data sets and real face image data sets experimental results show that p lda outperforms the popular fisher s lda based algorithms in the undersampled case

structural minimax probability machine minimax probability machine mpm is an interesting discriminative classifier based on generative prior knowledge it can directly estimate the probabilistic accuracy bound by minimizing the maximum probability of misclassification the structural information of data is an effective way to represent prior knowledge and has been found to be vital for designing classifiers in real world problems however mpm only considers the prior probability distribution of each class with a given mean and covariance matrix which does not efficiently exploit the structural information of data in this paper we use two finite mixture models to capture the structural information of the data from binary classification for each subdistribution in a finite mixture model only its mean and covariance matrix are assumed to be known based on the finite mixture models we propose a structural mpm smpm smpm can be solved effectively by a sequence of the second order cone programming problems moreover we extend a linear model of smpm to a nonlinear model by exploiting kernelization techniques we also show that the smpm can be interpreted as a large margin classifier and can be transformed to support vector machine and maxi min margin machine under certain special conditions experimental results on both synthetic and real world data sets demonstrate the effectiveness of smpm

brief robust adaptive compensation of biased sinusoidal disturbances with unknown frequency asymptotically stable observable linear systems of order n which are not required to be minimum phase and are affected by an additive noisy biased sinusoidal disturbance with unknown bias magnitude phase and frequency are considered the problem of designing an output feedback compensator which regulates the output to zero for any initial condition and for any biased sinusoidal disturbance with no noise is addressed under the assumption that the system parameters are known this problem is solved by a 2n 6 order compensator which generates asymptotically convergent estimates of the biased sinusoidal disturbance and of its parameters including frequency the robustness of the closed loop system with respect to sufficiently small additive unmodelled noise is characterized in terms of input to state stability

discriminative common vectors for face recognition in face recognition tasks the dimension of the sample space is typically larger than the number of the samples in the training set as a consequence the within class scatter matrix is singular and the linear discriminant analysis lda method cannot be applied directly this problem is known as the small sample size problem in this paper we propose a new face recognition method called the discriminative common vector method based on a variation of fisher s linear discriminant analysis for the small sample size case two different algorithms are given to extract the discriminative common vectors representing each person in the training set of the face database one algorithm uses the within class scatter matrix of the samples in the training set while the other uses the subspace methods and the gram schmidt orthogonalization procedure to obtain the discriminative common vectors then the discriminative common vectors are used for classification of new faces the proposed method yields an optimal solution for maximizing the modified fisher s linear discriminant criterion given in the paper our test results show that the discriminative common vector method is superior to other methods in terms of recognition accuracy efficiency and numerical stability

regulation of linear systems with unknown exosystems of uncertain order the problem of designing an output feedback control law which exponentially achieves output regulation is considered for known stabilizable and detectable linear systems which are allowed to be non minimum phase output references and or additive disturbances are both generated by a linear exosystem with unknown purely imaginary eigenvalues and uncertain order the novelty of this note is to require only an upper bound on the exosystem order to achieve a global solution which includes exponentially convergent estimates of the exosystem unknown frequencies

classification using geometric level sets a variational level set method is developed for the supervised classification problem nonlinear classifier decision boundaries are obtained by minimizing an energy functional that is composed of an empirical risk term with a margin based loss and a geometric regularization term new to machine learning the surface area of the decision boundary this geometric level set classifier is analyzed in terms of consistency and complexity through the calculation of its e entropy for multicategory classification an efficient scheme is developed using a logarithmic number of decision functions in the number of classes rather than the typical linear number of decision functions geometric level set classification yields performance results on benchmark data sets that are competitive with well established methods

disturbance cancellation for linear systems by adaptive internal models abstract the adaptive disturbance cancellation problem is considered for single input single output linear systems when the linear exosystem which generates the disturbances to be rejected and or the reference signals to be tracked is unknown that is both its order 2 r 1 and its parameters 1 r are unknown the proposed regulator makes use of two observers and contains an adaptive internal model which tunes m internal parameters on the basis of the output regulation error it is shown that if m r that is the adaptive internal model can reproduce the required input the regulation error tends exponentially to zero if m r that is the adaptive internal model can only approximate the required input within an approximation error the regulation error tends exponentially into a closed ball whose radius is proportional to provided that is sufficiently small so that singularities are avoided

optimal discriminant plane for a small number of samples and design method of classifier on the plane abstract in a previous work zi quan hong and jing yu yang minimum distance classifier on the optimal discriminant plane we suggested and derived a method for constructing classifier on the optimal discriminant plane using minimum distance criteria in this paper the problem of solving optimal discriminant plane for a small number of samples is discussed which is based on the above paper by the same authors in the case of a small number of samples generalized eigenequation ax bx established for a large number of samples usually has no solution because within class scatter matrix is singular to obtain the solution of the generalized eigenequation a new method is suggested in which the singular value perturbation is added to the within class scatter matrix such that the matrix becomes a nonsingular matrix therefore the generalized eigenequation can be solved using existing algorithms we proved that the generalized eigenequations on the optimal discriminant plane are stable in respect of eigenvalues and the generalized eigenvectors are indeed the optimal discriminant directions if the perturbation is subject to some certain conditions the experimental results have shown that our method works well and the minimum distance classifier on the optimal discriminant plane can be constructed with high performance even in the case of a small number of samples

riemannian manifold learning recently manifold learning has been widely exploited in pattern recognition data analysis and machine learning this paper presents a novel framework called riemannian manifold learning rml based on the assumption that the input high dimensional data lie on an intrinsically low dimensional riemannian manifold the main idea is to formulate the dimensionality reduction problem as a classical problem in riemannian geometry that is how to construct coordinate charts for a given riemannian manifold we implement the riemannian normal coordinate chart which has been the most widely used in riemannian geometry for a set of unorganized data points first two input parameters the neighborhood size k and the intrinsic dimension d are estimated based on an efficient simplicial reconstruction of the underlying manifold then the normal coordinates are computed to map the input high dimensional data into a low dimensional space experiments on synthetic data as well as real world images demonstrate that our algorithm can learn intrinsic geometric structures of the data preserve radial geodesic distances and yield regular embeddings

robust stability of systems with integral control a number of necessary and sufficient conditions are derived which must be satisfied by the plant d c gain matrix of a linear time invariant system in order for an integral controller to exist for which the closed loop system is stable based on these results the robustness of integral control systems is analyzed i e the family of plants is defined which are stable when controlled with the same integral controller conditions for actuator sensor failure tolerance of systems with integral control are also given finally parallels are drawn between the results of this paper and the bifurcation theory of nonlinear systems

face recognition based on the uncorrelated discriminant transformation abstract the extraction of discriminant features is the most fundamental and important problem in face recognition this paper presents a method to extract optimal discriminant features for face images by using the uncorrelated discriminant transformation and kl expansion experiments on the orl database and the nust603 database have been performed experimental results show that the uncorrelated discriminant transformation is superior to the foley sammon discriminant transformation and the new method to extract uncorrelated discriminant features for face images is very effective an error rate of 2 5 is obtained with the experiments on the orl database an average error rate of 1 2 is obtained with the experiments on the nust603 database experiments show that by extracting uncorrelated discriminant features face recognition could be performed with higher accuracy on lower than 16 16 resolution mosaic images it is suggested that for the uncorrelated discriminant transformation the optimal face image resolution can be regarded as the resolution m n which makes the dimensionality n mn of the original image vector space be larger and closer to the number of known face classes

unsupervised clustering and feature weighting based on generalized dirichlet mixture modeling abstract we propose a possibilistic approach for generalized dirichlet mixture parameter estimation data clustering and feature weighting the proposed algorithm called robust and unsupervised learning of finite generalized dirichlet mixture models rule gdm exploits a property of the generalized dirichlet distributions that transforms the data to make the features independent and follow beta distributions then it learns optimal relevance weights for each feature within each cluster this property makes rule gdm suitable for noisy and high dimensional feature spaces in addition rule gdm associates two types of memberships with each data sample the first one is the posterior probability and indicates how well a sample fits each estimated distribution the second membership represents the degree of typicality and is used to identify and discard noise points and outliers rule gdm minimizes one objective function which combines learning the two membership functions distribution parameters and the relevance weights for each feature within each distribution we also extend our algorithm to find the optimal number of clusters in an unsupervised and efficient way by exploiting some properties of the possibilistic membership function the performance of rule gdm is illustrated and compared to similar algorithms we use synthetic data to illustrate its robustness to noisy and high dimensional features we also compare our approach to other relevant algorithms using several standard data sets

optimal fisher discriminant analysis using the rank decomposition abstract the fisher optimal discriminant vector is a very efficient means in high dimensional pattern analysis in the case of a small number of samples the within class scatter matrix s w is singular therefore the calculation of the fisher optimal discriminant vector becomes very important in this paper the conception of the rank decomposition of matrices is first introduced and a new method for calculating the fisher optimal discriminant vector is presented which is particularly well suited to the case of a small number of samples in the sense that the scatter matrices are rank deficient the new method is compared with the pseudoinverse and perturbation methods in tian j opt soc am a 5 1670 1672 1988 and hong and yang pattern recognition 24 317 324 1991 an important conclusion is proved in the three methods our method is the best one for calculating the fisher optimal discriminant vectors in the cases of both a large and a small number of samples the experimental results have also shown that our method can give the highest recognition rate in the case of a small number of samples among the three methods

conjugate relation between loss functions and uncertainty sets in classification problems there are two main approaches to binary classification problems the loss function approach and the uncertainty set approach the loss function approach is widely used in real world data analysis statistical decision theory has been used to elucidate its properties such as statistical consistency conditional probabilities can also be estimated by using the minimum solution of the loss function in the uncertainty set approach an uncertainty set is defined for each binary label from training samples the best separating hyperplane between the two uncertainty sets is used as the decision function although the uncertainty set approach provides an intuitive understanding of learning algorithms its statistical properties have not been sufficiently studied in this paper we show that the uncertainty set is deeply connected with the convex conjugate of a loss function on the basis of the conjugate relation we propose a way of revising the uncertainty set approach so that it will have good statistical properties such as statistical consistency we also introduce statistical models corresponding to uncertainty sets in order to estimate conditional probabilities finally we present numerical experiments verifying that the learning with revised uncertainty sets improves the prediction accuracy

discrete data clustering using finite mixture models finite mixture models have been applied for different computer vision image processing and pattern recognition tasks the majority of the work done concerning finite mixture models has focused on mixtures for continuous data however many applications involve and generate discrete data for which discrete mixtures are better suited in this paper we investigate the problem of discrete data modeling using finite mixture models we propose a novel well motivated mixture that we call the multinomial generalized dirichlet mixture the novel model is compared with other discrete mixtures we designed experiments involving spatial color image databases modeling and summarization and text classification to show the robustness flexibility and merits of our approach

a novel feature extraction method for image recognition based on similar discriminant function sdf abstract the extraction of image features is the most fundamental and important problem in image recognition in this paper a similarity measure of matrices is first presented and a similar discriminant function sdf of images is established based on the discriminant function we further propose a novel feature extraction method for image recognition for each class of training image samples an optimal projection axis maximizing the similarity among these training image samples for the class is calculated unlike the common methods of feature extraction we extract a projective feature vector for a training image sample by projecting the image on the optimal projection axis of the class itself and a set of projective feature vectors for a testing image sample by projecting the image on all the optimal projection axes finally a hierarchical classifier in the optimal discriminant space is designed to recognize images in order to test the efficiency of our method it is used to recognize human faces and english characters experimental results have shown that our method has good recognition performance and the extracted projective feature vectors contain more recognition information than commonly used image features

algebraic feature extraction of image for recognition abstract the extraction of image features is one of the fundamental tasks in image recognition up until now there have been several kinds of features to be used for the purpose of image recognition as follows 1 visual features 2 statistical features of pixel 3 transform coefficient features in addition there is another kind of feature which the author believes is very useful i e 4 algebraic features which represent intrinsic attributions of an image singular values sv of image are this kind of feature in this paper we prove that sv feature vector has some important properties of algebraic and geometric invariance and insensitiveness to noise these properties are very useful for the description and recognition of images as an example sv feature vector is used for the problem of recognizing human facial images in this paper using sv feature vector samples of facial images a normal pattern bayes classification model based on sammon s optimal descriminant plane is constructed the experimental result shows that sv feature vector has good performance of class separation

global adaptive observers for nonlinear systems via filtered transformations the problem of determining global adaptive observers for a class of single output nonlinear systems which are linear with respect to an unknown constant parameter vector is addressed sufficient conditions are given to observe asymptotically an equivalent state without persistency of excitation under additional geometric conditions the original state can be observed as well the results obtained are based on a nonlinear change of coordinates driven by auxiliary filters filtered transformations

applications of second order cone programming in a second order cone program socp a linear function is minimized over the intersection of an affine set and the product of second order quadratic cones socps are nonlinear convex problems that include linear and convex quadratic programs as special cases but are less general than semidefinite programs sdps several efficient primaldual interior point methods for socp have been developed in the last few years after reviewing the basic theory of socps we describe general families of problems that tan be recast as socps these include robust linear programming and robust leastsquares problems problems involving sums or maxima of norms or with convex hyperbolic constraints we discuss a variety of engineering applications such as filter design antenna array weight design truss design and grasping forte optimization in robotics we describe an efficient primaldual interior point method for solving socps which shares many of the features of primaldual interior point methods for linear program

cluster based probability model and its application to image and texture processing we develop analyze and apply a specific form of mixture modeling for density estimation within the context of image and texture processing the technique captures much of the higher order nonlinear statistical relationships present among vector elements by combining aspects of kernel estimation and cluster analysis experimental results are presented in the following applications image restoration image and texture compression and texture classification

using locally estimated geodesic distance to optimize neighborhood graph for isometric data embedding to deal with the highly twisted and folded manifold this paper propose a geodesic distance based approach to build the neighborhood graph for isometric embedding this approach assumes that the neighborhood of a point located at the highly twisted place of the manifold may not be linear so that its neighbors should be determined by geodesic distance this approach firstly determines the neighborhood for each point using euclidean distance and then applies the locally estimated geodesic distances to optimize the neighborhood it increases only linear time complexity furthermore the optimized neighborhood can speed up the subsequent embedding process the proposed approach is simple general and easy to deal with a wider range of data the conducted experiments on both synthetic and real data sets validate the approach

modified fuzzy c means algorithm for segmentation of t1 t2 weighted brain mri in the past decades fuzzy segmentation methods especially the fuzzy c means algorithms have been widely used in the segmentation of brain medical images because they can preserve more information from the original image than other segmentation methods this paper introduces a modified robust fuzzy c means algorithm with special weighted bias estimation for segmentation of brain magnetic resonance image in general the intensity inhomogeneities are endorsed to imperfections in the radio frequency coils or to the problems connected with the image acquisition the intensity inhomogeneities are seriously affecting the segmentation process when computer assisted segmentation algorithms used in differentiating borders between the tissues in medical images the proposed algorithm of this paper is considered the fact in the standard fuzzy c means with no spatial context information which makes it sensitive to noise hence the proposed method is capable to deal with the intensity inhomogeneities and noised image effectively further to reduce the number of iterations the proposed modified robust algorithm initializes the centroid using dist max initialization algorithm before the execution of algorithm iteratively experimental results carried out through the comparative studies of segmentation results of the proposed method and the segmentation results of other existed algorithms when these algorithms implemented on brain magnetic resonance image and benchmark lung cancer dataset the superiority of the proposed method has shown from the experimental results

hyperellipsoidal statistical classifications in a reproducing kernel hilbert space standard support vector machines svms have kernels based on the euclidean distance this brief extends standard svms to svms with kernels based on the mahalanobis distance the extended svms become a special case of the euclidean distance when the covariance matrix in a reproducing kernel hilbert space is degenerated to an identity the mahalanobis distance leads to hyperellipsoidal kernels and the euclidean distance results in hyperspherical ones in this brief the mahalanobis distance based kernel in a reproducing kernel hilbert space is developed systematically extensive experiments demonstrate that the hyperellipsoidal kernels slightly outperform the hyperspherical ones with fewer svs

face recognition using flda with single training image per person fisher linear discriminant analysis flda has been widely used for feature extraction in face recognition however it cannot be used when each object has only one training sample because the intra class variations cannot be statistically measured in this case in this paper a novel method is proposed to solve this problem by evaluating the within class scatter matrix from the available single training image by using singular value decomposition svd we decompose the face image into two complementary parts a smooth general appearance image and a difference image the later is used to approximately evaluate the within class scatter matrix and thus the flda can be applied to extract the discriminant face features experimental results show that the proposed method is efficient and it can achieve higher recognition accuracy than many existing schemes

a dirichlet process mixture of generalized dirichlet distributions for proportional data modeling in this paper we propose a clustering algorithm based on both dirichlet processes and generalized dirichlet distribution which has been shown to be very flexible for proportional data modeling our approach can be viewed as an extension of the finite generalized dirichlet mixture model to the infinite case the extension is based on nonparametric bayesian analysis this clustering algorithm does not require the specification of the number of mixture components to be given in advance and estimates it in a principled manner our approach is bayesian and relies on the estimation of the posterior distribution of clusterings using gibbs sampler through some applications involving real data classification and image databases categorization using visual words we show that clustering via infinite mixture models offers a more powerful and robust performance than classic finite mixtures

a hybrid sem algorithm for high dimensional unsupervised learning using a finite generalized dirichlet mixture this paper applies a robust statistical scheme to the problem of unsupervised learning of high dimensional data we develop analyze and apply a new finite mixture model based on a generalization of the dirichlet distribution the generalized dirichlet distribution has a more general covariance structure than the dirichlet distribution and offers high flexibility and ease of use for the approximation of both symmetric and asymmetric distributions we show that the mathematical properties of this distribution allow high dimensional modeling without requiring dimensionality reduction and thus without a loss of information this makes the generalized dirichlet distribution more practical and useful we propose a hybrid stochastic expectation maximization algorithm hsem to estimate the parameters of the generalized dirichlet mixture the algorithm is called stochastic because it contains a step in which the data elements are assigned randomly to components in order to avoid convergence to a saddle point the adjective hybrid is justified by the introduction of a newton raphson step moreover the hsem algorithm autonomously selects the number of components by the introduction of an agglomerative term the performance of our method is tested by the classification of several pattern recognition data sets the generalized dirichlet mixture is also applied to the problems of image restoration image object recognition and texture image database summarization for efficient retrieval for the texture image summarization problem results are reported for the vistex texture image database from the mit media lab

a robust minimax approach to classification when constructing a classifier the probability of correct classification of future data points should be maximized we consider a binary classification problem where the mean and covariance matrix of each class are assumed to be known no further assumptions are made with respect to the class conditional distributions misclassification probabilities are then controlled in a worst case setting that is under all possible choices of class conditional densities with given mean and covariance matrix we minimize the worst case maximum probability of misclassification of future data points for a linear decision boundary this desideratum is translated in a very direct way into a convex second order cone optimization problem with complexity similar to a support vector machine problem the minimax problem can be interpreted geometrically as minimizing the maximum of the mahalanobis distances to the two classes we address the issue of robustness with respect to estimation errors in the means and covariances of the classes via a simple modification of the input data we also show how to exploit mercer kernels in this setting to obtain nonlinear decision boundaries yielding a classifier which proves to be competitive with current methods including support vector machines an important feature of this method is that a worst case bound on the probability of misclassification of future data is always obtained explicitly

a collaborative representation based projections method for feature extraction in graph embedding based methods we usually need to manually choose the nearest neighbors and then compute the edge weights using the nearest neighbors via l2 norm e g lle it is difficult and unstable to manually choose the nearest neighbors in high dimensional space so how to automatically construct a graph is very important in this paper first we give a l2 graph like l1 graph l2 graph calculates the edge weights using the total samples avoiding manually choosing the nearest neighbors second a l2 graph based feature extraction method is presented called collaborative representation based projections crp like spp crp aims to preserve the collaborative representation based reconstruction relationship of data crp utilizes a l2 norm graph to characterize the local compactness information crp maximizes the ratio between the total separability information and the local compactness information to seek the optimal projection matrix crp is much faster than spp since crp calculates the objective function with l2 norm while spp calculate the objective function with l1 norm experimental results on feret ar yale face databases and the polyu finger knuckle print database demonstrate that crp works well in feature extraction and leads to a good recognition performance we give a l2 norm graph based on collaborative representation we propose a collaborative representation based projections crp for feature extraction crp is a rayleigh quotient form and can be calculated via generalized eigenvalue decomposition

feature extraction based on laplacian bidirectional maximum margin criterion maximum margin criterion mmc based feature extraction is more efficient than linear discriminant analysis lda for calculating the discriminant vectors since it does not need to calculate the inverse within class scatter matrix however mmc ignores the discriminative information within the local structures of samples and the structural information embedding in the images in this paper we develop a novel criterion namely laplacian bidirectional maximum margin criterion lbmmc to address the issue we formulate the image total laplacian matrix image within class laplacian matrix and image between class laplacian matrix using the sample similar weight that is widely used in machine learning the proposed lbmmc based feature extraction computes the discriminant vectors by maximizing the difference between image between class laplacian matrix and image within class laplacian matrix in both row and column directions experiments on the feret and yale face databases show the effectiveness of the proposed lbmmc based feature extraction method

optimal inequalities in probability theory a convex optimization approach we propose a semidefinite optimization approach to the problem of deriving tight moment inequalities for p x in s for a set s defined by polynomial inequalities and a random vector x defined on omega subseteq cal r n that has a given collection of up to kth order moments in the univariate case we provide optimal bounds on p x in s when the first k moments of x are given as the solution of a semidefinite optimization problem in k 1 dimensions in the multivariate case if the sets s and omega are given by polynomial inequalities we obtain an improving sequence of bounds by solving semidefinite optimization problems of polynomial size in n for fixed k we characterize the complexity of the problem of deriving tight moment inequalities we show that it is np hard to find tight bounds for k geq 4 and omega cal r n and for k geq 2 and omega cal r n when the data in the problem is rational for k 1 and omega cal r n we show that we can find tight upper bounds by solving n convex optimization problems when the set s is convex and we provide a polynomial time algorithm when s and omega are unions of convex sets over which linear functions can be optimized efficiently for the case k 2 and omega cal r n we present an efficient algorithm for finding tight bounds when s is a union of convex sets over which convex quadratic functions can be optimized efficiently

a new lda based face recognition system which can solve the small sample size problem a new lda based face recognition system is presented in this paper linear discriminant analysis lda is one of the most popular linear projection techniques for feature extraction the major drawback of applying lda is that it may encounter the small sample size problem in this paper we propose a new lda based technique which can solve the small sample size problem we also prove that the most expressive vectors derived in the null space of the within class scatter matrix using principal component analysis pca are equal to the optimal discriminant vectors derived in the original space using lda the experimental results show that the new lda process improves the performance of a face recognition system signi cantly 2000 pattern recognition society published by elsevier science ltd all rights reserved

novel fisher discriminant classifiers at the present several applications need to classify high dimensional points belonging to highly unbalanced classes unfortunately when the training set cardinality is small compared to the data dimensionality small sample size problem the classification performance of several well known classifiers strongly decreases similarly the classification accuracy of several discriminative methods decreases when non linearly separable and unbalanced classes are treated in this paper we firstly survey state of the art methods that employ improved versions of linear discriminant analysis lda to deal with the above mentioned problems secondly we propose a family of classifiers based on the fisher subspace estimation which efficiently deal with the small sample size problem non linearly separable classes and unbalanced classes the promising results obtained by the proposed techniques on benchmark datasets and the comparison with state of the art predictors show the efficacy of the proposed techniques

efficient and robust feature extraction by maximum margin criterion in pattern recognition feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information principal component analysis pca and linear discriminant analysis lda are the two most popular linear dimensionality reduction methods however pca is not very effective for the extraction of the most discriminant features and lda is not stable due to the small sample size problem in this paper we propose some new linear and nonlinear feature extractors based on maximum margin criterion mmc geometrically feature extractors based on mmc maximize the average margin between classes after dimensionality reduction it is shown that mmc can represent class separability better than pca as a connection to lda we may also derive lda from mmc by incorporating some constraints by using some other constraints we establish a new linear feature extractor that does not suffer from the small sample size problem which is known to cause serious stability problems for lda the kernelized nonlinear counterpart of this linear feature extractor is also established in the paper our extensive experiments demonstrate that the new feature extractors are effective stable and efficient

a hybrid feature extraction selection approach for high dimensional non gaussian data clustering this paper presents an unsupervised approach for feature selection and extraction in mixtures of generalized dirichlet gd distributions our method defines a new mixture model that is able to extract independent and non gaussian features without loss of accuracy the proposed model is learned using the expectation maximization algorithm by minimizing the message length of the data set experimental results show the merits of the proposed methodology in the categorization of object images

scale invariant image recognition based on higher order autocorrelation features we propose a framework and a complete implementation of a translation and scale invariant image recognition system for natural indoor scenes the system employs higher order autocorrelation features of scale space data which permit linear classification an optimal linear classification method is presented which is able to cope with a large number of classes represented by many as well as very few samples in the course of the analysis of our system we examine which numerical methods for feature transformation and classification show sufficient stability to fulfill these demands the implementation has been extensively tested we present the results of our own application and several classification benchmarks

optimal reference subset selection for nearest neighbor classification by tabu search this paper presents an approach to select the optimal reference subset ors for nearest neighbor classifier the optimal reference subset which has minimum sample size and satisfies a certain resubstitution error rate threshold is obtained through a tabu search ts algorithm when the error rate threshold is set to zero the algorithm obtains a near minimal consistent subset of a given training set while the threshold is set to a small appropriate value the obtained reference subset may have reasonably good generalization capacity a neighborhood exploration method and an aspiration criterion are proposed to improve the efficiency of ts experimental results based on a number of typical data sets are presented and analyzed to illustrate the benefits of the proposed method the performances of the result consistent and non consistent reference subsets are evaluated

rejection of sinusoidal disturbances for known lti systems in the presence of output delay abstract this paper focuses on estimation and cancellation of unknown sinusoidal disturbances in a known lti system with the presence of a known output delay parametrizing the disturbance and representing the delay as a transport pde the problem is converted to an adaptive control problem for ode pde cascade an existing state observer is used to estimate the ode system states the exponential stability of the equilibrium of the closed loop and error system is proved the perfect estimation of the disturbance and state is shown moreover the convergence of the state to zero as t is achieved in the closed loop system the effectiveness of the controller is demonstrated in a numerical simulation

brill noether loci and the gonality stratification of m g for an irreducible smooth projective complex curve c of genus g the gonality defined as gon c min d z 1 there exists a gd on c is perhaps the second most natural invariant it gives an indication of how far c is from being rational in a way different from what the genus does for g 3 we consider the stratification of the moduli space mg of smooth curves of genus g given by gonality mg 2 m 1 g 3 m 1 g k mg

the nearest subclass classifier a compromise between the nearest mean and nearest neighbor classifier we present the nearest subclass classifier nsc which is a classification algorithm that unifies the flexibility of the nearest neighbor classifier with the robustness of the nearest mean classifier the algorithm is based on the maximum variance cluster algorithm and as such it belongs to the class of prototype based classifiers the variance constraint parameter of the cluster algorithm serves to regularize the classifier that is to prevent overfitting with a low variance constraint value the classifier turns into the nearest neighbor classifier and with a high variance parameter it becomes the nearest mean classifier with the respective properties in other words the number of prototypes ranges from the whole training set to only one per class in the experiments we compared the nsc with regard to its performance and data set compression ratio to several other prototype based methods on several data sets the nsc performed similarly to the k nearest neighbor classifier which is a well established classifier in many domains also concerning storage requirements and classification speed the nsc has favorable properties so it gives a good compromise between classification performance and efficiency

generalized risk zone selecting observations for classification in this paper we extend the risk zone concept by creating the generalized risk zone the generalized risk zone is a model independent scheme to select key observations in a sample set the observations belonging to the generalized risk zone have shown comparable in some experiments even better classification performance when compared to the use of the whole sample the main tool that allows this extension is the cauchy schwartz divergence used as a measure of dissimilarity between probability densities to overcome the setback concerning pdf s estimation we used the ideas provided by the information theoretic learning allowing the calculation to be performed on the available observations only we used the proposed methodology with learning vector quantization feedforward neural networks support vector machines and nearest neighbors

brief paper an adaptive learning regulator for uncertain minimum phase systems with undermodeled unknown exosystems the design of an adaptive learning regulator is addressed for uncertain minimum phase linear systems with known bounds known upper bound on system order known relative degree known high frequency gain sign and for unknown exosystems with unknown order uncertain frequencies on the basis of a known bound on system uncertainties and a known bound on the modeled exosystem frequencies a new adaptive output error feedback control algorithm is proposed which guarantees exponential convergence of both the output and the control input errors into residual bounds which decrease as the exosystem modeling error decreases exponential convergence of both errors to zero is obtained when the regulator exactly models all exosystem excited frequencies while asymptotic convergence of both errors to zero is achieved when the actual exosystem is overmodeled by the regulator the new algorithm generalizes existing learning controllers since in the case of periodic references and or disturbances the knowledge of the period is not required

filtering for texture classification a comparative study in this paper we review most major filtering approaches to texture feature extraction and perform a comparative study filtering approaches included are laws masks 1980 ring wedge filters dyadic gabor filter banks wavelet transforms wavelet packets and wavelet frames quadrature mirror filters discrete cosine transform eigenfilters optimized gabor filters linear predictors and optimized finite impulse response filters the features are computed as the local energy of the filter responses the effect of the filtering is highlighted keeping the local energy function and the classification algorithm identical for most approaches for reference comparisons with two classical nonfiltering approaches co occurrence statistical and autoregressive model based features are given we present a ranking of the tested approaches based on extensive experiments

enhancing prototype reduction schemes with lvq3 type algorithms abstract various prototype reduction schemes have been reported in the literature foremost among these are the prototypes for nearest neighbor pnn the vector quantization vq and the support vector machines svm methods in this paper we shall show that these schemes can be enhanced by the introduction of a post processing phase that is related but not identical to the lvq3 process although the post processing with lvq3 has been reported for the som and the basic vq methods in this paper we shall show that an analogous philosophy can be used in conjunction with the svm and pnn rules our essential modification to lvq3 first entails a partitioning of the respective training sets into two sets called the placement set and the optimizing set which are instrumental in determining the lvq3 parameters such a partitioning is novel to the literature our experimental results demonstrate that the proposed enhancement yields the best reported prototype condensation scheme to date for both artificial data sets and for samples involving real life data sets

harmonic generation in adaptive feedforward cancellation schemes the paper investigates the generation of harmonics in adaptive feedforward cancellation schemes specifically it is shown that an algorithm designed to reject a certain number of frequency components may in fact be capable of reducing higher order harmonics as well this effect originates in the time variation of the adaptive parameters a consequence is that the adaptive system may perform better than the algorithm with the fixed ideal parameters surprisingly the response to higher order harmonics can be calculated precisely using a laplace transform analysis the origin of the numerical procedure is in the equivalence between the adaptive feedforward cancellation scheme and a control scheme based on the internal model principle the implication of the results on design and implementation issues is discussed

maxi min margin machine learning large margin classifiers locally and globally in this paper we propose a novel large margin classifier called the maxi min margin machine m4 this model learns the decision boundary both locally and globally in comparison other large margin classifiers construct separating hyperplanes only either locally or globally for example a state of the art large margin classifier the support vector machine svm considers data only locally while another significant model the minimax probability machine mpm focuses on building the decision hyperplane exclusively based on the global information as a major contribution we show that svm yields the same solution as m4 when data satisfy certain conditions and mpm can be regarded as a relaxation model of m4 moreover based on our proposed local and global view of data another popular model the linear discriminant analysis can easily be interpreted and extended as well we describe the m4 model definition provide a geometrical interpretation present theoretical justifications and propose a practical sequential conic programming method to solve the optimization problem we also show how to exploit mercer kernels to extend m4 for nonlinear classifications furthermore we perform a series of evaluations on both synthetic data sets and real world benchmark data sets comparison with svm and mpm demonstrates the advantages of our new model

adaptive algorithms for the rejection of sinusoidal disturbances with unknown frequency abstract two algorithms are presented for the rejection of sinusoidal disturbances with unknown frequency the first is an indirect algorithm where the frequency of the disturbance is estimated and the estimate is used in another adaptive algorithm that adjusts the magnitude and phase of the input needed to cancel the effect of the disturbance a direct algorithm that uses the concept of a phase locked loop is also presented in which frequency estimation and disturbance cancellation are performed simultaneously approximate analyses are presented for both schemes and the results are found useful for the selection of the design parameters simulations are given which demonstrate the validity of the analytical results and the ability of the algorithms to reject sinusoidal disturbances with unknown frequency the indirect algorithm is found to have a larger capture region for the parameter estimates whereas the direct algorithm has superior convergence properties locally about the optimum parameter estimates

a robust competitive clustering algorithm with applications in computer vision this paper addresses three major issues associated with conventional partitional clustering namely sensitivity to initialization difficulty in determining the number of clusters and sensitivity to noise and outliers the proposed robust competitive agglomeration rca algorithm starts with a large number of clusters to reduce the sensitivity to initialization and determines the actual number of clusters by a process of competitive agglomeration noise immunity is achieved by incorporating concepts from robust statistics into the algorithm rca assigns two different sets of weights for each data point the first set of constrained weights represents degrees of sharing and is used to create a competitive environment and to generate a fuzzy partition of the data set the second set corresponds to robust weights and is used to obtain robust estimates of the cluster prototypes by choosing an appropriate distance measure in the objective function rca can be used to find an unknown number of clusters of various shapes in noisy data sets as well as to fit an unknown number of parametric models simultaneously several examples such as clustering mixture decomposition line plane fitting segmentation of range images and estimation of motion parameters of multiple objects are shown

the minimum error minimax probability machine we construct a distribution free bayes optimal classifier called the minimum error minimax probability machine mempm in a worst case setting i e under all possible choices of class conditional densities with a given mean and covariance matrix by assuming no specific distributions for the data our model is thus distinguished from traditional bayes optimal approaches where an assumption on the data distribution is a must this model is extended from the minimax probability machine mpm a recently proposed novel classifier and is demonstrated to be the general case of mpm moreover it includes another special case named the biased minimax probability machine which is appropriate for handling biased classification one appealing feature of mempm is that it contains an explicit performance indicator i e a lower bound on the worst case accuracy which is shown to be tighter than that of mpm we provide conditions under which the worst case bayes optimal classifier converges to the bayes optimal classifier we demonstrate how to apply a more general statistical framework to estimate model input parameters robustly we also show how to extend our model to nonlinear classification by exploiting kernelization techniques a series of experiments on both synthetic data sets and real world benchmark data sets validates our proposition and demonstrates the effectiveness of our model

online learning of hierarchical pitman yor process mixture of generalized dirichlet distributions with feature selection in this paper a novel statistical generative model based on hierarchical pitman yor process and generalized dirichlet distributions gds is presented the proposed model allows us to perform joint clustering and feature selection thanks to the interesting properties of the gd distribution we develop an online variational inference algorithm formulated in terms of the minimization of a kullback leibler divergence of our resulting model that tackles the problem of learning from high dimensional examples this variational bayes formulation allows simultaneously estimating the parameters determining the model s complexity and selecting the appropriate relevant features for the clustering structure moreover the proposed online learning algorithm allows data instances to be processed in a sequential manner which is critical for large scale and real time applications experiments conducted using challenging applications namely scene recognition and video segmentation where our approach is viewed as an unsupervised technique for visual learning in high dimensional spaces showed that the proposed approach is suitable and promising

advances on adaptive learning control the case of non minimum phase linear systems abstract the adaptive learning control theory is extended to include the case of non minimum phase time invariant linear systems with uncertain parameters the existence of two approximate solutions to the related output tracking problem is proved exponential tracking of periodic output reference signals is guaranteed along with the exponential estimation of the constant system parameters and of the constant coefficients characterizing the truncated fourier series expansion for the periodic input reference signal

discriminative common vector method with kernels in some pattern recognition tasks the dimension of the sample space is larger than the number of samples in the training set this is known as the small sample size problem linear discriminant analysis lda techniques cannot be applied directly to the small sample size case the small sample size problem is also encountered when kernel approaches are used for recognition in this paper we attempt to answer the question of how should one choose the optimal projection vectors for feature extraction in the small sample size case based on our findings we propose a new method called the kernel discriminative common vector method in this method we first nonlinearly map the original input space to an implicit higher dimensional feature space in which the data are hoped to be linearly separable then the optimal projection vectors are computed in this transformed space the proposed method yields an optimal solution for maximizing a modified fisher s linear discriminant criterion discussed in the paper thus under certain conditions a 100 recognition rate is guaranteed for the training set samples experiments on test data also show that in many situations the generalization performance of the proposed method compares favorably with other kernel approaches

twin mahalanobis distance based support vector machines for pattern recognition twin support vector machines tsvms achieve fast training speed and good performance for data classification however tsvms do not take full advantage of the statistical information in data such as the covariance of each class of data this paper proposes a new twin mahalanobis distance based support vector machine tmsvm classifier in which two mahalanobis distance based kernels are constructed according to the covariance matrices of two classes of data for optimizing the nonparallel hyperplanes tmsvms have a special case of tsvms when the covariance matrices in a reproducing kernel hilbert space are degenerated to the identity ones tmsvms are suitable for many real problems especially for the case that the covariance matrices of two classes of data are obviously different the experimental results on several artificial and benchmark datasets indicate that tmsvms not only possess fast learning speed but also obtain better generalization than tsvms and other methods

a fast nearest neighbor search algorithm by filtration abstract classifying an unknown input is a fundamental problem in pattern recognition one standard method is finding its nearest neighbors in a reference set it would be very time consuming if computed feature by feature for all templates in the reference set this naive method is o nd where n is the number of templates in the reference set and d is the number of features or dimension for this reason we present a technique for quickly eliminating most templates from consideration as possible neighbors the remaining candidate templates are then evaluated feature by feature against the query vector we utilize frequencies of features as a pre processing to reduce query processing time burden our approach is simple to implement and achieves great speedup experimentally the most notable advantage of the new method over other existing techniques occurs where the number of features is large and the type of each feature is binary although it works for other type features we improved our ocr system at least twice without a threshold or faster with higher threshold value by using the new algorithm

a generalized optimal set of discriminant vectors abstract a generalized optimal set of discriminant vectors for linear feature extraction is presented first the criteria of selecting the generalized optimal discriminant vectors are introduced and then a unified solving method is derived to solve the vectors of the generalized optimal set in both cases of a large number of samples and a small number of samples the experimental results show that the present method is superior to the foley sammon method foley and sammon ieee trans comput 24 281 289 1975 the positive pseudoinverse method tian et al opt engng 25 7 834 839 1986 the perturbation method hong and yang pattern recognition 24 317 324 1991 and the matrix rank decomposition method cheng et al pattern recognition 25 101 111 1992 in terms of correct classification rate

output regulation for linear systems via adaptive internal model given a stabilizable and detectable linear system with additive disturbances and output references generated by a linear stable exosystem with unknown parameters and known order the problem of designing a global output feedback regulator which asymptotically achieves output regulation and disturbance rejection is considered the system is assumed to be known while the frequencies of the exosystem are unknown all exosystem oscillatory modes are assumed to be excited by the initial condition a global solution is proposed consisting of a dynamic output feedback controller which includes exponentially convergent estimates of the unknown frequencies

generalized dirichlet distribution in bayesian analysis generalized dirichlet distribution has a more general covariance structure than dirichlet distribution this makes the generalized dirichlet distribution to be more practical and useful the concept of complete neutrality will be used to derive the general moment function for the generalized dirichlet distribution and then some properties of the generalized dirichlet distribution will be established similar to the dirichlet distribution the generalized dirichlet distribution will be shown to conjugate to multinominal sampling two experiments are designed for studying the differences between the dirichlet and the generalized dirichlet distributions in bayesian analysis a method for generating samples from a generalized dirichlet in presented when a prior distribution is either a dirichlet or a generalized dirichlet distribution the way for constructing such a prior is discussed