pairwise sparsity preserving embedding for unsupervised subspace learning and classification two novel unsupervised dimensionality reduction techniques termed sparse distance preserving embedding sdpe and sparse proximity preserving embedding sppe are proposed for feature extraction and classification sdpe and sppe perform in the clean data space recovered by sparse representation and enhanced euclidean distances over noise removed data are employed to measure pairwise similarities of points in extracting informative features sdpe and sppe aim at preserving pairwise similarities between data points in addition to preserving the sparse characteristics this paper calculates the sparsest representation of all vectors jointly by a convex optimization the sparsest codes enable certain local information of data to be preserved and can endow sdpe and sppe a natural discriminating power adaptive neighborhood and robust characteristic against noise and errors in delivering low dimensional embeddings we also mathematically show sdpe and sppe can be effectively extended for discriminant learning in a supervised manner the validity of sdpe and sppe is examined by extensive simulations comparison with other related state of the art unsupervised algorithms show that promising results are delivered by our techniques

worst case linear discriminant analysis as scalable semidefinite feasibility problems in this paper we propose an efficient semidefinite programming sdp approach to worst case linear discriminant analysis wlda compared with the traditional lda wlda considers the dimensionality reduction problem from the worst case viewpoint which is in general more robust for classification however the original problem of wlda is non convex and difficult to optimize in this paper we reformulate the optimization problem of wlda into a sequence of semidefinite feasibility problems to efficiently solve the semidefinite feasibility problems we design a new scalable optimization method with a quasi newton method and eigen decomposition being the core components the proposed method is orders of magnitude faster than standard interior point sdp solvers experiments on a variety of classification problems demonstrate that our approach achieves better performance than standard lda our method is also much faster and more scalable than standard interior point sdp solvers based wlda the computational complexity for an sdp with m constraints and matrices of size d by d is roughly reduced from mathcal o m 3 md 3 m 2 d 2 to mathcal o d 3 m d in our case

modified principal component analysis an integration of multiple similarity subspace models we modify the conventional principal component analysis pca and propose a novel subspace learning framework modified pca mpca using multiple similarity measurements mpca computes three similarity matrices exploiting the similarity measurements 1 mutual information 2 angle information and 3 gaussian kernel similarity we employ the eigenvectors of similarity matrices to produce new subspaces referred to as similarity subspaces a new integrated similarity subspace is then generated using a novel feature selection approach this approach needs to construct a kind of vector set termed weak machine cell wmc which contains an appropriate number of the eigenvectors spanning the similarity subspaces combining the wrapper method and the forward selection scheme mpca selects a wmc at a time that has a powerful discriminative capability to classify samples mpca is very suitable for the application scenarios in which the number of the training samples is less than the data dimensionality mpca outperforms the other state of the art pca based methods in terms of both classification accuracy and clustering result in addition mpca can be applied to face image reconstruction mpca can use other types of similarity measurements extensive experiments on many popular real world data sets such as face databases show that mpca achieves desirable classification results as well as has a powerful capability to represent data

eigenfaces vs fisherfaces recognition using class specific linear projection we develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression taking a pattern classification approach we consider each pixel in an image as a coordinate in a high dimensional space we take advantage of the observation that the images of a particular face under varying illumination but fixed pose lie in a 3d linear subspace of the high dimensional image space if the face is a lambertian surface without shadowing however since faces are not truly lambertian surfaces and do indeed produce self shadowing images will deviate from this linear subspace rather than explicitly modeling this deviation we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation our projection method is based on fisher s linear discriminant and produces well separated classes in a low dimensional subspace even under severe variation in lighting and facial expressions the eigenface technique another method based on linearly projecting the image space to a low dimensional subspace has similar computational requirements yet extensive experimental results demonstrate that the proposed fisherface method has error rates that are lower than those of the eigenface technique for tests on the harvard and yale face databases

improved discriminate analysis for high dimensional data and its application to face recognition many pattern recognition applications involve the treatment of high dimensional data and the small sample size problem principal component analysis pca is a common used dimension reduction technique linear discriminate analysis lda is often employed for classification pca plus lda is a famous framework for discriminant analysis in high dimensional space and singular cases in this paper we examine the theory of this framework and find out that even if there is no small sample size problem the pca dimension reduction cannot guarantee the subsequent successful application of lda we thus develop an improved discriminate analysis method by introducing an inverse fisher criterion and adding a constrain in pca procedure so that the singularity phenomenon will not occur experiment results on face recognition suggest that this new approach works well and can be applied even when the number of training samples is one per class

probabilistic visual learning for object representation we present an unsupervised technique for visual learning which is based on density estimation in high dimensional spaces using an eigenspace decomposition two types of density estimates are derived for modeling the training data a multivariate gaussian for unimodal distributions and a mixture of gaussians model for multimodal distributions those probability densities are then used to formulate a maximum likelihood estimation framework for visual search and target detection for automatic object recognition and coding our learning technique is applied to the probabilistic visual modeling detection recognition and coding of human faces and nonrigid objects such as hands

approximate information discriminant analysis a computationally simple heteroscedastic feature extraction technique in this article we develop a novel linear dimensionality reduction technique for classification the technique utilizes the first two statistical moments of data and retains the computational simplicity characteristic of second order techniques such as linear discriminant analysis formally the technique maximizes a criterion that belongs to the class of probability dependence measures and is naturally defined for multiple classes the criterion is based on an approximation of an information theoretic measure and is capable of handling heteroscedastic data the performance of our method along with similar feature extraction approaches is demonstrated based on experimental results with real world datasets our method compares favorably to similar second order linear dimensionality techniques

comments on globally maximizing locally minimizing unsupervised discriminant projection with application to face and palm biometrics this paper develops an unsupervised discriminant projection udp technique for dimensionality reduction of high dimensional data in small sample size cases udp can be seen as a linear approximation of a multimanifolds based learning framework which takes into account both the local and nonlocal quantities udp characterizes the local scatter as well as the nonlocal scatter seeking to find a projection that simultaneously maximizes the nonlocal scatter and minimizes the local scatter this characteristic makes udp more intuitive and more powerful than the most up to date method locality preserving projection lpp which considers only the local scatter for clustering or classification tasks the proposed method is applied to face and palm biometrics and is examined using the yale feret and ar face image databases and the polyu palmprint database the experimental results show that udp consistently outperforms lpp and pca and outperforms lda when the training sample size per class is small this demonstrates that udp is a good choice for real world biometrics applications

face recognition using scale adaptive directional and textural features a novel approach to face recognition problem using directional and texture information from face images is proposed in this paper in order to capture the directionality specially designed using local polynomial approximation technique scale adaptive digital filters are used for texture features extraction a low dimensional and computationally effective local descriptor is utilized textural and directional features are captured at the holistic and part based levels resulting in a robust face descriptor the proposed method is tested on a number of standard test face datasets orl xm2vts extended yale cmu pie ar and feret for different scenarios and its performance is compared with several state of the art techniques

algebraic feature extraction for image recognition based on an optimal discriminant criterion abstract a novel algebraic feature extraction method for image recognition is presented the main difference between the present method and any other existing algebraic feature extraction method is that the present method uses an optimal discriminant criterion to extract algebraic features for the given training image samples a set of optimal discriminant projection vectors is calculated according to a generalized fisher criterion function on the basis of the optimal discriminant projection vector the projective feature vectors of an image i e algebraic features are extracted by projecting the image onto all optimal discriminant projection vectors the method of calculating the set of optimal discriminant projection vectors is discussed a minimum distance classifier is also proposed for classifying projective feature vectors experimental results showed that the present method has good recognition performance an important conclusion about the present method is that the foley sammon optimal set of discriminant vectors is a special case of the set of optimal discriminant projection vectors

information discriminant analysis feature extraction with an information theoretic objective using elementary information theoretic tools we develop a novel technique for linear transformation from the space of observations into a low dimensional feature subspace for the purpose of classification the technique is based on a numerical optimization of an information theoretic objective function which can be computed analytically the advantages of the proposed method over several other techniques are discussed and the conditions under which the method reduces to linear discriminant analysis are given we show that the novel objective function enjoys many of the properties of the mutual information and the bayes error and we give sufficient conditions for the method to be bayes optimal since the objective function is maximized numerically we show how the calculations can be accelerated to yield feasible solutions the performance of the method compares favorably to other linear discriminant based feature extraction methods on a number of simulated and real world data sets

random sampling for subspace face recognition subspace face recognition often suffers from two problems 1 the training sample set is small compared with the high dimensional feature vector 2 the performance is sensitive to the subspace dimension instead of pursuing a single optimal subspace we develop an ensemble learning framework based on random sampling on all three key components of a classification system the feature space training samples and subspace parameters fisherface and null space lda n lda are two conventional approaches to address the small sample size problem but in many cases these lda classifiers are overfitted to the training set and discard some useful discriminative information by analyzing different overfitting problems for the two kinds of lda classifiers we use random subspace and bagging to improve them respectively by random sampling on feature vectors and training samples multiple stabilized fisherface and n lda classifiers are constructed and the two groups of complementary classifiers are integrated using a fusion rule so nearly all the discriminative information is preserved in addition we further apply random sampling on parameter selection in order to overcome the difficulty of selecting optimal parameters in our algorithms then we use the developed random sampling framework for the integration of multiple features a robust random sampling face recognition system integrating shape texture and gabor responses is finally constructed

a multi manifold discriminant analysis method for image feature extraction in this paper we propose a multi manifold discriminant analysis mmda method for an image feature extraction and pattern recognition based on graph embedded learning and under the fisher discriminant analysis framework in an mmda the within class graph and between class graph are respectively designed to characterize the within class compactness and the between class separability seeking for the discriminant matrix to simultaneously maximize the between class scatter and minimize the within class scatter in addition in an mmda the within class graph can represent the sub manifold information while the between class graph can represent the multi manifold information the proposed mmda is extensively examined by using the feret ar and orl face databases and the polyu finger knuckle print databases the experimental results demonstrate that an mmda is effective in feature extraction leading to promising image recognition performance

perturbation lda learning the difference between the class empirical mean and its expectation fisher s linear discriminant analysis lda is popular for dimension reduction and extraction of discriminant features in many pattern recognition applications especially biometric learning in deriving the fisher s lda formulation there is an assumption that the class empirical mean is equal to its expectation however this assumption may not be valid in practice in this paper from the perturbation perspective we develop a new algorithm called perturbation lda p lda in which perturbation random vectors are introduced to learn the effect of the difference between the class empirical mean and its expectation in fisher criterion this perturbation learning in fisher criterion would yield new forms of within class and between class covariance matrices integrated with some perturbation factors moreover a method is proposed for estimation of the covariance matrices of perturbation random vectors for practical implementation the proposed p lda is evaluated on both synthetic data sets and real face image data sets experimental results show that p lda outperforms the popular fisher s lda based algorithms in the undersampled case

multiclass classifiers based on dimension reduction with generalized lda linear discriminant analysis lda has been widely used for dimension reduction of data sets with multiple classes the lda has been recently extended to various generalized lda methods that are applicable regardless of the relative sizes between the data dimension and the number of data items in this paper we propose several multiclass classifiers based on generalized lda glda algorithms taking advantage of the dimension reducing transformation matrix without requiring additional training or parameter optimization a marginal linear discriminant classifier mldc a bayesian linear discriminant classifier bldc and a one dimensional bldc are introduced for multiclass classification our experimental results illustrate that these classifiers produce higher ten fold cross validation accuracy than knn and centroid based classifiers in the reduced dimensional space obtained from glda

complexity reduced scheme for feature extraction with linear discriminant analysis owing to the singularity of the within class scatter linear discriminant analysis lda becomes ill posed for small sample size sss problems null space based lda nlda which is an extension of lda provides good discriminant performances for sss problems yet as the original scheme for the feature extractor fe of nlda suffers from a complexity burden a few modified schemes have since been proposed for complexity reduction in this brief by transforming the problem of finding the fe of nlda into a linear equation problem a novel scheme is derived offering a further reduction of the complexity

a computationally efficient scheme for feature extraction with kernel discriminant analysis the kernel discriminant analysis kda an extension of the linear discriminant analysis lda and null space based lda into the kernel space generally provides good pattern recognition pr performance for both small sample size sss and non sss pr problems due to the eigen decomposition technique adopted however the original scheme for the feature extraction with the kda suffers from a high complexity burden in this paper we derive a transformation of the kda into a linear equation problem and propose a novel scheme for the feature extraction with the kda the proposed scheme is shown to provide us with a reduction of complexity without degradation of pr performance in addition to enhance the pr performance further we address the incorporation of regularization into the proposed scheme highlightswe propose a complexity reduced scheme for the kernel discriminant analysis the proposed scheme does not cause degradation of recognition accuracy performance the core is the transformation of the original problem into a linear equation problem we further address the incorporation of regularization into the proposed scheme

wavelet based discriminant analysis for face recognition the linear fisher discriminant analysis lda is a well known and popular statistical method in pattern recognition and classification when applied to face recognition problem the small sample size problem occurs we investigate the nature of this phenomenon and use wavelet transform for dimension reduction moreover we propose a regularized scheme based face recognition system comparisons are made with the tikhonov regularization method and the infinity fisher index method we find out that when the small sample size problem appears optimizing the fisher index does not lead to good results

one qubit reduced states of a pure many qubit state polygon inequalities we show that a necessary and sufficient condition for a set of n one qubit mixed states to be the reduced states of a pure n qubit state is that their smaller eigenvalues should satisfy polygon inequalities each of them must be no greater than the sum of the others

fisher regularized support vector machine this paper proposes fisher regularized support vector machine fishersvm fishersvm is a graph based supervised learning method fishersvm has two regularization terms maximum margin and fisher regularizations fishersvm aims to maximize the margin and minimize the within class scatter support vector machine svm and fisher discriminant analysis fda are two commonly used methods in machine learning and pattern recognition a combined method of the linear svm and fda called svm lda linear discriminant analysis has been proposed only for the linear case this paper generalizes this combined method to the nonlinear case from the view of regularization a fisher regularization is defined and incorporated into svm to obtain a fisher regularized support vector machine fishersvm in fishersvm there are two regularization terms the maximum margin regularization and fisher regularization which allow fishersvm to maximize the classification margin and minimize the within class scatter roughly speaking fishersvm can approximatively fulfill the fisher criterion and obtain good statistical separability this paper also discusses the connections of fishersvm to graph based regularization methods and the mathematical programming method to kernel fisher discriminant analysis kfda it shows that fishersvm can be thought of as a graph based supervised learning method or a robust kfda experimental results on artificial and real world data show that fishersvm has a promising generalization performance

rapid and brief communication an efficient kernel discriminant analysis method small sample size and high computational complexity are two major problems encountered when traditional kernel discriminant analysis methods are applied to high dimensional pattern classification tasks such as face recognition in this paper we introduce a new kernel discriminant learning method which is able to effectively address the two problems by using regularization and subspace decomposition techniques experiments performed on real face databases indicate that the proposed method outperforms in terms of classification accuracy existing kernel methods such as kernel principal component analysis and kernel linear discriminant analysis at a significantly reduced computational cost

scatter balance an angle based supervised dimensionality reduction subspace selection is widely applied in data classification clustering and visualization the samples projected into subspace can be processed efficiently in this paper we research the linear discriminant analysis lda and maximum margin criterion mmc algorithms intensively and analyze the effects of scatters to subspace selection meanwhile we point out the boundaries of scatters in lda and mmc algorithms to illustrate the differences and similarities of subspace selection in different circumstances besides the effects of outlier classes on subspace selection are also analyzed according to the above analysis we propose a new subspace selection method called angle linear discriminant embedding alde on the basis of angle measurement alde utilizes the cosine of the angle to get new within class and between class scatter matrices and avoids the small sample size problem simultaneously to deal with high dimensional data we extend alde to a two stage alde ts alde the synthetic data experiments indicate that alde can balance the within class and between class scatters and be robust to outlier classes the experimental results based on uci machine learning repository and image databases show that ts alde has a lower time complexity than alde while processing high dimensional data

uncorrelated linear discriminant analysis based on weighted pairwise fisher criterion in this paper we propose a novel uncorrelated weighted linear discriminant analysis uwlda method for feature extraction and recognition the uwlda first introduces a weighting function to restrain the dominant role of the classes with larger distance and then searches the optimal discriminant vectors under the conjugative orthogonal constrains in the null space of the within class scatter matrix and its conjugative orthogonal complement space respectively as a result the proposed technique not only derive the optimal and lossless discriminative information but also guarantee that all extracted features are statistically uncorrelated experiments on feret face database and ar face database are performed to test and evaluate the proposed algorithm the results demonstrate the effectiveness of uwlda

optimal discriminant plane for a small number of samples and design method of classifier on the plane abstract in a previous work zi quan hong and jing yu yang minimum distance classifier on the optimal discriminant plane we suggested and derived a method for constructing classifier on the optimal discriminant plane using minimum distance criteria in this paper the problem of solving optimal discriminant plane for a small number of samples is discussed which is based on the above paper by the same authors in the case of a small number of samples generalized eigenequation ax bx established for a large number of samples usually has no solution because within class scatter matrix is singular to obtain the solution of the generalized eigenequation a new method is suggested in which the singular value perturbation is added to the within class scatter matrix such that the matrix becomes a nonsingular matrix therefore the generalized eigenequation can be solved using existing algorithms we proved that the generalized eigenequations on the optimal discriminant plane are stable in respect of eigenvalues and the generalized eigenvectors are indeed the optimal discriminant directions if the perturbation is subject to some certain conditions the experimental results have shown that our method works well and the minimum distance classifier on the optimal discriminant plane can be constructed with high performance even in the case of a small number of samples

an efficient discriminant based solution for small sample size problem classification of high dimensional statistical data is usually not amenable to standard pattern recognition techniques because of an underlying small sample size problem to address the problem of high dimensional data classification in the face of a limited number of samples a novel principal component analysis pca based feature extraction classification scheme is proposed the proposed method yields a piecewise linear feature subspace and is particularly well suited to difficult recognition problems where achievable classification rates are intrinsically low such problems are often encountered in cases where classes are highly overlapped or in cases where a prominent curvature in data renders a projection onto a single linear subspace inadequate the proposed feature extraction classification method uses class dependent pca in conjunction with linear discriminant feature extraction and performs well on a variety of real world datasets ranging from digit recognition to classification of high dimensional bioinformatics and brain imaging data

face recognition using discriminant locality preserving projections based on maximum margin criterion in this paper we propose a new discriminant locality preserving projections based on maximum margin criterion dlpp mmc dlpp mmc seeks to maximize the difference rather than the ratio between the locality preserving between class scatter and locality preserving within class scatter dlpp mmc is theoretically elegant and can derive its discriminant vectors from both the range of the locality preserving between class scatter and the range space of locality preserving within class scatter dlpp mmc can also derive its discriminant vectors from the null space of locality preserving within class scatter when the parameter of dlpp mmc approaches experiments on the orl yale feret and pie face databases show the effectiveness of the proposed dlpp mmc

optimal fisher discriminant analysis using the rank decomposition abstract the fisher optimal discriminant vector is a very efficient means in high dimensional pattern analysis in the case of a small number of samples the within class scatter matrix s w is singular therefore the calculation of the fisher optimal discriminant vector becomes very important in this paper the conception of the rank decomposition of matrices is first introduced and a new method for calculating the fisher optimal discriminant vector is presented which is particularly well suited to the case of a small number of samples in the sense that the scatter matrices are rank deficient the new method is compared with the pseudoinverse and perturbation methods in tian j opt soc am a 5 1670 1672 1988 and hong and yang pattern recognition 24 317 324 1991 an important conclusion is proved in the three methods our method is the best one for calculating the fisher optimal discriminant vectors in the cases of both a large and a small number of samples the experimental results have also shown that our method can give the highest recognition rate in the case of a small number of samples among the three methods

trace ratio problem revisited dimensionality reduction is an important issue in many machine learning and pattern recognition applications and the trace ratio tr problem is an optimization problem involved in many dimensionality reduction algorithms conventionally the solution is approximated via generalized eigenvalue decomposition due to the difficulty of the original problem however prior works have indicated that it is more reasonable to solve it directly than via the conventional way in this brief we propose a theoretical overview of the global optimum solution to the tr problem via the equivalent trace difference problem eigenvalue perturbation theory is introduced to derive an efficient algorithm based on the newton raphson method theoretical issues on the convergence and efficiency of our algorithm compared with prior literature are proposed and are further supported by extensive empirical results

locally linear discriminant embedding an efficient method for face recognition in this paper an efficient feature extraction method named as locally linear discriminant embedding llde is proposed for face recognition it is well known that a point can be linearly reconstructed by its neighbors and the reconstruction weights are under the sum to one constraint in the classical locally linear embedding lle so the constrained weights obey an important symmetry for any particular data point they are invariant to rotations rescalings and translations the latter two are introduced to the proposed method to strengthen the classification ability of the original lle the data with different class labels are translated by the corresponding vectors and those belonging to the same class are translated by the same vector in order to cluster the data with the same label closer they are also rescaled to some extent so after translation and rescaling the discriminability of the data will be improved significantly the proposed method is compared with some related feature extraction methods such as maximum margin criterion mmc as well as other supervised manifold learning based approaches for example ensemble unified lle and linear discriminant analysis en ullelda locally linear discriminant analysis llda experimental results on yale and cmu pie face databases convince us that the proposed method provides a better representation of the class information and obtains much higher recognition accuracies

ga based optimal selection of pzmi features for face recognition one of the key problems in automated face recognition system is that of handling the face image variation in terms of scale rotation in plane and translation one approach is fixing mentioned problems in recognition processes by extracting one linear transformation invariant feature this paper presents a novel method for face recognition pseudo zernike moment invariant pzmi which has linear transformation invariance properties and is robust in the presence of noise utilized to produce feature vectors for decreasing computational complexity of feature extraction step we use genetic algorithm ga to select the optimal feature set which contains optimal pzmi orders and corresponding repetitions in addition we have investigated the effect of pzmi orders on recognition rate in noisy images proposed scheme has been tested on the feret database experimental results prove the advantages of the proposed method when compared with other pzmi based face recognition systems

a comparative study of feature extraction methods and their application to p rbf nns in face recognition problem this study is concerned with a design of a face recognition algorithm realized based on feature extraction with the aid of the 2 directional 2 dimensional linear discriminant analysis referred to as 2d 2lda the 2dlda algorithm yields high accuracy but comes with an unresolved problem of handling large feature matrices present in face recognition problems the proposed 2d 2lda algorithm is computationally more efficient and produces more powerful discrimination decision the proposed p rbf nns is used as a recognition module the architecture of this module consists of three functional components the coefficients of the p rbf nns model are obtained by fuzzy inference method forming the inference part of fuzzy rules the essential design parameters including learning rate momentum fuzzification coefficient and the feature selection mechanism of the networks are optimized by means of differential evolution de the experimental results reported for benchmark face datasets the yale orl database and ic ci dataset demonstrate the effectiveness and efficiency of 2d 2lda algorithm compared with other pre processing approaches such as lpp 2d pca 2d lpp sr pca 2d 2pca and fusion of pca lda the experimental results show that 2d 2lda based p rbf nns achieves higher performance than being reported by other methods

invariant differential operators and eigenspace representations on an affine symmetric space let g h be an affine symmetric space of split rank r let d be a preferred polynomial algebra of g invariant differential operators on g h generated by r elements we show that the space of k finite joint eigenfunctions of d on g h form an admissible g k module which is called an eigenspace representation the main content of this paper is description of the algebras of invariant differential operators and determination of the eigenspace representations on g h we also obtain a poisson transform for t spherical eigenfunctions on g h by eisenstein integrals

hyperplane navigation a method to set individual scores in fmri group datasets abstract recent studies have demonstrated that spatial patterns of fmri bold activity distribution over the brain may be used to classify different groups or mental states these studies are based on the application of advanced pattern recognition approaches and multivariate statistical classifiers most published articles in this field are focused on improving the accuracy rates and many approaches have been proposed to accomplish this task nevertheless a point inherent to most machine learning methods and still relatively unexplored in neuroimaging is how the discriminative information can be used to characterize groups and their differences in this work we introduce the maximum uncertainty linear discrimination analysis mlda and show how it can be applied to infer groups patterns by discriminant hyperplane navigation in addition we show that it naturally defines a behavioral score i e an index quantifying the distance between the states of a subject from predefined groups we validate and illustrate this approach using a motor block design fmri experiment data with 35 subjects

classifiability based discriminatory projection pursuit fisher s linear discriminant fld is one of the most widely used linear feature extraction method especially in many visual computation tasks based on the analysis on several limitations of the traditional fld this paper attempts to propose a new computational paradigm for discriminative linear feature extraction named classifiability based discriminatory projection pursuit cdpp which is different from the traditional fld and its variants there are two steps in the proposed cdpp one is the construction of a candidate projection set cps and the other is the pursuit of discriminatory projections specifically in the former step candidate projections are generated by using the nearest between class boundary samples while the latter is efficiently achieved by classifiability based adaboost learning from the cps we show that the new projection pursuit paradigm not only does not suffer from the limitations of the traditional fld but also inherits good generalizability from the boundary attribute of candidate projections extensive experiments on both synthetic and real datasets validate the effectiveness of cdpp for discriminative linear feature extraction

image covariance based subspace method for face recognition this paper proposes a new subspace method that is based on image covariance obtained from windowed features of images a windowed input feature consists of a number of pixels and the dimension of input space is determined by the number of windowed features each element of an image covariance matrix can be obtained from the inner product of two windowed features the 2d pca and 2d lda methods are then obtained from principal component analysis and linear discriminant analysis respectively using the image covariance matrix in the case of 2d lda there is no need for pca preprocessing and the dimension of subspace can be greater than the number of classes because the within class and between class image covariance matrices have full ranks comparative experiments are performed using the feret cmu and orl databases of facial images the experimental results show that the proposed 2d lda provides the best recognition rate among several subspace methods in all of the tests

complete large margin linear discriminant analysis using mathematical programming approach in this paper we develop a novel dimensionality reduction dr framework coined complete large margin linear discriminant analysis clmlda inspired by several recently proposed dr methods clmlda constructs two mathematical programming models by maximizing the minimum distance between each class center and the total class center respectively in the null space of within class scatter matrix and its orthogonal complementary space in this way clmlda not only makes full use of the discriminative information contained in the whole feature space but also overcome the weakness of linear discriminant analysis lda in dealing with the class separation problem the solutions of clmlda follow from solving two nonconvex optimization problems each of which is transformed to a series of convex quadratic programming problems by using the constrained concave convex procedure first and then solved by off the shelf optimization toolbox experiments on both toy and several publicly available databases demonstrate its feasibility and effectiveness

a new lda based face recognition system which can solve the small sample size problem a new lda based face recognition system is presented in this paper linear discriminant analysis lda is one of the most popular linear projection techniques for feature extraction the major drawback of applying lda is that it may encounter the small sample size problem in this paper we propose a new lda based technique which can solve the small sample size problem we also prove that the most expressive vectors derived in the null space of the within class scatter matrix using principal component analysis pca are equal to the optimal discriminant vectors derived in the original space using lda the experimental results show that the new lda process improves the performance of a face recognition system signi cantly 2000 pattern recognition society published by elsevier science ltd all rights reserved

fast adaptive algorithms and networks for class separability features abstract in this article we introduce accelerated algorithms for linear discriminant analysis lda and feature extraction from unimodal multiclass gaussian data current adaptive methods based on the gradient descent optimization technique use a fixed or a monotonically decreasing step size in each iteration which results in a slow convergence rate here we use a variable step size optimally computed in each iteration using the steepest descent method in order to accelerate the convergence of the algorithm based on the new adaptive algorithm we present a self organizing neural network for adaptive computation of the square root of the inverse covariance matrix 1 2 and use it i in a network for optimal feature extraction from gaussian data and ii in cascaded form with a principal component analysis network for lda experimental results demonstrate fast convergence and high stability of the algorithm and justify its advantages for on line pattern recognition applications with stationary and non stationary input data

novel fisher discriminant classifiers at the present several applications need to classify high dimensional points belonging to highly unbalanced classes unfortunately when the training set cardinality is small compared to the data dimensionality small sample size problem the classification performance of several well known classifiers strongly decreases similarly the classification accuracy of several discriminative methods decreases when non linearly separable and unbalanced classes are treated in this paper we firstly survey state of the art methods that employ improved versions of linear discriminant analysis lda to deal with the above mentioned problems secondly we propose a family of classifiers based on the fisher subspace estimation which efficiently deal with the small sample size problem non linearly separable classes and unbalanced classes the promising results obtained by the proposed techniques on benchmark datasets and the comparison with state of the art predictors show the efficacy of the proposed techniques

efficient and robust feature extraction by maximum margin criterion in pattern recognition feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information principal component analysis pca and linear discriminant analysis lda are the two most popular linear dimensionality reduction methods however pca is not very effective for the extraction of the most discriminant features and lda is not stable due to the small sample size problem in this paper we propose some new linear and nonlinear feature extractors based on maximum margin criterion mmc geometrically feature extractors based on mmc maximize the average margin between classes after dimensionality reduction it is shown that mmc can represent class separability better than pca as a connection to lda we may also derive lda from mmc by incorporating some constraints by using some other constraints we establish a new linear feature extractor that does not suffer from the small sample size problem which is known to cause serious stability problems for lda the kernelized nonlinear counterpart of this linear feature extractor is also established in the paper our extensive experiments demonstrate that the new feature extractors are effective stable and efficient

regularized class specific subspace classifier in this paper we mainly focus on how to achieve the translated subspace representation for each class which could simultaneously indicate the distribution of the associated class and the differences from its complementary classes by virtue of the reconstruction problem the class specific subspace classifier cssc problem could be represented as a series of biobjective optimization problems which minimize and maximize the reconstruction errors of the related class and its complementary classes respectively besides the regularization term is specifically introduced to ensure the whole system s stability accordingly a regularized class specific subspace classifier rcssc method can be further proposed based on solving a general quadratic ratio problem the proposed rcssc method consistently converges to the global optimal subspace and translation under the variations of the regularization parameter furthermore the proposed rcssc method could be extended to the unregularized case which is known as unregularized cssc ucssc method via orthogonal decomposition technique as a result the effectiveness and the superiority of both proposed rcssc and ucssc methods can be verified analytically and experimentally

face recognition using kernel scatter difference based discriminant analysis there are two fundamental problems with the fisher linear discriminant analysis for face recognition one is the singularity problem of the within class scatter matrix due to small training sample size the other is that it cannot efficiently describe complex nonlinear variations of face images because of its linear property in this letter a kernel scatter difference based discriminant analysis is proposed to overcome these two problems we first use the nonlinear kernel trick to map the input data into an implicit feature space f then a scatter difference based discriminant rule is defined to analyze the data in f the proposed method can not only produce nonlinear discriminant features but also avoid the singularity problem of the within class scatter matrix extensive experiments show encouraging recognition performance of the new algorithm

kernel quadratic discriminant analysis for small sample size problem it is generally believed that quadratic discriminant analysis qda can better fit the data in practical pattern recognition applications compared to linear discriminant analysis lda method this is due to the fact that qda relaxes the assumption made by lda based methods that the covariance matrix for each class is identical however it still assumes that the class conditional distribution is gaussian which is usually not the case in many real world applications in this paper a novel kernel based qda method is proposed to further relax the gaussian assumption by using the kernel machine technique the proposed method solves the complex pattern recognition problem by combining the qda solution and the kernel machine technique and at the same time tackles the so called small sample size problem through a regularized estimation of the covariance matrix extensive experimental results indicate that the proposed method is a more sophisticated solution outperforming many traditional kernel based learning algorithms

inexact implementation using krylov subspace methods for large scale exponential discriminant analysis with applications to high dimensionality reduction problems abstract exponential discriminant analysis eda is a generalized discriminant analysis method based on matrix exponential it can essentially overcome the intrinsic difficulty of small sample size problem that exists in the classical linear discriminant analysis lda however for data with high dimensionality one has to solve a large matrix exponential eigenproblem in this method and the time complexity is dominated by the computation of exponential of large matrices in this paper we propose two inexact krylov subspace algorithms for solving the large matrix exponential eigenproblem efficiently the contribution of this work is threefold first we consider how to compute matrix exponential vector products efficiently which is the key step in the krylov subspace method second we compare the discriminant analysis criterion of eda and that of lda from a theoretical point of view third we establish the relationship between the accuracy of the approximate eigenvectors and the distance to nearest neighbor classifier and show why the matrix exponential eigenproblem can be solved approximately in practice numerical experiments on some real world databases show the superiority of our new algorithms over their original counterpart for face recognition

formulating robust linear regression estimation as a one class lda criterion discriminative hat matrix linear discriminant analysis such as fisher s criterion is a statistical learning tool traditionally devoted to separating a training dataset into two or even several classes by the way of linear decision boundaries in this paper we show that this tool can formalize the robust linear regression problem as a robust estimator will do more precisely we develop a one class fischer s criterion in which the maximization provides both the regression parameters and the separation of the data in two classes typical data and atypical data or outliers this new criterion is built on the statistical properties of the subspace decomposition of the hat matrix from this angle we improve the discriminative properties of the hat matrix which is traditionally used as outlier diagnostic measure in linear regression naturally we call this new approach discriminative hat matrix the proposed algorithm is fully nonsupervised and needs only the initialization of one parameter synthetic and real datasets are used to study the performance both in terms of regression and classification of the proposed approach we also illustrate its potential application to image recognition and fundamental matrix estimation in computer vision

enhanced fisher discriminant criterion for image recognition many previous studies have shown that image recognition can be significantly improved by fisher linear discriminant analysis flda technique however flda ignores the variation of data points from the same class which characterizes the most important modes of variability of patterns and helps to improve the generalization capability of flda thus the performance of flda on testing data is not good enough in this paper we propose an enhanced fisher discriminant criterion efdc efdc explicitly considers the intra class variation and incorporates the intra class variation into the fisher discriminant criterion to build a robust and efficient dimensionality reduction function efdc obtains a subspace which best detects the discriminant structure and simultaneously preserves the modes of variability of patterns which will result in stable intraclass representation experimental results on four image database show a clear improvement over the results of flda based methods

discriminative common vector method with kernels in some pattern recognition tasks the dimension of the sample space is larger than the number of samples in the training set this is known as the small sample size problem linear discriminant analysis lda techniques cannot be applied directly to the small sample size case the small sample size problem is also encountered when kernel approaches are used for recognition in this paper we attempt to answer the question of how should one choose the optimal projection vectors for feature extraction in the small sample size case based on our findings we propose a new method called the kernel discriminative common vector method in this method we first nonlinearly map the original input space to an implicit higher dimensional feature space in which the data are hoped to be linearly separable then the optimal projection vectors are computed in this transformed space the proposed method yields an optimal solution for maximizing a modified fisher s linear discriminant criterion discussed in the paper thus under certain conditions a 100 recognition rate is guaranteed for the training set samples experiments on test data also show that in many situations the generalization performance of the proposed method compares favorably with other kernel approaches

a generalized optimal set of discriminant vectors abstract a generalized optimal set of discriminant vectors for linear feature extraction is presented first the criteria of selecting the generalized optimal discriminant vectors are introduced and then a unified solving method is derived to solve the vectors of the generalized optimal set in both cases of a large number of samples and a small number of samples the experimental results show that the present method is superior to the foley sammon method foley and sammon ieee trans comput 24 281 289 1975 the positive pseudoinverse method tian et al opt engng 25 7 834 839 1986 the perturbation method hong and yang pattern recognition 24 317 324 1991 and the matrix rank decomposition method cheng et al pattern recognition 25 101 111 1992 in terms of correct classification rate