discriminant analysis by gaussian mixtures fisher rao linear discriminant analysis lda is a valuable tool for multigroup classification lda is equivalent to maximum likelihood classification assuming gaussian distributions for each class in this paper we fit gaussian mixtures to each class to facilitate effective classification in non normal settings especially when the classes are clustered low dimensional views are an important by product of lda our new techniques inherit this feature we can control the within class spread of the subclass centres relative to the between class spread our technique for fitting these models permits a natural blend with nonparametric versions of lda

percolation on infinite graphs and isoperimetric inequalities we consider the bernoulli bond percolation process with parameter p on infinite graphs and we give a general criterion for bounded degree graphs to exhibit a non trivial percolation threshold based either on a single isoperimetric inequality if the graph has a bi infinite geodesic or two isoperimetric inequalities if the graph has not a bi infinite geodesic this new criterion extends previous criteria and brings together a large class of amenable graphs such as regular lattices and non amenable graphs such trees we also study the finite connectivity in graphs satisfying the new general criterion and show that graphs in this class with a bi infinite geodesic always have finite connectivity functions with exponential decay when p is sufficiently close to one on the other hand we show that there are graphs in the same class with no bi infinite geodesic for which the finite connectivity decays sub exponentially down to polynomially in the highly supercritical phase even for p arbitrarily close to one

eigenfaces vs fisherfaces recognition using class specific linear projection we develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression taking a pattern classification approach we consider each pixel in an image as a coordinate in a high dimensional space we take advantage of the observation that the images of a particular face under varying illumination but fixed pose lie in a 3d linear subspace of the high dimensional image space if the face is a lambertian surface without shadowing however since faces are not truly lambertian surfaces and do indeed produce self shadowing images will deviate from this linear subspace rather than explicitly modeling this deviation we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation our projection method is based on fisher s linear discriminant and produces well separated classes in a low dimensional subspace even under severe variation in lighting and facial expressions the eigenface technique another method based on linearly projecting the image space to a low dimensional subspace has similar computational requirements yet extensive experimental results demonstrate that the proposed fisherface method has error rates that are lower than those of the eigenface technique for tests on the harvard and yale face databases

discriminant subspace analysis a fukunaga koontz approach the fisher linear discriminant fld is commonly used in pattern recognition it finds a linear subspace that maximally separates class patterns according to the fisher criterion several methods of computing the fld have been proposed in the literature most of which require the calculation of the so called scatter matrices in this paper we bring a fresh perspective to fld via the fukunaga koontz transform fkt we do this by decomposing the whole data space into four subspaces with different discriminabilities as measured by eigenvalue ratios by connecting the eigenvalue ratio with the generalized eigenvalue we show where the fisher criterion is maximally satisfied we prove the relationship between fld and fkt analytically and propose a unified framework to understanding some existing work furthermore we extend our theory to the multiple discriminant analysis mda this is done by transforming the data into intraclass and extraclass spaces followed by maximizing the bhattacharyya distance based on our fkt analysis we identify the discriminant subspaces of mda fkt and propose an efficient algorithm which works even when the scatter matrices are singular or too large to be formed our method is general and may be applied to different pattern recognition problems we validate our method by experimenting on synthetic and real data

learning a mahalanobis metric from equivalence constraints many learning algorithms use a metric defined over the input space as a principal tool and their performance critically depends on the quality of this metric we address the problem of learning metrics using side information in the form of equivalence constraints unlike labels we demonstrate that this type of side information can sometimes be automatically obtained without the need of human intervention we show how such side information can be used to modify the representation of the data leading to improved clustering and classification specifically we present the relevant component analysis rca algorithm which is a simple and efficient algorithm for learning a mahalanobis metric we show that rca is the solution of an interesting optimization problem founded on an information theoretic basis if dimensionality reduction is allowed within rca we show that it is optimally accomplished by a version of fisher s linear discriminant that uses constraints moreover under certain gaussian assumptions rca can be viewed as a maximum likelihood estimation of the within class covariance matrix we conclude with extensive empirical evaluations of rca showing its advantage over alternative methods

regularized coplanar discriminant analysis for dimensionality reduction abstract the dimensionality reduction methods based on linear embedding such as neighborhood preserving embedding npe sparsity preserving projections spp and collaborative representation based projections crp try to preserve a certain kind of linear representation for each sample after projection however in the transformed low dimensional space the linear relationship between the samples may be changed which cannot make the linear representation based classifiers such as sparse representation based classifier src to achieve higher recognition accuracy in this paper we propose a new linear dimensionality reduction algorithm called regularized coplanar discriminant analysis rcda to address this problem it simultaneously seeks a linear projection matrix and some linear representation coefficients that make the samples from the same class coplanar and the samples from different classes not coplanar the proposed regularization term balances the bias from the optimal linear representation and that from the class mean to avoid overfitting the training data and overcomes the matrix singularity in solving the linear representation coefficients an alternative optimization approach is proposed to solve the rcda model experiments are done on several benchmark face databases and hyperspectral image databases and results show that rcda can obtain better performance than other dimensionality reduction methods

pruning noisy bases in discriminant analysis the success of linear discriminant analysis lda is due in part to the simplicity of its formulation which reduces to a simultaneous diagonalization of two symmetric matrices a and b however a fundamental drawback of this approach is that it cannot be efficiently applied wherever the matrix a is singular or when some of the smallest variances in are due to noise in this paper we present a factorization a 1b of and a correlation based criterion that can be readily employed to solve these problems we provide detailed derivations for the linear and nonlinear classification problems the usefulness of the proposed approach is demonstrated thoroughly using a large variety of databases

graph embedding and extensions a general framework for dimensionality reduction a large family of algorithms supervised or unsupervised stemming from statistics or geometry theory has been designed to provide different solutions to the problem of dimensionality reduction despite the different motivations of these algorithms we present in this paper a general formulation known as graph embedding to unify them within a common framework in graph embedding each algorithm can be considered as the direct graph embedding or its linear kernel tensor extension of a specific intrinsic graph that describes certain desired statistical or geometric properties of a data set with constraints from scale normalization or a penalty graph that characterizes a statistical or geometric property that should be avoided furthermore the graph embedding framework can be used as a general platform for developing new dimensionality reduction algorithms by utilizing this framework as a tool we propose a new supervised dimensionality reduction algorithm called marginal fisher analysis in which the intrinsic graph characterizes the intraclass compactness and connects each data point with its neighboring points of the same class while the penalty graph connects the marginal points and characterizes the interclass separability we show that mfa effectively overcomes the limitations of the traditional linear discriminant analysis algorithm due to data distribution assumptions and available projection directions real face recognition experiments show the superiority of our proposed mfa in comparison to lda also for corresponding kernel and tensor extensions

subset selection using the total least squares approach in collinearity problems with errors in the variables abstract if errors occur in all variables of a linear model many classical techniques such as least squares can fail because they lose their optimal properties to face those problems the total least squares tls technique introduced by golub and van loan is proven to be very useful in rejecting noise and improving the solution accuracy this paper treats the problem of subset selection for linear models with collinearities and errors in all its variables a new subset selection algorithm based on tls is presented and compared with the algorithm of golub klema and stewart the selection properties and accuracy in parameter estimation and prediction in the presence of errors are investigated with respect to prediction only cases in which one needs to predict the response from exact values of the selected predictor variables e g in hypotheses or simulation studies are considered the tls concept typically applies to those cases it may be concluded that our subset selection algorithm guarantees better stability and reliability of its subset choice and better prediction accuracy in almost all cases a computationally very attractive variant of our algorithm which enables the same prediction accuracy is also presented

pareto models for discriminative multiclass linear dimensionality reduction we address the class masking problem in multiclass linear discriminant analysis lda in the multiclass setting lda does not maximize each pairwise distance between classes but rather maximizes the sum of all pairwise distances this results in serious overlaps between classes that are close to each other in the input space and degrades classification performance our research proposes pareto discriminant analysis parda an approach for multiclass discriminative analysis that builds over multiobjective optimizing models parda decomposes the multiclass problem to a set of objective functions each representing the distance between every pair of classes unlike existing lda extensions that maximize the sum of all distances parda maximizes each pairwise distance to maximally separate all class means while minimizing the class overlap in the lower dimensional space experimental results on various data sets show consistent and promising performance of parda when compared with well known multiclass lda extensions highlightswe address the class masking problem cmp in multiclass lda mlda we model mlda as a set of objective functions for pairwise distances between classes we optimize the compound objective function using multiobjective optimization our models overcome the cmp and are extended to multimodal non gaussian data empirical analysis shows consistent and promising performance in favor of our models

max min distance analysis by using sequential sdp relaxation for dimension reduction we propose a new criterion for discriminative dimension reduction max min distance analysis mmda given a data set with c classes represented by homoscedastic gaussians mmda maximizes the minimum pairwise distance of these c classes in the selected low dimensional subspace thus unlike fisher s linear discriminant analysis flda and other popular discriminative dimension reduction criteria mmda duly considers the separation of all class pairs to deal with general case of data distribution we also extend mmda to kernel mmda kmmda dimension reduction via mmda kmmda leads to a nonsmooth max min optimization problem with orthonormal constraints we develop a sequential convex relaxation algorithm to solve it approximately to evaluate the effectiveness of the proposed criterion and the corresponding algorithm we conduct classification and data visualization experiments on both synthetic data and real data sets experimental results demonstrate the effectiveness of mmda kmmda associated with the proposed optimization algorithm

some comments on cp i study the typical configuration of a cp plot when the number of variables in the regression problem is large and there are many weak effects i show that a particular configuration that is very commonly seen can arise in a simple way i give a formula by means of which the risk incurred by the minimum cp rule can be estimated

solution of minimax problems using equivalent differentiable functions abstract a method is proposed for the solution of minimax optimization problems in which the individual functions involved are convex the method consists of solving a problem with an objective function which is the sum of high powers or strong exponentials of the separate components of the original objective function the resulting objective function which is equivalent at the limit to the minimax one is shown to be smooth as well as convex any efficient nonlinear programming method can be utilized for solving the equivalent problem a number of examples are discussed

bayes optimality in linear discriminant analysis we present an algorithm that provides the one dimensional subspace where the bayes error is minimized for the c class problem with homoscedastic gaussian distributions our main result shows that the set of possible one dimensional spaces v for which the order of the projected class means is identical defines a convex region with associated convex bayes error function g v this allows for the minimization of the error function using standard convex optimization algorithms our algorithm is then extended to the minimization of the bayes error in the more general case of heteroscedastic distributions this is done by means of an appropriate kernel mapping function this result is further extended to obtain the d dimensional solution for any given d by iteratively applying our algorithm to the null space of the d l dimensional solution we also show how this result can be used to improve upon the outcomes provided by existing algorithms and derive a low computational cost linear approximation extensive experimental validations are provided to demonstrate the use of these algorithms in classification data analysis and visualization

subclass discriminant analysis over the years many discriminant analysis da algorithms have been proposed for the study of high dimensional data in a large variety of problems each of these algorithms is tuned to a specific type of data distribution that which best models the problem at hand unfortunately in most problems the form of each class pdf is a priori unknown and the selection of the da algorithm that best fits our data is done over trial and error ideally one would like to have a single formulation which can be used for most distribution types this can be achieved by approximating the underlying distribution of each class with a mixture of gaussians in this approach the major problem to be addressed is that of determining the optimal number of gaussians per class i e the number of subclasses in this paper two criteria able to find the most convenient division of each class into a set of subclasses are derived extensive experimental results are shown using five databases comparisons are given against linear discriminant analysis lda direct lda dlda heteroscedastic lda hlda nonparametric da nda and kernel based lda k lda we show that our method is always the best or comparable to the best

multiclass linear dimension reduction by weighted pairwise fisher criteria we derive a class of computationally inexpensive linear dimension reduction criteria by introducing a weighted variant of the well known k class fisher criterion associated with linear discriminant analysis lda it can be seen that lda weights contributions of individual class pairs according to the euclidean distance of the respective class means we generalize upon lda by introducing a different weighting function

a manifold of pure gibbs states of the ising model on a cayley tree we study the ising model on a cayley tree a wide class of new extreme gibbs states is exhibited

a decision cognizant kullback leibler divergence in decision making systems involving multiple classifiers there is the need to assess classifier in congruence that is to gauge the degree of agreement between their outputs a commonly used measure for this purpose is the kullback leibler kl divergence we propose a variant of the kl divergence named decision cognizant kullback leibler divergence dc kl to reduce the contribution of the minority classes which obscure the true degree of classifier incongruence we investigate the properties of the novel divergence measure analytically and by simulation studies the proposed measure is demonstrated to be more robust to minority class clutter its sensitivity to estimation noise is also shown to be considerably lower than that of the classical kl divergence these properties render the dc kl divergence a much better statistic for discriminating between classifier congruence and incongruence in pattern recognition systems

generalized discriminant analysis using a kernel approach we present a new method that we call generalized discriminant analysis gda to deal with nonlinear discriminant analysis using kernel function operator the underlying theory is close to the support vector machines svm insofar as the gda method provides a mapping of the input vectors into high dimensional feature space in the transformed space linear properties make it easy to extend and generalize the classical linear discriminant analysis lda to nonlinear discriminant analysis the formulation is expressed as an eigenvalue problem resolution using a different kernel one can cover a wide class of nonlinearities for both simulated data and alternate kernels we give classification results as well as the shape of the decision function the results are confirmed using real data to perform seed classification

face recognition using lda based algorithms low dimensional feature representation with enhanced discriminatory power is of paramount importance to face recognition fr systems most of traditional linear discriminant analysis lda based methods suffer from the disadvantage that their optimality criteria are not directly related to the classification ability of the obtained feature representation moreover their classification accuracy is affected by the small sample size sss problem which is often encountered in fr tasks in this paper we propose a new algorithm that deals with both of the shortcomings in an efficient and cost effective manner the proposed method is compared in terms of classification accuracy to other commonly used fr methods on two face databases results indicate that the performance of the proposed method is overall superior to those of traditional fr approaches such as the eigenfaces fisherfaces and d lda methods

fractional step dimensionality reduction linear projections for dimensionality reduction computed using linear discriminant analysis lda are commonly based on optimization of certain separability criteria in the output space the resulting optimization problem is linear but these separability criteria are not directly related to the classification accuracy in the output space consequently a trial and error procedure has to be invoked experimenting with different separability criteria that differ in the weighting function used and selecting the one that performed best on the training set often even the best weighting function among the trial choices results in poor classification of data in the subspace in this short paper we introduce the concept of fractional dimensionality and develop an incremental procedure called the fractional step lda f lda to reduce the dimensionality in fractional steps the f lda algorithm is more robust to the selection of weighting function and for any given weighting function it finds a subspace in which the classification accuracy is higher than that obtained using lda

linear dimensionality reduction via a heteroscedastic extension of lda the chernoff criterion we propose an eigenvector based heteroscedastic linear dimension reduction ldr technique for multiclass data the technique is based on a heteroscedastic two class technique which utilizes the so called chernoff criterion and successfully extends the well known linear discriminant analysis lda the latter which is based on the fisher criterion is incapable of dealing with heteroscedastic data in a proper way for the two class case the between class scatter is generalized so to capture differences in co variances it is shown that the classical notion of between class scatter can be associated with euclidean distances between class means from this viewpoint the between class scatter is generalized by employing the chernoff distance measure leading to our proposed heteroscedastic measure finally using the results from the two class case a multiclass extension of the chernoff criterion is proposed this criterion combines separation information present in the class mean as well as the class covariance matrices extensive experiments and a comparison with similar dimension reduction techniques are presented

the bivar criterion for selecting regressors a criterion is developed from the sum of weighted squared bias and variance for use in selecting regressors we show that it is a generalization of the cp criterion and describe a graphical analysis of it its relationship with other selection criteria is discussed examples illustrating its use are included

enhanced fisher discriminant criterion for image recognition many previous studies have shown that image recognition can be significantly improved by fisher linear discriminant analysis flda technique however flda ignores the variation of data points from the same class which characterizes the most important modes of variability of patterns and helps to improve the generalization capability of flda thus the performance of flda on testing data is not good enough in this paper we propose an enhanced fisher discriminant criterion efdc efdc explicitly considers the intra class variation and incorporates the intra class variation into the fisher discriminant criterion to build a robust and efficient dimensionality reduction function efdc obtains a subspace which best detects the discriminant structure and simultaneously preserves the modes of variability of patterns which will result in stable intraclass representation experimental results on four image database show a clear improvement over the results of flda based methods

linear dimensionality reduction using relevance weighted lda the linear discriminant analysis lda is one of the most traditional linear dimensionality reduction methods this paper incorporates the inter class relationships as relevance weights into the estimation of the overall within class scatter matrix in order to improve the performance of the basic lda method and some of its improved variants we demonstrate that in some specific situations the standard multi class lda almost totally fails to find a discriminative subspace if the proposed relevance weights are not incorporated in order to estimate the relevance weights of individual within class scatter matrices we propose several methods of which one employs the evolution strategies