a bound on the spectral radius of graphs with e edges the spectral radius p a of the adjacency matrix a of a graph g with e edges satisfiesp a i l equalityoccnrsifandonlyif e i andgisa disjoint union of the complete graph k and isolated vertices 0

eigenfaces vs fisherfaces recognition using class specific linear projection we develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression taking a pattern classification approach we consider each pixel in an image as a coordinate in a high dimensional space we take advantage of the observation that the images of a particular face under varying illumination but fixed pose lie in a 3d linear subspace of the high dimensional image space if the face is a lambertian surface without shadowing however since faces are not truly lambertian surfaces and do indeed produce self shadowing images will deviate from this linear subspace rather than explicitly modeling this deviation we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation our projection method is based on fisher s linear discriminant and produces well separated classes in a low dimensional subspace even under severe variation in lighting and facial expressions the eigenface technique another method based on linearly projecting the image space to a low dimensional subspace has similar computational requirements yet extensive experimental results demonstrate that the proposed fisherface method has error rates that are lower than those of the eigenface technique for tests on the harvard and yale face databases

enhanced graph based dimensionality reduction with repulsion laplaceans graph based methods for linear dimensionality reduction have recently attracted much attention and research efforts the main goal of these methods is to preserve the properties of a graph representing the affinity between data points in local neighborhoods of the high dimensional space it has been observed that in general supervised graph methods outperform their unsupervised peers in various classification tasks supervised graphs are typically constructed by allowing two nodes to be adjacent only if they are of the same class however such graphs are oblivious to the proximity of data from different classes in this paper we propose a novel methodology which builds on repulsion graphs i e graphs that model undesirable proximity between points the main idea is to repel points from different classes that are close by in the input high dimensional space the proposed methodology is generic and can be applied to any graph based method for linear dimensionality reduction we provide ample experimental evidence in the context of face recognition which shows that the proposed methodology i offers significant performance improvement to various graph based methods and ii outperforms existing solutions relying on repulsion forces

distance metric learning by minimal distance maximization classic linear dimensionality reduction ldr methods such as principal component analysis pca and linear discriminant analysis lda are known not to be robust against outliers following a systematic analysis of the multi class ldr problem in a unified framework we propose a new algorithm called minimal distance maximization mdm to address the non robustness issue the principle behind mdm is to maximize the minimal between class distance in the output space mdm is formulated as a semi definite program sdp and its dual problem reveals a close connection to weighted ldr methods a soft version of mdm in which lda is subsumed as a special case is also developed to deal with overlapping centroids finally we drop the homoscedastic gaussian assumption made in mdm by extending it in a non parametric way along with a gradient based convex approximation algorithm to significantly reduce the complexity of the original sdp the effectiveness of our proposed methods are validated on two uci datasets and two face datasets

a note on upper bounds for the spectral radius of weighted graphs let g v e be a simple connected weighted graph on n vertices in which the edge weights are positive definite matrices the eigenvalues of g are the eigenvalues of its adjacency matrix in this note we present a correction in equality part in theorem 2 s sorgun s buyukkose the new upper bounds on the spectral radius of weighted graphs appl math comput 218 2012 5231 5238 in addition some related results are also provided

a unified framework for semi supervised dimensionality reduction in practice many applications require a dimensionality reduction method to deal with the partially labeled problem in this paper we propose a semi supervised dimensionality reduction framework which can efficiently handle the unlabeled data under the framework several classical methods such as principal component analysis pca linear discriminant analysis lda maximum margin criterion mmc locality preserving projections lpp and their corresponding kernel versions can be seen as special cases for high dimensional data we can give a low dimensional embedding result for both discriminating multi class sub manifolds and preserving local manifold structure experiments show that our algorithms can significantly improve the accuracy rates of the corresponding supervised and unsupervised approaches

on the spectral radius of 0 1 matrices we determine the maximum spectral radius for 0 1 matrices with k2 andk2 1 1 s respectively and for symmetric 0 1 matrices with zero trace and e k21 s graphs with e edges in all cases equality is characterized

trace ratio problem revisited dimensionality reduction is an important issue in many machine learning and pattern recognition applications and the trace ratio tr problem is an optimization problem involved in many dimensionality reduction algorithms conventionally the solution is approximated via generalized eigenvalue decomposition due to the difficulty of the original problem however prior works have indicated that it is more reasonable to solve it directly than via the conventional way in this brief we propose a theoretical overview of the global optimum solution to the tr problem via the equivalent trace difference problem eigenvalue perturbation theory is introduced to derive an efficient algorithm based on the newton raphson method theoretical issues on the convergence and efficiency of our algorithm compared with prior literature are proposed and are further supported by extensive empirical results

max min distance analysis by using sequential sdp relaxation for dimension reduction we propose a new criterion for discriminative dimension reduction max min distance analysis mmda given a data set with c classes represented by homoscedastic gaussians mmda maximizes the minimum pairwise distance of these c classes in the selected low dimensional subspace thus unlike fisher s linear discriminant analysis flda and other popular discriminative dimension reduction criteria mmda duly considers the separation of all class pairs to deal with general case of data distribution we also extend mmda to kernel mmda kmmda dimension reduction via mmda kmmda leads to a nonsmooth max min optimization problem with orthonormal constraints we develop a sequential convex relaxation algorithm to solve it approximately to evaluate the effectiveness of the proposed criterion and the corresponding algorithm we conduct classification and data visualization experiments on both synthetic data and real data sets experimental results demonstrate the effectiveness of mmda kmmda associated with the proposed optimization algorithm

an efficient algorithm to solve the small sample size problem for lda in this paper we present an efficient algorithm to solve the most discriminant vectors of lda for high dimensional data set the experiments on orl face database confirm the effectiveness of the proposed method

kpca plus lda a complete kernel fisher discriminant framework for feature extraction and recognition this paper examines the theory of kernel fisher discriminant analysis kfd in a hilbert space and develops a two phase kfd framework i e kernel principal component analysis kpca plus fisher linear discriminant analysis lda this framework provides novel insights into the nature of kfd based on this framework the authors propose a complete kernel fisher discriminant analysis ckfd algorithm ckfd can be used to carry out discriminant analysis in double discriminant subspaces the fact that it can make full use of two kinds of discriminant information regular and irregular makes ckfd a more powerful discriminator the proposed algorithm was tested and evaluated using the feret face database and the cenparmi handwritten numeral database the experimental results show that ckfd outperforms other kfd algorithms

geometric mean for subspace selection subspace selection approaches are powerful tools in pattern classification and data visualization one of the most important subspace approaches is the linear dimensionality reduction step in the fisher s linear discriminant analysis flda which has been successfully employed in many fields such as biometrics bioinformatics and multimedia information management however the linear dimensionality reduction step in flda has a critical drawback for a classification task with c classes if the dimension of the projected subspace is strictly lower than c 1 the projection to a subspace tends to merge those classes which are close together in the original feature space if separate classes are sampled from gaussian distributions all with identical covariance matrices then the linear dimensionality reduction step in flda maximizes the mean value of the kullback leibler kl divergences between different classes based on this viewpoint the geometric mean for subspace selection is studied in this paper three criteria are analyzed 1 maximization of the geometric mean of the kl divergences 2 maximization of the geometric mean of the normalized kl divergences and 3 the combination of 1 and 2 preliminary experimental results based on synthetic data uci machine learning repository and handwriting digits show that the third criterion is a potential discriminative subspace selection method which significantly reduces the class separation problem in comparing with the linear dimensionality reduction step in flda and its several representative extensions

subclass discriminant analysis over the years many discriminant analysis da algorithms have been proposed for the study of high dimensional data in a large variety of problems each of these algorithms is tuned to a specific type of data distribution that which best models the problem at hand unfortunately in most problems the form of each class pdf is a priori unknown and the selection of the da algorithm that best fits our data is done over trial and error ideally one would like to have a single formulation which can be used for most distribution types this can be achieved by approximating the underlying distribution of each class with a mixture of gaussians in this approach the major problem to be addressed is that of determining the optimal number of gaussians per class i e the number of subclasses in this paper two criteria able to find the most convenient division of each class into a set of subclasses are derived extensive experimental results are shown using five databases comparisons are given against linear discriminant analysis lda direct lda dlda heteroscedastic lda hlda nonparametric da nda and kernel based lda k lda we show that our method is always the best or comparable to the best

complete large margin linear discriminant analysis using mathematical programming approach in this paper we develop a novel dimensionality reduction dr framework coined complete large margin linear discriminant analysis clmlda inspired by several recently proposed dr methods clmlda constructs two mathematical programming models by maximizing the minimum distance between each class center and the total class center respectively in the null space of within class scatter matrix and its orthogonal complementary space in this way clmlda not only makes full use of the discriminative information contained in the whole feature space but also overcome the weakness of linear discriminant analysis lda in dealing with the class separation problem the solutions of clmlda follow from solving two nonconvex optimization problems each of which is transformed to a series of convex quadratic programming problems by using the constrained concave convex procedure first and then solved by off the shelf optimization toolbox experiments on both toy and several publicly available databases demonstrate its feasibility and effectiveness

a new lda based face recognition system which can solve the small sample size problem a new lda based face recognition system is presented in this paper linear discriminant analysis lda is one of the most popular linear projection techniques for feature extraction the major drawback of applying lda is that it may encounter the small sample size problem in this paper we propose a new lda based technique which can solve the small sample size problem we also prove that the most expressive vectors derived in the null space of the within class scatter matrix using principal component analysis pca are equal to the optimal discriminant vectors derived in the original space using lda the experimental results show that the new lda process improves the performance of a face recognition system signi cantly 2000 pattern recognition society published by elsevier science ltd all rights reserved

multiclass linear dimension reduction by weighted pairwise fisher criteria we derive a class of computationally inexpensive linear dimension reduction criteria by introducing a weighted variant of the well known k class fisher criterion associated with linear discriminant analysis lda it can be seen that lda weights contributions of individual class pairs according to the euclidean distance of the respective class means we generalize upon lda by introducing a different weighting function

efficient and robust feature extraction by maximum margin criterion in pattern recognition feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information principal component analysis pca and linear discriminant analysis lda are the two most popular linear dimensionality reduction methods however pca is not very effective for the extraction of the most discriminant features and lda is not stable due to the small sample size problem in this paper we propose some new linear and nonlinear feature extractors based on maximum margin criterion mmc geometrically feature extractors based on mmc maximize the average margin between classes after dimensionality reduction it is shown that mmc can represent class separability better than pca as a connection to lda we may also derive lda from mmc by incorporating some constraints by using some other constraints we establish a new linear feature extractor that does not suffer from the small sample size problem which is known to cause serious stability problems for lda the kernelized nonlinear counterpart of this linear feature extractor is also established in the paper our extensive experiments demonstrate that the new feature extractors are effective stable and efficient

improved global convergence probability using multiple independent optimizations for some problems global optimization algorithms may have a significant probability of not converging to the global optimum or require an extremely large number of function evaluations to reach it for such problems the probability of finding the global optimum may be improved by performing multiple independent short searches rather than using the entire available budget of function evaluations on a single long search the main difficulty in adopting such a strategy is to decide how many searches to carry out for a given function evaluation budget the basic premise of this paper is that different searches may have substantially different outcomes but they all start with rapid initial improvement of the objective function followed by much slower progress later on furthermore we assume that the number of function evaluations to the end of the initial stage of rapid progress does not change drastically from one search to another for a given problem and algorithmic setting therefore we propose that the number of function evaluations required for this rapid progress stage be estimated with one or two runs and then the same number of function evaluations be allocated to all subsequent searches we show that these assumptions work well for the particle swarm optimization algorithm applied to a set of difficult analytical test problems with known global solutions for these problems we show that the proposed strategy can substantially improve the probability of obtaining the global optimum for a given budget of function evaluations we also test a bayesian criterion for estimating the probability of having reached the global optimum at the end of the series of searches and find that it can provide a conservative estimate for most problems finally we demonstrate the approach on a particularly challenging engineering design problem constructed so as to have at least 32 widely separated local optima copyright 2006 john wiley sons ltd

laplacian eigenmaps for dimensionality reduction and data representation one of the central problems in machine learning and pattern recognition is to develop appropriate representations for complex data we consider the problem of constructing a representation for data lying on a low dimensional manifold embedded in a high dimensional space drawing on the correspondence between the graph laplacian the laplace beltrami operator on the manifold and the connections to the heat equation we propose a geometrically motivated algorithm for representing the high dimensional data the algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality preserving properties and a natural connection to clustering some potential applications and illustrative examples are discussed

regularized max min linear discriminant analysis several dimensionality reduction methods based on the max min idea have been proposed in recent years and can obtain good classification performance in this paper inspired by the idea we develop max min linear discriminant analysis mmlda which maximizes the minimum ratio of each two class scatter measure to the within class scatter measure however the method ignores equal emphasis on the distances between class centers and there may be room to improve the classification performance we then propose regularized max min linear discriminant analysis rmmlda which introduces the shannon entropy and the corresponding distance difference regularization terms based on mmlda the changing trends of distances between class centers can be precisely controlled in optimization and the separation between classes can be emphasized approximately equally as a result rmmlda may obtain better classification performance experiments on synthetic data sets and three publicly available data sets demonstrate its effectiveness highlightswe propose max min linear discriminant analysis we present regularized max min linear discriminant analysis using shannon entropy the improved form can obtain better classification results for high dimensional data

extremal graph characterization from the bounds of the spectral radius of weighted graphs abstract we consider weighted graphs where the edge weights are positive definite matrices the eigenvalues of a graph are the eigenvalues of its adjacency matrix we obtain a lower bound and an upper bound on the spectral radius of the adjacency matrix of weighted graphs and characterize graphs for which the bounds are attained

fractional step dimensionality reduction linear projections for dimensionality reduction computed using linear discriminant analysis lda are commonly based on optimization of certain separability criteria in the output space the resulting optimization problem is linear but these separability criteria are not directly related to the classification accuracy in the output space consequently a trial and error procedure has to be invoked experimenting with different separability criteria that differ in the weighting function used and selecting the one that performed best on the training set often even the best weighting function among the trial choices results in poor classification of data in the subspace in this short paper we introduce the concept of fractional dimensionality and develop an incremental procedure called the fractional step lda f lda to reduce the dimensionality in fractional steps the f lda algorithm is more robust to the selection of weighting function and for any given weighting function it finds a subspace in which the classification accuracy is higher than that obtained using lda

linear dimensionality reduction via a heteroscedastic extension of lda the chernoff criterion we propose an eigenvector based heteroscedastic linear dimension reduction ldr technique for multiclass data the technique is based on a heteroscedastic two class technique which utilizes the so called chernoff criterion and successfully extends the well known linear discriminant analysis lda the latter which is based on the fisher criterion is incapable of dealing with heteroscedastic data in a proper way for the two class case the between class scatter is generalized so to capture differences in co variances it is shown that the classical notion of between class scatter can be associated with euclidean distances between class means from this viewpoint the between class scatter is generalized by employing the chernoff distance measure leading to our proposed heteroscedastic measure finally using the results from the two class case a multiclass extension of the chernoff criterion is proposed this criterion combines separation information present in the class mean as well as the class covariance matrices extensive experiments and a comparison with similar dimension reduction techniques are presented

a novel visual codebook model based on fuzzy geometry for large scale image classification the codebook model has been developed as an effective means for image classification however the inherent operation of assigning visual words to image feature vectors in traditional codebook approaches causes serious ambiguities in image classification in particular the nearest word may not be the best fit to a feature and multiple words may be equally appropriate for one specific feature to resolve these ambiguities we propose a novel visual codebook model based on the n dimensional fuzzy geometry n d fg theory where all visual words and features are modeled as fuzzy points in the n d fg space and appropriate uncertainty is introduced to each fuzzy point to enhance the representation capacity this n d fg codebook model not only inherits advantages from the fuzzy set theory but also facilitates the analysis and determination of the relationship between visual words and features in geometric form by explicitly taking into account the ambiguities we propose a novel measure of similarity between the visual words and fuzzy features following the proposed codebook model and the novel similarity measure we develop two useful image classification algorithms by modifying popular image coding algorithms i e spm and llc finally experimental results demonstrate that the classification accuracy of the proposed algorithms is dramatically improved for a standard large scale image database for example with a codebook size of 256 the proposed algorithms achieve similar performance as traditional algorithms with a codebook size of 1024 indicating that the proposed algorithms reduce the computational cost by 75 while achieving almost identical classification accuracy to traditional algorithms thus the proposed algorithms represent a more efficient and appropriate scheme for big image data this paper aims to overcome the drawbacks of traditional visual words fuzzy visual word and fuzzy feature are defined using n dimensional fuzzy geometry a new similarity measure between fuzzy features and fuzzy visual words is designed two modified image classification frameworks based on fuzzy codebook are proposed experimental results demonstrated their advantages against traditional algorithms

principal manifolds and nonlinear dimensionality reduction via tangent space alignment we present a new algorithm for manifold learning and nonlinear dimensionality reduction based on a set of unorganized data points sampled with noise from a parameterized manifold the local geometry of the manifold is learned by constructing an approximation for the tangent space at each data point and those tangent spaces are then aligned to give the global coordinates of the data points with respect to the underlying manifold we also present an error analysis of our algorithm showing that reconstruction errors can be quite small in some cases we illustrate our algorithm using curves and surfaces both in two dimensional three dimensional 2d 3d euclidean spaces and in higher dimensional euclidean spaces we also address several theoretical and algorithmic issues for further research and improvements