discriminant analysis by gaussian mixtures fisher rao linear discriminant analysis lda is a valuable tool for multigroup classification lda is equivalent to maximum likelihood classification assuming gaussian distributions for each class in this paper we fit gaussian mixtures to each class to facilitate effective classification in non normal settings especially when the classes are clustered low dimensional views are an important by product of lda our new techniques inherit this feature we can control the within class spread of the subclass centres relative to the between class spread our technique for fitting these models permits a natural blend with nonparametric versions of lda

discriminant subspace analysis a fukunaga koontz approach the fisher linear discriminant fld is commonly used in pattern recognition it finds a linear subspace that maximally separates class patterns according to the fisher criterion several methods of computing the fld have been proposed in the literature most of which require the calculation of the so called scatter matrices in this paper we bring a fresh perspective to fld via the fukunaga koontz transform fkt we do this by decomposing the whole data space into four subspaces with different discriminabilities as measured by eigenvalue ratios by connecting the eigenvalue ratio with the generalized eigenvalue we show where the fisher criterion is maximally satisfied we prove the relationship between fld and fkt analytically and propose a unified framework to understanding some existing work furthermore we extend our theory to the multiple discriminant analysis mda this is done by transforming the data into intraclass and extraclass spaces followed by maximizing the bhattacharyya distance based on our fkt analysis we identify the discriminant subspaces of mda fkt and propose an efficient algorithm which works even when the scatter matrices are singular or too large to be formed our method is general and may be applied to different pattern recognition problems we validate our method by experimenting on synthetic and real data

feature extraction by non parametric mutual information maximization we present a method for learning discriminative feature transforms using as criterion the mutual information between class labels and transformed features instead of a commonly used mutual information measure based on kullback leibler divergence we use a quadratic divergence measure which allows us to make an efficient non parametric implementation and requires no prior assumptions about class densities in addition to linear transforms we also discuss nonlinear transforms that are implemented as radial basis function networks extensions to reduce the computational complexity are also presented and a comparison to greedy feature selection is made

discriminative sparsity preserving projections for image recognition previous works have demonstrated that image classification performance can be significantly improved by manifold learning however performance of manifold learning heavily depends on the manual selection of parameters resulting in bad adaptability in real world applications in this paper we propose a new dimensionality reduction method called discriminative sparsity preserving projections dspp different from the existing sparse subspace algorithms which manually construct a penalty adjacency graph dspp employs sparse representation model to adaptively build both intrinsic adjacency graph and penalty graph with weight matrix and then integrates global within class structure into the discriminant manifold learning objective function for dimensionality reduction extensive experimental results on four image databases demonstrate the effectiveness of the proposed approach we analyzed the vertex points of penalty graph using sparse representation we adaptively constructed both penalty adjacency graph and intrinsic graph our method simultaneously considered the global and local geometric structure of data

feature extraction using constrained approximation and suppression in this paper we systematize a family of constrained quadratic classifiers that belong to the class of one class classifiers one class classifiers such as the single class support vector machine or the subspace methods are widely used for pattern classification and detection problems because they have many advantages over binary classifiers we interpret subspace methods as rank constrained quadratic classifiers in the framework we also introduce two constraints and a method of suppressing the effect of competing classes to make them more accurate and retain their advantages over binary classifiers experimental results demonstrate the advantages of our methods over conventional classifiers

conjugate priors for exponential families having quadratic variance functions abstract consider a natural exponential family parameterized by it is well known that the standard conjugate prior on is characterized by a condition of posterior linearity for the expectation of the model mean parameter often however this family is not parameterized in terms of but rather in terms of a more usual parameter such at the mean the main question we address is under what conditions does a standard conjugate prior on induce a linear posterior expectation on itself we prove that essentially this happens iff the exponential family has quadratic variance function a consequence of this result is that the standard conjugate on coincides with the prior on induced by the standard conjugate on iff the variance function is quadratic the rest of the article covers more specific issues related to conjugate priors for exponential families in particular we analyze the monotonicity of the expected posterior variance for with respect to the sample size and the hyperparameter

conjugate parameterizations for natural exponential families abstract recently consonni and veronese have shown that the form of the standard conjugate distribution for the mean parameter of a univariate natural exponential family f coincides with that of the distribution induced on by the standard conjugate distribution for the canonical parameter if and only if f has a quadratic variance function in this article we present significant extensions of this result identifying conditions under which transformations of the canonical or mean parameter preserve the form of the standard conjugate family generalizations to the multivariate case are also considered and results relating jeffreys s prior to the standard conjugate family are presented the variance function is seen to play an important role throughout

clustering of multivariate binary data with dimension reduction via l1 regularized likelihood maximization clustering methods with dimension reduction have been receiving considerable wide interest in statistics lately and a lot of methods to simultaneously perform clustering and dimension reduction have been proposed this work presents a novel procedure for simultaneously determining the optimal cluster structure for multivariate binary data and the subspace to represent that cluster structure the method is based on a finite mixture model of multivariate bernoulli distributions and each component is assumed to have a low dimensional representation of the cluster structure this method can be considered as an extension of the traditional latent class analysis sparsity is introduced to the loading values which produces the low dimensional subspace for enhanced interpretability and more stable extraction of the subspace an em based algorithm is developed to efficiently solve the proposed optimization problem we demonstrate the effectiveness of the proposed method by applying it to a simulation study and real datasets highlightswe propose a novel clustering method of multivariate binary data the proposed method provides a low dimensional representation of clusters the proposed method overcomes the conventional tandem analysis the proposed method conducts feature selection for clustering

single observation unbiased priors this paper studies a class of default priors which we call single observation unbiased priors soup a prior for a parameter is a soup if the corresponding posterior mean of the parameter based on a single observation is an unbiased estimator of the parameter we prove that under mild regularity conditions a default prior for a convolution parameter is noninformative in the sense of yielding a posterior inference invariant under amalgamation only if it is a soup therefore when amalgamation invariance is desirable as in our motivating example of performing imputation for census undercount soup is the only possible coherent noninformative prior for bayesian predictions that will be utilized under aggregation the use of soup also mutually calibrates bayesian and frequentist inferences for aggregates of convolution parameters across many small areas we describe approaches that identify soups in many common models in particular a constructive duality method that identifies soups in pairs of distribution families we introduce o completeness a necessary and sufficient condition for a prior distribution to be uniquely characterized by the corresponding posterior mean uniqueness of the soup is determined by the o completeness of the dual family o completeness of a natural exponential family is implied by its completeness hence the diaconis ylvisaker characterization of the conjugate prior for natural exponential families via linear posterior expectation is a direct consequence of the completeness of such families for most of the examples we have examined the inverse of the variance function is the soup for the mean parameter of the corresponding family suggesting that hartigan s results on asymptotic unbiasedness can be generalized to some families with discrete parameters we also discuss a possible extension of berger s result on the inadmissibility of unbiased estimators as the nonexistence of soup can be a first step in establishing such inadmissibility

separation of non negative mixture of non negative sources using a bayesian approach and mcmc sampling this paper addresses blind source separation in the case where both the source signals and the mixing coefficients are non negative the problem is referred to as non negative source separation and the main application concerns the analysis of spectrometric data sets the separation is performed in a bayesian framework by encoding non negativity through the assignment of gamma priors on the distributions of both the source signals and the mixing coefficients a markov chain monte carlo mcmc sampling procedure is proposed to simulate the resulting joint posterior density from which marginal posterior mean estimates of the source signals and mixing coefficients are obtained results obtained with synthetic and experimental spectra are used to discuss the problem of non negative source separation and to illustrate the effectiveness of the proposed method

sanov results for glauber spin glass dynamics in this paper we prove a sanov result i e a large deviation principle ldp for the distribution of the empirical measure for the annealed glauber dynamics of the sherrington kirkpatrick spin glass without restrictions on time or temperature we prove a full ldp for the asymmetric dynamics and the crucial upper large deviations bound for the symmetric dynamics in the symmetric model a new order parameter arises corresponding to the response function in sozi83 in the asymmetric case we show that the corresponding rate function has a unique minimum given as the solution of a self consistent equation the key argument used in the proofs is a general result for mixing of what is known as large deviation systems lds with measures obeying an independent ldp

symmetric langevin spin glass dynamics we study the asymptotic behavior of symmetric spin glass dynamics in the sherrington kirkpatrick model as proposed by sompolinsky zippelius we prove that the averaged law of the empirical measure on the path space of these dynamics satisfies a large deviation upper bound in the high temperature regime we study the rate function which governs this large deviation upper bound and prove that it achieves its minimum value at a unique probability measure q which is not markovian we deduce an averaged and a quenched law of large numbers we then study the evolution of the gibbs measure of a spin glass under sompolinsky zippelius dynamics we also prove a large deviation upper bound for the law of the empirical measure and describe the asymptotic behavior of a spin on path space under this dynamic in the high temperature regime

normalized cuts and image segmentation we propose a novel approach for solving the perceptual grouping problem in vision rather than focusing on local features and their consistencies in the image data our approach aims at extracting the global impression of an image we treat image segmentation as a graph partitioning problem and propose a novel global criterion the normalized cut for segmenting the graph the normalized cut criterion measures both the total dissimilarity between the different groups as well as the total similarity within the groups we show that an efficient computational technique based on a generalized eigenvalue problem can be used to optimize this criterion we applied this approach to segmenting static images as well as motion sequences and found the results to be very encouraging

an algorithm for pole assignment in high order multivariable systems an efficient computational method is presented for solving the problem of pole assignment by state feedback in linear multivariable systems with large dimensions a given multiinput system is first transformed to an upper block hessenberg form by means of orthogonal state coordinate transformations the state feedback problem is then reformulated in terms of the sylvester equation the transformed system matrices along with certain assumed block forms for unknown matrices enable the sylvester equation to be decomposed and solved effectively a distinct point of the proposed algorithm is that the solution procedure can be tailored to parallel implementation and is therefore fast computational aspects of the algorithm are discussed and numerical examples are provided

hybrid dimensionality reduction method based on support vector machine and independent component analysis this paper presents a new hybrid dimensionality reduction method to seek projection through optimization of both structural risk supervised criterion and data independence unsupervised criterion classification accuracy is used as a metric to evaluate the performance of the method by minimizing the structural risk projection originated from the decision boundaries directly improves the classification performance from a supervised perspective from an unsupervised perspective projection can also be obtained based on maximum independence among features or attributes in data to indirectly achieve better classification accuracy over more intrinsic representation of the data orthogonality interrelates the two sets of projections such that minimum redundancy exists between the projections leading to more effective dimensionality reduction experimental results show that the proposed hybrid dimensionality reduction method that satisfies both criteria simultaneously provides higher classification performance especially for noisy data sets in relatively lower dimensional space than various existing methods

the 2d 4 simple quadratic natural exponential families on bf r sp d the present paper describes all the natural exponential families on r d whose variance function is of the form v m am o m b m c with m o m m and b linear in m there are 2d 4 types of such families which are built from particular mixtures of families of normal poisson gamma hyperbolic on r and negative multinomial distributions the proof of this result relies mainly on techniques used in the elementary theory of lie algebras

modified quadratic discriminant functions and the application to chinese character recognition issues in the quadratic discriminant functions qdf are discussed and two types of modified quadratic disriminant functions mqdf1 mqdf2 which are less sensitive to the estimation error of the covariance matrices are proposed the mqdf1 is a function which employs a kind of a pseudo bayesian estimate of the covariance matrix instead of the maximum likelihood estimate ordinarily used in the qdf the mqdf2 is a variation of the mqdf1 to save the required computation time and storage two discriminant functions were applied to chinese character recognition to evaluate their effectiveness and remarkable improvement was observed in their performance

learning a probabilistic topology discovering model for scene categorization a recent advance in scene categorization prefers a topological based modeling to capture the existence and relationships among different scene components to that effect local features are typically used to handle photographing variances such as occlusions and clutters however in many cases the local features alone cannot well capture the scene semantics since they are extracted from tiny regions e g 4 times 4 patches within an image in this paper we mine a discriminative topology and a low redundant topology from the local descriptors under a probabilistic perspective which are further integrated into a boosting framework for scene categorization in particular by decomposing a scene image into basic components a graphlet model is used to describe their spatial interactions accordingly scene categorization is formulated as an intergraphlet matching problem the above procedure is further accelerated by introducing a probabilistic based representative topology selection scheme that makes the pairwise graphlet comparison trackable despite their exponentially increasing volumes the selected graphlets are highly discriminative and independent characterizing the topological characteristics of scene images a weak learner is subsequently trained for each topology which are boosted together to jointly describe the scene image in our experiment the visualized graphlets demonstrate that the mined topological patterns are representative to scene categories and our proposed method beats state of the art models on five popular scene data sets

simple exponential family pca principal component analysis pca is a widely used model for dimensionality reduction in this paper we address the problem of determining the intrinsic dimensionality of a general type data population by selecting the number of principal components for a generalized pca model in particular we propose a generalized bayesian pca model which deals with general type data by employing exponential family distributions model selection is realized by empirical bayesian inference of the model we name the model as simple exponential family pca sepca since it embraces both the principal of using a simple model for data representation and the practice of using a simplified computational procedure for the inference our analysis shows that the empirical bayesian inference in sepca formally realizes an intuitive criterion for pca model selection a preserved principal component must sufficiently correlate to data variance that is uncorrelated to the other principal components experiments on synthetic and real data sets demonstrate effectiveness of sepca and exemplify its characteristics for model selection

regularized discriminant analysis abstract linear and quadratic discriminant analysis are considered in the small sample high dimensional setting alternatives to the usual maximum likelihood plug in estimates for the covariance matrices are proposed these alternatives are characterized by two parameters the values of which are customized to individual situations by jointly minimizing a sample based estimate of future misclassification risk computationally fast implementations are presented and the efficacy of the approach is examined through simulation studies and application to data these studies indicate that in many circumstances dramatic gains in classification accuracy can be achieved

feature extraction with deep neural networks by a generalized discriminant analysis we present an approach to feature extraction that is a generalization of the classical linear discriminant analysis lda on the basis of deep neural networks dnns as for lda discriminative features generated from independent gaussian class conditionals are assumed this modeling has the advantages that the intrinsic dimensionality of the feature space is bounded by the number of classes and that the optimal discriminant function is linear unfortunately linear transformations are insufficient to extract optimal discriminative features from arbitrarily distributed raw measurements the generalized discriminant analysis gerda proposed in this paper uses nonlinear transformations that are learnt by dnns in a semisupervised fashion we show that the feature extraction based on our approach displays excellent performance on real world recognition and detection tasks such as handwritten digit recognition and face detection in a series of experiments we evaluate gerda features with respect to dimensionality reduction visualization classification and detection moreover we show that gerda dnns can preprocess truly high dimensional input data to low dimensional representations that facilitate accurate predictions even if simple linear predictors or measures of similarity are used

transformations of solvents and spectral factors of matrix polynomials and their applications the relationships between solvents and spectral factors of a high degree matrix polynomial are explored various new transformations are developed to convert right left solvents into spectral factors and vice versa the transformation of a right left solvent to a left right solvent is also established the newly established algorithms are then applied to determine the spectral factorization of a matrix polynomial for optimal control problems the developed algebraic theory enhances the capability of the analysis and synthesis of a system described by a high degree matrix differential equation

the sherrington kirkpatrick model of spin glasses and stochastic calculus the high temperature case we study the fluctuations of free energy energy and entropy in the high temperature regime for the sherrington kirkpatrick model of spin glasses we introduce here a new dynamical method with the help of brownian motions and continuous martingales indexed by the square root of the inverse temperature as parameter thus formulating the thermodynamic formalism in terms of random processes the well established technique of stochastic calculus leads us naturally to prove that these fluctuations are simple gaussian processes with independent increments a generalization of a result proved by aizenman lebowitz and ruelle 1

bootstrap based regularization for low rank matrix estimation we develop a flexible framework for low rank matrix estimation that allows us to transform noise models into regularization schemes via a simple bootstrap algorithm effectively our procedure seeks an autoencoding basis for the observed matrix that is stable with respect to the specified noise model we call the resulting procedure a stable autoencoder in the simplest case with an isotropic noise model our method is equivalent to a classical singular value shrinkage estimator for non isotropic noise models e g poisson noise the method does not reduce to singular value shrinkage and instead yields new estimators that perform well in experiments moreover by iterating our stable autoencoding scheme we can automatically generate low rank estimates without specifying the target rank as a tuning parameter

regularized gaussian discriminant analysis through eigenvalue decomposition abstract friedman proposed a regularization technique rda of discriminant analysis in the gaussian framework rda uses two regularization parameters to design an intermediate classifier between the linear the quadratic the nearest means classifiers in this article we propose an alternative approach called edda that is based on the reparameterization of the covariance matrix k of a group gk in terms of its eigenvalue decomposition k k d k a k d k where k specifies the volume of density contours of gk the diagonal matrix of eigenvalues specifies its shape the eigenvectors specify its orientation variations on constraints concerning volumes shapes orientations k a k and d k lead to 14 discrimination models of interest for each model we derived the normal theory maximum likelihood parameter estimates our approach consists of selecting a model by minimizing the sample based estimate of future misclassification risk by cross validation numerical experiments on simulated and rea

unsupervised learning of finite mixture models this paper proposes an unsupervised algorithm for learning a finite mixture model from multivariate data the adjective unsupervised is justified by two properties of the algorithm 1 it is capable of selecting the number of components and 2 unlike the standard expectation maximization em algorithm it does not require careful initialization the proposed method also avoids another drawback of em for mixture fitting the possibility of convergence toward a singular estimate at the boundary of the parameter space the novelty of our approach is that we do not use a model selection criterion to choose one among a set of preestimated candidate models instead we seamlessly integrate estimation and model selection in a single algorithm our technique can be applied to any type of parametric mixture model for which it is possible to write an em algorithm in this paper we illustrate it with experiments involving gaussian mixtures these experiments testify for the good performance of our approach

block power method for computing solvents and spectral factors of matrix polynomials abstract this paper is concerned with the extension of the power method used for finding the largest eigenvalue and associated eigenvector of a matrix to its block from for computing the largest block eigenvalue and associated block eigenvector of a non symmetric matrix based on the developed block power method several algorithms are developed for solving the complete set of solvents and spectral factors of a matrix polynomial without prior knowledge of the latent roots of the matrix polynomial moreover when any right left solvent of a matrix polynomial is given the proposed method can be used to determine the corresponding left right solvent such that both right and left solvents have the same eigenspectra the matrix polynomial of interest must have distinct block solvents and a corresponding non singular polynomial matrix the established algorithms can be applied in the analysis and or design of systems described by high degree vector differential equations and or matrix fraction descriptions

the block partial fraction expansion of a matrix fraction description with repeated block poles a state space technique is developed for finding the block partial fraction expansion of an irreducible right matrix fraction description with distinct and or repeated block poles the developed method is the generalized version of the existing methods which consider only distinct block poles

computation of the principal nth roots of complex matrices a new computational algorithm for finding the principal n th roots and associated roots of complex matrices is developed via a recursive solution of a block discrete state equation

semisupervised learning based on generalized point charge models the recent years have witnessed a surge of interest in semisupervised learning numerous methods have been proposed for learning from partially labeled data in this brief a novel semisupervised learning approach based on an electrostatic field model is proposed we treat the labeled data points as point charges therefore the remaining unlabeled data points are placed in the electrostatic fields generated by these charges the labels of these unlabeled data points can be regarded as the electric potentials of the electrostatic field at their corresponding places moreover we also develop an efficient way to extend our method for out of sample data and analyze theoretically the relationship between our method and the traditional graph based methods finally the experimental results on both toy and real world data sets are provided to show the effectiveness of our method

discriminant adaptive nearest neighbor classification nearest neighbour classification expects the class conditional probabilities to be locally constant and suffers from bias in high dimensions we propose a locally adaptive form of nearest neighbour classification to try to ameliorate this curse of dimensionality we use a local linear discriminant analysis to estimate an effective metric for computing neighbourhoods we determine the local decision boundaries from centroid information and then shrink neighbourhoods in directions orthogonal to these local decision boundaries and elongate them parallel to the boundaries thereafter any neighbourhood based classifier can be employed using the modified neighbourhoods the posterior probabilities tend to be more homogeneous in the modified neighbourhoods we also propose a method for global dimension reduction that combines local dimension information in a number of examples the methods demonstrate the potential for substantial improvements over nearest neighbour classification

cascade decomposition and realization of multivariable systems via block pole and block zero placement a technique is presented for state feedback and state feedforward block decomposition of a class of multivariable systems into a block cascaded form structure having the assigned block poles and block zeros a state space technique is also presented for minimal realization of a multivariable compensator represented by a voltage transfer function matrix using the decoupled rc cascaded networks and summers

a bayesian hierarchical factorization model for vector fields factorization based techniques explain arrays of observations using a relatively small number of factors and provide an essential arsenal for multi dimensional data analysis most factorization models are however developed on general arrays of scalar values for a class of practical data arising from observing spatial signals including images it is desirable for a model to consider general observations e g handling a vector field and non exchangeable factors e g handling spatial connections between the columns and the rows of the data in this paper a probabilistic model for factorization is proposed we adopt bayesian hierarchical modeling and treat the factors as latent random variables a markov structure is imposed on the distribution of factors to account for the spatial connections the model is designed to represent vector arrays sampled from fields of continuous domains therefore a tailored observation model is developed to represent the link between the factor product and the data the proposed technique has been shown effective in analyzing optical flow fields computed on both synthetic images and real life videoclips

nenmf an optimal gradient method for nonnegative matrix factorization nonnegative matrix factorization nmf is a powerful matrix decomposition technique that approximates a nonnegative matrix by the product of two low rank nonnegative matrix factors it has been widely applied to signal processing computer vision and data mining traditional nmf solvers include the multiplicative update rule mur the projected gradient method pg the projected nonnegative least squares pnls and the active set method as however they suffer from one or some of the following three problems slow convergence rate numerical instability and nonconvergence in this paper we present a new efficient nenmf solver to simultaneously overcome the aforementioned problems it applies nesterov s optimal gradient method to alternatively optimize one factor with another fixed in particular at each iteration round the matrix factor is updated by using the pg method performed on a smartly chosen search point where the step size is determined by the lipschitz constant since nenmf does not use the time consuming line search and converges optimally at rate in optimizing each matrix factor it is superior to mur and pg in terms of efficiency as well as approximation accuracy compared to pnls and as that suffer from numerical instability problem in the worst case nenmf overcomes this deficiency in addition nenmf can be used to solve norm norm and manifold regularized nmf with the optimal convergence rate numerical experiments on both synthetic and real world datasets show the efficiency of nenmf for nmf and its variants comparing to representative nmf solvers extensive experiments on document clustering suggest the effectiveness of nenmf

noise level estimation for model selection in kernel pca denoising one of the main challenges in unsupervised learning is to find suitable values for the model parameters in kernel principal component analysis kpca for example these are the number of components the kernel and its parameters this paper presents a model selection criterion based on distance distributions mdds this criterion can be used to find the number of components and the sigma 2 parameter of radial basis function kernels by means of spectral comparison between information and noise the noise content is estimated from the statistical moments of the distribution of distances in the original dataset this allows for a type of randomization of the dataset without actually having to permute the data points or generate artificial datasets after comparing the eigenvalues computed from the estimated noise with the ones from the input dataset information is retained and maximized by a set of model parameters in addition to the model selection criterion this paper proposes a modification to the fixed size method and uses the incomplete cholesky factorization both of which are used to solve kpca in large scale applications these two approaches together with the model selection mdd were tested in toy examples and real life applications and it is shown that they outperform other known algorithms

generalized lyapunov equation and factorization of matrix polynomials abstract this paper presents an algorithm for the construction of a solution of the generalized lyapunov equation it is proved that the polynomial matrix factorization relative to the imaginary axis may be reduced to the successive solution of lyapunov equations i e the factorization is reduced to the solution of a sequence of generalized lyapunov equations not to the solution of generalized riccati equation

natural real exponential families with cubic variance functions pursuing the classification initiated by morris 1982 we describe all the natural exponential families on the real line such that the variance is a polynomial function of the mean with degree less than or equal to 3 we get twelve different types the first six appear in the fundamental paper by morris most of the other six appear as distributions of first passage times in the literature an explanation of this occurrence of stopping times is provided by the introduction of the notion of reciprocity between two measures of between two natural exponential families and by classical fluctuation theory

discriminative embedded clustering a framework for grouping high dimensional data in many real applications of machine learning and data mining we are often confronted with high dimensional data how to cluster high dimensional data is still a challenging problem due to the curse of dimensionality in this paper we try to address this problem using joint dimensionality reduction and clustering different from traditional approaches that conduct dimensionality reduction and clustering in sequence we propose a novel framework referred to as discriminative embedded clustering which alternates them iteratively within this framework we are able not only to view several traditional approaches and reveal their intrinsic relationships but also to be stimulated to develop a new method we also propose an effective approach for solving the formulated nonconvex optimization problem comprehensive analyses including convergence behavior parameter determination and computational complexity together with the relationship to other related approaches are also presented plenty of experimental results on benchmark data sets illustrate that the proposed method outperforms related state of the art clustering approaches and existing joint dimensionality reduction and clustering methods

a unified self stabilizing neural network algorithm for principal and minor components extraction recently many unified learning algorithms have been developed for principal component analysis and minor component analysis these unified algorithms can be used to extract principal components and if altered simply by the sign can also serve as a minor component extractor this is of practical significance in the implementations of algorithms this paper proposes a unified self stabilizing neural network learning algorithm for principal and minor components extraction and studies the stability of the proposed unified algorithm via the fixed point analysis method the proposed unified self stabilizing algorithm for principal and minor components extraction is extended for tracking the principal subspace ps and minor subspace ms the averaging differential equation and the energy function associated with the unified algorithm for tracking ps and ms are given it is shown that the averaging differential equation will globally asymptotically converge to an invariance set and the corresponding energy function exhibit a unique global minimum attained if and only if its state matrices span the ps or ms of the autocorrelation matrix of a vector data stream it is concluded that the proposed unified algorithm for tracking ps and ms can efficiently track an orthonormal basis of the ps or ms simulations are carried out to further illustrate the theoretical results achieved

non negative matrix factorization with sparseness constraints non negative matrix factorization nmf is a recently developed technique for finding parts based linear representations of non negative data although it has successfully been applied in several applications it does not always result in parts based representations in this paper we show how explicitly incorporating the notion of sparseness improves the found decompositions additionally we provide complete matlab code both for standard nmf and for our extension our hope is that this will further the application of these methods to solving novel data analysis problems

max min distance analysis by using sequential sdp relaxation for dimension reduction we propose a new criterion for discriminative dimension reduction max min distance analysis mmda given a data set with c classes represented by homoscedastic gaussians mmda maximizes the minimum pairwise distance of these c classes in the selected low dimensional subspace thus unlike fisher s linear discriminant analysis flda and other popular discriminative dimension reduction criteria mmda duly considers the separation of all class pairs to deal with general case of data distribution we also extend mmda to kernel mmda kmmda dimension reduction via mmda kmmda leads to a nonsmooth max min optimization problem with orthonormal constraints we develop a sequential convex relaxation algorithm to solve it approximately to evaluate the effectiveness of the proposed criterion and the corresponding algorithm we conduct classification and data visualization experiments on both synthetic data and real data sets experimental results demonstrate the effectiveness of mmda kmmda associated with the proposed optimization algorithm

state feedback decomposition of multivariable systems via block pole placement some recently developed algebraic theorems of matrices and applications to matrix fraction descriptions of a class of multivariable systems are introduced also a new block pole placement is developed for the state feedback block decomposition of multivariable systems

mixtures of probabilistic principal component analyzers principal component analysis pca is one of the most popular techniques for processing compressing and visualizing data although its effectiveness is limited by its global linearity while nonlinear variants of pca have been proposed an alternative paradigm is to capture data complexity by a combination of local linear pca projections however conventional pca does not correspond to a probability density and so there is no unique way to combine pca models therefore previous attempts to formulate mixture models for pca have been ad hoc to some extent in this article pca is formulated within a maximum likelihood framework based on a specific form of gaussian latent variable model this leads to a well defined mixture model for probabilistic principal component analyzers whose parameters can be determined using an expectationmaximization algorithm we discuss the advantages of this model in the context of clustering density modeling and local dimensionality reduction and we demonstrate its application to image compression and handwritten digit recognition

geometric mean for subspace selection subspace selection approaches are powerful tools in pattern classification and data visualization one of the most important subspace approaches is the linear dimensionality reduction step in the fisher s linear discriminant analysis flda which has been successfully employed in many fields such as biometrics bioinformatics and multimedia information management however the linear dimensionality reduction step in flda has a critical drawback for a classification task with c classes if the dimension of the projected subspace is strictly lower than c 1 the projection to a subspace tends to merge those classes which are close together in the original feature space if separate classes are sampled from gaussian distributions all with identical covariance matrices then the linear dimensionality reduction step in flda maximizes the mean value of the kullback leibler kl divergences between different classes based on this viewpoint the geometric mean for subspace selection is studied in this paper three criteria are analyzed 1 maximization of the geometric mean of the kl divergences 2 maximization of the geometric mean of the normalized kl divergences and 3 the combination of 1 and 2 preliminary experimental results based on synthetic data uci machine learning repository and handwriting digits show that the third criterion is a potential discriminative subspace selection method which significantly reduces the class separation problem in comparing with the linear dimensionality reduction step in flda and its several representative extensions

matrix variate factor analysis and its applications factor analysis fa seeks to reveal the relationship between an observed vector variable and a latent variable of reduced dimension it has been widely used in many applications involving high dimensional data such as image representation and face recognition an intrinsic limitation of fa lies in its potentially poor performance when the data dimension is high a problem known as curse of dimensionality motivated by the fact that images are inherently matrices we develop in this brief an fa model for matrix variate variables and present an efficient parameter estimation algorithm experiments on both toy and real world image data demonstrate that the proposed matrix variant fa model is more efficient and accurate than the classical fa approach especially when the observed variable is high dimensional and the samples available are limited

enriched conjugate and reference priors for the wishart family on symmetric cones a general wishart family on a symmetric cone is a natural exponential family nef having a homogeneous quadratic variance function using results in the abstract theory of euclidean jordan algebras the structure of conditional reducibility is shown to hold for such a family and we identify the associated parameterization and analyze its properties the enriched standard conjugate family for and the mean parameter are defined and discussed this family is considerably more flexible than the standard conjugate one the reference priors for and are obtained and shown to belong to the enriched standard conjugate family in particular this allows us to verify that reference posteriors are always proper the above results extend those available for nefs having a simple quadratic variance function specifications of the theory to the cone of real symmetric and positive definite matrices are discussed in detail and allow us to perform bayesian inference on the covariance matrix of a multivariate normal model under the enriched standard conjugate family in particular commonly employed bayes estimates such as the posterior expectation of and 1 are provided in closed form

dimensionality reduction of multimodal labeled data by local fisher discriminant analysis reducing the dimensionality of data without losing intrinsic information is an important preprocessing step in high dimensional data analysis fisher discriminant analysis fda is a traditional technique for supervised dimensionality reduction but it tends to give undesired results if samples in a class are multimodal an unsupervised dimensionality reduction method called locality preserving projection lpp can work well with multimodal data due to its locality preserving property however since lpp does not take the label information into account it is not necessarily useful in supervised learning scenarios in this paper we propose a new linear supervised dimensionality reduction method called local fisher discriminant analysis lfda which effectively combines the ideas of fda and lpp lfda has an analytic form of the embedding transformation and the solution can be easily computed just by solving a generalized eigenvalue problem we demonstrate the practical usefulness and high scalability of the lfda method in data visualization and classification tasks through extensive simulation studies we also show that lfda can be extended to non linear dimensionality reduction scenarios by applying the kernel trick

a new method for block partial fraction expansion of matrix fraction descriptions the block observability form is used to develop a method for finding the block partial fraction expansion of an irreducible right or left matrix fraction description with distinct and or repeated solvents of the denominator matrix polynomials the new formulas ar very simple and only require that the block vandermonde matrix be formed from the right or left solvents all the matrix residues of the expansion are found simultaneously by a simple matrix manipulation and the formulas are suitable for digital computer application

transformations of a class of multivariable control systems to block companion forms similarity block transformations are developed to transform a class of linear time invariant mimo state equations in arbitrary coordinates into block companion forms so that the classical lines of thought for siso systems can be extended to mimo systems

ga sift a new scale invariant feature transform for multispectral image using geometric algebra feature analysis plays an important role in many multispectral image applications and scale invariant feature transform sift has been successfully applied for extraction of image features however the existing sift algorithms cannot extract features from multispectral images directly this paper puts forward a novel algorithmic framework based on the sift for multispectral images firstly with the theory of the geometric algebra ga a new representation of multispectral image including spatial and spectral information is put forward and discussed secondly a new method for obtaining the scale space of the multispectral image is proposed thirdly following the procedures of the sift the ga based difference of gaussian images are computed and the keypoints can be detected in the ga space fourthly the feature points are finally detected and described in the mathematical framework of the ga finally the comparison results show that the ga sift outperforms some previously reported sift algorithms in the feature extraction from a multispectral image and it is comparable with its counterparts in the feature extraction of color images indicating good performance in various applications of image analysis

approximate orthogonal sparse embedding for dimensionality reduction locally linear embedding lle is one of the most well known manifold learning methods as the representative linear extension of lle orthogonal neighborhood preserving projection onpp has attracted widespread attention in the field of dimensionality reduction in this paper a unified sparse learning framework is proposed by introducing the sparsity or l 1 norm learning which further extends the lle based methods to sparse cases theoretical connections between the onpp and the proposed sparse linear embedding are discovered the optimal sparse embeddings derived from the proposed framework can be computed by iterating the modified elastic net and singular value decomposition we also show that the proposed model can be viewed as a general model for sparse linear and nonlinear kernel subspace learning based on this general model sparse kernel embedding is also proposed for nonlinear sparse feature extraction extensive experiments on five databases demonstrate that the proposed sparse learning framework performs better than the existing subspace learning algorithm particularly in the cases of small sample sizes

bilinear probabilistic principal component analysis probabilistic principal component analysis ppca is a popular linear latent variable model for performing dimension reduction on 1 d data in a probabilistic manner however when used on 2 d data such as images ppca suffers from the curse of dimensionality due to the subsequently large number of model parameters to overcome this problem we propose in this paper a novel probabilistic model on 2 d data called bilinear ppca bppca this allows the establishment of a closer tie between bppca and its nonprobabilistic counterpart moreover two efficient parameter estimation algorithms for fitting bppca are also developed experiments on a number of 2 d synthetic and real world data sets show that bppca is more accurate than existing probabilistic and nonprobabilistic dimension reduction methods

image categorization by learning a propagated graphlet path spatial pyramid matching is a standard architecture for categorical image retrieval however its performance is largely limited by the prespecified rectangular spatial regions when pooling local descriptors in this paper we propose to learn object shaped and directional receptive fields for image categorization in particular different objects in an image are seamlessly constructed by superpixels while the direction captures human gaze shifting path by generating a number of superpixels in each image we construct graphlets to describe different objects they function as the object shaped receptive fields for image comparison due to the huge number of graphlets in an image a saliency guided graphlet selection algorithm is proposed a manifold embedding algorithm encodes graphlets with the semantics of training image tags then we derive a manifold propagation to calculate the postembedding graphlets by leveraging visual saliency maps the sequentially propagated graphlets constitute a path that mimics human gaze shifting finally we use the learned graphlet path as receptive fields for local image descriptor pooling the local descriptors from similar receptive fields of pairwise images more significantly contribute to the final image kernel thorough experiments demonstrate the advantage of our approach

large deviations for exchangeable random vectors satisfies a large deviation principle with rate function a x inf a6 x o e s u 1 where s 4 is the support of the mixing measure 4 if x1 x2 is a sequence of i i d random vectors xn the corresponding sequence of sample means and pown po x1 then pon is exponentially continuous if the classical rate function ar v is jointly lower semicontinuous and a uniform integrability condition introduced by de acosta is satisfied these results are applied in section 4 to derive a large deviation theory for exchangeable random variables the resulting rate functions are typically nonconvex if the parameter space e is not compact then examples can be constructed where a full large deviation principle is not satisfied because the upper bound fails for a noncompact set

nonlinear regularization path for quadratic loss support vector machines regularization path algorithms have been proposed to deal with model selection problem in several machine learning approaches these algorithms allow computation of the entire path of solutions for every value of regularization parameter using the fact that their solution paths have piecewise linear form in this paper we extend the applicability of regularization path algorithm to a class of learning machines that have quadratic loss and quadratic penalty term this class contains several important learning machines such as squared hinge loss support vector machine svm and modified huber loss svm we first show that the solution paths of this class of learning machines have piecewise nonlinear form and piecewise segments between two breakpoints are characterized by a class of rational functions then we develop an algorithm that can efficiently follow the piecewise nonlinear path by solving these rational equations to solve these rational equations we use rational approximation technique with quadratic convergence rate and thus our algorithm can follow the nonlinear path much more precisely than existing approaches such as predictor corrector type nonlinear path approximation we show the algorithm performance on some artificial and real data sets

mahalanobis distance on extended grassmann manifolds for variational pattern analysis in pattern classification problems pattern variations are often modeled as a linear manifold or a low dimensional subspace conventional methods use such models and define a measure of similarity or dissimilarity however these similarity measures are deterministic and do not take into account the distribution of linear manifolds or low dimensional subspaces therefore if the distribution is not isotopic the distance measurements are not reliable as well as vector based distance measurement in the euclidean space we previously systematized the representations of variational patterns using the grassmann manifold and introduce the mahalanobis distance to the grassmann manifold as a natural extension of euclidean case in this paper we present two methods that flexibly extend the mahalanobis distance on the extended grassmann manifolds these methods can be used to measure pattern dis similarity on the basis of the pattern structure experimental evaluation of the performance of the proposed methods demonstrated that they exhibit a lower error classification rate

an optimization criterion for generalized discriminant analysis on undersampled problems an optimization criterion is presented for discriminant analysis the criterion extends the optimization criteria of the classical linear discriminant analysis lda through the use of the pseudoinverse when the scatter matrices are singular it is applicable regardless of the relative sizes of the data dimension and sample size overcoming a limitation of classical lda the optimization problem can be solved analytically by applying the generalized singular value decomposition gsvd technique the pseudoinverse has been suggested and used for undersampled problems in the past where the data dimension exceeds the number of data points the criterion proposed in this paper provides a theoretical justification for this procedure an approximation algorithm for the gsvd based approach is also presented it reduces the computational complexity by finding subclusters of each cluster and uses their centroids to capture the structure of each cluster this reduced problem yields much smaller matrices to which the gsvd can be applied efficiently experiments on text data with up to 7 000 dimensions show that the approximation algorithm produces results that are close to those produced by the exact algorithm

design of state feedback for large scale multivariable systems this note presents a general method which reduces the computational requirements in the state feedback design of large scale multivariable systems the given system is first transformed into a general block canonical form by using simple equivalent transformation the state feedback problem is then reformulated in terms of a sylvester equation finally the transformed system matrices along with certain assumed block forms for unknown matrices enable the sylvester equation to be decomposed and solved effectively

large deviations for lattice systems i parametrized independent fields we prove large deviation theorems for empirical measures of independent random fields whose distributions depend measurably on an auxiliary parameter this dependence respects the action of the shift group and a large deviation principle holds whenever a certain ergodicity condition is satisfied we also investigate the entropy functions for these processes especially in relation to the usual relative entropy

manifold regularized discriminative nonnegative matrix factorization with fast gradient descent nonnegative matrix factorization nmf has become a popular data representation method and has been widely used in image processing and pattern recognition problems this is because the learned bases can be interpreted as a natural parts based representation of data and this interpretation is consistent with the psychological intuition of combining parts to form a whole for practical classification tasks however nmf ignores both the local geometry of data and the discriminative information of different classes in addition existing research results show that the learned basis is unnecessarily parts based because there is neither explicit nor implicit constraint to ensure the representation parts based in this paper we introduce the manifold regularization and the margin maximization to nmf and obtain the manifold regularized discriminative nmf md nmf to overcome the aforementioned problems the multiplicative update rule mur can be applied to optimizing md nmf but it converges slowly in this paper we propose a fast gradient descent fgd to optimize md nmf fgd contains a newton method that searches the optimal step length and thus fgd converges much faster than mur in addition fgd includes mur as a special case and can be applied to optimizing nmf and its variants for a problem with 165 samples in r1600 fgd converges in 28 s while mur requires 282 s we also apply fgd in a variant of md nmf and experimental results confirm its efficiency experimental results on several face image datasets suggest the effectiveness of md nmf

generalized latent trait models in this paper we discuss a general model framework within which manifest variables with different distributions in the exponential family can be analyzed with a latent trait model a unified maximum likelihood method for estimating the parameters of the generalized latent trait model will be presented we discuss in addition the scoring of individuals on the latent dimensions the general framework presented allows not only the analysis of manifest variables all of one type but also the simultaneous analysis of a collection of variables with different distributions the approach used analyzes the data as they are by making assumptions about the distribution of the manifest variables directly

partial fraction expansions for transfer matrices and matrix fraction descriptions of multivariable systems a general formula originally to find the scalar partial fraction expansion pfe of a scalar rational function and the matrix pfe of a rational transfer matrix is generalized to find the block pfe of a right coprime matrix fraction description mfd with simple and or repeated block poles this formula expresses the block residues explicitly in terms of the coefficient matrices of the numerator and the denominator matrix polynomials and the left solvents of the denominator matrix polynomial it is therefore straightforward and easy to apply as compared with the existing methods

generalized linear discriminant analysis a unified framework and efficient model selection high dimensional data are common in many domains and dimensionality reduction is the key to cope with the curse of dimensionality linear discriminant analysis lda is a well known method for supervised dimensionality reduction when dealing with high dimensional and low sample size data classical lda suffers from the singularity problem over the years many algorithms have been developed to overcome this problem and they have been applied successfully in various applications however there is a lack of a systematic study of the commonalities and differences of these algorithms as well as their intrinsic relationships in this paper a unified framework for generalized lda is proposed which elucidates the properties of various algorithms and their relationships based on the proposed framework we show that the matrix computations involved in lda based algorithms can be simplified so that the cross validation procedure for model selection can be performed efficiently we conduct extensive experiments using a collection of high dimensional data sets including text documents face images gene expression data and gene expression pattern images to evaluate the proposed theories and algorithms

face recognition using lda based algorithms low dimensional feature representation with enhanced discriminatory power is of paramount importance to face recognition fr systems most of traditional linear discriminant analysis lda based methods suffer from the disadvantage that their optimality criteria are not directly related to the classification ability of the obtained feature representation moreover their classification accuracy is affected by the small sample size sss problem which is often encountered in fr tasks in this paper we propose a new algorithm that deals with both of the shortcomings in an efficient and cost effective manner the proposed method is compared in terms of classification accuracy to other commonly used fr methods on two face databases results indicate that the performance of the proposed method is overall superior to those of traditional fr approaches such as the eigenfaces fisherfaces and d lda methods

simplifying neural networks by soft weight sharing one way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights we propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians a set of weights is simple if the weights have high probability density under the mixture model this can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values since we do not know the appropriate means or variances of the clusters in advance we allow the parameters of the mixture model to adapt at the same time as the network learns simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms

data dimensionality estimation methods a survey in this paper data dimensionality estimation methods are reviewed the estimation of the dimensionality of a data set is a classical problem of pattern recognition there are some good reviews algorithms for clustering data prentice hall englewood cliffs nj 1988 in literature but they do not include more recent developments based on fractal techniques and neural autoassociators the aim of this paper is to provide an up to date survey of the dimensionality estimation methods of a data set paying special attention to the fractal based methods

estimating the support of a high dimensional distribution suppose you are given some data set drawn from an underlying probability distribution p and you want to estimate a simple subset s of input space such that the probability that a test point drawn from p lies outside of s equals some a priori specified value between 0 and 1 we propose a method to approach this problem by trying to estimate a function f that is positive on s and negative on the complement the functional form of f is given by a kernel expansion in terms of a potentially small subset of the training data it is regularized by controlling the length of the weight vector in an associated feature space the expansion coefficients are found by solving a quadratic programming problem which we do by carrying out sequential optimization over pairs of input patterns we also provide a theoretical analysis of the statistical performance of our algorithm the algorithm is a natural extension of the support vector algorithm to the case of unlabeled data

large deviations for langevin spin glass dynamics we study the asymptotic behaviour of asymmetrical spin glass dynamics in a sherrington kirkpatrick model as proposed by sompolinsky zippelius we prove that the annealed law of the empirical measure on path space of these dynamics satisfy a large deviation principle in the high temperature regime we study the rate function of this large deviation principle and prove that it achieves its minimum value at a unique probability measureq which is not markovian we deduce that the quenched law of the empirical measure converges to q extending then the preceeding results to replicated dynamics we investigate the quenched behavior of a single spin we get quenched convergence toq in the case of a symmetric initial law and even potential for the free spin

probabilistic principal component analysis principal component analysis pca is a ubiquitous technique for data analysis and processing but one which is not based upon a probability model in this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model closely related to factor analysis we consider the properties of the associated likelihood function giving an em algorithm for estimating the principal subspace iteratively and discuss the advantages conveyed by the definition of a probability density function for pca

penalized discriminant analysis fisher s linear discriminant analysis lda is a popular data analytic tool for studying the relationship between a set of predictors and a categorical response in this paper we describe a penalized version of lda it is designed for situations in which there are many highly correlated predictors such as those obtained by discretizing a function or the grey scale values of the pixels in a series of images in cases such as these it is natural efficient and sometimes essential to impose a spatial smoothness constraint on the coefficients both for improved prediction performance and interpretability we cast the classification problem into a regression framework via optimal scoring using this our proposal facilitates the use of any penalized regression technique in the classification setting the technique is illustrated with examples in speech recognition and handwritten character recognition

compatible prior distributions for directed acyclic graph models summary the application of certain bayesian techniques such as the bayes factor and model averaging requires the specification of prior distributions on the parameters of alternative models we propose a new method for constructing compatible priors on the parameters of models nested in a given directed acyclic graph model using a conditioning approach we define a class of parameterizations that is consistent with the modular structure of the directed acyclic graph and derive a procedure that is invariant within this class which we name reference conditioning

kernel projection classifiers with suppressing features of other classes we propose a new classification method based on a kernel technique called suppressed kernel sample space projection classifier sksp which is extended from kernel sample space projection classifier ksp in kernel methods samples are classified after they are mapped from an input space to a high dimensional space called a feature space the space that is spanned by samples of a class in a feature space is defined as a kernel sample space in ksp an unknown input vector is classified to the class of which projection norm onto the kernel sample space is maximized ksp can be interpreted as a special type of kernel principal component analysis kpca kpca is also used in classification problems however ksp has more useful properties compared with kpca and its accuracy is almost the same as or better than that of kpca classifier since ksp is a single class classifier it uses only self class samples for learning thus for a multiclass classification problem even though there are very many classes the computational cost does not change for each class however we expect that more useful features can be obtained for classification if samples from other classes are used by extending ksp to sksp the effects of other classes are suppressed and useful features can be extracted with an oblique projection experiments on two class classification problems indicate that sksp shows high accuracy in many classification problems

exponential family factors for bayesian factor analysis expressing data as linear functions of a small number of unknown variables is a useful approach employed by several classical data analysis methods e g factor analysis principal component analysis or latent semantic indexing these models represent the data using the product of two factors in practice one important concern is how to link the learned factors to relevant quantities in the context of the application to this end various specialized forms of the factors have been proposed to improve interpretability toward developing a unified view and clarifying the statistical significance of the specialized factors we propose a bayesian model family we employ exponential family distributions to specify various types of factors which provide a unified probabilistic formulation a gibbs sampling procedure is constructed as a general computation routine we verify the model by experiments in which the proposed model is shown to be effective in both emulating existing models and motivating new model designs for particular problem settings

input space versus feature space in kernel based methods this paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector sv kernel functions we first discuss the geometry of feature space in particular we review what is known about the shape of the image of input space under the feature space map and how this influences the capacity of sv methods following this we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel using the example of the class of inhomogeneous polynomial kernels which are often used in sv pattern recognition we then discuss the connection between feature space and input space by dealing with the question of how one can given some vector in feature space find a preimage exact or approximate in input space we describe algorithms to tackle this issue and show their utility in two applications of kernel methods first we use it to reduce the computational complexity of sv decision functions second we combine it with the kernel pca algorithm thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real world data