a connection formula for the second painleve transcendent we consider a particular case of the second painleve transcendent y xy 2y 3 it is known that if y x kai x as x then if 0 k 1 y x sim d left x right raise0 5ex hbox scriptstyle 1 kern 0 1em kern 0 15em lower0 25ex hbox scriptstyle 4 sin frac 2 3 left x right 3 2 frac 3 4 d2 ell n left x right c asx to infty where d k and c k are the connection formulae for this nonlinear ordinary differential equation

scaled rotation regularization abstract a new regularization method a scaled rotation is proposed and compared with the standard linear regularized discriminant analysis a sense of the method consists in the singular value decomposition s tdt of a sample covariance matrix s and a use of the following representation of an inverse of the covariance matrix s 1 t d i 1 t for certain data structures the scaled rotation helps to reduce the generalization error in small learning set and high dimensionality cases efficacy of the scaled rotation increases if one transforms the data by y d i 1 2 t x and uses an optimally stopped single layer perceptron classifier afterwards

constrained linear quadratic regulation the paper is a contribution to the theory of the infinite horizon linear quadratic regulator lqr problem subject to inequality constraints on the inputs and states extending an approach first proposed by sznaier and damborg 1987 a solution algorithm is presented which requires solving a finite number of finite dimensional positive definite quadratic programs the constrained lqr outlined does not feature the undesirable mismatch between open loop and closed loop nominal system trajectories which is present in the other popular forms of model predictive control mpc that can be implemented with a finite quadratic programming algorithm the constrained lqr is shown to be both optimal and stabilizing the solution algorithm is guaranteed to terminate in finite time with a computational cost that has a reasonable upper bound compared to the minimal cost for computing the optimal solution inherent to the approach is the removal of a tuning parameter the control horizon which is present in other mpc approaches and for which no reliable tuning guidelines are available two examples are presented that compare constrained lqr and two other popular forms of mpc the examples demonstrate that constrained lqr achieves significantly better performance than the other forms of mpc on some plants and the computational cost is not prohibitive for online implementation

improved discriminate analysis for high dimensional data and its application to face recognition many pattern recognition applications involve the treatment of high dimensional data and the small sample size problem principal component analysis pca is a common used dimension reduction technique linear discriminate analysis lda is often employed for classification pca plus lda is a famous framework for discriminant analysis in high dimensional space and singular cases in this paper we examine the theory of this framework and find out that even if there is no small sample size problem the pca dimension reduction cannot guarantee the subsequent successful application of lda we thus develop an improved discriminate analysis method by introducing an inverse fisher criterion and adding a constrain in pca procedure so that the singularity phenomenon will not occur experiment results on face recognition suggest that this new approach works well and can be applied even when the number of training samples is one per class

corrigendum corrigendum to the explicit linear quadratic regulator for constrained systems automatica 38 1 2002 3 20 we apologize that example 7 1 as publishedin bemporad morari dua andpistikopoulos 2002 is incorrect due to a miscalculation of the weight matrix p on the terminal state

ridge regression biased estimation for nonorthogonal problems in multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory if not incorrect if the prediction vectors are not orthogonal proposed is an estimation procedure based on adding small positive quantities to the diagonal of x x introduced is the ridge trace a method for showing in two dimensions the effects of nonorthogonality it is then shown how to augment x x to obtain biased estimates with smaller mean square error

estimation of the mean of a multivariate normal distribution estimation of the means of independent normal random variables is considered using sum of squared errors as loss an unbiased estimate of risk is obtained for an arbitrary estimate and certain special classes of estimates are then discussed the results are applied to smoothing by use of moving averages and to trimmed analogs of the james stein estimate a suggestion is made for calculating approximate confidence sets for the mean vector centered at an arbitrary estimate

optimal output regulation for discrete time switched and markovian jump linear systems first three different but related output regulation performance criteria for the linear time varying system are defined in the discrete time domain namely the peak impulse response peak output variance and average output variance per unit time then they are extended for switched linear systems and markovian jump linear systems and characterized by an increasing union of finite dimensional linear matrix inequality conditions finally the infinite horizon suboptimal lqg control problem which aims to maintain the average output variance below a given level subject to the uniform exponential stability of the closed loop system is solved for both switched linear systems and markovian jump linear systems the solution is given by a dynamic linear output feedback controller that not only perfectly observes the present mode but also recalls a finite number of past modes

towards a generalized singular value decomposition we suggest a form for and give a constructive derivation of the generalized singular value decomposition of any two matrices having the same number of columns we outline its desirable characteristics and compare it to an earlier suggestion by van loan siam j numer anal 13 1976 pp 76 83 the present form largely follows from the work of van loan but is slightly more general and computationally more amenable than that in the paper cited we also prove a useful extension of a theorem of stewart siam rev 19 1977 pp 634 662 on unitary decompositions of submatrices of a unitary matrix

sparse orthogonal linear discriminant analysis in this paper sparse orthogonal linear discriminant analysis olda is studied the main contributions of the present work include the following i all minimum frobenius norm dimension solutions of the optimization problem used for establishing olda are characterized explicitly and ii this explicit characterization leads to two numerical algorithms for computing a sparse linear transformation for olda the first is based on the gradient flow approach while the second is a sequential linear bregman method we experiment with real world datasets to illustrate that the sequential linear bregman method is much better than the gradient flow approach the sequential linear bregman method always achieves comparable classification accuracy with the normal olda satisfactory sparsity and orthogonality and acceptable cpu times

inferring hidden states in langevin dynamics on large networks average case performance we present average performance results for dynamical inference problems in large networks where a set of nodes is hidden while the time trajectories of the others are observed examples of this scenario can occur in signal transduction and gene regulation networks we focus on the linear stochastic dynamics of continuous variables interacting via random gaussian couplings of generic symmetry we analyze the inference error given by the variance of the posterior distribution over hidden paths in the thermodynamic limit and as a function of the system parameters and the ratio ensuremath alpha between the number of hidden and observed nodes by applying kalman filter recursions we find that the posterior dynamics is governed by an effective drift that incorporates the effect of the observations we present two approaches for characterizing the posterior variance that allow us to tackle respectively equilibrium and nonequilibrium dynamics the first appeals to random matrix theory and reveals average spectral properties of the inference error and typical posterior relaxation times the second is based on dynamical functionals and yields the inference error as the solution of an algebraic equation

painleve equations nonlinear special functions the six painleve equations pi pvi were first discovered about a hundred years ago by painleve and his colleagues in an investigation of nonlinear second order ordinary differential equations recently there has been considerable interest in the painleve equations primarily due to the fact that they arise as reductions of the soliton equations which are solvable by inverse scattering consequently the painleve equations can be regarded as completely integrable equations and possess solutions which can be expressed in terms of solutions of linear integral equations despite being nonlinear equations although first discovered from strictly mathematical considerations the painleve equations have arisen in a variety of important physical applications including statistical mechanics plasma physics nonlinear waves quantum gravity quantum field theory general relativity nonlinear optics and fibre optics the painleve equations may be thought of a nonlinear analogues of the classical special functions they possess hierarchies of rational solutions and one parameter families of solutions expressible in terms of the classical special functions for special values of the parameters further the painleve equations admit symmetries under affine weyl groups which are related to the associated backlund transformations in this paper i discuss some of the remarkable properties which the painleve equations possess including connection formulae backlund transformations associated discrete equations and hierarchies of exact solutions in particular the second painleve equation pii is used to illustrate these properties and some of the applications of pii are also discussed

the discrimination subspace model abstract for a p variate normal random vector measured in two populations we propose a method of discrimination under the constraint that all differences between the two populations occur in a subspace of dimension q p this method of classification is based on the discrimination subspace model denoted by dsm q and is intermediate between linear and quadratic discrimination it combines the ideas of dimension reduction and constraints on the parameter space thus substantially reducing the number of parameters to be estimated the maximum likelihood estimators of the model are presented and the performance of dsm q versus quadratic and linear discrimination is assessed via simulation it is generally shown that discrimination based on dsm q consistently yields noticeably lower expected actual error rates relative to the traditional methods the method is illustrated with a real data example and is compared to linear and quadratic discrimination using a leave one out method the example confirms t

spectral density of sparse sample covariance matrices applying the replica method of statistical mechanics we evaluate the eigenvalue density of the large random matrix sample covariance matrix of the form j a t a where a is an m n real sparse random matrix the difference from a dense random matrix is the most significant in the tail region of the spectrum we compare the results of several approximation schemes focusing on the behaviour in the tail region

survey constrained model predictive control stability and optimality model predictive control is a form of control in which the current control action is obtained by solving at each sampling instant a finite horizon open loop optimal control problem using the current state of the plant as the initial state the optimization yields an optimal control sequence and the first control in this sequence is applied to the plant an important advantage of this type of control is its ability to cope with hard constraints on controls and states it has therefore been widely applied in petro chemical and related industries where satisfaction of constraints is particularly important because efficiency demands operating points on or close to the boundary of the set of admissible states and controls in this review we focus on model predictive control of constrained systems both linear and nonlinear and discuss only briefly model predictive control of unconstrained nonlinear and or time varying systems we concentrate our attention on research dealing with stability and optimality in these areas the subject has developed in our opinion to a stage where it has achieved sufficient maturity to warrant the active interest of researchers in nonlinear control we distill from an extensive literature essential principles that ensure stability and use these to present a concise characterization of most of the model predictive controllers that have been proposed in the literature in some cases the finite horizon optimal control problem solved on line is exactly equivalent to the same problem with an infinite horizon in other cases it is equivalent to a modified infinite horizon optimal control problem in both situations known advantages of infinite horizon optimal control accrue

shrinkage based diagonal discriminant analysis and its applications in high dimensional data high dimensional data such as microarrays have brought us new statistical challenges for example using a large number of genes to classify samples based on a small number of microarrays remains a difficult problem diagonal discriminant analysis support vector machines and k nearest neighbor have been suggested as among the best methods for small sample size situations but none was found to be superior to others in this article we propose an improved diagonal discriminant approach through shrinkage and regularization of the variances the performance of our new approach along with the existing methods is studied through simulations and applications to real data these studies show that the proposed shrinkage based and regularization diagonal discriminant methods have lower misclassification rates than existing methods in many cases

regularized discriminant analysis abstract linear and quadratic discriminant analysis are considered in the small sample high dimensional setting alternatives to the usual maximum likelihood plug in estimates for the covariance matrices are proposed these alternatives are characterized by two parameters the values of which are customized to individual situations by jointly minimizing a sample based estimate of future misclassification risk computationally fast implementations are presented and the efficacy of the approach is examined through simulation studies and application to data these studies indicate that in many circumstances dramatic gains in classification accuracy can be achieved

cardinality constrained linear quadratic optimal control as control implementation often incurs not only a variable cost associated with the magnitude or energy of the control but also a setup cost we consider a discrete time linear quadratic lq optimal control problem with a limited number of control implementations termed in this technical note the cardinality constrained linear quadratic optimal control cclq we first derive a semi analytical feedback policy for cclq problems using dynamic programming dp due to the exponential growth of the complexity in calculating the action regions however dp procedure is only efficient for cclq problems with a scalar state space recognizing this fact we develop then two lower bounding schemes and integrate them into a branch and bound bnb solution framework to offer an efficient algorithm in solving general cclq problems adopting the devised solution algorithm for cclq problems we can solve efficiently the linear quadratic optimal control problem with setup costs

joint estimation of inverse covariance matrices lying in an unknown subspace we consider the problem of joint estimation of inverse covariance matrices lying in an unknown subspace of the linear space of symmetric matrices we perform the estimation using groups of measurements with different covariances assuming the inverse covariances span a low dimensional subspace our aim is to determine this subspace and to exploit this knowledge in order to improve the estimation we develop a novel optimization algorithm discovering and exploiting the underlying low dimensional subspace we provide a computationally efficient algorithm and derive a tight upper performance bound numerical simulations on synthetic and real world data are presented to illustrate the performance benefits of the algorithm

information loss of the mahalanobis distance in high dimensions application to feature selection when an infinite training set is used the mahalanobis distance between a pattern measurement vector of dimensionality d and the center of the class it belongs to is distributed as a chi2 with d degrees of freedom however the distribution of mahalanobis distance becomes either fisher or beta depending on whether cross validation or resubstitution is used for parameter estimation in finite training sets the total variation between chi2 and fisher as well as between chi2 and beta allows us to measure the information loss in high dimensions the information loss is exploited then to set a lower limit for the correct classification rate achieved by the bayes classifier that is used in subset feature selection

constrained infinite horizon linear quadratic regulation of discrete time systems we are concerned with the output norm constrained infinite horizon linear quadratic regulation problem where the underlying state control constraints are specified by curved rather than polyhedral surfaces each suboptimal problem admits an exact convex synthesis condition expressed by an increasing union of linear matrix inequalities and results in an eventually time invariant controller satisfying the desired performance level subject to given constraints and closed loop stability the output norm constrained minimum time control problem is also dealt with as a corollary

model sparsity and brain pattern interpretation of classification models in neuroimaging interest is increasing in applying discriminative multivariate analysis techniques to the analysis of functional neuroimaging data model interpretation is of great importance in the neuroimaging context and is conventionally based on a brain map derived from the classification model in this study we focus on the relative influence of model regularization parameter choices on both the model generalization the reliability of the spatial patterns extracted from the classification model and the ability of the resulting model to identify relevant brain networks defining the underlying neural encoding of the experiment for a support vector machine logistic regression and fisher s discriminant analysis we demonstrate that selection of model regularization parameters has a strong but consistent impact on the generalizability and both the reproducibility and interpretable sparsity of the models for both 2 and 1 regularization importantly we illustrate a trade off between model spatial reproducibility and prediction accuracy we show that known parts of brain networks can be overlooked in pursuing maximization of classification accuracy alone with either 2 and or 1 regularization this supports the view that the quality of spatial patterns extracted from models cannot be assessed purely by focusing on prediction accuracy our results instead suggest that model regularization parameters must be carefully selected so that the model and its visualization enhance our ability to interpret the brain

on constrained infinite time linear quadratic optimal control abstract this work presents a solution to the infinite time linear quadratic optimal control itlqoc problem with state and control constraints it is shown that a single finite dimensional convex program of known size can yield this solution properties of the resulting value function with respect to initial conditions are also established and are shoen to be useful in determining the aforementioned problem size an example illustrating the method is finally presented

covariance matrix estimation and classification with limited training data a new covariance matrix estimator useful for designing classifiers with limited training data is developed in experiments this estimator achieved higher classification accuracy than the sample covariance matrix and common covariance matrix estimates in about half of the experiments it achieved higher accuracy than regularized discriminant analysis but required much less computation

computation of minimum time feedback control laws for discrete time systems with state control constraints the problem of finding a feedback law that drives the state of a linear discrete time system to the origin in minimum time subject to state control constraints is considered algorithms are given to obtain facial descriptions of the m step admissible sets these descriptions are then used to characterize the complete class of minimum time feedback laws moreover the characterization leads to a conceptually simple on line implementation the main ideas are illustrated with two simple examples

robust receding horizon control of constrained nonlinear systems we present a method for the construction of a robust dual mode receding horizon controller which can be employed for a wide class of nonlinear systems with state and control constraints and model error the controller is dual mode in a neighborhood of the origin the control action is generated by a linear feedback controller designed for the linearized system outside this neighborhood receding horizon control is employed existing receding horizon controllers for nonlinear continuous time systems which are guaranteed to stabilize the nonlinear system to which they are applied require the exact solution at every instant of an optimal control problem with terminal equality constraints these requirements are considerably relaxed in the dual mode receding horizon controller presented in this paper stability is achieved by imposing a terminal inequality rather than an equality constraint only approximate minimization is required a variable time horizon is permitted robustness is achieved by employing conservative state and stability constraint sets thereby permitting a margin of error the resultant dual mode controller requires considerably less online computation than existing receding horizon controllers for nonlinear constrained systems

person re identification by dual regularized kiss metric learning person re identification aims to match the images of pedestrians across different camera views from different locations this is a challenging intelligent video surveillance problem that remains an active area of research due to the need for performance improvement person re identification involves two main steps feature representation and metric learning although the keep it simple and straightforward kiss metric learning method for discriminative distance metric learning has been shown to be effective for the person re identification the estimation of the inverse of a covariance matrix is unstable and indeed may not exist when the training set is small resulting in poor performance here we present dual regularized kiss dr kiss metric learning by regularizing the two covariance matrices dr kiss improves on kiss by reducing overestimation of large eigenvalues of the two estimated covariance matrices and in doing so guarantees that the covariance matrix is irreversible furthermore we provide theoretical analyses for supporting the motivations specifically we first prove why the regularization is necessary then we prove that the proposed method is robust for generalization we conduct extensive experiments on three challenging person re identification datasets viper grid and cuhk 01 and show that dr kiss achieves new state of the art performance

estimates of optimal storage conditions in neural network memories based on random matrix theory the authors formulate a method for estimating the critical conditions for storage of sets of data in neural network memory this variational method is based on random matrix theory and calculating the average spectrum of a matrix whose elements are given by overlaps of the stored patterns several generic cases of random overlap matrices are considered they investigate the cases of simply uncorrelated random patterns and spatially and semantically correlated ones they obtain bounds of the critical curve in the control parameters space which determine the stability of the stored data sets

a study of regularized gaussian classifier in high dimension small sample set case based on mdl principle with application to spectrum recognition in classifying high dimensional patterns such as stellar spectra by a gaussian classifier the covariance matrix estimated with a small number sample set becomes unstable leading to degraded classification accuracy in this paper we investigate the covariance matrix estimation problem for small number samples with high dimension setting based on minimum description length mdl principle a new covariance matrix estimator is developed and a formula for fast estimation of regularization parameters is derived experiments on spectrum pattern recognition are conducted to investigate the classification accuracy with the developed covariance matrix estimator higher classification accuracy results are obtained and demonstrated in our new approach

sparse canonical correlation analysis new formulation and algorithm in this paper we study canonical correlation analysis cca which is a powerful tool in multivariate data analysis for finding the correlation between two sets of multidimensional variables the main contributions of the paper are 1 to reveal the equivalent relationship between a recursive formula and a trace formula for the multiple cca problem 2 to obtain the explicit characterization for all solutions of the multiple cca problem even when the corresponding covariance matrices are singular 3 to develop a new sparse cca algorithm and 4 to establish the equivalent relationship between the uncorrelated linear discriminant analysis and the cca problem we test several simulated and real world datasets in gene classification and cross language document retrieval to demonstrate the effectiveness of the proposed algorithm the performance of the proposed method is competitive with the state of the art sparse cca algorithms

cross validation the jackknife and the bootstrap excess error estimation in forward logistic regression abstract given a prediction rule based on a set of patients what is the probability of incorrectly predicting the outcome of a new patient call this probability the true error an optimistic estimate is the apparent error or the proportion of incorrect predictions on the original set of patients and it is the goal of this article to study estimates of the excess error or the difference between the true and apparent errors i consider three estimates of the excess error cross validation the jackknife and the bootstrap using simulations and real data the three estimates for a specific prediction rule are compared when the prediction rule is allowed to be complicated overfitting becomes a real danger and excess error estimation becomes important the prediction rule chosen here is moderately complicated involving a variable selection procedure based on forward logistic regression

accuracy of pseudo inverse covariance learning a random matrix theory analysis for many learning problems estimates of the inverse population covariance are required and often obtained by inverting the sample covariance matrix increasingly for modern scientific data sets the number of sample points is less than the number of features and so the sample covariance is not invertible in such circumstances the moore penrose pseudo inverse sample covariance matrix constructed from the eigenvectors corresponding to nonzero sample covariance eigenvalues is often used as an approximation to the inverse population covariance matrix the reconstruction error of the pseudo inverse sample covariance matrix in estimating the true inverse covariance can be quantified via the frobenius norm of the difference between the two the reconstruction error is dominated by the smallest nonzero sample covariance eigenvalues and diverges as the sample size becomes comparable to the number of features for high dimensional data we use random matrix theory techniques and results to study the reconstruction error for a wide class of population covariance matrices we also show how bagging and random subspace methods can result in a reduction in the reconstruction error and can be combined to improve the accuracy of classifiers that utilize the pseudo inverse sample covariance matrix we test our analysis on both simulated and benchmark data sets

joint deconvolution and classification with applications to passive acoustic underwater multipath this paper addresses the problem of classifying signals that have been corrupted by noise and unknown linear time invariant lti filtering such as multipath given labeled uncorrupted training signals a maximum a posteriori approach to the deconvolution and classification is considered which produces estimates of the desired signal the unknown channel and the class label for cases in which only a class label is needed the classification accuracy can be improved by not committing to an estimate of the channel or signal a variant of the quadratic discriminant analysis qda classifier is proposed that probabilistically accounts for the unknown lti filtering and which avoids deconvolution the proposed qda classifier can work either directly on the signal or on features whose transformation by lti filtering can be analyzed as an example a classifier for subband power features is derived results on simulated data and real bowhead whale vocalizations show that jointly considering deconvolution with classification can dramatically improve classification performance over traditional methods over a range of signal to noise ratios

subclass graph embedding and a marginal fisher analysis paradigm subspace learning techniques have been extensively used for dimensionality reduction dr in many pattern classification problem domains recently methods like subclass discriminant analysis sda and clustering based discriminant analysis cda which use subclass information for the discrimination between the data classes have attracted much attention in parallel important work has been accomplished on graph embedding ge which is a general framework unifying several subspace learning techniques in this paper ge has been extended in order to integrate subclass discriminant information resulting to the novel subclass graph embedding sge framework which is the main contribution of our work it is proven that sge encapsulates a diversity of both supervised and unsupervised unimodal methods like locality preserving projections lpp principal component analysis pca and linear discriminant analysis lda the theoretical link of sda and cda methods with sge is also established along these lines it is shown that sge comprises a generalization of the typical ge framework including subclass dr methods moreover it allows for an easy utilization of kernels for confronting non linearly separable data employing sge in this paper a novel dr algorithm which uses subclass discriminant information called subclass marginal fisher analysis smfa has been proposed through a series of experiments on various real world datasets it is shown that smfa outperforms in most of the cases the state of the art demonstrating the efficacy and power of sge as a platform to develop new methods highlightsgraph embedding is extended in order to integrate subclass informationthe novel subclass graph embedding framework is proposed the kernelized version of the new framework is presentedsubclass graph embedding encapsulates various subspace learning methods a novel subclass marginal fisher analysis method is proposed

lmi tools for eventually periodic systems abstract this paper is focused on the concept of an eventually periodic linear discrete time system we derive a necessary and sufficient analysis condition for checking open loop stability and performance of such systems and use this to derive exact controller synthesis conditions given eventually periodic plants all the conditions derived are provided in terms of semi definite programming problems the motivation for this work is controlling nonlinear systems along prespecified trajectories notably those which eventually settle down into periodic orbits and those with uncertain initial states

uniform stabilization of discrete time switched and markovian jump linear systems the uniform stability of discrete time switched linear systems possibly with a strongly connected switching path constraint and the existence of finite path dependent dynamic output feedback controllers uniformly stabilizing such a system are both shown to be characterized by the existence of a finite dimensional feasible system of linear matrix inequalities this characterization is based on the observation that a linear time varying system is uniformly stable only if there exists a finite path dependent quadratic lyapunov function the synthesis of a uniformly stabilizing controller is done without conservatism by solving any feasible system of linear matrix inequalities among an increasing family of systems of linear matrix inequalities the result carries over to the almost sure uniform stabilization of markovian jump linear systems

first order tree type dependence between variables and classification performance structuralization of the covariance matrix reduces the number of parameters to be estimated from the training data and does not affect an increase in the generalization error asymptotically as both the number of dimensions and training sample size grow a method to benefit from approximately correct assumptions about the first order tree dependence between components of the feature vector is proposed we use a structured estimate of the covariance matrix to decorrelate and scale the data and to train a single layer perceptron in the transformed feature space we show that training the perceptron can reduce negative effects of inexact a priori information experiments performed with 13 artificial and 10 real world data sets show that the first order tree type dependence model is the most preferable one out of two dozen of the covariance matrix structures investigated

optimal disturbance attenuation for discrete time switched and markovian jump linear systems an exact condition for uniform stabilization and disturbance attenuation for switched linear systems is given in the discrete time domain via the union of an increasing family of linear matrix inequality conditions associated with each markovian jump linear system is a switched linear system so we obtain a necessary and sufficient condition for almost sure uniform stabilization and disturbance attenuation for markovian jump linear systems as well the results lead to semidefinite programming based controller synthesis techniques from which optimal finite path dependent linear dynamic output feedback controllers arise naturally in particular under the notion of path by path optimal disturbance attenuation finite path dependent controllers can outperform the usual mode dependent ones

estimating the error rate of a prediction rule improvement on cross validation abstract we construct a prediction rule on the basis of some data and then wish to estimate the error rate of this rule in classifying future observations cross validation provides a nearly unbiased estimate using only the original data cross validation turns out to be related closely to the bootstrap estimate of the error rate this article has two purposes to understand better the theoretical basis of the prediction problem and to investigate some related estimators which seem to offer considerably improved estimation in small samples

functional linear discriminant analysis for irregularly sampled curves we introduce a technique for extending the classical method of linear discriminant analysis lda to data sets where the predictor variables are curves or functions this procedure which we call functional linear discriminant analysis flda is particularly useful when only fragments of the curves are observed all the techniques associated with lda can be extended for use with flda in particular flda can be used to produce classifications on new test curves give an estimate of the discriminant function between classes and provide a one or two dimensional pictorial representation of a set of curves we also extend this procedure to provide generalizations of quadratic and regularized discriminant analysis

towards a theory of early visual processing we propose a theory of the early processing in the mammalian visual pathway the theory is formulated in the language of information theory and hypothesizes that the goal of this processing is to recode in order to reduce a generalized redundancy subject to a constraint that specifies the amount of average information preserved in the limit of no noise this theory becomes equivalent to barlow s redundancy reduction hypothesis but it leads to very different computational strategies when noise is present a tractable approach for finding the optimal encoding is to solve the problem in successive stages where at each stage the optimization is performed within a restricted class of transfer functions we explicitly find the solution for the class of encodings to which the parvocellular retinal processing belongs namely linear and nondivergent transformations the solution shows agreement with the experimentally observed transfer functions at all levels of signal to noise

on the distributions of the lengths of the longest monotone subsequences in random words we consider the distributions of the lengths of the longest weakly increasing and strongly decreasing subsequences in words of length n from an alphabet of k letters in the limit as k these become the corresponding distributions for permutations on n letters we find toeplitz determinant representations for the exponential generating functions on n of these distribution functions and show that they are expressible in terms of solutions of painleve v equations we show further that in the weakly increasing case the generating unction gives the distribution of the smallest eigenvalue in the k k laguerre random matrix ensemble and that the distribution itself has after centering and normalizing an n limit which is equal to the distribution function for the largest eigenvalue in the gaussian unitary ensemble of k k hermitian matrices of trace zero

efficient integration of sufficient dimension reduction and prediction in discriminant analysis abstractsufficient dimension reduction sdr methods are popular model free tools for preprocessing and data visualization in regression problems where the number of variables is large unfortunate

estimation of a covariance matrix under stein s loss on decrit la technique generale de stein pour ameliorer l estimateur sans biais invariant le meilleur et l estimateur minimax de la matrice de covariance normale

bilinear probabilistic principal component analysis probabilistic principal component analysis ppca is a popular linear latent variable model for performing dimension reduction on 1 d data in a probabilistic manner however when used on 2 d data such as images ppca suffers from the curse of dimensionality due to the subsequently large number of model parameters to overcome this problem we propose in this paper a novel probabilistic model on 2 d data called bilinear ppca bppca this allows the establishment of a closer tie between bppca and its nonprobabilistic counterpart moreover two efficient parameter estimation algorithms for fitting bppca are also developed experiments on a number of 2 d synthetic and real world data sets show that bppca is more accurate than existing probabilistic and nonprobabilistic dimension reduction methods

robust singular value decompositions a new approach to projection pursuit abstract robust location and covariance estimators are developed via general m estimation for covariance matrix eigenvectors and eigenvalues the solution to this gm estimation problem is obtained by transforming it into a series of robust regression problems based on a new algorithm for the singular value decomposition it is shown here that the singular value decomposition can be represented as an iteration of two steps a least squares regression fit of the data matrix followed by a rotation to the regression hyperplanes an algorithm to obtain the solution to this gm estimation problem is presented along with results of a monte carlo study and examples of its application in addition it is shown how the output of this algorithm can be used to numerically search for multivariate outliers which is especially useful in exploratory data analysis with high dimensional data and large sample sizes where standard graphical techniques are difficult to implement because the algorithm computes robust estima

an optimization criterion for generalized discriminant analysis on undersampled problems an optimization criterion is presented for discriminant analysis the criterion extends the optimization criteria of the classical linear discriminant analysis lda through the use of the pseudoinverse when the scatter matrices are singular it is applicable regardless of the relative sizes of the data dimension and sample size overcoming a limitation of classical lda the optimization problem can be solved analytically by applying the generalized singular value decomposition gsvd technique the pseudoinverse has been suggested and used for undersampled problems in the past where the data dimension exceeds the number of data points the criterion proposed in this paper provides a theoretical justification for this procedure an approximation algorithm for the gsvd based approach is also presented it reduces the computational complexity by finding subclusters of each cluster and uses their centroids to capture the structure of each cluster this reduced problem yields much smaller matrices to which the gsvd can be applied efficiently experiments on text data with up to 7 000 dimensions show that the approximation algorithm produces results that are close to those produced by the exact algorithm

robust time optimal control of constrained linear systems a version of dynamic programming which computes level sets of the value function rather than the value function set itself is used to design robust non linear controllers for linear discrete time dynamical systems subject to hard constraints on controls and states the controller stabilizes the system and steers all trajectories emanating in a prescribed set to a control invariant set in minimum time for the robust regulator problem the control invariant terminal set is a neighborhood preferably small of the origin for the robust tracking problem the control invariant terminal set is a neighborhood of the invariant set in which the tracking error is zero two non linear controllers which utilize the level sets of the value function are described the first requires the controller to solve on line a modest linear program whose dimension is approximately the same as that of the control variable the second decomposes each level set into a set of simplices a piecewise linear control law affine in each simplex is then constructed

novel fisher discriminant classifiers at the present several applications need to classify high dimensional points belonging to highly unbalanced classes unfortunately when the training set cardinality is small compared to the data dimensionality small sample size problem the classification performance of several well known classifiers strongly decreases similarly the classification accuracy of several discriminative methods decreases when non linearly separable and unbalanced classes are treated in this paper we firstly survey state of the art methods that employ improved versions of linear discriminant analysis lda to deal with the above mentioned problems secondly we propose a family of classifiers based on the fisher subspace estimation which efficiently deal with the small sample size problem non linearly separable classes and unbalanced classes the promising results obtained by the proposed techniques on benchmark datasets and the comparison with state of the art predictors show the efficacy of the proposed techniques

computation of the constrained infinite time linear quadratic regulator this paper presents an efficient algorithm for computing the solution to the constrained infinite time linear quadratic regulator clqr problem for discrete time systems the algorithm combines multi parametric quadratic programming with reachability analysis to obtain the optimal piecewise affine pwa feedback law the algorithm reduces the time necessary to compute the pwa solution for the clqr when compared to other approaches it also determines the minimal finite horizon email protected s such that the constrained finite horizon lqr problem equals the clqr problem for a compact set of states s the on line computational effort for the implementation of the clqr can be significantly reduced as well either by evaluating the pwa solution or by solving the finite dimensional quadratic program associated with the clqr for a horizon of email protected s

generalized consistent error estimator of linear discriminant analysis a classifier is epistemologically vacuous without an accurate estimate of its true error rate in situations where the number of sample points is of the same order of magnitude as the dimension of observations serious issues arise with respect to the performance of error estimators in this paper we place the problem of synthesizing an error rate estimator of a common linear classifier in an asymptotic setting in which the number of sample points is kept comparable in magnitude to the dimension of observations double asymptotic we construct a generalized consistent estimator of the true error rate for linear discriminant analysis in the multivariate gaussian model under the assumption of a common covariance matrix in other words the estimator converges to true error rate in the double asymptotic sense we employ simulations using both synthetic and real data to compare the performance of the new estimator to the classical consistent estimator of the true error plug in estimator as well as other well known estimators we observe that the constructed estimator can outperform other estimators of the true error in many situations in terms of bias and root mean square rms error

efficient and robust feature extraction by maximum margin criterion in pattern recognition feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information principal component analysis pca and linear discriminant analysis lda are the two most popular linear dimensionality reduction methods however pca is not very effective for the extraction of the most discriminant features and lda is not stable due to the small sample size problem in this paper we propose some new linear and nonlinear feature extractors based on maximum margin criterion mmc geometrically feature extractors based on mmc maximize the average margin between classes after dimensionality reduction it is shown that mmc can represent class separability better than pca as a connection to lda we may also derive lda from mmc by incorporating some constraints by using some other constraints we establish a new linear feature extractor that does not suffer from the small sample size problem which is known to cause serious stability problems for lda the kernelized nonlinear counterpart of this linear feature extractor is also established in the paper our extensive experiments demonstrate that the new feature extractors are effective stable and efficient

covariance regularized regression and classification for high dimensional problems in recent years many methods have been developed for regression in high dimensional settings we propose covariance regularized regression a family of methods that use a shrunken estimate of the inverse covariance matrix of the features in order to achieve superior prediction an estimate of the inverse covariance matrix is obtained by maximizing its log likelihood under a multivariate normal model subject to a constraint on its elements this estimate is then used to estimate coefficients for the regression of the response onto the features we show that ridge regression the lasso and the elastic net are special cases of covariance regularized regression and we demonstrate that certain previously unexplored forms of covariance regularized regression can outperform existing methods in a range of situations the covariance regularized regression framework is extended to generalized linear models and linear discriminant analysis and is used to analyze gene expression data sets with multiple class and survival outcomes

distribution of eigenvalues of ensembles of asymmetrically diluted hopfield matrices using grassmann variables and an analogy with two dimensional electrostatics we obtain the average eigenvalue distribution of ensembles of asymmetrically diluted hopfield matrices in the limit we found that in the limit of strong dilution the distribution is uniform in a circle in the complex plane

brief an algorithm for multi parametric quadratic programming and explicit mpc solutions explicit solutions to constrained linear model predictive control problems can be obtained by solving multi parametric quadratic programs mp qp where the parameters are the components of the state vector we study the properties of the polyhedral partition of the state space induced by the multi parametric piecewise affine solution and propose a new mp qp solver compared to existing algorithms our approach adopts a different exploration strategy for subdividing the parameter space avoiding unnecessary partitioning and qp problem solving with a significant improvement of efficiency

appearance models based on kernel canonical correlation analysis this paper introduces a new approach to constructing appearance models based on kernel canonical correlation analysis kernel cca kernel cca is a non linear extension of cca where a non linear transformation of the input data is performed implicitly using kernel methods although in this respect it is similar to other generalized linear methods kernel cca is especially well suited for relating two sets of measurements the benefits of our method compared to standard feature extraction methods based on pca will be illustrated experimentally for the task of estimating an object s pose from raw brightness images

the explicit linear quadratic regulator for constrained systems we present a technique to compute the explicit state feedback solution to both the finite and infinite horizon linear quadratic optimal control problem subject to state and input constraints we show that this closed form solution is piecewise linear and continuous as a practical consequence of the result constrained linear quadratic regulation becomes attractive also for systems with high sampling rates as on line quadratic programming solvers are no more required for the implementation

dynamics of batch training in a perceptron early stopping and weight decay are studied in a linear perceptron using a new simplified approach to the dynamics in the thermodynamical limit the approach is directly deduced from the gradient descent weight update it allows an exact description of the dynamics of the batch training process the results are compared with a recent study of early stopping and weight decay based on the equilibrium statistical mechanics approach it is shown that the equilibrium results for early stopping are good approximations and are exact for weight decay furthermore in the dynamical approach it is possible to determine the necessary number of training steps to fulfil certain termination conditions it can be shown that asymptotically i e if the number of examples is large only two batch steps are required to reach the optimal convergence if the learning rate is optimally chosen

eigenvalues of covariance matrices application to neural network learning the learing time of a simple neural network model is obtained through an analytic computation of the eigenvalue spectrum for the hessian matrix which describes the second order properties of the objective function in the space of coupling coefficients the results are generic for symmetric matrices obtained by summing outer products of random vectors the form of the eigenvalue distribution suggests new techniques for accelerating the learning process and provides a theoretical justification for the choice of centered versus biased state variables

kernel quadratic discriminant analysis for small sample size problem it is generally believed that quadratic discriminant analysis qda can better fit the data in practical pattern recognition applications compared to linear discriminant analysis lda method this is due to the fact that qda relaxes the assumption made by lda based methods that the covariance matrix for each class is identical however it still assumes that the class conditional distribution is gaussian which is usually not the case in many real world applications in this paper a novel kernel based qda method is proposed to further relax the gaussian assumption by using the kernel machine technique the proposed method solves the complex pattern recognition problem by combining the qda solution and the kernel machine technique and at the same time tackles the so called small sample size problem through a regularized estimation of the covariance matrix extensive experimental results indicate that the proposed method is a more sophisticated solution outperforming many traditional kernel based learning algorithms

inexact implementation using krylov subspace methods for large scale exponential discriminant analysis with applications to high dimensionality reduction problems abstract exponential discriminant analysis eda is a generalized discriminant analysis method based on matrix exponential it can essentially overcome the intrinsic difficulty of small sample size problem that exists in the classical linear discriminant analysis lda however for data with high dimensionality one has to solve a large matrix exponential eigenproblem in this method and the time complexity is dominated by the computation of exponential of large matrices in this paper we propose two inexact krylov subspace algorithms for solving the large matrix exponential eigenproblem efficiently the contribution of this work is threefold first we consider how to compute matrix exponential vector products efficiently which is the key step in the krylov subspace method second we compare the discriminant analysis criterion of eda and that of lda from a theoretical point of view third we establish the relationship between the accuracy of the approximate eigenvectors and the distance to nearest neighbor classifier and show why the matrix exponential eigenproblem can be solved approximately in practice numerical experiments on some real world databases show the superiority of our new algorithms over their original counterpart for face recognition

rapid and brief communications modified linear discriminant analysis in this paper a modified fisher linear discriminant analysis flda is proposed and aims to not only overcome the rank limitation of flda that is at most only finding a discriminant vector for 2 class problem based on fisher discriminant criterion but also relax singularity of the within class scatter matrix and finally improves classification performance of flda experiments on nine publicly available datasets show that the proposed method has better or comparable performance on all the datasets than flda

penalized discriminant analysis fisher s linear discriminant analysis lda is a popular data analytic tool for studying the relationship between a set of predictors and a categorical response in this paper we describe a penalized version of lda it is designed for situations in which there are many highly correlated predictors such as those obtained by discretizing a function or the grey scale values of the pixels in a series of images in cases such as these it is natural efficient and sometimes essential to impose a spatial smoothness constraint on the coefficients both for improved prediction performance and interpretability we cast the classification problem into a regression framework via optimal scoring using this our proposal facilitates the use of any penalized regression technique in the classification setting the technique is illustrated with examples in speech recognition and handwritten character recognition

enhanced fisher discriminant criterion for image recognition many previous studies have shown that image recognition can be significantly improved by fisher linear discriminant analysis flda technique however flda ignores the variation of data points from the same class which characterizes the most important modes of variability of patterns and helps to improve the generalization capability of flda thus the performance of flda on testing data is not good enough in this paper we propose an enhanced fisher discriminant criterion efdc efdc explicitly considers the intra class variation and incorporates the intra class variation into the fisher discriminant criterion to build a robust and efficient dimensionality reduction function efdc obtains a subspace which best detects the discriminant structure and simultaneously preserves the modes of variability of patterns which will result in stable intraclass representation experimental results on four image database show a clear improvement over the results of flda based methods

self supervised learning based on discriminative nonlinear features for image classification for learning based tasks such as image classification the feature dimension is usually very high the learning is afflicted by the curse of dimensionality as the search space grows exponentially with the dimension discriminant em dem proposed a framework by applying self supervised learning in a discriminating subspace this paper extends the linear dem to a nonlinear kernel algorithm kernel dem kdem and evaluates kdem extensively on benchmark image databases and synthetic data various comparisons with other state of the art learning techniques are investigated for several tasks of image classification extensive results show the effectiveness of our approach

asymptotics of plancherel measures for symmetric groups 1 1 plancherel measures given a finite group g by the corresponding plancherel measure we mean the probability measure on the set g of irreducible representations of g which assigns to a representation g the weight dim g for the symmetric group s n the set s n is the set of partitions of the number n which we shall identify with young diagrams with n squares throughout this paper the plancherel measure on partitions arises naturally in representation theoretic combinatorial and probabilistic problems for example the plancherel distribution of the first part of a partition coincides with the distribution of the longest increasing subsequence of a uniformly distributed random permutation 31 we denote the plancherel measure on partitions of n by mn