prediction and classification in nonlinear data analysis something old something new something borrowed something blue prediction and classification are two very active areas in modern data analysis in this paper prediction with nonlinear optimal scaling transformations of the variables is reviewed and extended to the use of multiple additive components much in the spirit of statistical learning techniques that are currently popular among other areas in data mining also a classification clustering method is described that is particularly suitable for analyzing attribute value data from systems biology genomics proteomics and metabolomics and which is able to detect groups of objects that have similar values on small subsets of the attributes

discriminant analysis by gaussian mixtures fisher rao linear discriminant analysis lda is a valuable tool for multigroup classification lda is equivalent to maximum likelihood classification assuming gaussian distributions for each class in this paper we fit gaussian mixtures to each class to facilitate effective classification in non normal settings especially when the classes are clustered low dimensional views are an important by product of lda our new techniques inherit this feature we can control the within class spread of the subclass centres relative to the between class spread our technique for fitting these models permits a natural blend with nonparametric versions of lda

learning efficiency of very simple grammars from positive data the class of very simple grammars is known to be polynomial time identifiable in the limit from positive data this paper gives an even more general discussion on the efficiency of identification of very simple grammars from positive data which includes both positive and negative results in particular we present an alternative efficient inconsistent learning algorithm for very simple grammars

flexible discriminant analysis by optimal scoring abstract fisher s linear discriminant analysis is a valuable tool for multigroup classification with a large number of predictors one can find a reduced number of discriminant coordinate functions that are optimal for separating the groups with two such functions one can produce a classification map that partitions the reduced space into regions that are identified with group membership and the decision boundaries are linear this article is about richer nonlinear classification schemes linear discriminant analysis is equivalent to multiresponse linear regression using optimal scorings to represent the groups in this paper we obtain nonparametric versions of discriminant analysis by replacing linear regression by any nonparametric regression method in this way any multiresponse regression technique such as mars or neural networks can be postprocessed to improve its classification performance

svm decision boundary based discriminative subspace induction we study the problem of linear dimension reduction for classification with a focus on sufficient dimension reduction i e finding subspaces without loss of discrimination power first we formulate the concept of sufficient subspace for classification in parallel terms as for regression then we present a new method to estimate the smallest sufficient subspace based on an improvement of decision boundary analysis dba the main idea is to combine dba with support vector machines svm to overcome the inherent difficulty of dba in small sample size situations while keeping dba s estimation simplicity the compact representation of svm boundary results in a significant gain in both speed and accuracy over previous dba implementations alternatively this technique can be viewed as a way to reduce the run time complexity of svm itself comparative experiments on one simulated and four real world benchmark datasets highlight the superior performance of the proposed approach

sparse multinomial kernel discriminant analysis smkda dimensionality reduction via canonical variate analysis cva is important for pattern recognition and has been extended variously to permit more flexibility e g by kernelizing the formulation this can lead to over fitting usually ameliorated by regularization here a method for sparse multinomial kernel discriminant analysis smkda is proposed using a sparse basis to control complexity it is based on the connection between cva and least squares and uses forward selection via orthogonal least squares to approximate a basis generalizing a similar approach for binomial problems classification can be performed directly via minimum mahalanobis distance in the canonical variates smkda achieves state of the art performance in terms of accuracy and sparseness on 11 benchmark datasets

the ii p method for estimating multivariate functions from noisy data the method for estimating an underlying smooth function of m variables x l xm using noisy data is based on approximating it by a sum of products of the form m m x m the problem is then reduced to estimating the univariate functions in the products a convergent algorithm is described the method keeps tight control on the degrees of freedom used in the fit many examples are given the quality of fit given by the method is excellent usually only a few products are enough to fit even fairly complicated functions the coding into products of univariate functions allows a relatively understandable interpretation of the multivariate fit

on fitting cox s proportional hazards models to survey data summary binder 1992 proposed a method of fitting cox s proportional hazards models to survey data with complex sampling designs he defined the regression parameter of interest as the solution to the partial likelihood score equation based on all the data values of the survey population under study and developed heuristically a procedure to estimate the regression parameter and the corresponding variance in this paper we provide a formal justification of binder s method furthermore we present an alternative approach which regards the survey population as a random sample from an infinite universe and accounts for this randomness in the statistical inference under the alternative approach the regression parameter retains its original interpretation as the log hazard ratio and the statistical conclusion applies to other populations the related problem of survival function estimation is also studied

the discrimination subspace model abstract for a p variate normal random vector measured in two populations we propose a method of discrimination under the constraint that all differences between the two populations occur in a subspace of dimension q p this method of classification is based on the discrimination subspace model denoted by dsm q and is intermediate between linear and quadratic discrimination it combines the ideas of dimension reduction and constraints on the parameter space thus substantially reducing the number of parameters to be estimated the maximum likelihood estimators of the model are presented and the performance of dsm q versus quadratic and linear discrimination is assessed via simulation it is generally shown that discrimination based on dsm q consistently yields noticeably lower expected actual error rates relative to the traditional methods the method is illustrated with a real data example and is compared to linear and quadratic discrimination using a leave one out method the example confirms t

do we need hundreds of classifiers to solve real world classification problems we evaluate 179 classifiers arising from 17 families discriminant analysis bayesian neural networks support vector machines decision trees rule based classifiers boosting bagging stacking random forests and other ensembles generalized linear models nearest neighbors partial least squares and principal component regression logistic and multinomial regression multiple adaptive regression splines and other methods implemented in weka r with and without the caret package c and matlab including all the relevant classifiers available today we use 121 data sets which represent the whole uci data base excluding the large scale problems and other own real problems in order to achieve significant conclusions about the classifier behavior not dependent on the data set collection the classifiers most likely to be the bests are the random forest rf versions the best of which implemented in r and accessed via caret achieves 94 1 of the maximum accuracy overcoming 90 in the 84 3 of the data sets however the difference is not statistically significant with the second best the svm with gaussian kernel implemented in c using libsvm which achieves 92 3 of the maximum accuracy a few models are clearly better than the remaining ones random forest svm with gaussian and polynomial kernels extreme learning machine with gaussian kernel c5 0 and avnnet a committee of multi layer perceptrons implemented in r with the caret package the random forest is clearly the best family of classifiers 3 out of 5 bests classifiers are rf followed by svm 4 classifiers in the top 10 neural networks and boosting ensembles 5 and 3 members in the top 20 respectively

rich classes inferable from positive data inductive inference from positive data is shown to be remarkably powerful using the framework of elementary formal systems an elementary formal system efs for short is a kind of logic program on consisting of finitely many axioms any context sensitive language is definable by a restricted efs called a length bounded efs length bounded efss with at most n axioms are considered and it is shown that inductive inference from positive data works successfully for their models as well as for their languages from this it follows that any class of logic programs such as prolog programs corresponding to length bounded efss can be inferred from positive facts

a multi manifold discriminant analysis method for image feature extraction in this paper we propose a multi manifold discriminant analysis mmda method for an image feature extraction and pattern recognition based on graph embedded learning and under the fisher discriminant analysis framework in an mmda the within class graph and between class graph are respectively designed to characterize the within class compactness and the between class separability seeking for the discriminant matrix to simultaneously maximize the between class scatter and minimize the within class scatter in addition in an mmda the within class graph can represent the sub manifold information while the between class graph can represent the multi manifold information the proposed mmda is extensively examined by using the feret ar and orl face databases and the polyu finger knuckle print databases the experimental results demonstrate that an mmda is effective in feature extraction leading to promising image recognition performance

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

tree structured classification via generalized discriminant analysis abstract the problem of constructing classification rules that can be represented as decision trees is considered each object to be classified has an associated x vector containing possibly incomplete covariate information tree construction is based on the information provided in a learning sample of objects with known class identities the x vectors in the learning sample may have missing values as well procedures are proposed for each of the components of classifier construction such as split selection tree size determination treatment of missing values and ranking of variables the main idea is recursive application of linear discriminant analysis with the variables at each stage being appropriately chosen according to the data and the type of splits desired standard statistical techniques used as basic building blocks include analysis of variance linear and canonical discriminant analysis and principal component analysis a new method of tree structured classification is obtained by assem

predicting multivariate responses in multiple linear regression we look at the problem of predicting several response variables from the same set of explanatory variables the question is how to take advantage of correlations between the response variables to improve predictive accuracy compared with the usual procedure of doing individual regressions of each response variable on the common set of predictor variables a new procedure is introduced called the curds and whey method its use can substantially reduce prediction errors when there are correlations between responses while maintaining accuracy even if the responses are uncorrelated in extensive simulations the new procedure is compared with several previously proposed methods for predicting multiple responses including partial least squares and exhibits superior accuracy one version can be easily implemented in the context of standard statistical packages

iterative learning of simple external contextual languages it is investigated for which choice of a parameter q denoting the number of contexts the class of simple external contextual languages is iteratively learnable on the one hand the class admits for all values of q polynomial time learnability provided an adequate choice of the hypothesis space is given on the other hand additional constraints like consistency and conservativeness or the use of a one one hypothesis space changes the picture iterative learning limits the long term memory of the learner to the current hypothesis and these constraints further hinder storage of information via padding of this hypothesis it is shown that if q 3 then simple external contextual languages are not iteratively learnable using a class preserving one one hypothesis space while for q 1 it is iteratively learnable even in polynomial time it is also investigated for which choice of the parameters the simple external contextual languages can be learnt by a consistent and conservative iterative learner

linear smoothers and additive models we study linear smoothers and their use in building nonparametric regression models in the first part of this paper we examine certain aspects of linear smoothers for scatterplots examples of these are the running mean and running line kernel and cubic spline smoothers the eigenvalue and singular value decompositions of the corresponding smoother matrix are used to describe qualitatively a smoother and several other topics such as the number of degrees of freedom of a smoother are discussed in the second part of the paper we describe how linear smoothers can be used to estimate the additive model a powerful nonparametric regression model using the backfitting algorithm we show that backfitting is the gauss seidel iterative method for solving a set of normal equations associated with the additive model we provide conditions for consistency and nondegeneracy and prove convergence for the backfitting and related algorithms for a class of smoothers that includes cubic spline smoothers

classification trees with unbiased multiway splits two univariate split methods and one linear combination split method are proposed for the construction of classification trees with multiway splits examples are given where the trees are more compact and hence easier to interpret than binary trees a major strength of the univariate split methods is that they have negligible bias in variable selection both when the variables differ in the number of splits they offer and when they differ in the number of missing values this is an advantage because inferences from the tree structures can be adversely affected by selection bias the new methods are shown to be highly competitive in terms of computational speed and classification accuracy of future observations

shape adaptive radial basis functions radial basis functions for discrimination and regression have been used with some success in a wide variety of applications here we investigate the optimal choice for the form of the basis functions and present an iterative strategy for obtaining the function in a regression context using a conjugate gradient based algorithm together with a nonparametric smoother this is developed in a discrimination framework using the concept of optimal scaling results are presented for a range of simulated and real data sets

classification by pairwise coupling we discuss a strategy for polychotomous classification that involves estimating class probabilities for each pair of classes and then coupling the estimates together the coupling model is similar to the bradley terry method for paired comparisons we study the nature of the class probability estimates that arise and examine the performance of the procedure in real and simulated data sets classifiers used include linear discriminants nearest neighbors adaptive nonlinear methods and the support vector machine

shape descriptors for classification of functional data curve discrimination is an important task in engineering and other sciences we propose several shape descriptors for classifying functional data inspired by form analysis from the image analysis field statistical moments coefficients of the components of independent component analysis and two mathematical morphology descriptors morphological covariance and spatial size distributions these are applied to three problems an artificial problem a speech recognition problem and a biomechanical application shape descriptors are compared with other methods in the literature with better or similar performance obtaining

penalized discriminant methods for the classification of tumors from gene expression data en raison de l avancee des technologies de micropuces haut debit il est devenu possible de developper des systemes de classifications moleculaires pour differents types de cancer dans cet article nous proposons une methode fondee sur les modeles de regression regularisee pour la classification des tumeurs dans les experiences de micropuces les performances des modeles en composantes principales des moindres carres partiels et de la ridge regression sont etudiees ces procedures de regression sont adaptees aux techniques de classification utilisant l algorithme de score optimal nous developpons egalement une methode de classement des genes fondee sur les modeles de regression ajustee les methodes proposees sont appliquees a deux etudes de cancerologie utilisant les micropuces

on the role of procrastination in machine learning the traditional model of inductive inference is enhanced to allow learning machines to procrastinate about how many trials they will need to complete an inference or about how accurate their solution will be hierarchies of classes of learnable phenomena represented by sets of recursive functions isomorphic to the structure of the constructive ordinals are revealed the existence of such hierarchies indicates a potential advantage of procrastination as a learning technique tradeoffs between the two types of procrastination are shown not to exist in general one of the new classes of sets of inferrible functions introduced in this paper turns out to have the somewhat unusual property of being closed under finite unions a new technique based on a hardest to lear set relative to a class is used

flexible parsimonious smoothing and additive modeling a simple method is presented for fitting regression models that are nonlinear in the explanatory variables despite its simplicity or perhaps because of it the method has some powerful characteristics that cause it to be competitive with and often superior to more sophisticated techniques especially for small data sets in the presence of high noise

fitting cox s proportional hazards models from survey data summary the proportional hazards model makes a number of assumptions which may not always be completely satisfied however fitting such models can have both descriptive and analytical value we discuss the implications of the survey design for large scale studies we propose a weighted estimator and derive a design based estimator for the variance of the estimated parameters the derivation requires using methods not previously used in the context of analysis of survey data we conduct a small simulation study comparing the proposed method with others in the literature

functional classification in hilbert spaces let x be a random variable taking values in a separable hilbert space x with label y spl isin 0 1 we establish universal weak consistency of a nearest neighbor type classifier based on n independent copies x sub i y sub i of the pair x y extending the classical result of stone to infinite dimensional hilbert spaces under a mild condition on the distribution of x we also prove strong consistency we reduce the infinite dimension of x by considering only the first d coefficients of a fourier series expansion of each x sub i and then we perform k nearest neighbor classification in spl ropf sup d both the dimension and the number of neighbors are automatically selected from the data using a simple data splitting device an application of this technique to a signal discrimination problem involving speech recordings is presented

polynomial splines and their tensor products in extended linear modeling 1994 wald memorial lecture analysis of variance type models are considered for a regression function or for the logarithm of a probability function conditional probability function density function conditional density function hazard function conditional hazard function or spectral density function polynomial splines are used to model the main effects and their tensor products are used to model any interaction components that are included in the special context of survival analysis the baseline hazard function is modeled and nonproportionality is allowed in general the theory involves the l 2 rate of convergence for the fitted model and its components the methodology involves least squares and maximum likelihood estimation stepwise addition of basis functions using rao statistics stepwise deletion using wald statistics and model selection using the bayesian information criterion cross validation or an independent test set publicly available software written in c and interfaced to s s plus is used to apply this methodology to real data

discriminant analysis through a semiparametric model summary we consider a semiparametric generalisation of normal theory discriminant analysis the semiparametric model assumes that after unspecified univariate monotone transformations the class distributions are multivariate normal we introduce an estimation procedure based on the distribution quantiles in which the parameters of the semiparametric model are estimated directly without estimating the nonparametric transformations the procedure is computationally fast and the estimation accuracy is shown to have the usual parametric rate the relationship between the method and more general nonparametric discriminant analysis is discussed the semiparametric specification of the class densities is a submodel of the nonparametric log density functional analysis of variance model in which the main effects are completely nonparametric but the interaction terms are specified semiparametrically simulations and real examples are used to illustrate the procedure

classification via kernel product estimators summary multivariate kernel density estimation is often used as the basis for a nonparametric classification technique however the multivariate kernel classifier suffers from the curse of dimensionality requiring inordinately large sample sizes to achieve a reasonable degree of accuracy in high dimensional settings a variance stabilising approach to kernel classification can be motivated through an alternative interpretation of linear and quadratic discriminant analysis in which rotations of the coordinate axes are employed to obtain an assumed mutual independence among the components of the rotated data this alternative method which we call the method of kernel product estimators performs well in a variety of examples including a 20 dimensional target recognition problem

exploratory projection pursuit abstract a new projection pursuit algorithm for exploring multivariate data is presented that has both statistical and computational advantages over previous methods a number of practical issues concerning its application are addressed a connection to multivariate density estimation is established and its properties are investigated through simulation studies and application to real data the goal of exploratory projection pursuit is to use the data to find low one two or three dimensional projections that provide the most revealing views of the full dimensional data with these views the human gift for pattern recognition can be applied to help discover effects that may not have been anticipated in advance since linear effects are directly captured by the covariance structure of the variable pairs which are straightforward to estimate the emphasis here is on the discovery of nonlinear effects such as clustering or other general nonlinear associations among the variables although arbitrary

models for nonresponse in sample surveys abstract the literature on the analysis of incomplete data using models is reviewed in the context of nonresponse in sample surveys the modeling approach provides a large body of methods for handling unit and item nonresponse some of which cannot be derived from the randomization theory of inference for surveys key concepts from the literature on incomplete data such as factorizations of the likelihood for special data patterns the em algorithm for general data patterns and ignorability of the response mechanism are discussed within the survey context model based procedures are related to common methods for handling nonresponse in surveys such as weighting or imputation of means within subclasses of the population

recursive concave convex fisher linear discriminant with applications to face handwritten digit and terrain recognition in classification previous studies have shown that an eigenvalue based technique can be cast as an related svm type problem and that by solving this svm type problem the performance can be improved significantly in this paper we develop a recursive concave convex fisher linear discriminant dr rpfld for dimension reduction technique of high dimensional data to extract as many meaningful features as possible which incorporates the fundamental idea behind fisher linear discriminant and casts the fisher linear discriminant as a concave convex programming problem based on the hinge loss the solution of our method follows from solving the related svm type optimization problems iteratively which means the proposed method can be viewed as the combination of multiple related svm type problems the special formulation of our method provides convenience for constructing sparse multi class fisher linear discriminant directly due to use of a recursive procedure the number of features available from rpfld is independent of the number of classes meaning that in contrast to the original fisher linear discriminant the number of features available from our method has no upper bound we evaluate our algorithm on the yale and orl face image databases handwritten digit database and terrain image dataset experimental results show that rpfld outperforms other fisher linear discriminant algorithms

breast cancer prediction using the isotonic separation technique a recently developed data separation classification method called isotonic separation is applied to breast cancer prediction two breast cancer data sets one with clean and sufficient data and the other with insufficient data are used for the study and the results are compared against those of decision tree induction methods linear programming discrimination methods learning vector quantization support vector machines adaptive boosting and other methods the experiment results show that isotonic separation is a viable and useful tool for data classification in the medical domain

natural discriminant analysis using interactive potts models natural discriminant analysis based on interactive potts models is developed in this work a generative model composed of piece wise multivariate gaussian distributions is used to characterize the input space exploring the embedded clustering and mixing structures and developing proper internal representations of input parameters the maximization of a log likelihood function measuring the fitness of all input parameters to the generative model and the minimization of a design cost summing up square errors between posterior outputs and desired outputs constitutes a mathematical framework for discriminant analysis we apply a hybrid of the mean field annealing and the gradient descent methods to the optimization of this framework and obtain multiple sets of interactive dynamics which realize coupled potts models for discriminant analysis the new learning process is a whole process of component analysis clustering analysis and labeling analysis its major improvement compared to the radial basis function and the support vector machine is described by using some artificial examples and a real world application to breast cancer diagnosis

on the intrinsic complexity of learning a new view of learning is presented the basis of this view is a natural notion of reduction we prove completeness and relative difficulty results an infinite hierarchy of intrinsically more and more difficult to learn concepts is presented our results indicate that the complexity notion captured by our new notion of reduction differs dramatically from the traditional studies of the complexity of the algorithms performing learning tasks

a model for markers and latent health status summary we extend the bivariate wiener process considered by whitmore and co workers and model the joint process of a marker and health status the health status process is assumed to be latent or unobservable the time to reach the primary end point or failure death onset of disease etc is the time when the latent health status process first crosses a failure threshold level inferences for the model are based on two kinds of data censored survival data and marker measurements covariates such as treatment variables risk factors and base line conditions are related to the model parameters through generalized linear regression functions the model offers a much richer potential for the study of treatment efficacy than do conventional models treatment effects can be assessed in terms of their influence on both the failure threshold and the health status process parameters we derive an explicit formula for the prediction of residual failure times given the current marker level also we discuss model validation this model does not require the proportional hazards assumption and hence can be widely used to demonstrate the usefulness of the model we apply the methods in analysing data from protocol 116a of the aids clinical trials group

feature extraction based on laplacian bidirectional maximum margin criterion maximum margin criterion mmc based feature extraction is more efficient than linear discriminant analysis lda for calculating the discriminant vectors since it does not need to calculate the inverse within class scatter matrix however mmc ignores the discriminative information within the local structures of samples and the structural information embedding in the images in this paper we develop a novel criterion namely laplacian bidirectional maximum margin criterion lbmmc to address the issue we formulate the image total laplacian matrix image within class laplacian matrix and image between class laplacian matrix using the sample similar weight that is widely used in machine learning the proposed lbmmc based feature extraction computes the discriminant vectors by maximizing the difference between image between class laplacian matrix and image within class laplacian matrix in both row and column directions experiments on the feret and yale face databases show the effectiveness of the proposed lbmmc based feature extraction method

generalization performance of fisher linear discriminant based on markov sampling fisher linear discriminant fld is a well known method for dimensionality reduction and classification that projects high dimensional data onto a low dimensional space where the data achieves maximum class separability the previous works describing the generalization ability of fld have usually been based on the assumption of independent and identically distributed i i d samples in this paper we go far beyond this classical framework by studying the generalization ability of fld based on markov sampling we first establish the bounds on the generalization performance of fld based on uniformly ergodic markov chain u e m c samples and prove that fld based on u e m c samples is consistent by following the enlightening idea from markov chain monto carlo methods we also introduce a markov sampling algorithm for fld to generate u e m c samples from a given data of finite size through simulation studies and numerical studies on benchmark repository using fld we find that fld based on u e m c samples generated by markov sampling can provide smaller misclassification rates compared to i i d samples

the efficiency of multinomial logistic regression compared with multiple group discriminant analysis abstract multinomial logistic regression also referred to as polychotomous logistic regression is frequently used for the analysis of categorical response data with continuous or categorical explanatory variables parameter estimates are usually obtained through direct maximum likelihood estimation normal discriminant analysis has been used as an alternative approach to this methodology although it is strictly appropriate only when the usual normal discriminant assumptions concerning the explanatory variables are valid comparative evaluations of the two procedures including the investigation of bias and efficiency in parameter estimation have been almost entirely limited to the case of two response groups for both normal and nonnormal explanatory variables bull and donner 1987 extended the comparison of logistic regression and normal discrimination in the normal case to more than two response groups deriving the large sample distribution of the slope parameter estimates for each of the two proc

classification using kernel density estimates multiscale analysis and visualization the use of kernel density estimates in discriminant analysis is quite well known among scientists and engineers interested in statistical pattern recognition using a kernel density estimate involves properly selecting the scale of smoothing namely the bandwidth parameter the bandwidth that is optimum for the mean integrated square error of a class density estimator may not always be good for discriminant analysis where the main emphasis is on the minimization of misclassification rates on the other hand cross validation based methods for bandwidth selection which try to minimize estimated misclassification rates may require huge computation when there are several competing populations besides such methods usually allow only one bandwidth for each population density estimate whereas in a classification problem the optimum bandwidth for a class density estimate may vary significantly depending on its competing class densities and their prior probabilities therefore in a multiclass problem it

hinging hyperplanes for regression classification and function approximation a hinge function y h x consists of two hyperplanes continuously joined together at a hinge in regression prediction classification pattern recognition and noiseless function approximation use of sums of hinge functions gives a powerful and efficient alternative to neural networks with computation times several orders of magnitude less than is obtained by fitting neural networks with a comparable number of parameters a simple and effective method for finding good hinges is presented

efficient and robust feature extraction by maximum margin criterion in pattern recognition feature extraction techniques are widely employed to reduce the dimensionality of data and to enhance the discriminatory information principal component analysis pca and linear discriminant analysis lda are the two most popular linear dimensionality reduction methods however pca is not very effective for the extraction of the most discriminant features and lda is not stable due to the small sample size problem in this paper we propose some new linear and nonlinear feature extractors based on maximum margin criterion mmc geometrically feature extractors based on mmc maximize the average margin between classes after dimensionality reduction it is shown that mmc can represent class separability better than pca as a connection to lda we may also derive lda from mmc by incorporating some constraints by using some other constraints we establish a new linear feature extractor that does not suffer from the small sample size problem which is known to cause serious stability problems for lda the kernelized nonlinear counterpart of this linear feature extractor is also established in the paper our extensive experiments demonstrate that the new feature extractors are effective stable and efficient

discriminant analysis for fast multiclass data classification through regularized kernel function approximation in this brief we have proposed the multiclass data classification by computationally inexpensive discriminant analysis through vector valued regularized kernel function approximation vvrkfa vvrkfa being an extension of fast regularized kernel function approximation frkfa provides the vector valued response at single step the vvrkfa finds a linear operator and a bias vector by using a reduced kernel that maps a pattern from feature space into the low dimensional label space the classification of patterns is carried out in this low dimensional label subspace a test pattern is classified depending on its proximity to class centroids the effectiveness of the proposed method is experimentally verified and compared with multiclass support vector machine svm on several benchmark data sets as well as on gene microarray data for multi category cancer classification the results indicate the significant improvement in both training and testing time compared to that of multiclass svm with comparable testing accuracy principally in large data sets experiments in this brief also serve as comparison of performance of vvrkfa with stratified random sampling and sub sampling

generalized discriminant analysis using a kernel approach we present a new method that we call generalized discriminant analysis gda to deal with nonlinear discriminant analysis using kernel function operator the underlying theory is close to the support vector machines svm insofar as the gda method provides a mapping of the input vectors into high dimensional feature space in the transformed space linear properties make it easy to extend and generalize the classical linear discriminant analysis lda to nonlinear discriminant analysis the formulation is expressed as an eigenvalue problem resolution using a different kernel one can cover a wide class of nonlinearities for both simulated data and alternate kernels we give classification results as well as the shape of the decision function the results are confirmed using real data to perform seed classification

classification based on hybridization of parametric and nonparametric classifiers parametric methods of classification assume specific parametric models for competing population densities e g gaussian population densities can lead to linear and quadratic discriminant analysis and they work well when these model assumptions are valid violation in one or more of these parametric model assumptions often leads to a poor classifier on the other hand nonparametric classifiers e g nearest neighbor and kernel based classifiers are more flexible and free from parametric model assumptions but the statistical instability of these classifiers may lead to poor performance when we have small numbers of training sample observations nonparametric methods however do not use any parametric structure of population densities therefore even when one has some additional information about population densities that important information is not used to modify the nonparametric classification rule this paper makes an attempt to overcome these limitations of parametric and nonparametric approaches and combines their strengths to develop some hybrid classification methods we use some simulated examples and benchmark data sets to examine the performance of these hybrid discriminant analysis tools asymptotic results on their misclassification rates have been derived under appropriate regularity conditions

penalized discriminant analysis fisher s linear discriminant analysis lda is a popular data analytic tool for studying the relationship between a set of predictors and a categorical response in this paper we describe a penalized version of lda it is designed for situations in which there are many highly correlated predictors such as those obtained by discretizing a function or the grey scale values of the pixels in a series of images in cases such as these it is natural efficient and sometimes essential to impose a spatial smoothness constraint on the coefficients both for improved prediction performance and interpretability we cast the classification problem into a regression framework via optimal scoring using this our proposal facilitates the use of any penalized regression technique in the classification setting the technique is illustrated with examples in speech recognition and handwritten character recognition

extensions of lda by pca mixture model and class wise features linear discriminant analysis lda is a data discrimination technique that seeks transformation to maximize the ratio of the between class scatter and the within class scatter while it has been successfully applied to several applications it has two limitations both concerning the underfitting problem first it fails to discriminate data with complex distributions since all data in each class are assumed to be distributed in the gaussian manner second it can lose class wise information since it produces only one transformation over the entire range of classes we propose three extensions of lda to overcome the above problems the first extension overcomes the first problem by modelling the within class scatter using a pca mixture model that can represent more complex distribution the second extension overcomes the second problem by taking different transformation for each class in order to provide class wise features the third extension combines these two modifications by representing each class in terms of the pca mixture model and taking different transformation for each mixture component it is shown that all our proposed extensions of lda outperform lda concerning classification errors for synthetic data classification hand written digit recognition and alphabet recognition

using threshold regression to analyze survival data from complex surveys with application to mortality linked nhanes iii phase ii genetic data the cox proportional hazards ph model is a common statistical technique used for analyzing time to event data the assumption of ph however is not always appropriate in real applications in cases where the assumption is not tenable threshold regression tr and other survival methods which do not require the ph assumption are available and widely used these alternative methods generally assume that the study data constitute simple random samples in particular tr has not been studied in the setting of complex surveys that involve 1 differential selection probabilities of study subjects and 2 intracluster correlations induced by multistage cluster sampling in this paper we extend tr procedures to account for complex sampling designs the pseudo maximum likelihood estimation technique is applied to estimate the tr model parameters computationally efficient taylor linearization variance estimators that consider both the intracluster correlation and the differential selection probabilities are developed the proposed methods are evaluated by using simulation experiments with various complex designs and illustrated empirically by using mortality linked third national health and nutrition examination survey phase ii genetic data

a note on the connection and equivalence of three sparse linear discriminant analysis methods in this article we reveal the connection between and equivalence of three sparse linear discriminant analysis methods the l1 fisher s discriminant analysis proposed by wu et al in 2008 the sparse optimal scoring proposed by clemmensen et al in 2011 and the direct sparse discriminant analysis dsda proposed by mai et al in 2012 it is shown that for any sequence of penalization parameters the normalized solutions of dsda equal the normalized solutions of the other two methods at different penalization parameters a prostate cancer dataset is used to demonstrate the theory