boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

a comprehensive evaluation of random vector functional link networks with randomly generated weights between input and hidden layers a random vector functional link network is a universal approximator for continuous functions on compact sets with fast learning property though it was proposed two decades ago the classification ability of this family of networks has not been fully investigated yet through a very comprehensive evaluation by using 121 uci datasets the effect of bias in the output layer direct links from the input layer to the output layer and type of activation functions in the hidden layer scaling of parameter randomization as well as the solution procedure for the output weights are investigated in this work surprisingly we found that the direct link plays an important performance enhancing role in rvfl while the bias term in the output neuron had no significant effect the ridge regression based closed form solution was better than those with moore penrose pseudoinverse instead of using a uniform randomization in 1 1 for all datasets tuning the scaling of the uniform randomization range for each dataset enhances the overall performance six commonly used activation functions were investigated in this work and we found that hardlim and sign activation functions degenerate the overall performance these basic conclusions can serve as general guidelines for designing rvfl networks based classifiers

stabilized nearest neighbor classifier and its statistical properties abstractthe stability of statistical analysis is an important indicator for reproducibility which is one main principle of the scientific method it entails that similar statistical conclusions can be reached based on independent samples from the same underlying population in this article we introduce a general measure of classification instability cis to quantify the sampling variability of the prediction made by a classification method interestingly the asymptotic cis of any weighted nearest neighbor classifier turns out to be proportional to the euclidean norm of its weight vector based on this concise form we propose a stabilized nearest neighbor snn classifier which distinguishes itself from other nearest neighbor classifiers by taking the stability into consideration in theory we prove that snn attains the minimax optimal convergence rate in risk and a sharp convergence rate in cis the latter rate result is established for general plug in classifiers under a low noise condition ext

a comparison of several vector quantization codebook generation approaches a review and a performance comparison of several often used vector quantization vq codebook generation algorithms are presented the codebook generation algorithms discussed include the linde buzo gray lbg binary splitting algorithm the pairwise nearest neighbor algorithm the simulated annealing algorithm and the fuzzy c means clustering analysis algorithm a new directed search binary splitting method which reduces the complexity of the lbg algorithm is presented also a new initial codebook selection method which can obtain a good initial codebook is presented by using this initial codebook selection algorithm the overall lbg codebook generation time can be reduced by a factor of 1 5 2

building forests of local trees a novel approach in the field of classifier ensembles is proposed the approach uses an ensemble of random decision trees each decision tree is trained on a different area of the input space areas can overlap and a good coverage of the input space is ensured experimental results confirm the validity of the approach ensemble methods have shown to be more effective than monolithic classifiers in particular when diversity holds among their components how to enforce diversity in classifier ensembles has received much attention from machine learning researchers yielding a variety of different techniques and algorithms in this paper a novel algorithm for ensemble classifiers is proposed in which ensemble components are trained with focus on different regions of the sample space in so doing diversity is mainly a consequence of the intention to limit the scope of base classifiers the algorithm proposed in this paper shares roots with several ensemble paradigms in particular with random forests as it generates forests of decision trees as well as decision trees are trained with focus on specific subsets of the sample space the resulting ensemble is in fact a forest of local trees comparative experimental results highlight that on average these ensembles perform better than other relevant kinds of ensemble classifiers including random forests

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

bagging for linear classifiers abstract classifiers built on small training sets are usually biased or unstable different techniques exist to construct more stable classifiers it is not clear which ones are good and whether they really stabilize the classifier or just improve the performance in this paper bagging bootstrapping and aggregating l breiman bagging predictors machine learning j 24 2 123 140 1996 is studied for a number of linear classifiers a measure for the instability of classifiers is introduced the influence of regularization and bagging on this instability and the generalization error of linear classifiers is investigated in a simulation study it is shown that in general bagging is not a stabilizing technique it is also demonstrated that one can consider the instability of the classifier to predict how useful bagging will be finally it is shown experimentally that bagging might improve the performance of the classifier only for very unstable situations

classification by evolutionary ensembles this paper is about building an ensemble of classifiers each of which is trained based on a particular weighting over the training examples a weighting is a set of weights associated with the examples the task concerns search in a tremendous weighting space in this view we propose to incorporate a genetic algorithm ga it performs a wide yet efficient search for appropriate weightings chromosomes the difference from a traditional ga is that all the weightings throughout evolution will be exploited to form the final ensemble not just the best weighting our algorithm is tested on the uci benchmark data sets and used to design a face detection system robust and consistently accurate classification is experienced comparative results with two other algorithms i e adaboost and bagging are also given

least squares quantization in pcm it has long been realized that in pulse code modulation pcm with a given ensemble of signals to handle the quantum values should be spaced more closely in the voltage regions where the signal amplitude is more likely to fall it has been shown by panter and dite that in the limit as the number of quanta becomes infinite the asymptotic fractional density of quanta per unit voltage should vary as the one third power of the probability density per unit voltage of signal amplitudes in this paper the corresponding result for any finite number of quanta is derived that is necessary conditions are found that the quanta and associated quantization intervals of an optimum finite quantization scheme must satisfy the optimization criterion used is that the average quantization noise power be a minimum it is shown that the result obtained here goes over into the panter and dite result as the number of quanta become large the optimum quautization schemes for 2 b quanta b 1 2 cdots 7 are given numerically for gaussian and for laplacian distribution of signal amplitudes

improvement of the k means clustering filtering algorithm in this paper we present a modified filtering algorithm mfa by making use of center variations to speed up clustering process our method first divides clusters into static and active groups we use the information of cluster displacements to reject unlikely cluster centers for all nodes in the kd tree we reduce the computational complexity of filtering algorithm fa through finding candidates for each node mainly from the set of active cluster centers two conditions for determining the set of candidate cluster centers for each node from active clusters are developed our approach is different from the major available algorithm which passes no information from one stage of iteration to the next theoretical analysis shows that our method can reduce the computational complexity in terms of the number of distance calculations of fa at each stage of iteration by a factor of fc ac where fc and ac are the numbers of total clusters and active clusters respectively compared with the fa our algorithm can effectively reduce the computing time and number of distance calculations it is noted that our proposed algorithm can generate the same clusters as that produced by hard k means clustering the superiority of our method is more remarkable when a larger data set with higher dimension is used

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

analysis of boosting algorithms using the smooth margin function we introduce a useful tool for analyzing boosting algorithms called the smooth margin function a differentiable approximation of the usual margin for boosting algorithms we present two boosting algorithms based on this smooth margin coordinate ascent boosting and approximate coordinate ascent boosting which are similar to freund and schapire s adaboost algorithm and breiman s arc gv algorithm we give convergence rates to the maximum margin solution for both of our algorithms and for arc gv we then study adaboost s convergence properties using the smooth margin function we precisely bound the margin attained by adaboost when the edges of the weak classifiers fall within a specified range this shows that a previous bound proved by ratsch and warmuth is exactly tight furthermore we use the smooth margin to capture explicit properties of adaboost in cases where cyclic behavior occurs

on visualization and aggregation of nearest neighbor classifiers nearest neighbor classification is one of the simplest and most popular methods for statistical pattern recognition a major issue in k nearest neighbor classification is how to find an optimal value of the neighborhood parameter k in practice this value is generally estimated by the method of cross validation however the ideal value of k in a classification problem not only depends on the entire data set but also on the specific observation to be classified instead of using any single value of k this paper studies results for a finite sequence of classifiers indexed by k along with the usual posterior probability estimates a new measure called the bayesian measure of strength is proposed and investigated in this paper as a measure of evidence for different classes the results of these classifiers and their corresponding estimated misclassification probabilities are visually displayed using shaded strips these plots provide an effective visualization of the evidence in favor of different classes when a given data point is to be classified we also propose a simple weighted averaging technique that aggregates the results of different nearest neighbor classifiers to arrive at the final decision based on the analysis of several benchmark data sets the proposed method is found to be better than using a single value of k

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

support vector machines with the ramp loss and the hard margin loss in the interest of deriving classifiers that are robust to outlier observations we present integer programming formulations of vapnik s support vector machine svm with the ramp loss and hard margin loss the ramp loss allows a maximum error of 2 for each training observation while the hard margin loss calculates error by counting the number of training observations that are in the margin or misclassified outside of the margin svm with these loss functions is shown to be a consistent estimator when used with certain kernel functions in computational studies with simulated and real world data svm with the robust loss functions ignores outlier observations effectively providing an advantage over svm with the traditional hinge loss when using the linear kernel despite the fact that training svm with the robust loss functions requires the solution of a quadratic mixed integer program qmip and is np hard while traditional svm requires only the solution of a continuous quadratic program qp we are able to find good solutions and prove optimality for instances with up to 500 observations solution methods are presented for the new formulations that improve computational performance over industry standard integer programming solvers alone

ensemble pruning using spectral coefficients ensemble pruning aims to increase efficiency by reducing the number of base classifiers without sacrificing and preferably enhancing performance in this brief a novel pruning paradigm is proposed two class supervised learning problems are pruned using a combination of first and second order walsh coefficients a comparison is made with other ordered aggregation pruning methods using multilayer perceptron base classifiers the walsh pruning method is analyzed with the help of a model that shows the relationship between second order coefficients and added classification error with respect to bayes error

a fast vq codebook generation algorithm using codeword displacement in this paper we present a fast codebook generation algorithm called cgaucd codebook generation algorithm using codeword displacement by making use of the codeword displacement between successive partition processes by implementing a fast search algorithm named mfaupi modified fast algorithm using projection and inequality for vq encoding in the partition step of cgaucd the codebook generation time can be further reduced significantly using mfaupi the computing time of cgaucd can be reduced by a factor of 4 7 7 6 compared to generalized lloyd algorithm gla our proposed method can reduce the codebook generation time by a factor of 35 9 121 2 compared to the best codebook generation algorithm to our knowledge our approach can further reduce the corresponding computing time by 26 0 32 8 it is noted that our proposed algorithm can generate the same codebook as that produced by the gla the superiority of our method is more remarkable when a larger codebook is generated

classification trees with unbiased multiway splits two univariate split methods and one linear combination split method are proposed for the construction of classification trees with multiway splits examples are given where the trees are more compact and hence easier to interpret than binary trees a major strength of the univariate split methods is that they have negligible bias in variable selection both when the variables differ in the number of splits they offer and when they differ in the number of missing values this is an advantage because inferences from the tree structures can be adversely affected by selection bias the new methods are shown to be highly competitive in terms of computational speed and classification accuracy of future observations

uncorrelated multilinear discriminant analysis with regularization and aggregation for tensor object recognition this paper proposes an uncorrelated multilinear discriminant analysis umlda framework for the recognition of multidimensional objects known as tensor objects uncorrelated features are desirable in recognition tasks since they contain minimum redundancy and ensure independence of features the umlda aims to extract uncorrelated discriminative features directly from tensorial data through solving a tensor to vector projection the solution consists of sequential iterative processes based on the alternating projection method and an adaptive regularization procedure is incorporated to enhance the performance in the small sample size sss scenario a simple nearest neighbor classifier is employed for classification furthermore exploiting the complementary information from differently initialized and regularized umlda recognizers an aggregation scheme is adopted to combine them at the matching score level resulting in enhanced generalization performance while alleviating the regularization parameter selection problem the umlda based recognition algorithm is then empirically shown on face and gait recognition tasks to outperform four multilinear subspace solutions mpca dater gtda tr1da and four linear subspace solutions bayesian lda ulda r jd lda

consistency of data driven histogram methods for density estimation and classification we present general sufficient conditions for the almost surel1 consistency of histogram density estimates based on data dependent partitions analogous conditions guarantee the almost sure risk consistency of histogram classification schemes based on data dependent partitions multivariate data is considered throughout in each case the desired consistency requires shrinking cells subexponential growth of a combinatorial complexity measure and sub linear growth of the number of cells it is not required that the cells of every partition be rectangles with sides paralles to the coordinate axis or that each cell contain a minimum number of points no assumptions are made concerning the common distribution of the training vectors we apply the results to establish the consistency of several known partitioning estimates including the kn spacing density estimate classifiers based on statistically equivalent blocks and classifiers based on multivariate clustering schemes

robust visual tracking via co trained kernelized correlation filters abstract recent advances in visual tracking have witnessed the importance of discriminative classifiers tasked with distinguishing the target from the background however a single classifier may fail to cope with complex surrounding environment and large appearance variations of the target motivated by multi view learning we equip a basic framework to train a pool of discriminative classifiers jointly in a closed form fashion in this paper it poses an extra regularization term in ridge regression which interacts with other base models in the ensemble through a simple realization of this approach we show co trained kernelized correlation filters cokcf which consist of two kcf trackers are able to outperform the kcf tracker by a larger margin and perform favorably against other state of the art trackers on 63 benchmark video sequences

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

any discrimination rule can have an arbitrarily bad probability of error for finite sample size consider the basic discrimination problem based on a sample of size n drawn from the distribution of x y on the borel sets of rdx 0 1 if 0 thus any attempt to find a nontrivial distribution free upper bound for rn will fail and any results on the rate of convergence of rn to r must use assumptions about the distribution of x y

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

constructing l2 svm based fuzzy classifiers in high dimensional space with automatic model selection and fuzzy rule ranking in this paper a new scheme for constructing parsimonious fuzzy classifiers is proposed based on the l2 support vector machine l2 svm technique with model selection and feature ranking performed simultaneously in an integrated manner in which fuzzy rules are optimally generated from data by l2 svm learning in order to identify the most influential fuzzy rules induced from the svm learning two novel indexes for fuzzy rule ranking are proposed and named as alpha values and omega values of fuzzy rules in this paper the alpha values are defined as the lagrangian multipliers of the l2 svm and adopted to evaluate the output contribution of fuzzy rules while the omega values are developed by considering both the rule base structure and the output contribution of fuzzy rules as a prototype based classifier the l2 svm based fuzzy classifier evades the curse of dimensionality in high dimensional space in the sense that the number of support vectors which equals the number of induced fuzzy rules is not related to the dimensionality experimental results on high dimensional benchmark problems have shown that by using the proposed scheme the most influential fuzzy rules can be effectively induced and selected and at the same time feature ranking results can also be obtained to construct parsimonious fuzzy classifiers with better generalization performance than the well known algorithms in literature

optimal weighted nearest neighbour classifiers we derive an asymptotic expansion for the excess risk regret of a weighted nearest neighbour classifier this allows us to find the asymptotically optimal vector of nonnegative weights which has a rather simple form we show that the ratio of the regret of this classifier to that of an unweighted k nearest neighbour classifier depends asymptotically only on the dimension d of the feature vectors and not on the underlying populations the improvement is greatest when d 4 but thereafter decreases as d rightarrow infty the popular bagged nearest neighbour classifier can also be regarded as a weighted nearest neighbour classifier and we show that its corresponding weights are somewhat suboptimal when d is small in particular worse than those of the unweighted k nearest neighbour classifier when d 1 but are close to optimal when d is large finally we argue that improvements in the rate of convergence are possible under stronger smoothness assumptions provided we allow negative weights our findings are supported by an empirical performance comparison on both simulated and real data sets

improvements on cross validation the 632 bootstrap method abstract a training set of data has been used to construct a rule for predicting future responses what is the error rate of this rule this is an important question both for comparing models and for assessing a final selected model the traditional answer to this question is given by cross validation the cross validation estimate of prediction error is nearly unbiased but can be highly variable here we discuss bootstrap estimates of prediction error which can be thought of as smoothed versions of cross validation we show that a particular bootstrap method the 632 rule substantially outperforms cross validation in a catalog of 24 simulation experiments besides providing point estimates we also consider estimating the variability of an error rate estimate all of the results here are nonparametric and apply to any possible prediction rule however we study only classification problems with 0 1 loss in detail our simulations include smooth prediction rules like fisher s linear discriminant fun

variable selection with error control another look at stability selection summary stability selection was recently introduced by meinshausen and buhlmann as a very general technique designed to improve the performance of a variable selection algorithm it is based on aggregating the results of applying a selection procedure to subsamples of the data we introduce a variant called complementary pairs stability selection and derive bounds both on the expected number of variables included by complementary pairs stability selection that have low selection probability under the original procedure and on the expected number of high selection probability variables that are excluded these results require no e g exchangeability assumptions on the underlying model or on the quality of the original selection procedure under reasonable shape restrictions the bounds can be further tightened yielding improved error control and therefore increasing the applicability of the methodology

minimax nonparametric classification i rates of convergence this paper studies minimax aspects of nonparametric classification we first study minimax estimation of the conditional probability of a class label given the feature variable this function say f is assumed to be in a general nonparametric class we show the minimax rate of convergence under square l sub 2 loss is determined by the massiveness of the class as measured by metric entropy the second part of the paper studies minimax classification the loss of interest is the difference between the probability of misclassification of a classifier and that of the bayes decision as is well known an upper bound on risk for estimating f gives an upper bound on the risk for classification but the rate is known to be suboptimal for the class of monotone functions this suggests that one does not have to estimate f well in order to classify well however we show that the two problems are in fact of the same difficulty in terms of rates of convergence under a sufficient condition which is satisfied by many function classes including besov sobolev lipschitz and bounded variation this is somewhat surprising in view of a result of devroye gorfi and lugosi see a probabilistic theory of pattern recognition new york springer verlag 1996

on the rate of convergence of the bagged nearest neighbor estimate bagging is a simple way to combine estimates in order to improve their performance this method suggested by breiman in 1996 proceeds by resampling from the original data set constructing a predictor from each subsample and decide by combining by bagging an n sample the crude nearest neighbor regression estimate is turned into a consistent weighted nearest neighbor regression estimate which is amenable to statistical analysis letting the resampling size kn grows appropriately with n it is shown that this estimate may achieve optimal rate of convergence independently from the fact that resampling is done with or without replacement since the estimate with the optimal rate of convergence depends on the unknown distribution of the observations adaptation results by data splitting are presented

the theorem of grauert mulich spindler the boundedness of semi stable sheaves on p is a simple corollary to the above see 13 if one looks into the proof of this theorem by spindler and the techniques by van de ven 14 and barth 1 he can generalize immediately the theorem in the following way let e be a torsion free coherent sheaf on a non singular projective variety x over an algebraically closed field k and ox 1 a very ample invertible sheaf on x if g l y n dimx are sufficiently general member of a very ample linear subsystem l of iox 1 l then y yl i1 is a non singular subvariety of codimension n in x and elf is torsion free

a fast k means clustering algorithm using cluster center displacement in this paper we present a fast k means clustering algorithm fkmcucd using the displacements of cluster centers to reject unlikely candidates for a data point the computing time of our proposed algorithm increases linearly with the data dimension d whereas the computational complexity of major available kd tree based algorithms increases exponentially with the value of d theoretical analysis shows that our method can reduce the computational complexity of full search by a factor of sf and sf is independent of vector dimension the experimental results show that compared to full search our proposed method can reduce computational complexity by a factor of 1 37 4 39 using the data set from six real images compared with the filtering algorithm which is among the available best algorithms of k means clustering our algorithm can effectively reduce the computing time it is noted that our proposed algorithm can generate the same clusters as that produced by hard k means clustering the superiority of our method is more remarkable when a larger data set with higher dimension is used

an efficient modified boosting method for solving classification problems based on the adaboost algorithm a modified boosting method is proposed in this paper for solving classification problems this method predicts the class label of an example as the weighted majority voting of an ensemble of classifiers each classifier is obtained by applying a given weak learner to a subsample with size smaller than that of the original training set which is drawn from the original training set according to the probability distribution maintained over the training set a parameter is introduced into the reweighted scheme proposed in adaboost to update the probabilities assigned to training examples so that the algorithm can be more accurate than adaboost the experimental results on synthetic and several real world data sets available from the uci repository show that the proposed method improves the prediction accuracy the execution speed as well as the robustness to classification noise of adaboost furthermore the diversity accuracy patterns of the ensemble classifiers are investigated by kappa error diagrams

how large should ensembles of classifiers be we propose to determine the size of a parallel ensemble by estimating the minimum number of classifiers that are required to obtain stable aggregate predictions assuming that majority voting is used a statistical description of the convergence of the ensemble prediction to its asymptotic infinite size limit is given the analysis of the voting process shows that for most test instances the ensemble prediction stabilizes after only a few classifiers are polled by contrast a small but non negligible fraction of these instances require large numbers of classifier queries to reach stable predictions specifically the fraction of instances whose stable predictions require more than t classifiers for email protected 1 has a universal form and is proportional to t 1 2 the ensemble size is determined as the minimum number of classifiers that are needed to estimate the infinite ensemble prediction at an average confidence level a close to one this approach differs from previous proposals which are based on determining the size for which the prediction error not the predictions themselves stabilizes in particular it does not require estimates of the generalization performance of the ensemble which can be unreliable it has general validity because it is based solely on the statistical description of the convergence of majority voting to its asymptotic limit extensive experiments using representative parallel ensembles bagging and random forest illustrate the application of the proposed framework in a wide range of classification problems these experiments show that the optimal ensemble size is very sensitive to the particular classification problem considered

reducing variability of crossvalidation for smoothing parameter choice one of the attractions of crossvalidation as a tool for smoothing parameter choice is its applicability to a wide variety of estimator types and contexts however its detractors comment adversely on the relatively high variance of crossvalidatory smoothing parameters noting that this compromises the performance of the estimators in which those parameters are used we show that the variability can be reduced simply significantly and reliably by employing bootstrap aggregation or bagging we establish that in theory when bagging is implemented using an adaptively chosen resample size the variability of crossvalidation can be reduced by an order of magnitude however it is arguably more attractive to use a simpler approach based for example on half sample bagging which can reduce variability by approximately 50

hyperbolicity of generic surfaces of high degree in projective 3 space the main goal of this work is to prove that a very generic surface of degree at least 21 in complex projective 3 dimensional space is hyperbolic in the sense of kobayashi this means that every entire holomorphic map f inline graphic xmlns xlink http www w3 org 1999 xlink xlink href 01i x to the surface is constant in 1970 kobayashi conjectured more generally that a very generic hypersurface of sufficiently high degree in projective space is hyperbolic our technique follows the stream of ideas initiated by green and griffiths in 1979 which consists of considering jet differentials and their associated base loci however a key ingredient is the use of a different kind of jet bundle namely the semple jet bundles previously studied by the first named author the base locus calculation is achieved through a sequence of riemann roch formulas combined with a suitable generic vanishing theorem for order 2 jets our method covers the case of surfaces of general type with picard group inline graphic xmlns xlink http www w3 org 1999 xlink xlink href 02i and 13 12 2 c 2 1 9 c 2 0 where 2 is the 2 jet threshold bounded below by 1 6 for surfaces in inline graphic xmlns xlink http www w3 org 1999 xlink xlink href 03i 3 the final conclusion is obtained by using recent results of mcquillan on holomorphic foliations

on weak base hypotheses and their implications for boosting regression and classification when studying the training error and the prediction error for boosting it is often assumed that the hypotheses returned by the base learner are weakly accurate or are able to beat a random guesser by a certain amount of difference it has been an open question how much this difference can be whether it will eventually disappear in the boosting process or be bounded by a positive amount this question is crucial for the behavior of both the training error and the prediction error in this paper we study this problem and show affirmatively that the amount of improvement over the random guesser will be at least a positive amount for almost all possible sample realizations and for most of the commonly used base hypotheses this has a number of implications for the prediction error including for example that boosting forever may not be good and regularization may be necessary the problem is studied by first considering an analog of adaboost in regression where we study similar properties and find that for good performance one cannot hope to avoid regularization by just adopting the boosting device to regression

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

a robust minimax approach to classification when constructing a classifier the probability of correct classification of future data points should be maximized we consider a binary classification problem where the mean and covariance matrix of each class are assumed to be known no further assumptions are made with respect to the class conditional distributions misclassification probabilities are then controlled in a worst case setting that is under all possible choices of class conditional densities with given mean and covariance matrix we minimize the worst case maximum probability of misclassification of future data points for a linear decision boundary this desideratum is translated in a very direct way into a convex second order cone optimization problem with complexity similar to a support vector machine problem the minimax problem can be interpreted geometrically as minimizing the maximum of the mahalanobis distances to the two classes we address the issue of robustness with respect to estimation errors in the means and covariances of the classes via a simple modification of the input data we also show how to exploit mercer kernels in this setting to obtain nonlinear decision boundaries yielding a classifier which proves to be competitive with current methods including support vector machines an important feature of this method is that a worst case bound on the probability of misclassification of future data is always obtained explicitly

vote counting measures for ensemble classifiers various measures such as margin and bias variance have been proposed with the aim of gaining a better understanding of why multiple classifier systems mcs perform as well as they do while these measures provide different perspectives for mcs analysis it is not clear how to use them for mcs design in this paper a different measure based on a spectral representation is proposed for two class problems it incorporates terms representing positive and negative correlation of pairs of training patterns with respect to class labels experiments employing mlp base classifiers in which parameters are fixed but systematically varied demonstrate the sensitivity of the proposed measure to base classifier complexity

arcing classifier with discussion and a rejoinder by the author recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error one of the more effective is bagging here modified training sets are formed by resampling from the original training set classifiers constructed using these training sets and then combined by voting freund and schapire propose an algorithm the basis of which is to adaptively resample and combine hence the acronym arcing so that the weights in the resampling are increased for those cases most often misclassified and the combining is done by weighted voting arcing is more successful than bagging in test set error reduction we explore two arcing algorithms compare them to each other and to bagging and try to understand how arcing works we introduce the definitions of bias and variance for a classifier as components of the test set error unstable classifiers can have low bias on a large range of data sets their problem is high variance combining multiple versions either through bagging or arcing reduces variance significantly

vector fields on the total space of hypersurfaces in the projective space and hyperbolicity the results we obtain in this article concern the hyperbolicity of very generic hypersurfaces in the 3 dimensional projective space we show that the kobayashi conjecture is true in this setting as long as the degree of the hypersurface is greater than 18

sequential combination of weighted and nonparametric bagging for classification we propose a simple sequential procedure for bagged classification which modifies nonparametric bagging by randomizing class labels of resampled data points the random labelling feature of the procedure also enables us to undertake unsupervised classification with the benefit of supervised learning theoretical properties are given for the nearest neighbour classifier in the case of supervised learning and a hard thresholding indicator in the case of unsupervised learning showing that sequential bagging accelerates convergence of the bagged predictor to the bayes rule simulation results are provided in support of the proposed method

properties of bagged nearest neighbour classifiers summary it is shown that bagging a computationally intensive method asymptotically improves the performance of nearest neighbour classifiers provided that the resample size is less than 69 of the actual sample size in the case of with replacement bagging or less than 50 of the sample size for without replacement bagging however for larger sampling fractions there is no asymptotic difference between the risk of the regular nearest neighbour classifier and its bagged version in particular neither achieves the large sample performance of the bayes classifier in contrast when the sampling fractions converge to 0 but the resample sizes diverge to 1 the bagged classifier converges to the optimal bayes rule and its risk converges to the risk of the latter these results are most readily seen when the two populations have well defined densities but they may also be derived in other cases where densities exist in only a relative sense cross validation can be used effectively to choose the sampling fraction numerical calculation is used to illustrate these theoretical properties

estimating the error rate of a prediction rule improvement on cross validation abstract we construct a prediction rule on the basis of some data and then wish to estimate the error rate of this rule in classifying future observations cross validation provides a nearly unbiased estimate using only the original data cross validation turns out to be related closely to the bootstrap estimate of the error rate this article has two purposes to understand better the theoretical basis of the prediction problem and to investigate some related estimators which seem to offer considerably improved estimation in small samples

on the structure of vector quantizers vector quantization is intrinsically superior to predictive coding transform coding and other suboptimal and em ad hoc procedures since it achieves optimal rate distortion performance subject only to a constraint on memory or block length of the observable signal segment being encoded the key limitation of existing techniques is the very large randomly generated code books which must be stored and the computational complexity of the associated encoding procedures the quantization operation is decomposed into its rudimentary structural components this leads to a simple and elegant approach to derive analytical properties of optimal quantizers some useful properties of quantizers and algorithmic approaches are given which are relevant to the complexity of both storage and processing in the encoding operation highly disordered quantizers which have been designed using a clustering algorithm are considered finally lattice quantizers are examined which circumvent the need for a code book by using a highly structured code based on lattices the code vectors are algorithmically generated in a simple manner rather than stored in a code book and fast algorithms perform the encoding algorithm with negligible complexity

boosting for high dimensional linear models we prove that boosting with the squared error loss l 2 boosting is consistent for very high dimensional linear models where the number of predictor variables is allowed to grow essentially as fast as o exp sample size assuming that the true underlying regression function is sparse in terms of the 11 norm of the regression coefficients in the language of signal processing this means consistency for de noising using a strongly overcomplete dictionary if the underlying signal is sparse in terms of the l 1 norm we also propose here an aic based method for tuning namely for choosing the number of boosting iterations this makes l 2 boosting computationally attractive since it is not required to run the algorithm multiple times for cross validation as commonly used so far we demonstrate l 2 boosting for simulated data in particular where the predictor dimension is large in comparison to sample size and for a difficult tumor classification problem with gene expression microarray data

classification using kernel density estimates multiscale analysis and visualization the use of kernel density estimates in discriminant analysis is quite well known among scientists and engineers interested in statistical pattern recognition using a kernel density estimate involves properly selecting the scale of smoothing namely the bandwidth parameter the bandwidth that is optimum for the mean integrated square error of a class density estimator may not always be good for discriminant analysis where the main emphasis is on the minimization of misclassification rates on the other hand cross validation based methods for bandwidth selection which try to minimize estimated misclassification rates may require huge computation when there are several competing populations besides such methods usually allow only one bandwidth for each population density estimate whereas in a classification problem the optimum bandwidth for a class density estimate may vary significantly depending on its competing class densities and their prior probabilities therefore in a multiclass problem it

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

on fusers that perform better than best sensor in a multiple sensor system sensor s sub i i 1 2 n outputs y sup i spl isin 0 1 according to an unknown probability distribution p y sup i x in response to input x spl isin 0 1 we choose a fuser that combines the outputs of sensors from a function class f f 0 1 sup n spl rarr 0 1 by minimizing empirical error based on an i i d sample if f satisfies the isolation property we show that the fuser performs at least as well as the best sensor in a probably approximately correct sense several well known fusers such as linear combinations special potential functions and certain feedforward networks satisfy the isolation property

bias of nearest neighbor error estimates the bias of the finite sample nearest neighbor nn error from its asymptotic value is examined expressions are obtained which relate the bias of the nn and 2 nn errors to sample size dimensionality metric and distributions these expressions isolate the effect of sample size from that of the distributions giving an explicit relation showing how the bias changes as the sample size is increased experimental results are given which suggest that the expressions accurately predict the bias it is shown that when the dimensionality of the data is high it may not be possible to estimate the asymptotic error simply by increasing the sample size a new procedure is suggested to alleviate this problem this procedure involves measuring the mean nn errors at several sample sizes and using our derived relationship between the bias and the sample size to extrapolate an estimate of the asymptotic nn error the results are extended to the multiclass problem the choice of an optimal metric to minimize the bias is also discussed

out of bag estimation of the optimal sample size in bagging the performance of m out of n bagging with and without replacement in terms of the sampling ratio m n is analyzed standard bagging uses resampling with replacement to generate bootstrap samples of equal size as the original training set m w o r n without replacement methods typically use half samples m w r n 2 these choices of sampling sizes are arbitrary and need not be optimal in terms of the classification performance of the ensemble we propose to use the out of bag estimates of the generalization accuracy to select a near optimal value for the sampling ratio ensembles of classifiers trained on independent samples whose size is such that the out of bag error of the ensemble is as low as possible generally improve the performance of standard bagging and can be efficiently built

risk bounds for cart classifiers under a margin condition non asymptotic risk bounds for classification and regression trees cart classifiers are obtained in the binary supervised classification framework under a margin assumption on the joint distribution of the covariates and the labels these risk bounds are derived conditionally on the construction of the maximal binary tree and allow to prove that the linear penalty used in the cart pruning algorithm is valid under the margin condition it is also shown that conditionally on the construction of the maximal tree the final selection by test sample does not alter dramatically the estimation accuracy of the bayes classifier

a complete fuzzy decision tree technique in this paper a new method of fuzzy decision trees called soft decision trees sdt is presented this method combines tree growing and pruning to determine the structure of the soft decision tree with refitting and backfitting to improve its generalization capabilities the method is explained and motivated and its behavior is first analyzed empirically on 3 large databases in terms of classification error rate model complexity and cpu time a comparative study on 11 standard uci repository databases then shows that the soft decision trees produced by this method are significantly more accurate than standard decision trees moreover a global model variance study shows a much lower variance for soft decision trees than for standard trees as a direct cause of the improved accuracy

image coding using vector quantization a review a review of vector quantization techniques used for encoding digital images is presented first the concept of vector quantization is introduced then its application to digital images is explained spatial predictive transform hybrid binary and subband vector quantizers are reviewed the emphasis is on the usefulness of the vector quantization when it is combined with conventional image coding techniques or when it is used in different domains

an empirical study of using rotation forest to improve regressors this paper investigates the performance of rotation forest ensemble method in improving the generalization ability of a base predictor for solving regression problems through conducting experiments on several benchmark data sets which is also compared with that of bagging random forest adaboost r2 and a single regression tree the sensitivity of rotation forest to the choice of parameters included in it is also studied on the considered regression data sets adaboost r2 is seen to generally outperform rotation forest and both of them are better than random forest and a single tree with respect to bagging and rotation forest it seems that there is not a clear winner between them furthermore pruning the tree seems to have some bad effect on the performance of all the considered methods

different paradigms for choosing sequential reweighting algorithms analyses of the success of ensemble methods in classification have pointed out the important role played by the margin distribution function on the training and test sets while it is acknowledged that one should generally try to achieve high margins on the training set the more precise shape of the empirical margin distribution function one should favor in practice is subject to different approaches we first present two concurrent philosophies for choosing the empirical margin profile the minimax margin paradigm and the mean and variance paradigm the best known representative of the first paradigm is the adaboost algorithm and this philosophy has been shown by several other authors to be closely related to the principle of the support vector machine we show that the second paradigm is very close in spirit to fisher s linear discriminant in a feature space we construct two boosting type algorithms very similar in their form dedicated to one or the other philosophy we consequently derive by interpolation a very simple family of iterative reweighting algorithms that can be understood as different trade offs between the two paradigms and argue from experiments that this can allow for a suitable adaptivity to different classification problems particularly in the presence of noise or excessive complexity of the base classifiers

double bagging combining classifiers by bootstrap aggregation the combination of classifiers leads to substantial reduction of misclassification error in a wide range of applications and benchmark problems we suggest using an out of bag sample for combining different classifiers in our setup a linear discriminant analysis is performed using the observations in the out of bag sample and the corresponding discriminant variables computed for the observations in the bootstrap sample are used as additional predictors for a classification tree two classifiers are combined and therefore method and variable selection bias is no problem for the corresponding estimate of misclassification error the need of an additional test sample disappears moreover the procedure performs comparable to the best classifiers used in a number of artificial examples and applications

fast sparse approximation for least squares support vector machine in this paper we present two fast sparse approximation schemes for least squares support vector machine ls svm named fsals svm and pfsals svm to overcome the limitation of ls svm that it is not applicable to large data sets and to improve test speed fsals svm iteratively builds the decision function by adding one basis function from a kernel based dictionary at one time the process is terminated by using a flexible and stable epsilon insensitive stopping criterion a probabilistic speedup scheme is employed to further improve the speed of fsals svm and the resulting classifier is named pfsals svm our algorithms are of two compelling features low complexity and sparse solution experiments on benchmark data sets show that our algorithms obtain sparse classifiers at a rather low cost without sacrificing the generalization performance

clustering ensembles models of consensus and weak partitions clustering ensembles have emerged as a powerful method for improving both the robustness as well as the stability of unsupervised classification solutions however finding a consensus clustering from multiple partitions is a difficult problem that can be approached from graph based combinatorial or statistical perspectives this study extends previous research on clustering ensembles in several respects first we introduce a unified representation for multiple clusterings and formulate the corresponding categorical clustering problem second we propose a probabilistic model of consensus using a finite mixture of multinomial distributions in a space of clusterings a combined partition is found as a solution to the corresponding maximum likelihood problem using the em algorithm third we define a new consensus function that is related to the classical intraclass variance criterion using the generalized mutual information definition finally we demonstrate the efficacy of combining partitions generated by weak clustering algorithms that use data projections and random data splits a simple explanatory model is offered for the behavior of combinations of such weak clustering components combination accuracy is analyzed as a function of several parameters that control the power and resolution of component partitions as well as the number of partitions we also analyze clustering ensembles with incomplete information and the effect of missing cluster labels on the quality of overall consensus experimental results demonstrate the effectiveness of the proposed methods on several real world data sets

a kernel based two class classifier for imbalanced data sets many kernel classifier construction algorithms adopt classification accuracy as performance metrics in model evaluation moreover equal weighting is often applied to each data sample in parameter estimation these modeling practices often become problematic if the data sets are imbalanced we present a kernel classifier construction algorithm using orthogonal forward selection ofs in order to optimize the model generalization for imbalanced two class data sets this kernel classifier identification algorithm is based on a new regularized orthogonal weighted least squares rowls estimator and the model selection criterion of maximal leave one out area under curve loo auc of the receiver operating characteristics rocs it is shown that owing to the orthogonalization procedure the loo auc can be calculated via an analytic formula based on the new regularized orthogonal weighted least squares parameter estimator without actually splitting the estimation data set the proposed algorithm can achieve minimal computational expense via a set of forward recursive updating formula in searching model terms with maximal incremental loo auc value numerical examples are used to demonstrate the efficacy of the algorithm

optimal rates of convergence to bayes risk in nonparametric discrimination consider the multiclassification discrimination problem with known prior probabilities and a multi dimensional vector of observations assume the underlying densities corresponding to the various classes are unknown but a training sample of size n is available from each class rates of convergence to bayes risk are investigated under smoothness conditions on the underlying densities of the type often seen in nonparametric density estimation these rates can be drastically affected by a small change in the prior probabilities so the error criterion used here is bayes risk averaged uniformly over all prior probabilities then it is shown that a certain rate n r is optimal in the sense that no rule can do better uniformly over the class of smooth densities and a rule is exhibited which does that well the optimal value of r depends on the smoothness of the distributions and the dimensionality of the observations in the same way as for nonparametric density estimation with integrated square error loss

a new vector quantization clustering algorithm the pairwise nearest neighbor pnn algorithm is presented as an alternative to the linde buzo gray 1980 lbg generalized lloyd 1982 algorithm for vector quantization clustering the pnn algorithm derives a vector quantization codebook in a diminishingly small fraction of the time previously required without sacrificing performance in addition the time needed to generate a codebook grows only o n log n in training set size and is independent of the number of code words desired using this method one can either minimize the number of code words needed subject to a maximum rate the pnn algorithm can be used with squared error and weighted squared error distortion measure simulations on a variety of images encoded at 1 2 b pixel indicate that pnn codebooks can be developed in roughly 5 of the time required by the lbg algorithm

detection and characterization of cluster substructure i linear structure fuzzy c lines in part i a generalization of the fuzzy c means or fuzzy isodata clustering algorithms is developed necessary conditions for minimization of a generalized total weighted squared orthogonal error objective function lead to a picard iteration scheme which generates simultaneously i c fuzzy clusters in the data ii a set of c prototypical straight lines in feature space which best fit the data in a well defined sense iii a set of c prototpyical centers of mass on the c lines which characterize the core of each linear fuzzy cluster theoretical optimization is achieved using principal components of generalized within cluster fuzzy scatter matrices a convergence theorem for each algorithm in the infinite family is given the algorithms are exemplified by five numerical examples using both real and artificial data sets having essentially linear substructure in part ii the fuzzy c means and fuzzy c lines algorithms are shown to be special cases of a more general class of fuzzy algorithms the

model based classification trees the construction of classification trees is nearly always top down locally optimal and data driven such recursive designs are often globally inefficient for instance in terms of the mean depth necessary to reach a given classification rate we consider statistical models for which exact global optimization is feasible and thereby demonstrate that recursive and global procedures may result in very different tree graphs and overall performance

tree based prediction on incomplete data using imputation or surrogate decisions the goal is to investigate the prediction performance of tree based techniques when the available training data contains features with missing values also the future test cases may contain missing values and thus the methods should be able to generate predictions for such test cases the missing values are handled either by using surrogate decisions within the trees or by the combination of an imputation method with a tree based method missing values generated according to missing completely at random mcar missing at random mar and missing not at random mnar mechanisms are considered with various fractions of missing data imputation models are built in the learning phase and do not make use of the response variable so that the resulting procedures allow to predict individual incomplete test cases in the empirical comparison both classification and regression problems are considered using a simulated and real life datasets the performance is evaluated by misclassification rate of predictions and mean squared prediction error respectively overall our results show that for smaller fractions of missing data an ensemble method combined with surrogates or single imputation suffices for moderate to large fractions of missing values ensemble methods based on conditional inference trees combined with multiple imputation show the best performance while conditional bagging using surrogates is a good alternative for high dimensional prediction problems theoretical results confirm the potential better prediction performance of multiple imputation ensembles

some theory for generalized boosting algorithms we give a review of various aspects of boosting clarifying the issues through a few simple results and relate our work and that of others to the minimax paradigm of statistics we consider the population version of the boosting algorithm and prove its convergence to the bayes classifier as a corollary of a general result about gauss southwell optimization in hilbert space we then investigate the algorithmic convergence of the sample version and give bounds to the time until perfect separation of the sample we conclude by some results on the statistical optimality of the l2 boosting

the random subspace method for constructing decision forests much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes the dilemma between overfitting and achieving maximum accuracy is seldom resolved a method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity the classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector that is trees constructed in randomly chosen subspaces the subspace method is compared to single tree classifiers and other forest construction methods by experiments on publicly available datasets where the method s superiority is demonstrated we also discuss independence between trees in a forest and relate that to the combined classification accuracy