boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

ls svr based solving volterra integral equations abstract in this paper a novel hybrid method is presented for solving the second kind linear volterra integral equations due to the powerful regression ability of least squares support vector regression ls svr we approximate the unknown function of integral equations by using ls svr in intervals with known numerical solutions the trapezoid quadrature is used to approximate subsequent integrations in intervals with unknown numerical solutions the feasibility of the proposed method is examined on some integral equations experimental results of comparison with analytic and repeated modified trapezoid quadrature method s solutions show that the proposed algorithm could reach a very high accuracy the proposed algorithm could be a good tool for solving the second kind linear volterra integral equations

building forests of local trees a novel approach in the field of classifier ensembles is proposed the approach uses an ensemble of random decision trees each decision tree is trained on a different area of the input space areas can overlap and a good coverage of the input space is ensured experimental results confirm the validity of the approach ensemble methods have shown to be more effective than monolithic classifiers in particular when diversity holds among their components how to enforce diversity in classifier ensembles has received much attention from machine learning researchers yielding a variety of different techniques and algorithms in this paper a novel algorithm for ensemble classifiers is proposed in which ensemble components are trained with focus on different regions of the sample space in so doing diversity is mainly a consequence of the intention to limit the scope of base classifiers the algorithm proposed in this paper shares roots with several ensemble paradigms in particular with random forests as it generates forests of decision trees as well as decision trees are trained with focus on specific subsets of the sample space the resulting ensemble is in fact a forest of local trees comparative experimental results highlight that on average these ensembles perform better than other relevant kinds of ensemble classifiers including random forests

posterior probability support vector machines for unbalanced data this paper proposes a complete framework of posterior probability support vector machines ppsvms for weighted training samples using modified concepts of risks linear separability margin and optimal hyperplane within this framework a new optimization problem for unbalanced classification problems is formulated and a new concept of support vectors established furthermore a soft ppsvm with an interpretable parameter spl nu is obtained which is similar to the spl nu svm developed by scho spl uml lkopf et al and an empirical method for determining the posterior probability is proposed as a new approach to determine spl nu the main advantage of an ppsvm classifier lies in that fact that it is closer to the bayes optimal without knowing the distributions to validate the proposed method two synthetic classification examples are used to illustrate the logical correctness of ppsvms and their relationship to regular svms and bayesian methods several other classification experiments are conducted to demonstrate that the performance of ppsvms is better than regular svms in some cases compared with fuzzy support vector machines fsvms the proposed ppsvm is a natural and an analytical extension of regular svms based on the statistical learning theory

stopping criterion for boosting based data reduction techniques from binary to multiclass problem so far boosting has been used to improve the quality of moderately accurate learning algorithms by weighting and combining many of their weak hypotheses into a final classifier with theoretically high accuracy in a recent work sebban nock and lallich 2001 we have attempted to adapt boosting properties to data reduction techniques in this particular context the objective was not only to improve the success rate but also to reduce the time and space complexities due to the storage requirements of some costly learning algorithms such as nearest neighbor classifiers in that framework each weak hypothesis which is usually built and weighted from the learning set is replaced by a single learning instance the weight given by boosting defines in that case the relevance of the instance and a statistical test allows one to decide whether it can be discarded without damaging further classification tasks in sebban nock and lallich 2001 we addressed problems with two classes it is the aim of the present paper to relax the class constraint and extend our contribution to multiclass problems beyond data reduction experimental results are also provided on twenty three datasets showing the benefits that our boosting derived weighting rule brings to weighted nearest neighbor classifiers

random subspace based ensemble sparse representation a new random subspace based joint sparse representation model is proposed the sparse representations in multiple subspaces are integrated into an ensemble sparse representation the spectral clustering algorithm with random subspace based ensemble sparse representation is shown the semi supervised classification method with random subspace based ensemble sparse representation is described in this paper a new random subspace based ensemble sparse representation rs esr algorithm is proposed where the random subspace is introduced into sparse representation model for high dimensional data the random subspace method can not only reduce dimension of data but also make full use of effective information of data it is not like traditional dimensionality reduction methods that may lose some information of original data additionally a joint sparse representation model is emloyed to obtain the sparse representation of a sample set in the low dimensional random subspace then the sparse representations in multiple random subspaces are integrated as an ensemble sparse representation moreover the obtained rs esr is applied in classical clustering and semi supervised classification the experimental results on different real world data sets show the superiority of rs esr over traditional methods

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

learn mf a random subspace approach for the missing feature problem we introduce learn mf an ensemble of classifiers based algorithm that employs random subspace selection to address the missing feature problem in supervised classification unlike most established approaches learn mf does not replace missing values with estimated ones and hence does not need specific assumptions on the underlying data distribution instead it trains an ensemble of classifiers each on a random subset of the available features instances with missing values are classified by the majority voting of those classifiers whose training data did not include the missing features we show that learn mf can accommodate substantial amount of missing data and with only gradual decline in performance as the amount of missing data increases we also analyze the effect of the cardinality of the random feature subsets and the ensemble size on algorithm performance finally we discuss the conditions under which the proposed approach is most effective

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

numerical solution of volterra integral equations of the second kind with convolution kernel by using taylor series expansion method in this paper we present a taylor series expansion method for a class of volterra integral equations of second kind with smooth or weakly singular kernels this method use taylor series approximation method for integral equation and transform the integral equation to an nth order linear differential equation boundary conditions for differential equation produce in easy way this method gives an approximate simple and closed form solution for integral equation some numerical example use to illustrate the accuracy of method

causal inference by using invariant prediction identification and confidence intervals summary what is the difference between a prediction that is made with a causal model and that with a non causal model suppose that we intervene on the predictor variables or change the whole environment the predictions from a causal model will in general work as well under interventions as for observational data in contrast predictions from a non causal model can potentially be very wrong if we actively intervene on variables here we propose to exploit this invariance of a prediction under a causal model for causal inference given different experimental settings e g various interventions we collect all models that do show invariance in their predictive accuracy across settings and interventions the causal model will be a member of this set of models with high probability this approach yields valid confidence intervals for the causal relationships in quite general scenarios we examine the example of structural equation models in more detail and provide sufficient assumptions under which the set of causal predictors becomes identifiable we further investigate robustness properties of our approach under model misspecification and discuss possible extensions the empirical properties are studied for various data sets including large scale gene perturbation experiments

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

high performance rotation invariant multiview face detection rotation invariant multiview face detection mvfd aims to detect faces with arbitrary rotation in plane rip and rotation off plane rop angles in still images or video sequences mvfd is crucial as the first step in automatic face processing for general applications since face images are seldom upright and frontal unless they are taken cooperatively in this paper we propose a series of innovative methods to construct a high performance rotation invariant multiview face detector including the width first search wfs tree detector structure the vector boosting algorithm for learning vector output strong classifiers the domain partition based weak learning method the sparse feature in granular space and the heuristic search for sparse feature selection as a result of that our multiview face detector achieves low computational complexity broad detection scope and high detection accuracy on both standard testing sets and real life images

a study on relationship between generalization abilities and fuzziness of base classifiers in ensemble learning we investigate essential relationships between generalization capabilities and fuzziness of fuzzy classifiers viz the classifiers whose outputs are vectors of membership grades of a pattern to the individual classes the study makes a claim and offers sound evidence behind the observation that higher fuzziness of a fuzzy classifier may imply better generalization aspects of the classifier especially for classification data exhibiting complex boundaries this observation is not intuitive with a commonly accepted position in traditional pattern recognition the relationship that obeys the conditional maximum entropy principle is experimentally confirmed furthermore the relationship can be explained by the fact that samples located close to classification boundaries are more difficult to be correctly classified than the samples positioned far from the boundaries this relationship is expected to provide some guidelines as to the improvement of generalization aspects of fuzzy classifiers

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

two dimensional polar harmonic transforms for invariant image representation this paper introduces a set of 2d transforms based on a set of orthogonal projection bases to generate a set of features which are invariant to rotation we call these transforms polar harmonic transforms phts unlike the well known zernike and pseudo zernike moments the kernel computation of phts is extremely simple and has no numerical stability issue whatsoever this implies that phts encompass the orthogonality and invariance advantages of zernike and pseudo zernike moments but are free from their inherent limitations this also means that phts are well suited for application where maximal discriminant information is needed furthermore phts make available a large set of features for further feature selection in the process of seeking for the best discriminative or representative features for a particular application

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

on the doubt about margin explanation of boosting margin theory provides one of the most popular explanations to the success of adaboost where the central point lies in the recognition that margin is the key for characterizing the performance of adaboost this theory has been very influential e g it has been used to argue that adaboost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero previously the minimum margin bound was established for adaboost however breiman 1999 9 pointed out that maximizing the minimum margin does not necessarily lead to a better generalization later reyzin and schapire 2006 37 emphasized that the margin distribution rather than minimum margin is crucial to the performance of adaboost in this paper we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and emargin bound then we improve the previous empirical bernstein bounds audibert et al 2009 maurer and pontil 2009 2 30 and based on such findings we defend the margin based explanation against breiman s doubts by proving a new generalization error bound that considers exactly the same factors as schapire et al 1998 39 but is sharper than breiman s 1999 9 minimum margin bound by incorporating factors such as average margin and variance we present a generalization error bound that is heavily related to the whole margin distribution we also provide margin distribution bounds for generalization error of voting classifiers in finite vc dimension space

on the perturbations of spectra of upper triangular operator matrices abstract in this paper we investigate perturbations of the regular spectrum of an upper triangular operator matrix such as m c a c 0 b acting on a hilbert space h k

bandwidth choice for nonparametric classification it is shown that for kernel based classification with univariate distributions and two populations optimal bandwidth choice has a dichotomous character if the two densities cross at just one point where their curvatures have the same signs then minimum bayes risk is achieved using bandwidths which are an order of magnitude larger than those which minimize pointwise estimation error on the other hand if the curvature signs are different or if there are multiple crossing points then bandwidths of conventional size are generally appropriate the range of different modes of behavior is narrower in multivariate settings there the optimal size of bandwidth is generally the same as that which is appropriate for pointwise density estimation these properties motivate empirical rules for bandwidth choice

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

attribute bagging improving accuracy of classifier ensembles by using random feature subsets we present attribute bagging ab a technique for improving the accuracy and stability of classifier ensembles induced using random subsets of features ab is a wrapper method that can be used with any learning algorithm it establishes an appropriate attribute subset size and then randomly selects subsets of features creating projections of the training set on which the ensemble classifiers are built the induced classifiers are then used for voting this article compares the performance of our ab method with bagging and other algorithms on a hand pose recognition dataset it is shown that ab gives consistently better results than bagging both in accuracy and stability the performance of ensemble voting in bagging and the ab method as a function of the attribute subset size and the number of voters for both weighted and unweighted voting is tested and discussed we also demonstrate that ranking the attribute subsets by their classification accuracy and voting using only the best subsets further improves the resulting performance of the ensemble

theory and algorithm for learning with dissimilarity functions we study the problem of classification when only a dissimilarity function between objects is accessible that is data samples are represented not by feature vectors but in terms of their pairwise dissimilarities we establish sufficient conditions for dissimilarity functions to allow building accurate classifiers the theory immediately suggests a learning paradigm construct an ensemble of simple classifiers each depending on a pair of examples then find a convex combination of them to achieve a large margin we next develop a practical algorithm referred to as dissimilarity based boosting dboost for learning with dissimilarity functions under theoretical guidance experiments on a variety of databases demonstrate that the dboost algorithm is promising for several dissimilarity measures widely used in practice

continuous spectrum point spectrum and residual spectrum of operator matrices abstract let m c denote a 2 2 upper triangular operator matrix of the form m c a c 0 b which is acting on the sum of banach spaces x y or hilbert spaces h k in this paper the sets c b y x c m c c b k h p m c and c b k h r m c are respectively characterized completely where c denotes the continuous spectrum p denotes the point spectrum and r denotes the residual spectrum moreover some corresponding counterexamples are given

bregman divergences and surrogates for learning bartlett et al 2006 recently proved that a ground condition for surrogates classification calibration ties up their consistent minimization to that of the classification risk and left as an important problem the algorithmic questions about their minimization in this paper we address this problem for a wide set which lies at the intersection of classification calibrated surrogates and those of murata et al 2004 this set coincides with those satisfying three common assumptions about surrogates equivalent expressions for the members sometimes well known follow for convex and concave surrogates frequently used in the induction of linear separators and decision trees most notably they share remarkable algorithmic features for each of these two types of classifiers we give a minimization algorithm provably converging to the minimum of any such surrogate while seemingly different we show that these algorithms are offshoots of the same ldquomasterrdquo algorithm this provides a new and broad unified account of different popular algorithms including additive regression with the squared loss the logistic loss and the top down induction performed in cart c4 5 moreover we show that the induction enjoys the most popular boosting features regardless of the surrogate experiments are provided on 40 readily available domains

the rate of convergence of adaboost the adaboost algorithm was designed to combine many weak hypotheses that perform slightly better than random guessing into a strong hypothesis that has very low error we study the rate at which adaboost iteratively converges to the minimum of the exponential loss unlike previous work our proofs do not require a weak learning assumption nor do they require that minimizers of the exponential loss are finite our first result shows that the exponential loss of adaboost s computed parameter vector will be at most e more than that of any parameter vector of l1 norm bounded by b in a number of rounds that is at most a polynomial in b and 1 e we also provide lower bounds showing that a polynomial dependence is necessary our second result is that within c e iterations adaboost achieves a value of the exponential loss that is at most e more than the best possible value where c depends on the data set we show that this dependence of the rate on e is optimal up to constant factors that is at least 1 e rounds are necessary to achieve within e of the optimal exponential loss

conjugate relation between loss functions and uncertainty sets in classification problems there are two main approaches to binary classification problems the loss function approach and the uncertainty set approach the loss function approach is widely used in real world data analysis statistical decision theory has been used to elucidate its properties such as statistical consistency conditional probabilities can also be estimated by using the minimum solution of the loss function in the uncertainty set approach an uncertainty set is defined for each binary label from training samples the best separating hyperplane between the two uncertainty sets is used as the decision function although the uncertainty set approach provides an intuitive understanding of learning algorithms its statistical properties have not been sufficiently studied in this paper we show that the uncertainty set is deeply connected with the convex conjugate of a loss function on the basis of the conjugate relation we propose a way of revising the uncertainty set approach so that it will have good statistical properties such as statistical consistency we also introduce statistical models corresponding to uncertainty sets in order to estimate conditional probabilities finally we present numerical experiments verifying that the learning with revised uncertainty sets improves the prediction accuracy

complexities of convex combinations and bounding the generalization error in classification we introduce and study several measures of complexity of functions from the convex hull of a given base class these complexity measures take into account the sparsity of the weights of a convex combination as well as certain clustering properties of the base functions involved in it we prove new upper confidence bounds on the generalization error of ensemble voting classification algorithms that utilize the new complexity measures along with the empirical distributions of classification margins providing a better explanation of generalization performance of large margin classification methods

an iterative solution for the second kind integral equations using radial basis functions the purpose of this paper is to present a new method based on radial basis functions rbfs and iterative procedure for the solution of linear integral equations of fredholm and volterra types of the second kind to obtain a primary approximate solution the collocation method with a small number of nodes based on the rbfs and their first order derivatives can be adapted moreover we present an iterative procedure from the integral equation to improve the accuracy of solution numerical examples show that our approach has the potentiality to become an efficient method

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

classification of faces in man and machine we attempt to shed light on the algorithms humans use to classify images of human faces according to their gender for this a novel methodology combining human psychophysics and machine learning is introduced we proceed as follows first we apply principal component analysis pca on the pixel information of the face stimuli we then obtain a data set composed of these pca eigenvectors combined with the subjects gender estimates of the corresponding stimuli second we model the gender classification process on this data set using a separating hyperplane sh between both classes this sh is computed using algorithms from machine learning the support vector machine svm the relevance vector machine the prototype classifier and the k means classifier the classification behavior of humans and machines is then analyzed in three steps first the classification errors of humans and machines are compared for the various classifiers and we also assess how well machines can recreate the subjects internal decision boundary by studying the training errors of the machines second we study the correlations between the rank order of the subjects responses to each stimulus the gender estimate with its reaction time and confidence rating and the rank order of the distance of these stimuli to the sh finally we attempt to compare the metric of the representations used by humans and machines for classification by relating the subjects gender estimate of each stimulus and the distance of this stimulus to the sh while we show that the classification error alone is not a sufficient selection criterion between the different algorithms humans might use to classify face stimuli the distance of these stimuli to the sh is shown to capture essentials of the internal decision space of humans furthermore algorithms such as the prototype classifier using stimuli in the center of the classes are shown to be less adapted to model human classification behavior than algorithms such as the svm based on stimuli close to the boundary between the classes

solving second kind integral equations by galerkin methods with hybrid legendre and block pulse functions in this paper we use a combination of legendre and block pulse functions on the interval 0 1 to solve the linear integral equation of the second kind we convert the integral equation to a system of linear equations we show that our estimates have a good degree of accuracy

essential approximate point spectra and weyl s theorem for operator matrices abstract when a b h and b b k are given we denote by m c the operator acting on the infinite dimensional separable hilbert space h k of the form m c a c 0 b in this paper it is shown that a 2 2 operator matrix m c is upper semi fredholm and ind m c 0 for some c b k h if and only if a is upper semi fredholm and n b and n a n b d a d b or n b d a if r b is closed d a if r b is not closed we also give the necessary and sufficient conditions for which m c is weyl or m c is lower semi fredholm with nonnegative index for some c b k h in addition we explore how weyl s theorem browder s theorem a weyl s theorem and a browder s theorem survive for 2 2 upper triangular operator matrices on the hilbert space

optimal third root asymptotic bounds in the statistical estimation of thresholds this paper is concerned with estimating the intersection point of two densities given a sample of both of the densities this problem arises in classification theory the main results provide lower bounds for the probability of the estimation errors to be large on a scale determined by the inverse cube root of the sample size as corollaries we obtain probabilistic bounds for the prediction error in a classification problem the key to the proof is an entropy estimate the lower bounds are based on bounds for general estimators which are applicable in other contexts as well furthermore we introduce a class of optimal estimators whose errors asymptotically meet the border permitted by the lower bounds

arcing classifier with discussion and a rejoinder by the author recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error one of the more effective is bagging here modified training sets are formed by resampling from the original training set classifiers constructed using these training sets and then combined by voting freund and schapire propose an algorithm the basis of which is to adaptively resample and combine hence the acronym arcing so that the weights in the resampling are increased for those cases most often misclassified and the combining is done by weighted voting arcing is more successful than bagging in test set error reduction we explore two arcing algorithms compare them to each other and to bagging and try to understand how arcing works we introduce the definitions of bias and variance for a classifier as components of the test set error unstable classifiers can have low bias on a large range of data sets their problem is high variance combining multiple versions either through bagging or arcing reduces variance significantly

vote counting measures for ensemble classifiers various measures such as margin and bias variance have been proposed with the aim of gaining a better understanding of why multiple classifier systems mcs perform as well as they do while these measures provide different perspectives for mcs analysis it is not clear how to use them for mcs design in this paper a different measure based on a spectral representation is proposed for two class problems it incorporates terms representing positive and negative correlation of pairs of training patterns with respect to class labels experiments employing mlp base classifiers in which parameters are fixed but systematically varied demonstrate the sensitivity of the proposed measure to base classifier complexity

boosting as a regularized path to a maximum margin classifier in this paper we study boosting methods from a new perspective we build on recent work by efron et al to show that boosting approximately and in some cases exactly minimizes its loss criterion with an l1 constraint on the coefficient vector this helps understand the success of boosting with early stopping as regularized fitting of the loss criterion for the two most commonly used criteria exponential and binomial log likelihood we further show that as the constraint is relaxed or equivalently as the boosting iterations proceed the solution converges in the separable case to an l1 optimal separating hyper plane we prove that this l1 optimal separating hyper plane has the property of maximizing the minimal l1 margin of the training data as defined in the boosting literature an interesting fundamental similarity between boosting and kernel support vector machines emerges as both can be described as methods for regularized optimization in high dimensional predictor space using a computational trick to make the calculation practical and converging to margin maximizing solutions while this statement describes svms exactly it applies to boosting only approximately

properties of bagged nearest neighbour classifiers summary it is shown that bagging a computationally intensive method asymptotically improves the performance of nearest neighbour classifiers provided that the resample size is less than 69 of the actual sample size in the case of with replacement bagging or less than 50 of the sample size for without replacement bagging however for larger sampling fractions there is no asymptotic difference between the risk of the regular nearest neighbour classifier and its bagged version in particular neither achieves the large sample performance of the bayes classifier in contrast when the sampling fractions converge to 0 but the resample sizes diverge to 1 the bagged classifier converges to the optimal bayes rule and its risk converges to the risk of the latter these results are most readily seen when the two populations have well defined densities but they may also be derived in other cases where densities exist in only a relative sense cross validation can be used effectively to choose the sampling fraction numerical calculation is used to illustrate these theoretical properties

a re weighting strategy for improving margins we present a simple general scheme for improving margins that is inspired on well known margin theory principles the scheme is based on a sample re weighting strategy the very basic idea is in fact to add to the training set new replicas of samples which are not classified with a sufficient margin as a study case we present a new algorithm namely tvq which is an instance of the proposed scheme and involves a tangent distance based 1 nn classifier implementing a sort of quantization of the tangent distance prototypes the tangent distance models created in this way have shown a significant improvement in generalization power with respect to standard tangent models moreover the obtained models were able to outperform other state of the art algorithms such as svm in an ocr task

essential point spectra of operator matrices trough local spectral theory abstract for a b x b b y and c b y x let m c be the operator defined on x y by a c 0 b in this paper we study defect set a b m c where is the browder spectrum the essential approximate point spectrum and browder essential approximate point spectrum we then give application for weyl s and browder s theorems

classification of gene expression data the manifold based metric learning way classification of microarray gene expression data can potentially help medical diagnosis and becomes an important topic in bioinformatics however microarray data sets are usually of small sample size relative to an overwhelming number of genes this makes the classification problem fairly challenging instance based learning ibl algorithms such as nearest neighbor k nn are usually the baseline algorithm due to their simplicity however practices show that k nn performs not very well in this field this paper introduces manifold based metric learning to improve the performance of ibl methods a novel metric learning algorithm is proposed by utilizing both local manifold structural information and local discriminant information in addition a random subspace extension is also presented we apply the proposed algorithm to the gene classification problem in three ways one in the original feature space another in the reduced feature space and the third via the random subspace extension statistical evaluation shows that the proposed algorithm can achieve promising results and gain significant performance improvement over traditional ibl algorithms

constructing ensembles of classifiers by means of weighted instance selection in this paper we approach the problem of constructing ensembles of classifiers from the point of view of instance selection instance selection is aimed at obtaining a subset of the instances available for training capable of achieving at least the same performance as the whole training set in this way instance selection algorithms try to keep the performance of the classifiers while reducing the number of instances in the training set meanwhile boosting methods construct an ensemble of classifiers iteratively focusing each new member on the most difficult instances by means of a biased distribution of the training instances in this work we show how these two methodologies can be combined advantageously we can use instance selection algorithms for boosting using as objective to optimize the training error weighted by the biased distribution of the instances given by the boosting method our method can be considered as boosting by instance selection instance selection has mostly been developed and used for k nearest neighbor k nn classifiers so as a first step our methodology is suited to construct ensembles of k nn classifiers constructing ensembles of classifiers by means of instance selection has the important feature of reducing the space complexity of the final ensemble as only a subset of the instances is selected for each classifier however the methodology is not restricted to k nn classifier other classifiers such as decision trees and support vector machines svms may also benefit from a smaller training set as they produce simpler classifiers if an instance selection algorithm is performed before training in the experimental section we show that the proposed approach is able to produce better and simpler ensembles than random subspace method rsm method for k nn and standard ensemble methods for c4 5 and svms

ssc a classifier combination method based on signal strength we propose a new classifier combination method the signal strength based combining ssc approach to combine the outputs of multiple classifiers to support the decision making process in classification tasks as ensemble learning methods have attracted growing attention from both academia and industry recently it is critical to understand the fundamental issues of the combining rule motivated by the signal strength concept our proposed ssc algorithm can effectively integrate the individual vote from different classifiers in an ensemble learning system comparative studies of our method with nine major existing combining rules namely geometric average rule arithmetic average rule median value rule majority voting rule borda count max and min rule weighted average and weighted majority voting rules is presented furthermore we also discuss the relationship of the proposed method with respect to margin based classifiers including the boosting method adaboost m1 and adaboost m2 and support vector machines by margin analysis detailed analyses of margin distribution graphs are presented to discuss the characteristics of the proposed method simulation results for various real world datasets illustrate the effectiveness of the proposed method

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

numerical solution of the second kind integral equations using radial basis function networks in this paper a new method using radial basis function rbf networks is presented for solving the linear second kind integral equations of fredholm and volterra types this method employs a growing neural network as the approximate solution of the integral equations whose the activation functions are rbfs the parameters weights centers and widths of the approximate solution are adjusted by using an unconstrained optimization problem numerical results show that our method has the potentiality to become an efficient approach for solving integral equations

a class of logistic type discriminant functions summary in two group discriminant analysis the neyman pearson lemma establishes that the roc receiver operating characteristic curve for an arbitrary linear function is everywhere below the roc curve for the true likelihood ratio the weighted area between these two curves can be used as a risk function for finding good discriminant functions the weight function corresponds to the objective of the analysis for example to minimise the expected cost of misclassification or to maximise the area under the roc the resulting discriminant functions can be estimated by iteratively reweighted logistic regression we investigate some asymptotic properties in the near logistic setting where we assume the covariates have been chosen such that a linear function gives a reasonable but not necessarily exact

maximum relative margin and data dependent regularization leading classification methods such as support vector machines svms and their counterparts achieve strong generalization performance by maximizing the margin of separation between data classes while the maximum margin approach has achieved promising performance this article identifies its sensitivity to affine transformations of the data and to directions with large data spread maximum margin solutions may be misled by the spread of data and preferentially separate classes along large spread directions this article corrects these weaknesses by measuring margin not in the absolute sense but rather only relative to the spread of data in any projection direction maximum relative margin corresponds to a data dependent regularization on the classification function while maximum absolute margin corresponds to an l2 norm constraint on the classification function interestingly the proposed improvements only require simple extensions to existing maximum margin formulations and preserve the computational efficiency of svms through the maximization of relative margin surprising performance gains are achieved on real world problems such as digit text classification and on several other benchmark data sets in addition risk bounds are derived for the new formulation based on rademacher averages

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

self adjoint perturbations of spectra for upper triangular operator matrices abstract this paper is concerned with the self adjoint perturbations of the spectra for the upper triangular partial operator matrix with given diagonal entries a necessary and sufficient condition is given under which such operator matrix admits a weyl fredholm operator completion by choosing some bounded self adjoint operator it is shown that the self adjoint perturbation of the weyl essential spectrum can be the proper set of the general perturbation combining the spectral properties we further characterize the perturbation of the weyl essential spectrum for hamiltonian operators

generalization bounds and complexities based on sparsity and clustering for convex combinations of functions from random classes a unified approach is taken for deriving new generalization data dependent bounds for several classes of algorithms explored in the existing literature by different approaches this unified approach is based on an extension of vapnik s inequality for vc classes of sets to random classes of sets that is classes depending on the random data invariant under permutation of the data and possessing the increasing property generalization bounds are derived for convex combinations of functions from random classes with certain properties algorithms such as svms support vector machines boosting with decision stumps radial basis function networks some hierarchies of kernel machines or convex combinations of indicator functions over sets with finite vc dimension generate classifier functions that fall into the above category we also explore the individual complexities of the classifiers such as sparsity of weights and weighted variance over clusters from the convex combination introduced by koltchinskii and panchenko 2004 and show sparsity type and cluster variance type generalization bounds for random classes

classification based on hybridization of parametric and nonparametric classifiers parametric methods of classification assume specific parametric models for competing population densities e g gaussian population densities can lead to linear and quadratic discriminant analysis and they work well when these model assumptions are valid violation in one or more of these parametric model assumptions often leads to a poor classifier on the other hand nonparametric classifiers e g nearest neighbor and kernel based classifiers are more flexible and free from parametric model assumptions but the statistical instability of these classifiers may lead to poor performance when we have small numbers of training sample observations nonparametric methods however do not use any parametric structure of population densities therefore even when one has some additional information about population densities that important information is not used to modify the nonparametric classification rule this paper makes an attempt to overcome these limitations of parametric and nonparametric approaches and combines their strengths to develop some hybrid classification methods we use some simulated examples and benchmark data sets to examine the performance of these hybrid discriminant analysis tools asymptotic results on their misclassification rates have been derived under appropriate regularity conditions

double bagging combining classifiers by bootstrap aggregation the combination of classifiers leads to substantial reduction of misclassification error in a wide range of applications and benchmark problems we suggest using an out of bag sample for combining different classifiers in our setup a linear discriminant analysis is performed using the observations in the out of bag sample and the corresponding discriminant variables computed for the observations in the bootstrap sample are used as additional predictors for a classification tree two classifiers are combined and therefore method and variable selection bias is no problem for the corresponding estimate of misclassification error the need of an additional test sample disappears moreover the procedure performs comparable to the best classifiers used in a number of artificial examples and applications

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

consistency of random forests and other averaging classifiers in the last years of his life leo breiman promoted random forests for use in classification he suggested using averaging as a means of obtaining good discrimination rules the base classifiers used for averaging are simple and randomized often based on random samples from the data he left a few questions unanswered regarding the consistency of such rules in this paper we give a number of theorems that establish the universal consistency of averaging rules we also show that some popular classifiers including one suggested by breiman are not universally consistent

random rotation ensembles in machine learning ensemble methods combine the predictions of multiple base learners to construct more accurate aggregate predictions established supervised learning algorithms inject randomness into the construction of the individual base learners in an effort to promote diversity within the resulting ensembles an undesirable side effect of this approach is that it generally also reduces the accuracy of the base learners in this paper we introduce a method that is simple to implement yet general and effective in improving ensemble diversity with only modest impact on the accuracy of the individual base learners by randomly rotating the feature space prior to inducing the base learners we achieve favorable aggregate predictions on standard data sets compared to state of the art ensemble methods most notably for tree based ensembles which are particularly sensitive to rotation

the random subspace method for constructing decision forests much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes the dilemma between overfitting and achieving maximum accuracy is seldom resolved a method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity the classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector that is trees constructed in randomly chosen subspaces the subspace method is compared to single tree classifiers and other forest construction methods by experiments on publicly available datasets where the method s superiority is demonstrated we also discuss independence between trees in a forest and relate that to the combined classification accuracy

perturbation of spectra of operator matrices and local spectral theory abstract for a l x b l y and c l y x we denote by m c the operator defined on x y by a c 0 b in this article we study defect set d a b m c for different spectra including the spectrum the essential spectrum weyl spectrum and the approximate point spectrum we then apply the obtained results to the stability of such spectra d and the classes of operators c for which stability holds of m c using local spectral theory

the importance of convexity in learning with squared loss we show that if the closure of a function class f under the metric induced by some probability distribution is not convex then the sample complexity for agnostically learning f with squared loss using only hypotheses in f is spl omega ln 1 spl delta spl epsiv sup 2 where 1 spl delta is the probability of success and spl epsiv is the required accuracy in comparison if the class f is convex and has finite pseudodimension then the sample complexity is o 1 spl epsiv ln 1 spl epsiv ln 1 b if a nonconvex class f has finite pseudodimension then the sample complexity for agnostically learning the closure of the convex hull of f is o 1 spl epsiv 1 spl epsiv ln 1 spl epsiv ln 1 spl delta hence for agnostic learning learning the convex hull provides better approximation capabilities with little sample complexity penalty

the implicit bias of gradient descent on separable data we examine gradient descent on unregularized logistic regression problems with homogeneous linear predictors on linearly separable datasets we show the predictor converges to the direction of the max margin hard margin svm solution the result also generalizes to other monotone decreasing loss functions with an infimum at infinity to multi class problems and to training a weight layer in a deep network in a certain restricted setting furthermore we show this convergence is very slow and only logarithmic in the convergence of the loss itself this can help explain the benefit of continuing to optimize the logistic or cross entropy loss even after the training error is zero and the training loss is extremely small and as we show even if the validation loss increases our methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods