boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

sparse ensembles using weighted combination methods based on linear programming an ensemble of multiple classifiers is widely considered to be an effective technique for improving accuracy and stability of a single classifier this paper proposes a framework of sparse ensembles and deals with new linear weighted combination methods for sparse ensembles sparse ensemble is to sparsely combine the outputs of multiple classifiers by using a sparse weight vector when the continuous outputs of multiple classifiers are provided in our methods the problem of solving sparse weight vector can be formulated as linear programming problems in which the hinge loss or and the 1 norm regularization are exploited both the hinge loss and the 1 norm regularization are techniques inducing sparsity used in machine learning we only ensemble classifiers with nonzero weight coefficients in these lp based methods the ensemble training error is minimized while the weight vector of ensemble learning is controlled which can be thought as implementing the structure risk minimization rule and naturally explains good performance of these methods the promising experimental results over uci data sets and the radar high resolution range profile data are presented

inference on the prediction of ensembles of infinite size in this paper we introduce a framework for making statistical inference on the asymptotic prediction of parallel classification ensembles the validity of the analysis is fairly general it only requires that the individual classifiers are generated in independent executions of some randomized learning algorithm and that the final ensemble prediction is made via majority voting given an unlabeled test instance the predictions of the classifiers in the ensemble are obtained sequentially as the individual predictions become known bayes theorem is used to update an estimate of the probability that the class predicted by the current ensemble coincides with the classification of the corresponding ensemble of infinite size using this estimate the voting process can be halted when the confidence on the asymptotic prediction is sufficiently high an empirical investigation in several benchmark classification problems shows that most of the test instances require querying only a small number of classifiers to converge to the infinite ensemble prediction with a high degree of confidence for these instances the difference between the generalization error of the finite ensemble and the infinite ensemble limit is very small often negligible

triangularizability of trace class operators with increasing spectrum abstract for any measurable set e of a measure space x let p e be the orthogonal projection on the hilbert space l 2 x with the range ran p e f l 2 x f 0 a e on e c that is called a standard subspace of l 2 x let t be an operator on l 2 x having increasing spectrum relative to standard compressions that is for any measurable sets e and f with e f the spectrum of the operator p e t ran p e is contained in the spectrum of the operator p f t ran p f in 2009 marcoux mastnak and radjavi asked whether the operator t has a non trivial invariant standard subspace they answered this question affirmatively when either the measure space x is discrete or the operator t has finite rank we study this problem in the case of trace class kernel operators we also slightly strengthen the above mentioned result for finite rank operators

ensemble pruning via semi definite programming an ensemble is a group of learning models that jointly solve a problem however the ensembles generated by existing techniques are sometimes unnecessarily large which can lead to extra memory usage computational costs and occasional decreases in effectiveness the purpose of ensemble pruning is to search for a good subset of ensemble members that performs as well as or better than the original ensemble this subset selection problem is a combinatorial optimization problem and thus finding the exact optimal solution is computationally prohibitive various heuristic methods have been developed to obtain an approximate solution however most of the existing heuristics use simple greedy search as the optimization method which lacks either theoretical or empirical quality guarantees in this paper the ensemble subset selection problem is formulated as a quadratic integer programming problem by applying semi definite programming sdp as a solution technique we are able to get better approximate solutions computational experiments show that this sdp based pruning algorithm outperforms other heuristics in the literature its application in a classifier sharing study also demonstrates the effectiveness of the method

posterior probability support vector machines for unbalanced data this paper proposes a complete framework of posterior probability support vector machines ppsvms for weighted training samples using modified concepts of risks linear separability margin and optimal hyperplane within this framework a new optimization problem for unbalanced classification problems is formulated and a new concept of support vectors established furthermore a soft ppsvm with an interpretable parameter spl nu is obtained which is similar to the spl nu svm developed by scho spl uml lkopf et al and an empirical method for determining the posterior probability is proposed as a new approach to determine spl nu the main advantage of an ppsvm classifier lies in that fact that it is closer to the bayes optimal without knowing the distributions to validate the proposed method two synthetic classification examples are used to illustrate the logical correctness of ppsvms and their relationship to regular svms and bayesian methods several other classification experiments are conducted to demonstrate that the performance of ppsvms is better than regular svms in some cases compared with fuzzy support vector machines fsvms the proposed ppsvm is a natural and an analytical extension of regular svms based on the statistical learning theory

on the general ber expression of one and two dimensional amplitude modulations quadrature amplitude modulation qam is an attractive technique to achieve high rate transmission without increasing the bandwidth a great deal of attention has been devoted to the study of bit error rate ber performance of qam and approximate expressions for the bit error probability of qam have been developed in many places in the literature however the exact and general ber expression of qam with an arbitrary constellation size has not been derived yet we provide an exact and general closed form expression of the ber for one dimensional and two dimensional amplitude modulations i e pam and qam under an additive white gaussian noise awgn channel when gray code bit mapping is employed the provided ber expressions offer a convenient way to evaluate the performance of pam and qam systems for various cases of practical interest moreover simple approximations can be found from our expressions which are the same as the well known approximations if only the dominant terms are considered

stopping criterion for boosting based data reduction techniques from binary to multiclass problem so far boosting has been used to improve the quality of moderately accurate learning algorithms by weighting and combining many of their weak hypotheses into a final classifier with theoretically high accuracy in a recent work sebban nock and lallich 2001 we have attempted to adapt boosting properties to data reduction techniques in this particular context the objective was not only to improve the success rate but also to reduce the time and space complexities due to the storage requirements of some costly learning algorithms such as nearest neighbor classifiers in that framework each weak hypothesis which is usually built and weighted from the learning set is replaced by a single learning instance the weight given by boosting defines in that case the relevance of the instance and a statistical test allows one to decide whether it can be discarded without damaging further classification tasks in sebban nock and lallich 2001 we addressed problems with two classes it is the aim of the present paper to relax the class constraint and extend our contribution to multiclass problems beyond data reduction experimental results are also provided on twenty three datasets showing the benefits that our boosting derived weighting rule brings to weighted nearest neighbor classifiers

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

decision combination in multiple classifier systems a multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem the rankings can be combined by methods that either reduce or rerank a given set of classes an intersection method and union method are proposed for class set reduction three methods based on the highest rank the borda count and logistic regression are proposed for class set reranking these methods have been tested in applications of degraded machine printed characters and works from large lexicons resulting in substantial improvement in overall correctness

causal inference by using invariant prediction identification and confidence intervals summary what is the difference between a prediction that is made with a causal model and that with a non causal model suppose that we intervene on the predictor variables or change the whole environment the predictions from a causal model will in general work as well under interventions as for observational data in contrast predictions from a non causal model can potentially be very wrong if we actively intervene on variables here we propose to exploit this invariance of a prediction under a causal model for causal inference given different experimental settings e g various interventions we collect all models that do show invariance in their predictive accuracy across settings and interventions the causal model will be a member of this set of models with high probability this approach yields valid confidence intervals for the causal relationships in quite general scenarios we examine the example of structural equation models in more detail and provide sufficient assumptions under which the set of causal predictors becomes identifiable we further investigate robustness properties of our approach under model misspecification and discuss possible extensions the empirical properties are studied for various data sets including large scale gene perturbation experiments

random forests with ensemble of feature spaces random forests receive much attention from researchers because of their excellent performance as breiman suggested the performance of random forests depends on the strength of the weak learners in the forests and the diversity among them however in the literature many researchers only considered pre processing of the data or post processing of the random forests models in this paper we propose a new method to increase the diversity of each tree in the forests and thereby improve the overall accuracy during the training process of each individual tree in the forest different rotation spaces are concatenated into a higher space at the root node then the best split is exhaustively searched within this higher space the location where the best split lies decides which rotation method to be used for all subsequent nodes the performance of the proposed method here is evaluated on 42 benchmark data sets from various research fields and compared with the standard random forests the results show that the proposed method improves the performance of the random forests in most cases

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

learning similarity with cosine similarity ensemble this paper proposes a cosine similarity ensemble cse method to learn similarity cse is a selective ensemble and combines multiple cosine similarity learners a learner redefines the pattern vectors and determines its threshold adaptively experimental results show the superiority of cse there is no doubt that similarity is a fundamental notion in the field of machine learning and pattern recognition how to represent and measure similarity appropriately is a pursuit of many researchers many tasks such as classification and clustering can be accomplished perfectly when a similarity metric is well defined cosine similarity is a widely used metric that is both simple and effective this paper proposes a cosine similarity ensemble cse method for learning similarity in cse diversity is guaranteed by using multiple cosine similarity learners each of which makes use of a different initial point to define the pattern vectors used in its similarity measures the cse method is not limited to measuring similarity using only pattern vectors that start at the origin in addition the thresholds of these separate cosine similarity learners are adaptively determined the idea of using a selective ensemble is also implemented in cse and the proposed cse method outperforms other compared methods on various data sets

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

ensemble pruning using spectral coefficients ensemble pruning aims to increase efficiency by reducing the number of base classifiers without sacrificing and preferably enhancing performance in this brief a novel pruning paradigm is proposed two class supervised learning problems are pruned using a combination of first and second order walsh coefficients a comparison is made with other ordered aggregation pruning methods using multilayer perceptron base classifiers the walsh pruning method is analyzed with the help of a model that shows the relationship between second order coefficients and added classification error with respect to bayes error

two dimensional polar harmonic transforms for invariant image representation this paper introduces a set of 2d transforms based on a set of orthogonal projection bases to generate a set of features which are invariant to rotation we call these transforms polar harmonic transforms phts unlike the well known zernike and pseudo zernike moments the kernel computation of phts is extremely simple and has no numerical stability issue whatsoever this implies that phts encompass the orthogonality and invariance advantages of zernike and pseudo zernike moments but are free from their inherent limitations this also means that phts are well suited for application where maximal discriminant information is needed furthermore phts make available a large set of features for further feature selection in the process of seeking for the best discriminative or representative features for a particular application

a simple lemma on greedy approximation in hilbert space and convergence rates for projection pursuit regression and neural network training a general convergence criterion for certain iterative sequences in hilbert space is presented for an important subclass of these sequences estimates of the rate of convergence are given under very mild assumptions these results establish an 0 1 f4n nonsampling convergence rate for projection pursuit regression and neural network training where n represents the number of ridge functions neurons or coefficients in a greedy basis expansion

bandwidth choice for nonparametric classification it is shown that for kernel based classification with univariate distributions and two populations optimal bandwidth choice has a dichotomous character if the two densities cross at just one point where their curvatures have the same signs then minimum bayes risk is achieved using bandwidths which are an order of magnitude larger than those which minimize pointwise estimation error on the other hand if the curvature signs are different or if there are multiple crossing points then bandwidths of conventional size are generally appropriate the range of different modes of behavior is narrower in multivariate settings there the optimal size of bandwidth is generally the same as that which is appropriate for pointwise density estimation these properties motivate empirical rules for bandwidth choice

progressive subspace ensemble learning there are not many classifier ensemble approaches which investigate the data sample space and the feature space at the same time and this multi pronged approach will be helpful for constructing more powerful learning models for example the adaboost approach only investigates the data sample space while the random subspace technique only focuses on the feature space to address this limitation we propose the progressive subspace ensemble learning approach psel which takes into account the data sample space and the feature space at the same time specifically psel first adopts the random subspace technique to generate a set of subspaces then a progressive selection process based on new cost functions that incorporate current and long term information to select the classifiers sequentially will be introduced finally a weighted voting scheme is used to summarize the predicted labels and obtain the final result we also adopt a number of non parametric tests to compare psel and its competitors over multiple datasets the results of the experiments show that psel works well on most of the real datasets and outperforms a number of state of the art classifier ensemble approaches highlightsprogressive subspace ensemble learning it takes into account the data sample space and the feature space at the same time a progressive selection process based on new cost functions that incorporate current and long term information to select the classifiers sequentially will be introduced

wavelet decompositions of random forests smoothness analysis sparse approximation and applications in this paper we introduce in the setting of machine learning a generalization of wavelet analysis which is a popular approach to low dimensional structured signal analysis the wavelet decomposition of a random forest provides a sparse approximation of any regression or classification high dimensional function at various levels of detail with a concrete ordering of the random forest nodes from significant elements to nodes capturing only insignificant noise motivated by function space theory we use the wavelet decomposition to compute numerically a weak type smoothness index that captures the complexity of the underlying function as we show through extensive experimentation this sparse representation facilitates a variety of applications such as improved regression for difficult datasets a novel approach to feature importance resilience to noisy or irrelevant features compression of ensembles etc

closed form error analysis of variable gain multihop systems in nakagami m fading channels in this paper infinite integrals involving the product of bessel functions of different arguments are solved in closed form the obtained solutions form a framework for the error probability analysis of wireless amplify and forward af systems with an arbitrary number of variable gain relays operating over independent but not necessarily identical nakagami m fading channels here we show that the error probability can be described by generalized hypergeometric functions namely gauss s and lauricella s multivariate hypergeometric functions this work represents a significant improvement over previous contributions and extends previous formulas pertaining to dual hop transmissions over identical nakagami m fading channels numerical examples show an excellent match between simulation and theoretical results

theory and algorithm for learning with dissimilarity functions we study the problem of classification when only a dissimilarity function between objects is accessible that is data samples are represented not by feature vectors but in terms of their pairwise dissimilarities we establish sufficient conditions for dissimilarity functions to allow building accurate classifiers the theory immediately suggests a learning paradigm construct an ensemble of simple classifiers each depending on a pair of examples then find a convex combination of them to achieve a large margin we next develop a practical algorithm referred to as dissimilarity based boosting dboost for learning with dissimilarity functions under theoretical guidance experiments on a variety of databases demonstrate that the dboost algorithm is promising for several dissimilarity measures widely used in practice

incremental construction of classifier and discriminant ensembles we discuss approaches to incrementally construct an ensemble the first constructs an ensemble of classifiers choosing a subset from a larger set and the second constructs an ensemble of discriminants where a classifier is used for some classes only we investigate criteria including accuracy significant improvement diversity correlation and the role of search direction for discriminant ensembles we test subset selection and trees fusion is by voting or by a linear model using 14 classifiers on 38 data sets incremental search finds small accurate ensembles in polynomial time the discriminant ensemble uses a subset of discriminants and is simpler interpretable and accurate we see that an incremental ensemble has higher accuracy than bagging and random subspace method and it has a comparable accuracy to adaboost but fewer classifiers

bregman divergences and surrogates for learning bartlett et al 2006 recently proved that a ground condition for surrogates classification calibration ties up their consistent minimization to that of the classification risk and left as an important problem the algorithmic questions about their minimization in this paper we address this problem for a wide set which lies at the intersection of classification calibrated surrogates and those of murata et al 2004 this set coincides with those satisfying three common assumptions about surrogates equivalent expressions for the members sometimes well known follow for convex and concave surrogates frequently used in the induction of linear separators and decision trees most notably they share remarkable algorithmic features for each of these two types of classifiers we give a minimization algorithm provably converging to the minimum of any such surrogate while seemingly different we show that these algorithms are offshoots of the same ldquomasterrdquo algorithm this provides a new and broad unified account of different popular algorithms including additive regression with the squared loss the logistic loss and the top down induction performed in cart c4 5 moreover we show that the induction enjoys the most popular boosting features regardless of the surrogate experiments are provided on 40 readily available domains

conjugate relation between loss functions and uncertainty sets in classification problems there are two main approaches to binary classification problems the loss function approach and the uncertainty set approach the loss function approach is widely used in real world data analysis statistical decision theory has been used to elucidate its properties such as statistical consistency conditional probabilities can also be estimated by using the minimum solution of the loss function in the uncertainty set approach an uncertainty set is defined for each binary label from training samples the best separating hyperplane between the two uncertainty sets is used as the decision function although the uncertainty set approach provides an intuitive understanding of learning algorithms its statistical properties have not been sufficiently studied in this paper we show that the uncertainty set is deeply connected with the convex conjugate of a loss function on the basis of the conjugate relation we propose a way of revising the uncertainty set approach so that it will have good statistical properties such as statistical consistency we also introduce statistical models corresponding to uncertainty sets in order to estimate conditional probabilities finally we present numerical experiments verifying that the learning with revised uncertainty sets improves the prediction accuracy

exploration of classification confidence in ensemble learning ensemble learning has attracted considerable attention owing to its good generalization performance the main issues in constructing a powerful ensemble include training a set of diverse and accurate base classifiers and effectively combining them ensemble margin computed as the difference of the vote numbers received by the correct class and the another class received with the most votes is widely used to explain the success of ensemble learning this definition of the ensemble margin does not consider the classification confidence of base classifiers in this work we explore the influence of the classification confidence of the base classifiers in ensemble learning and obtain some interesting conclusions first we extend the definition of ensemble margin based on the classification confidence of the base classifiers then an optimization objective is designed to compute the weights of the base classifiers by minimizing the margin induced classification loss several strategies are tried to utilize the classification confidences and the weights it is observed that weighted voting based on classification confidence is better than simple voting if all the base classifiers are used in addition ensemble pruning can further improve the performance of a weighted voting ensemble we also compare the proposed fusion technique with some classical algorithms the experimental results also show the effectiveness of weighted voting with classification confidence

how large should ensembles of classifiers be we propose to determine the size of a parallel ensemble by estimating the minimum number of classifiers that are required to obtain stable aggregate predictions assuming that majority voting is used a statistical description of the convergence of the ensemble prediction to its asymptotic infinite size limit is given the analysis of the voting process shows that for most test instances the ensemble prediction stabilizes after only a few classifiers are polled by contrast a small but non negligible fraction of these instances require large numbers of classifier queries to reach stable predictions specifically the fraction of instances whose stable predictions require more than t classifiers for email protected 1 has a universal form and is proportional to t 1 2 the ensemble size is determined as the minimum number of classifiers that are needed to estimate the infinite ensemble prediction at an average confidence level a close to one this approach differs from previous proposals which are based on determining the size for which the prediction error not the predictions themselves stabilizes in particular it does not require estimates of the generalization performance of the ensemble which can be unreliable it has general validity because it is based solely on the statistical description of the convergence of majority voting to its asymptotic limit extensive experiments using representative parallel ensembles bagging and random forest illustrate the application of the proposed framework in a wide range of classification problems these experiments show that the optimal ensemble size is very sensitive to the particular classification problem considered

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

classification of faces in man and machine we attempt to shed light on the algorithms humans use to classify images of human faces according to their gender for this a novel methodology combining human psychophysics and machine learning is introduced we proceed as follows first we apply principal component analysis pca on the pixel information of the face stimuli we then obtain a data set composed of these pca eigenvectors combined with the subjects gender estimates of the corresponding stimuli second we model the gender classification process on this data set using a separating hyperplane sh between both classes this sh is computed using algorithms from machine learning the support vector machine svm the relevance vector machine the prototype classifier and the k means classifier the classification behavior of humans and machines is then analyzed in three steps first the classification errors of humans and machines are compared for the various classifiers and we also assess how well machines can recreate the subjects internal decision boundary by studying the training errors of the machines second we study the correlations between the rank order of the subjects responses to each stimulus the gender estimate with its reaction time and confidence rating and the rank order of the distance of these stimuli to the sh finally we attempt to compare the metric of the representations used by humans and machines for classification by relating the subjects gender estimate of each stimulus and the distance of this stimulus to the sh while we show that the classification error alone is not a sufficient selection criterion between the different algorithms humans might use to classify face stimuli the distance of these stimuli to the sh is shown to capture essentials of the internal decision space of humans furthermore algorithms such as the prototype classifier using stimuli in the center of the classes are shown to be less adapted to model human classification behavior than algorithms such as the svm based on stimuli close to the boundary between the classes

an analysis of ensemble pruning techniques based on ordered aggregation several pruning strategies that can be used to reduce the size and increase the accuracy of bagging ensembles are analyzed these heuristics select subsets of complementary classifiers that when combined can perform better than the whole ensemble the pruning methods investigated are based on modifying the order of aggregation of classifiers in the ensemble in the original bagging algorithm the order of aggregation is left unspecified when this order is random the generalization error typically decreases as the number of classifiers in the ensemble increases if an appropriate ordering for the aggregation process is devised the generalization error reaches a minimum at intermediate numbers of classifiers this minimum lies below the asymptotic error of bagging pruned ensembles are obtained by retaining a fraction of the classifiers in the ordered ensemble the performance of these pruned ensembles is evaluated in several benchmark classification tasks under different training conditions the results of this empirical investigation show that ordered aggregation can be used for the efficient generation of pruned ensembles that are competitive in terms of performance and robustness of classification with computationally more costly methods that directly select optimal or near optimal subensembles

optimal third root asymptotic bounds in the statistical estimation of thresholds this paper is concerned with estimating the intersection point of two densities given a sample of both of the densities this problem arises in classification theory the main results provide lower bounds for the probability of the estimation errors to be large on a scale determined by the inverse cube root of the sample size as corollaries we obtain probabilistic bounds for the prediction error in a classification problem the key to the proof is an entropy estimate the lower bounds are based on bounds for general estimators which are applicable in other contexts as well furthermore we introduce a class of optimal estimators whose errors asymptotically meet the border permitted by the lower bounds

triangularizability of operators with increasing spectrum we establish finite and infinite dimensional versions of the following assertion if m is a matrix with the property that whenever p and q are diagonal projections with p q the spectrum of pmp considered as an operator on the range of p is contained in that of qmq considered as an operator on the range of q then there is a permutation matrix u such that u 1mu is triangular

sequential combination of weighted and nonparametric bagging for classification we propose a simple sequential procedure for bagged classification which modifies nonparametric bagging by randomizing class labels of resampled data points the random labelling feature of the procedure also enables us to undertake unsupervised classification with the benefit of supervised learning theoretical properties are given for the nearest neighbour classifier in the case of supervised learning and a hard thresholding indicator in the case of unsupervised learning showing that sequential bagging accelerates convergence of the bagged predictor to the bayes rule simulation results are provided in support of the proposed method

boosting as a regularized path to a maximum margin classifier in this paper we study boosting methods from a new perspective we build on recent work by efron et al to show that boosting approximately and in some cases exactly minimizes its loss criterion with an l1 constraint on the coefficient vector this helps understand the success of boosting with early stopping as regularized fitting of the loss criterion for the two most commonly used criteria exponential and binomial log likelihood we further show that as the constraint is relaxed or equivalently as the boosting iterations proceed the solution converges in the separable case to an l1 optimal separating hyper plane we prove that this l1 optimal separating hyper plane has the property of maximizing the minimal l1 margin of the training data as defined in the boosting literature an interesting fundamental similarity between boosting and kernel support vector machines emerges as both can be described as methods for regularized optimization in high dimensional predictor space using a computational trick to make the calculation practical and converging to margin maximizing solutions while this statement describes svms exactly it applies to boosting only approximately

irreducible compact operators the purpose of the present paper is to show that a positive compact ideal irreducible operator t in a banach lattice of dimension greater than one has spectral radius r t 0 i e t is not quasi nilpotent it was shown in 6 that there exist positive irreducible operators which are quasi nilpotent and the question under what conditions a positive irreducible operator t satisfies r t 0 has been studied extensively see e g 6 and 7 sect v 6 in particular we mention the ando krieger theorem see e g 9 theorem 136 9 which states that a positive irreducible kernel operator t in a banach function space has r t 0 recently in 8 some special situations are discussed in which a compact positive irreducible operator has a strictly positive spectral radius let l be a real or complex banach lattice by 2 we denote the banach space of all bounded linear operators in l as usual we write s t in l whenever su tu for all o uel given 0 t l the closed ideal j in l is called t invariant if t j j and t is called ideal irreducible if the only closed t invariant ideals are 0 and l see 7 iii 8 for the general theory of banach lattices and positive operators we refer to the books 7 and 9 in the proof of the main result of the paper we will use some properties of the center of a banach lattice we recall some elementary facts let l be a real banach lattice the center z l of l is the subspace of 5 l consisting of all operators z for which there exists 0 2 ir such that f l 2 f for all f l the center has the structure of a vector lattice with nl v n2 u n l u v n2u and ozla zz u lu a nzu for all o u l and all l n2ez l moreover z l is a commutative algebra with respect to composition as multiplication and the identity operator i in l is a strong unit i e for any rc z l there exists 0 2eir such that zi 2i see 9 chap 20 note that if l c k for some compact hausdorff space k then z l can be identified with c k acting on itself by multiplication if l is a dedekind complete banach lattice and 0 u v in l then it is an easy consequence of the freudenthal spectral theorem 4 theorem 40 2 that there exists n z l such that zcv u and o n i however in an arbitrary banach lattice such a 7z need not exist as is shown by the banach lattice l c 0 1 the next lemma shows that under rather mild conditions we can

constructing ensembles of classifiers by means of weighted instance selection in this paper we approach the problem of constructing ensembles of classifiers from the point of view of instance selection instance selection is aimed at obtaining a subset of the instances available for training capable of achieving at least the same performance as the whole training set in this way instance selection algorithms try to keep the performance of the classifiers while reducing the number of instances in the training set meanwhile boosting methods construct an ensemble of classifiers iteratively focusing each new member on the most difficult instances by means of a biased distribution of the training instances in this work we show how these two methodologies can be combined advantageously we can use instance selection algorithms for boosting using as objective to optimize the training error weighted by the biased distribution of the instances given by the boosting method our method can be considered as boosting by instance selection instance selection has mostly been developed and used for k nearest neighbor k nn classifiers so as a first step our methodology is suited to construct ensembles of k nn classifiers constructing ensembles of classifiers by means of instance selection has the important feature of reducing the space complexity of the final ensemble as only a subset of the instances is selected for each classifier however the methodology is not restricted to k nn classifier other classifiers such as decision trees and support vector machines svms may also benefit from a smaller training set as they produce simpler classifiers if an instance selection algorithm is performed before training in the experimental section we show that the proposed approach is able to produce better and simpler ensembles than random subspace method rsm method for k nn and standard ensemble methods for c4 5 and svms

classification of gene expression data the manifold based metric learning way classification of microarray gene expression data can potentially help medical diagnosis and becomes an important topic in bioinformatics however microarray data sets are usually of small sample size relative to an overwhelming number of genes this makes the classification problem fairly challenging instance based learning ibl algorithms such as nearest neighbor k nn are usually the baseline algorithm due to their simplicity however practices show that k nn performs not very well in this field this paper introduces manifold based metric learning to improve the performance of ibl methods a novel metric learning algorithm is proposed by utilizing both local manifold structural information and local discriminant information in addition a random subspace extension is also presented we apply the proposed algorithm to the gene classification problem in three ways one in the original feature space another in the reduced feature space and the third via the random subspace extension statistical evaluation shows that the proposed algorithm can achieve promising results and gain significant performance improvement over traditional ibl algorithms

a probabilistic model of classifier competence for dynamic ensemble selection the concept of a classifier competence is fundamental to multiple classifier systems mcss in this study a method for calculating the classifier competence is developed using a probabilistic model in the method first a randomised reference classifier rrc whose class supports are realisations of the random variables with beta probability distributions is constructed the parameters of the distributions are chosen in such a way that for each feature vector in a validation set the expected values of the class supports produced by the rrc and the class supports produced by a modelled classifier are equal this allows for using the probability of correct classification of the rrc as the competence of the modelled classifier the competences calculated for a validation set are then generalised to an entire feature space by constructing a competence function based on a potential function model or regression three systems based on a dynamic classifier selection and a dynamic ensemble selection des were constructed using the method developed the des based system had statistically significant higher average rank than the ones of eight benchmark mcss for 22 data sets and a heterogeneous ensemble the results obtained indicate that the full vector of class supports should be used for evaluating the classifier competence as this potentially improves performance of mcss

ssc a classifier combination method based on signal strength we propose a new classifier combination method the signal strength based combining ssc approach to combine the outputs of multiple classifiers to support the decision making process in classification tasks as ensemble learning methods have attracted growing attention from both academia and industry recently it is critical to understand the fundamental issues of the combining rule motivated by the signal strength concept our proposed ssc algorithm can effectively integrate the individual vote from different classifiers in an ensemble learning system comparative studies of our method with nine major existing combining rules namely geometric average rule arithmetic average rule median value rule majority voting rule borda count max and min rule weighted average and weighted majority voting rules is presented furthermore we also discuss the relationship of the proposed method with respect to margin based classifiers including the boosting method adaboost m1 and adaboost m2 and support vector machines by margin analysis detailed analyses of margin distribution graphs are presented to discuss the characteristics of the proposed method simulation results for various real world datasets illustrate the effectiveness of the proposed method

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

classification based on hybridization of parametric and nonparametric classifiers parametric methods of classification assume specific parametric models for competing population densities e g gaussian population densities can lead to linear and quadratic discriminant analysis and they work well when these model assumptions are valid violation in one or more of these parametric model assumptions often leads to a poor classifier on the other hand nonparametric classifiers e g nearest neighbor and kernel based classifiers are more flexible and free from parametric model assumptions but the statistical instability of these classifiers may lead to poor performance when we have small numbers of training sample observations nonparametric methods however do not use any parametric structure of population densities therefore even when one has some additional information about population densities that important information is not used to modify the nonparametric classification rule this paper makes an attempt to overcome these limitations of parametric and nonparametric approaches and combines their strengths to develop some hybrid classification methods we use some simulated examples and benchmark data sets to examine the performance of these hybrid discriminant analysis tools asymptotic results on their misclassification rates have been derived under appropriate regularity conditions

double bagging combining classifiers by bootstrap aggregation the combination of classifiers leads to substantial reduction of misclassification error in a wide range of applications and benchmark problems we suggest using an out of bag sample for combining different classifiers in our setup a linear discriminant analysis is performed using the observations in the out of bag sample and the corresponding discriminant variables computed for the observations in the bootstrap sample are used as additional predictors for a classification tree two classifiers are combined and therefore method and variable selection bias is no problem for the corresponding estimate of misclassification error the need of an additional test sample disappears moreover the procedure performs comparable to the best classifiers used in a number of artificial examples and applications

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

bounds for multihop relayed communications in nakagami m fading we present closed form lower bounds for the performance of multihop transmissions with nonregenerative relays over not necessarily identically distributed nakagami m fading channels the end to end signal to noise ratio is formulated and upper bounded by using an inequality between harmonic and geometric means of positive random variables rvs novel closed form expressions are derived for the moment generating function the probability density function and the cumulative distribution function of the product of rational powers of statistically independent gamma rvs these statistical results are then applied to studying the outage probability and the average bit error probability for phase and frequency modulated signaling numerical examples compare analytical and simulation results verifying the tightness of the proposed bounds

harmonic mean and end to end performance of transmission systems with relays closed form expressions for the statistics of the harmonic mean of two independent and identically distributed gamma variates are presented the probability density function of the harmonic mean of two f variates is also derived these statistical results are then applied to study the performance of wireless communication systems with nonregenerative relays over flat nakagami fading channels more specifically outage probability formulas for noise limited systems as well as systems affected by interference are obtained furthermore general expressions for average bit error rates are also derived finally comparisons between regenerative and nonregenerative systems are presented numerical results show that the former systems clearly outperform the latter ones for low average signal to noise ratios snrs they also show that the two systems have similar performance at high average snrs

the importance of convexity in learning with squared loss we show that if the closure of a function class f under the metric induced by some probability distribution is not convex then the sample complexity for agnostically learning f with squared loss using only hypotheses in f is spl omega ln 1 spl delta spl epsiv sup 2 where 1 spl delta is the probability of success and spl epsiv is the required accuracy in comparison if the class f is convex and has finite pseudodimension then the sample complexity is o 1 spl epsiv ln 1 spl epsiv ln 1 b if a nonconvex class f has finite pseudodimension then the sample complexity for agnostically learning the closure of the convex hull of f is o 1 spl epsiv 1 spl epsiv ln 1 spl epsiv ln 1 spl delta hence for agnostic learning learning the convex hull provides better approximation capabilities with little sample complexity penalty

the implicit bias of gradient descent on separable data we examine gradient descent on unregularized logistic regression problems with homogeneous linear predictors on linearly separable datasets we show the predictor converges to the direction of the max margin hard margin svm solution the result also generalizes to other monotone decreasing loss functions with an infimum at infinity to multi class problems and to training a weight layer in a deep network in a certain restricted setting furthermore we show this convergence is very slow and only logarithmic in the convergence of the loss itself this can help explain the benefit of continuing to optimize the logistic or cross entropy loss even after the training error is zero and the training loss is extremely small and as we show even if the validation loss increases our methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods