boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

ambiguity based multiclass active learning most existing works on active learning al focus on binary classification problems which limit their applications in various real world scenarios one solution to multiclass al mal is evaluating the informativeness of unlabeled samples by an uncertainty model and selecting the most uncertain one for query in this paper an ambiguity based strategy is proposed to tackle this problem by applying a possibility approach first the possibilistic memberships of unlabeled samples in the multiple classes are calculated from the one against all based support vector machine model then by employing fuzzy logic operators these memberships are aggregated into a new concept named k order ambiguity which estimates the risk of labeling a sample among k classes afterward the k order ambiguities are used to form an overall ambiguity measure to evaluate the uncertainty of the unlabeled samples finally the sample with the maximum ambiguity is selected for query and a new mal strategy is developed experiments demonstrate the feasibility and effectiveness of the proposed method

alternative c means clustering algorithms abstract in this paper we propose a new metric to replace the euclidean norm in c means clustering procedures on the basis of the robust statistic and the influence function we claim that the proposed new metric is more robust than the euclidean norm we then create two new clustering methods called the alternative hard c means ahcm and alternative fuzzy c means afcm clustering algorithms these alternative types of c means clustering have more robustness than c means clustering numerical results show that ahcm has better performance than hcm and afcm is better than fcm we recommend afcm for use in cluster analysis recently this afcm algorithm has successfully been used in segmenting the magnetic resonance image of ophthalmology to differentiate the abnormal tissues from the normal tissues

estimation of parameters in latent class models using fuzzy clustering algorithms abstract a mixture approach to clustering is an important technique in cluster analysis a mixture of multivariate multinomial distributions is usually used to analyze categorical data with latent class model the parameter estimation is an important step for a mixture distribution described here are four approaches to estimating the parameters of a mixture of multivariate multinomial distributions the first approach is an extended maximum likelihood ml method the second approach is based on the well known expectation maximization em algorithm the third approach is the classification maximum likelihood cml algorithm in this paper we propose a new approach using the so called fuzzy class model and then create the fuzzy classification maximum likelihood fcml approach for categorical data the accuracy robustness and effectiveness of these four types of algorithms for estimating the parameters of multivariate binomial mixtures are compared using real empirical data and samples drawn from the multivariate binomial mixtures of two classes the results show that the proposed fcml algorithm presents better accuracy robustness and effectiveness overall the fcml algorithm has the superiority over the ml em and cml algorithms thus we recommend fcml as another good tool for estimating the parameters of mixture multivariate multinomial models

general c means clustering model partitional clustering is an important part of cluster analysis based on various theories numerous clustering algorithms have been developed and new clustering algorithms continue to appear in the literature it is known that occam s razor plays a pivotal role in data based models and partitional clustering is categorized as a data based model however no relation had previously been discovered between occam s razor and partitional clustering as we discuss in this paper the three main contributions of this paper can be summarized as follows 1 according to a novel definition of the mean a unifying generative framework for partitional clustering algorithms called a general c means clustering model gcm is presented and studied and 2 based on the local optimality test of the gcm the connection between occam s razor and partitional clustering is established for the first time as its application a comprehensive review of the existing objective function based clustering algorithms is presented based on gcm 3 under a common assumption about partitional clustering a theoretical guide for devising and implementing clustering algorithm is discovered these conclusions are verified by numerical experimental results

multicategory support vector machines two category support vector machines svm have been very popular in the machine learning community for classification problems solving multicategory problems by a series of binary classifiers is quite common in the svm paradigm however this approach may fail under various circumstances we propose the multicategory support vector machine msvm which extends the binary svm to the multicategory case and has good theoretical properties the proposed method provides a unifying framework when there are either equal or unequal misclassification costs as a tuning criterion for the msvm an approximate leave one out cross validation function called generalized approximate cross validation is derived analogous to the binary case the effectiveness of the msvm is demonstrated through the applications to cancer classification using microarray data and cloud classification with satellite radiance profiles

a multiclass classification method based on decoding of binary classifiers in this letter we present new methods of multiclass classification that combine multiple binary classifiers misclassification of each binary classifier is formulated as a bit inversion error with probabilistic models by making an analogy to the context of information transmission theory dependence between binary classifiers is incorporated into our model which makes a decoder a type of boltzmann machine we performed experimental studies using a synthetic data set data sets from the uci repository and bioinformatics data sets and the results show that the proposed methods are superior to the existing multiclass classification methods

clustering construction on a multimodal probability model in 2009 yu et al proposed a multimodal probability model mpm for clustering this paper makes advanced clustering constructions on the mpm we first reconstruct most existing clustering algorithms such as the k means fuzzy c means possibilistic c means mean shift classification maximum likelihood and latent class methods by establishing the relationships between these clustering algorithms and the mpm under our clustering construction we find that the mpm can be seen as a basic probability model for most existing clustering algorithms we then construct new clustering frameworks based on the mpm one of the frameworks develops new penalized type clustering algorithms another one induces entropy type clustering algorithms especially with sample weighted clustering several numerical and real data sets are made for comparisons these experimental results show that our clustering constructions based on the mpm can produce useful and effective clustering algorithms

a tree search algorithm for the p median problem abstract in this paper we present two lower bounds for the p median problem the problem of locating p facilities medians on a network these bounds are based on two separate lagrangean relaxations of a zero one formulation of the problem with subgradient optimisation being used to maximise these bounds penalty tests based on these lower bounds and a heuristically determined upper bound to the problem are developed and shown to result in a large reduction in problem size the incorporation of the lower bounds and the penalty tests into a tree search procedure is described and computational results are given for problems with an arbitrary number of medians and having up to 200 vertices a comparison is also made between these algorithms and the dual based algorithm of erlenkotter

classification by pairwise coupling we discuss a strategy for polychotomous classification that involves estimating class probabilities for each pair of classes and then coupling the estimates together the coupling model is similar to the bradley terry method for paired comparisons we study the nature of the class probability estimates that arise and examine the performance of the procedure in real and simulated data sets classifiers used include linear discriminants nearest neighbors adaptive nonlinear methods and the support vector machine

classification using geometric level sets a variational level set method is developed for the supervised classification problem nonlinear classifier decision boundaries are obtained by minimizing an energy functional that is composed of an empirical risk term with a margin based loss and a geometric regularization term new to machine learning the surface area of the decision boundary this geometric level set classifier is analyzed in terms of consistency and complexity through the calculation of its e entropy for multicategory classification an efficient scheme is developed using a logarithmic number of decision functions in the number of classes rather than the typical linear number of decision functions geometric level set classification yields performance results on benchmark data sets that are competitive with well established methods

vector quantization in dct domain using fuzzy possibilistic c means based on penalized and compensated constraints in this paper fuzzy possibilistic c means fpcm approach based on penalized and compensated constraints are proposed to vector quantization vq in discrete cosine transform dct for image compression these approaches are named penalized fuzzy possibilistic c means pfpcm and compensated fuzzy possibilistic c means cfpcm the main purpose is to modify the fpcm strategy with penalized or compensated constraints so that the cluster centroids can be updated with penalized or compensated terms iteratively in order to 4nd near global solution in optimal problem the information transformed by dct was separated into dc and ac coe5cients then the ac coe5cients are trained by using the proposed methods to generate better codebook based on vq the compression performances using the proposed approaches are compared with fpcm and conventional vq method from the experimental results the promising performances can be obtained using the proposed approaches 2002 pattern recognition society published by elsevier science ltd all rights reserved

on a class of fuzzy classification maximum likelihood procedures classification maximum likelihood cml procedure is a remarkable mixture of maximum likelihood approach to clustering this has been well documented in the book of mclachlan and basford in this paper we make the fuzzy extension of the cml procedure based on this class of fuzzy cml procedures we extend the fuzzy clustering algorithms of trauwaert kaufman and rousseeuw by adding a penalty term especially we derive a generalized type of fuzzy c means fcm clustering algorithms called the penalized fcm clustering algorithms then we create some asymptotic behaviors of these penalized fcm procedures by doing some numerical examples we find that the penalized fcm is more meaningful and effective than fcm

on convergence properties of the em algorithm for gaussian mixtures we build up the mathematical connection between the expectation maximization em algorithm and gradient based approaches for maximum likelihood learning of finite gaussian mixtures we show that the em step in parameter space is obtained from the gradient via a projection matrix p and we provide an explicit expression for the matrix we then analyze the convergence of em in terms of special properties of p and provide new results analyzing the effect that p has on the likelihood surface based on these mathematical results we present a comparative discussion of the advantages and disadvantages of em and other algorithms for the learning of gaussian mixture models

the analysis of contingency tables preface contingency tables and the chi square test 2 x 2 contingency tables r x c contingency tables multidimensional tables log linear models for contingency table linear logistic models contingency tables with ordered categories some special types of contingency tables appendices percentage points of the chi squared distribution computer software for fitting log linear and logistic models reference index

optimization approaches to supervised classification abstract the supervised classification problem one of the oldest and most recurrent problems in applied data analysis has always been analyzed from many different perspectives when the emphasis is placed on its overall goal of developing classification rules with minimal classification cost supervised classification can be understood as an optimization problem on the other hand when the focus is in modeling the uncertainty involved in the classification of future unknown entities it can be formulated as a statistical problem other perspectives that pay particular attention to pattern recognition and machine learning aspects of supervised classification have also a long history that has lead to influential insights and different methodologies in this review two approaches to supervised classification strongly related to optimization theory will be discussed and compared in particular we will review methodologies based on mathematical programming models that optimize observable criteria linked to the true objective of misclassification error or cost minimization and approaches derived from the minimization of known bounds on the true misclassification error the former approach is known as the mathematical programming approach to supervised classification while the latter is in the origin of the well known classification support vector machines throughout the review two group as well as general multi group problems will be considered and the review will conclude with a discussion of the most promising research directions in this area

reducing multiclass to binary a unifying approach for margin classifiers we present a unifying framework for studying the solution of multiclass categorization problems by reducing them to multiple binary problems that are then solved using a margin based binary learning algorithm the proposed framework unifies some of the most popular approaches in which each class is compared against all others or in which all pairs of classes are compared to each other or in which output codes with error correcting properties are used we propose a general method for combining the classifiers generated on the binary problems and we prove a general empirical multiclass loss bound given the empirical loss of the individual binary learning algorithms the scheme and the corresponding bounds apply to many popular classification learning algorithms including support vector machines adaboost regression logistic regression and decision tree algorithms we also give a multiclass generalization error analysis for general output codes with adaboost as the binary learner experimental results with svm and adaboost show that our scheme provides a viable alternative to the most commonly used multiclass algorithms

in defense of one vs all classification we consider the problem of multiclass classification our main thesis is that a simple one vs all scheme is as accurate as any other approach assuming that the underlying binary classifiers are well tuned regularized classifiers such as support vector machines this thesis is interesting in that it disagrees with a large body of recent published work on multiclass classification we support our position by means of a critical review of the existing literature a substantial collection of carefully controlled experimental work and theoretical arguments

stability and generalization we define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave one out error the methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real valued function we study the stability properties of large classes of learning algorithms such as regularization based algorithms in particular we focus on hilbert space regularization and kullback leibler regularization we demonstrate how to apply the results to svm for regression and classification

on the algorithmic implementation of multiclass kernel based vector machines in this paper we describe the algorithmic implementation of multiclass kernel based vector machines our starting point is a generalized notion of the margin to multiclass problems using this notion we cast multiclass categorization problems as a constrained optimization problem with a quadratic objective function unlike most of previous approaches which typically decompose a multiclass problem into multiple independent binary classification tasks our notion of margin yields a direct method for training multiclass predictors by using the dual of the optimization problem we are able to incorporate kernels with a compact set of constraints and decompose the dual problem into multiple optimization problems of reduced size we describe an efficient fixed point algorithm for solving the reduced optimization problems and prove its convergence we then discuss technical details that yield significant running time improvements for large datasets finally we describe various experiments with our approach comparing it to previously studied kernel based methods our experiments indicate that for multiclass problems we attain state of the art accuracy

on the convergence properties of the em algorithm two convergence aspects of the em algorithm are studied i does the em algorithm find a local maximum or a stationary value of the incomplete data likelihood function ii does the sequence of parameter estimates generated by em converge several convergence results are obtained under conditions that are applicable to many practical situations two useful special cases are a if the unobserved complete data specification can be described by a curved exponential family with compact parameter space all the limit points of any em sequence are stationary points of the likelihood function b if the likelihood function is unimodal and a certain differentiability condition is satisfied then any em sequence converges to the unique maximum likelihood estimate a list of key properties of the algorithm is included

a probabilistic and statistical view of fuzzy methods fuzzy set theory has primarily been associated with control theory and with the representation of uncertainty in applications in artificial intelligence more recently fuzzy methods have been proposed as alternatives to traditional statistical methods in statistical quality control linear regression and forecasting among other areas we review some basic concepts of fuzzy methods point out some philosophical and practical problems and offer simpler alternatives based on traditional probability and statistical theory applications in control theory and statistical quality control serve as our primary examples

sparse multinomial kernel discriminant analysis smkda dimensionality reduction via canonical variate analysis cva is important for pattern recognition and has been extended variously to permit more flexibility e g by kernelizing the formulation this can lead to over fitting usually ameliorated by regularization here a method for sparse multinomial kernel discriminant analysis smkda is proposed using a sparse basis to control complexity it is based on the connection between cva and least squares and uses forward selection via orthogonal least squares to approximate a basis generalizing a similar approach for binomial problems classification can be performed directly via minimum mahalanobis distance in the canonical variates smkda achieves state of the art performance in terms of accuracy and sparseness on 11 benchmark datasets

optimality test for generalized fcm and its application to parameter selection in cluster analysis the fuzzy c means fcm clustering algorithm is the best known and most widely used method it was proven to converge to either a local minimum or saddle points by bezdek et al wei and mendel produced efficient optimality tests for fcm fixed points recently a weighting exponent selection for fcm was proposed by yu et al inspired by these results we unify several alternative fcm algorithms into one model called the generalized fuzzy c means gfcm this gfcm model presents a wide variation of fcm algorithms and can easily lead to new and interesting clustering algorithms moreover we construct a general optimality test for gfcm fixed points this is applied to theoretically choose the parameters in the gfcm model the experimental results demonstrate the precision of the theoretical analysis