dissimilarity metric learning in the belief function framework the evidential k nearest neighbor ek nn method provided a global treatment of imperfect knowledge regarding the class membership of training patterns it has outperformed traditional k nn rules in many applications but still shares some of their basic limitations e g 1 classification accuracy depends heavily on how to quantify the dissimilarity between different patterns and 2 no guarantee for satisfactory performance when training patterns contain unreliable imprecise and or uncertain input features in this paper we propose to address these issues by learning a suitable metric using a low dimensional transformation of the input space so as to maximize both the accuracy and efficiency of the ek nn classification to this end a novel loss function to learn the dissimilarity metric is constructed it consists of two terms the first one quantifies the imprecision regarding the class membership of each training pattern while by means of feature selection the second one controls the influence of unreliable input features on the output linear transformation the proposed method has been compared with some other metric learning methods on several synthetic and real datasets it consistently led to comparable performance with regard to testing accuracy and class structure visualization

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

implementing dempster s rule for hierarchial evidence this article gives an algorithm for the exact implementation of dempster s rule in the case of hierarchical evidence this algorithm is computationally efficient and it makes the approximation suggested by gordon and shortliffe unnecessary the algorithm itself is simple but its derivation depends on a detailed understanding of the interaction of hierarchical evidence

inequality systems and minimax theorems abstract this paper is a comprehensive study of the results about the consistency of inequality systems the minimax theorems are the main tools used in proceeding to this survey most of the known results are quoted and some of them are weakened or generalized to infinite dimensional spaces applications to programming are given

ensemble pruning via semi definite programming an ensemble is a group of learning models that jointly solve a problem however the ensembles generated by existing techniques are sometimes unnecessarily large which can lead to extra memory usage computational costs and occasional decreases in effectiveness the purpose of ensemble pruning is to search for a good subset of ensemble members that performs as well as or better than the original ensemble this subset selection problem is a combinatorial optimization problem and thus finding the exact optimal solution is computationally prohibitive various heuristic methods have been developed to obtain an approximate solution however most of the existing heuristics use simple greedy search as the optimization method which lacks either theoretical or empirical quality guarantees in this paper the ensemble subset selection problem is formulated as a quadratic integer programming problem by applying semi definite programming sdp as a solution technique we are able to get better approximate solutions computational experiments show that this sdp based pruning algorithm outperforms other heuristics in the literature its application in a classifier sharing study also demonstrates the effectiveness of the method

feature combination and the knn framework in object classification in object classification feature combination can usually be used to combine the strength of multiple complementary features and produce better classification results than any single one while multiple kernel learning mkl is a popular approach to feature combination in object classification it does not always perform well in practical applications on one hand the optimization process in mkl usually involves a huge consumption of computation and memory space on the other hand in some cases mkl is found to perform no better than the baseline combination methods this observation motivates us to investigate the underlying mechanism of feature combination with average combination and weighted average combination as a result we empirically find that in average combination it is better to use a sample of the most powerful features instead of all whereas in one type of weighted average combination the best classification accuracy comes from a nearly sparse combination we integrate these observations into the k nearest neighbors knns framework based on which we further discuss some issues related to sparse solution and mkl finally by making use of the knn framework we present a new weighted average combination method which is shown to perform better than mkl in both accuracy and efficiency in experiments we believe that the work in this paper is helpful in exploring the mechanism underlying feature combination

on the impact of fusion strategies on classification errors for large ensembles of classifiers the growing availability of sensor networks brings practical situations where a large number of classifiers can be used for building a classifier ensemble in the most general case involving sensor networks the classifiers are fed with multiple inputs collected at different locations however classifier fusion is often studied within an idealized formulation where each classifier is fed with the same point in the feature space and estimate the posterior class probability given this input we first expand this formulation to situations where classifiers are fed with multiple inputs demonstrating the relevance of the formulation to situations involving sensor networks and a large number of classifiers following that we determine the rate of convergence of the classification error of a classifier ensemble for three fusion strategies average median and maximum when the number of classifiers becomes large as the size of the ensemble increases the best strategy is defined as the one that results in fastest convergence of the classification error to zero the best strategy is analytically shown to depend on the distribution of the individual classification errors average is the best for normal distributions maximum is the best for uniform distributions and median is the best for cauchy distributions the general effect of heavy tailedness is also analytically investigated for the average and median strategies the median strategy is shown to be robust to heavy tailedness while performance of the average strategy is shown to degrade as heavy tailedness becomes more pronounced the combined effects of bimodality and heavy tailedness are also investigated when the number of classifiers become large

local classifier weighting by quadratic programming it has been widely accepted that the classification accuracy can be improved by combining outputs of multiple classifiers however how to combine multiple classifiers with various potentially conflicting decisions is still an open problem a rich collection of classifier combination procedures many of which are heuristic in nature have been developed for this goal in this brief we describe a dynamic approach to combine classifiers that have expertise in different regions of the input space to this end we use local classifier accuracy estimates to weight classifier outputs specifically we estimate local recognition accuracies of classifiers near a query sample by utilizing its nearest neighbors and then use these estimates to find the best weights of classifiers to label the query the problem is formulated as a convex quadratic optimization problem which returns optimal nonnegative classifier weights with respect to the chosen objective function and the weights ensure that locally most accurate classifiers are weighted more heavily for labeling the query sample experimental results on several data sets indicate that the proposed weighting scheme outperforms other popular classifier combination schemes particularly on problems with complex decision boundaries hence the results indicate that local classification accuracy based combination techniques are well suited for decision making when the classifiers are trained by focusing on different regions of the input space

bagging for linear classifiers abstract classifiers built on small training sets are usually biased or unstable different techniques exist to construct more stable classifiers it is not clear which ones are good and whether they really stabilize the classifier or just improve the performance in this paper bagging bootstrapping and aggregating l breiman bagging predictors machine learning j 24 2 123 140 1996 is studied for a number of linear classifiers a measure for the instability of classifiers is introduced the influence of regularization and bagging on this instability and the generalization error of linear classifiers is investigated in a simulation study it is shown that in general bagging is not a stabilizing technique it is also demonstrated that one can consider the instability of the classifier to predict how useful bagging will be finally it is shown experimentally that bagging might improve the performance of the classifier only for very unstable situations

drcw ovo one vs one strategy is a common and established technique in machine learning to deal with multi class classification problems it consists of dividing the original multi class problem into easier to solve binary subproblems considering each possible pair of classes since several classifiers are learned their combination becomes crucial in order to predict the class of new instances due to the division procedure a series of difficulties emerge at this stage such as the non competence problem each classifier is learned using only the instances of its corresponding pair of classes and hence it is not competent to classify instances belonging to the rest of the classes nevertheless at classification time all the outputs of the classifiers are taken into account because the competence cannot be known a priori the classification problem would be solved on this account we develop a distance based combination strategy which weights the competence of the outputs of the base classifiers depending on the closeness of the query instance to each one of the classes our aim is to reduce the effect of the non competent classifiers enhancing the results obtained by the state of the art combinations for one vs one strategy we carry out a thorough experimental study supported by the proper statistical analysis showing that the results obtained by the proposed method outperform both in terms of accuracy and kappa measures the previous combinations for one vs one strategy highlightsthe non competence is an important problem in one vs one strategy we develop a distance based combination strategy based on dynamic classifier weighting strategies weights are settled depending on the closeness of the test instance to each one of the classesthe effect of the non competent classifiers is reduced the new strategy enhances the results obtained w r t the state of the art aggregations

decision combination in multiple classifier systems a multiple classifier system is a powerful solution to difficult pattern recognition problems involving large class sets and noisy input because it allows simultaneous use of arbitrary feature descriptors and classification procedures decisions by the classifiers can be represented as rankings of classifiers and different instances of a problem the rankings can be combined by methods that either reduce or rerank a given set of classes an intersection method and union method are proposed for class set reduction three methods based on the highest rank the borda count and logistic regression are proposed for class set reranking these methods have been tested in applications of degraded machine printed characters and works from large lexicons resulting in substantial improvement in overall correctness

analysis of decision boundaries in linearly combined neural classifiers combining or integrating the outputs of several pattern classifiers has led to improved performance in a multitude of applications this paper provides an analytical framework to quantify the improvements in classification results due to combining we show that combining networks linearly in output space reduces the variance of the actual decision region boundaries around the optimum boundary this result is valid under the assumption that the a posteriori probability distributions for each class are locally monotonic around the bayes optimum boundary in the absence of classifier bias the error is shown to be proportional to the boundary variance resulting in a simple expression for error rate improvements in the presence of bias the error reduction expressed in terms of a bias reduction factor is shown to be less than or equal to the reduction obtained in the absence of bias the analysis presented here facilitates the understanding of the relationships among error rates classifier boundary distributions and combining in output space

extended multimodal eigenclassifiers and criteria for fusion model selection diversity among base classifiers is one of the key issues in classifier combination although the eigenclassifiers method proposed by ulas et al 2012 aims to create uncorrelated base classifier outputs however for multiclass classification problems correlation among base classifier outputs arise due to the redundant features in the transformed classifier output space which causes higher estimator variance and lower prediction accuracy in this paper we extend eigenclassifiers method to obtain truly uncorrelated base classifiers we also generalize the distribution on base classifier outputs from unimodal to multimodal which lets us handle the class imbalance problem we also aim to answer the question of which classifier fusion method should be used for a given dataset in order to answer this question we generate a dataset by calculating the performances of ten different fusion methods on 38 different datasets we investigate accuracy diversity relationship of ensembles on this experimental dataset by using eigenvalue distributions and divergence metrics defined by kuncheva and whitaker 2001 we obtain basic rules which can be used to decide on a fusion method given a dataset

combined 5 2 cv f test for comparing supervised classification learning algorithms dietterich 1998 reviews five statistical tests and proposes the 5 2 cv t test for determining whether there is a significant difference between the error rates of two classifiers in our experiments we noticed that the 5 2 cv t test result may vary depending on factors that should not affect the test and we propose a variant the combined 5 2 cv f test that combines multiple statistics to get a more robust test simulation results show that this combined version of the test has lower type i error and higher power than 5 2 cv proper

classifier combination based on confidence transformation this paper investigates the effects of confidence transformation in combining multiple classifiers using various combination rules the combination methods were tested in handwritten digit recognition by combining varying classifier sets the classifier outputs are transformed to confidence measures by combining three scaling functions global normalization gaussian density modeling and logistic regression and three confidence types linear sigmoid and evidence the combination rules include fixed rules sum rule product rule median rule etc and trained rules linear discriminants and weighted combination with various parameter estimation techniques the experimental results justify that confidence transformation benefits the combination performance of either fixed rules or trained rules trained rules mostly outperform fixed rules especially when the classifier set contains weak classifiers among the trained rules the support vector machine with linear kernel linear svm performs best while the weighted combination with optimized weights performs comparably well i have also attempted the joint optimization of confidence parameters and combination weights but its performance was inferior to that of cascaded confidence transformation combination this justifies that the cascaded strategy is a right way of multiple classifier combination

averaging maximum penalized likelihood and bayesian estimation for improving gaussian mixture probability density estimates we apply the idea of averaging ensembles of estimators to probability density estimation in particular we use gaussian mixture models which are important components in many neural network applications we investigate the performance of averaging using three data sets for comparison we employ two traditional regularization approaches i e a maximum penalized likelihood approach and a bayesian approach in the maximum penalized likelihood approach we use penalty functions derived from conjugate bayesian priors such that an expectation maximization em algorithm can be used for training in all experiments the maximum penalized likelihood approach and averaging improved performance considerably if compared to a maximum likelihood approach in two of the experiments the maximum penalized likelihood approach outperformed averaging in one experiment averaging was clearly superior our conclusion is that maximum penalized likelihood gives good results if the penalty term in the cost function is appropriate for the particular problem if this is not the case averaging is superior since it shows greater robustness by not relying on any particular prior assumption the bayesian approach worked very well on a low dimensional toy problem but failed to give good performance in higher dimensional problems

hierarchical mixtures of experts and the em algorithm we present a tree structured architecture for supervised learning the statistical model underlying the architecture is a hierarchical mixture model in which both the mixture coefficients and the mixture components are generalized linear models glim s learning is treated as a maximum likelihood problem in particular we present an expectation maximization em algorithm for adjusting the parameters of the architecture we also develop an on line learning algorithm in which the parameters are updated incrementally comparative simulation results are presented in the robot dynamics domain

the combination of evidence in the transferable belief model a description of the transferable belief model which is used to quantify degrees of belief based on belief functions is given the impact of open and closed world assumption on conditioning is discussed the nature of the frame of discernment on which a degree of belief will be established is discussed a set of axioms justifying dempster s rule for the combination of belief functions induced by two distinct evidences is presented

positive continuous linear functionals on riesz spaces and applications to minimax theorems on deduit des egalites minimax pour des fonctions prenant des valeurs dans un espace topologique de riesz complet ordonne e a partir d egalites minimax faibles et en fin de compte par des theoremes minimax classiques pour des fonctions a valeurs reelles

training a support vector machine in the primal most literature on support vector machines svms concentrates on the dual optimization problem in this letter we point out that the primal problem can also be solved efficiently for both linear and nonlinear svms and that there is no reason for ignoring this possibility on the contrary from the primal point of view new families of algorithms for large scale svm training can be investigated

a theoretical study on six classifier fusion strategies we look at a single point in feature space two classes and l classifiers estimating the posterior probability for class spl omega sub 1 assuming that the estimates are independent and identically distributed normal or uniform we give formulas for the classification error for the following fusion methods average minimum maximum median majority vote and oracle

theoretical bounds of majority voting performance for a binary classification problem a number of earlier studies that have attempted a theoretical analysis of majority voting assume independence of the classifiers we formulate the majority voting problem as an optimization problem with linear constraints no assumptions on the independence of classifiers are made for a binary classification problem given the accuracies of the classifiers in the team the theoretical upper and lower bounds for performance obtained by combining them through majority voting are shown to be solutions of the corresponding optimization problem the objective function of the optimization problem is nonlinear in the case of an even number of classifiers when rejection is allowed for the other cases the objective function is linear and hence the problem is a linear program lp using the framework we provide some insights and investigate the relationship between two candidate classifier diversity measures and majority voting performance

supervised subspace projections for constructing ensembles of classifiers we present a method for constructing ensembles of classifiers using supervised projections of random subspaces the method combines the philosophy of boosting focusing on difficult instances with the improved accuracy achieved by supervised projection methods to obtain very good results in terms of testing error to achieve both accuracy and diversity random subspaces are created at each step and within each random subspace a supervised projection is obtained using only the misclassified instances the next classifier is trained using all available examples in the space given by the supervised projections the method is compared with adaboost and other ensemble methods showing improved performance on a set of 32 problems from the uci machine learning repository in terms of testing error it obtains results that are significantly better than adaboost and random subspace method using a decision tree as base learner furthermore the robustness of the method in the presence of class label noise is above the results obtained with adaboost a study performed using k error diagrams shows that the proposed method improves the results of boosting by obtaining diverse and more accurate classifiers the decomposition of testing error into bias and variance terms shows that our method performs better than bagging in terms of reducing the bias term of the error and better than adaboost in terms of reducing the variance term of the error

inequality systems and minimax results without linear structure inequality systems and their connections with minimax results for real valued functions have been studied in different ways in several papers 1 2 4 5 7 9 133 in particular pomerol 131 takes into consideration the equivalence between the consistency of inequality systems and the resolvability of each inequality of the system it should be remarked that a minimax theorem can be directly derived from this equivalence the aim of this paper is to investigate the relationships between the con sistency of inequality systems and minimax results for functions defined on spaces without linear structure and taking values in a riesz space for real valued functions minimax results without linear structure go back to fan 3 throughout this paper

a feature based robust digital image watermarking scheme a robust digital image watermarking scheme that combines image feature extraction and image normalization is proposed the goal is to resist both geometric distortion and signal processing attacks we adopt a feature extraction method called mexican hat wavelet scale interaction the extracted feature points can survive a variety of attacks and be used as reference points for both watermark embedding and detection the normalized image of an image object is nearly invariant with respect to rotations as a result the watermark detection task can be much simplified when it is applied to the normalized image however because image normalization is sensitive to image local variation we apply image normalization to nonoverlapped image disks separately the disks are centered at the extracted feature points several copies of a 16 bit watermark sequence are embedded in the original image to improve the robustness of watermarks simulation results show that our scheme can survive low quality jpeg compression color reduction sharpening gaussian filtering median filtering row or column removal shearing rotation local warping cropping and linear geometric transformations

randomboost simplified multiclass boosting through randomization we propose a novel boosting approach to multiclass classification problems in which multiple classes are distinguished by a set of random projection matrices in essence the approach uses random projections to alleviate the proliferation of binary classifiers typically required to perform multiclass classification the result is a multiclass classifier with a single vector valued parameter irrespective of the number of classes involved two variants of this approach are proposed the first method randomly projects the original data into new spaces while the second method randomly projects the outputs of learned weak classifiers these methods are not only conceptually simple but also effective and easy to implement a series of experiments on synthetic machine learning and visual recognition data sets demonstrate that our proposed methods could be compared favorably with existing multiclass boosting algorithms in terms of both the convergence rate and classification accuracy

decision templates for multiple classifier fusion an experimental comparison abstract multiple classifier fusion may generate more accurate classification than each of the constituent classifiers fusion is often based on fixed combination rules like the product and average only under strict probabilistic conditions can these rules be justified we present here a simple rule for adapting the class combiner to the application c decision templates one per class are estimated with the same training set that is used for the set of classifiers these templates are then matched to the decision profile of new incoming objects by some similarity measure we compare 11 versions of our model with 14 other techniques for classifier fusion on the satimage and phoneme datasets from the database elena our results show that decision templates based on integral type measures of similarity are superior to the other schemes on both data sets

a theoretical analysis of bagging as a linear combination of classifiers we apply an analytical framework for the analysis of linearly combined classifiers to ensembles generated by bagging this provides an analytical model of bagging misclassification probability as a function of the ensemble size which is a novel result in the literature experimental results on real data sets confirm the theoretical predictions this allows us to derive a novel and theoretically grounded guideline for choosing bagging ensemble size furthermore our results are consistent with explanations of bagging in terms of classifier instability and variance reduction support the optimality of the simple average over the weighted average combining rule for ensembles generated by bagging and apply to other randomization based methods for constructing classifier ensembles although our results do not allow to compare bagging misclassification probability with the one of an individual classifier trained on the original training set we discuss how the considered theoretical framework could be exploited to this aim

classification with nonmetric distances image retrieval and class representation a key problem in appearance based vision is understanding how to use a set of labeled images to classify new images systems that model human performance or that use robust image matching methods often use nonmetric similarity judgments but when the triangle inequality is not obeyed most pattern recognition techniques are not applicable exemplar based nearest neighbor methods can be applied to a wide class of nonmetric similarity functions the key issue however is to find methods for choosing good representatives of a class that accurately characterize it we show that existing condensing techniques are ill suited to deal with nonmetric dataspaces we develop techniques for solving this problem emphasizing two points first we show that the distance between images is not a good measure of how well one image can represent another in nonmetric spaces instead we use the vector correlation between the distances from each image to other previously seen images second we show that in nonmetric spaces boundary points are less significant for capturing the structure of a class than in euclidean spaces we suggest that atypical points may be more important in describing classes we demonstrate the importance of these ideas to learning that generalizes from experience by improving performance we also suggest ways of applying parametric techniques to supervised learning problems that involve a specific nonmetric distance functions showing how to generalize the idea of linear discriminant functions in a way that may be more useful in nonmetric spaces

solving nonstationary classification problems with coupled support vector machines many learning problems may vary slowly over time in particular some critical real world applications when facing this problem it is desirable that the learning method could find the correct input output function and also detect the change in the concept and adapt to it we introduce the time adaptive support vector machine ta svm which is a new method for generating adaptive classifiers capable of learning concepts that change with time the basic idea of ta svm is to use a sequence of classifiers each one appropriate for a small time window but in contrast to other proposals learning all the hyperplanes in a global way we show that the addition of a new term in the cost function of the set of svms that penalizes the diversity between consecutive classifiers produces a coupling of the sequence that allows ta svm to learn as a single adaptive classifier we evaluate different aspects of the method using appropriate drifting problems in particular we analyze the regularizing effect of changing the number of classifiers in the sequence or adapting the strength of the coupling a comparison with other methods in several problems including the well known stagger dataset and the real world electricity pricing domain shows the good performance of ta svm in all tested situations

combining multiple classifiers by averaging or by multiplying abstract in classification tasks it may be wise to combine observations from different sources not only it decreases the training time but it can also increase the robustness and the performance of the classification combining is often done by just weighted averaging of the outputs of the different classifiers using equal weights for all classifiers then results in the mean combination rule this works very well in practice but the combination strategy lacks a fundamental basis as it cannot readily be derived from the joint probabilities this contrasts with the product combination rule which can be obtained from the joint probability under the assumption of independency in this paper we will show differences and similarities between this mean combination rule and the product combination rule in theory and in practice

geometric invariance in image watermarking surviving geometric attacks in image watermarking is considered to be of great importance in this paper the watermark is used in an authentication context two solutions are being proposed for such a problem both geometric and invariant moments are used in the proposed techniques an invariant watermark is designed and tested against attacks performed by stirmark using the invariant moments on the other hand an image normalization technique is also proposed which creates a normalized environment for watermark embedding and detection the proposed algorithms have the advantage of being robust computationally efficient and no overhead needs to be transmitted to the decoder side the proposed techniques have proven to be highly robust to all geometric manipulations filtering compression and slight cropping which are performed as part of stirmark attacks as well as noise addition both gaussian and salt pepper

combinations of weak classifiers to obtain classification systems with both good generalization performance and efficiency in space and time we propose a learning method based on combinations of weak classifiers where weak classifiers are linear classifiers perceptrons which can do a little better than making random guesses a randomized algorithm is proposed to find the weak classifiers they are then combined through a majority vote as demonstrated through systematic experiments the method developed is able to obtain combinations of weak classifiers with good generalization performance and a fast training time on a variety of test problems and real applications theoretical analysis on one of the test problems investigated in our experiments provides insights on when and why the proposed method works in particular when the strength of weak classifiers is properly chosen combinations of weak classifiers can achieve a good generalization performance with polynomial space and time complexity

maximal confidence intervals of the interval valued belief structure and applications an unsolved problem in the interval valued belief structure is the existence of maximal confidence intervals referring to the credible and maximal interval valued beliefs assigned to focal elements in which every point can be reached by a combination rule in this study the existence of maximal confidence intervals of the interval valued belief structure was investigated and validated in addition some applications of the maximal confidence interval in constructing valid and normalized interval valued belief structures are discussed a numerical experiment was conducted to illustrate the existence of maximal confidence intervals several propositions and a theorem were further proposed to demonstrate the validity of the maximal confidence interval by using the concept of the maximal confidence interval a procedure was proposed to construct valid and normalized interval valued belief structures and some meaningful guidance for assigning interval valued beliefs was also suggested to the experts finally a series of well designed examples was presented as applications of the maximal confidence interval the proposed procedure and the suggestions they indicated that valid and normalized interval valued belief structures can be achieved accurately and efficiently in practical applications with the proposed procedure and guidance on the basis of the concept of the maximal confidence interval

multiple instance learning as a classifier combining problem in multiple instance learning mil an object is represented as a bag consisting of a set of feature vectors called instances in the training set the labels of bags are given while the uncertainty comes from the unknown labels of instances in the bags in this paper we study mil with the assumption that instances are drawn from a mixture distribution of the concept and the non concept which leads to a convenient way to solve mil as a classifier combining problem it is shown that instances can be classified with any standard supervised classifier by re weighting the classification posteriors given the instance labels the label of a bag can be obtained as a classifier combining problem an optimal decision rule is derived that determines the threshold on the fraction of instances in a bag that is assigned to the concept class we provide estimators for the two parameters in the model the method is tested on a toy data set and various benchmark data sets and shown to provide results comparable to state of the art mil methods

a bayesian committee machine the bayesian committee machine bcm is a novel approach to combining estimators that were trained on different data sets although the bcm can be applied to the combination of any kind of estimators the main foci are gaussian process regression and related systems such as regularization networks and smoothing splines for which the degrees of freedom increase with the number of training data somewhat surprisingly we find that the performance of the bcm improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator the bcm also provides a new solution for on line learning with potential applications to data mining we apply the bcm to systems with fixed basis functions and discuss its relationship to gaussian process regression finally we show how the ideas behind the bcm can be applied in a non bayesian setting to extend the input dependent combination of estimators

dissimilarity based ensembles for multiple instance learning in multiple instance learning objects are sets bags of feature vectors instances rather than individual feature vectors in this paper we address the problem of how these bags can best be represented two standard approaches are to use dis similarities between bags and prototype bags or between bags and prototype instances the first approach results in a relatively low dimensional representation determined by the number of training bags whereas the second approach results in a relatively high dimensional representation determined by the total number of instances in the training set however an advantage of the latter representation is that the informativeness of the prototype instances can be inferred in this paper a third intermediate approach is proposed which links the two approaches and combines their strengths our classifier is inspired by a random subspace ensemble and considers subspaces of the dissimilarity space defined by subsets of instances as prototypes we provide insight into the structure of some popular multiple instance problems and show state of the art performances on these data sets

an analysis of ensemble pruning techniques based on ordered aggregation several pruning strategies that can be used to reduce the size and increase the accuracy of bagging ensembles are analyzed these heuristics select subsets of complementary classifiers that when combined can perform better than the whole ensemble the pruning methods investigated are based on modifying the order of aggregation of classifiers in the ensemble in the original bagging algorithm the order of aggregation is left unspecified when this order is random the generalization error typically decreases as the number of classifiers in the ensemble increases if an appropriate ordering for the aggregation process is devised the generalization error reaches a minimum at intermediate numbers of classifiers this minimum lies below the asymptotic error of bagging pruned ensembles are obtained by retaining a fraction of the classifiers in the ordered ensemble the performance of these pruned ensembles is evaluated in several benchmark classification tasks under different training conditions the results of this empirical investigation show that ordered aggregation can be used for the efficient generation of pruned ensembles that are competitive in terms of performance and robustness of classification with computationally more costly methods that directly select optimal or near optimal subensembles

mass function derivation and combination in multivariate data spaces in this paper we consider the problem of how to derive mass functions systematically from data samples we also consider the ensuing problem of how to combine different mass functions derived in this way we show that a mass function can be efficiently and systematically derived from multivariate data we also demonstrate that combining mass functions obtained in this manner can be done easily the way of deriving and combining mass functions is illustrated with a simple example

matrix games with payoffs of belief structures imprecise matrix games such as interval valued matrix games and fuzzy matrix games have attracted much interest for a long time most of the previous studies on imprecise matrix games mainly focus on the fuzzy uncertainty of payoffs however the uncertainties of nonspecificity and discord involved in payoffs are not well addressed so far the purpose of this paper is to study the matrix game with such types of uncertainties in order to achieve that purpose we present a matrix game model with payoffs of belief structures so as to integrate discord and nonspecificity the proposed model can be used to express more imprecise interactions between players in the reality besides as another main contribution of the study an effective method for solving matrix games with belief structures payoffs is developed to help us find the equilibrium points of the kind of games at last an example is given to illustrate the proposed model and method

the combination of multiple classifiers using an evidential reasoning approach in many domains when we have several competing classifiers available we want to synthesize them or some of them to get a more accurate classifier by a combination function in this paper we propose a class indifferent method for combining classifier decisions represented by evidential structures called triplet and quartet using dempster s rule of combination this method is unique in that it distinguishes important elements from the trivial ones in representing classifier decisions makes use of more information than others in calculating the support for class labels and provides a practical way to apply the theoretically appealing dempster shafer theory of evidence to the problem of ensemble learning we present a formalism for modelling classifier decisions as triplet mass functions and we establish a range of formulae for combining these mass functions in order to arrive at a consensus decision in addition we carry out a comparative study with the alternatives of simplet and dichotomous structure and also compare two combination methods dempster s rule and majority voting over the uci benchmark data to demonstrate the advantage our approach offers

digital watermarking robust to geometric distortions in this paper we present two watermarking approaches that are robust to geometric distortions the first approach is based on image normalization in which both watermark embedding and extraction are carried out with respect to an image normalized to meet a set of predefined moment criteria we propose a new normalization procedure which is invariant to affine transform attacks the resulting watermarking scheme is suitable for public watermarking applications where the original image is not available for watermark extraction the second approach is based on a watermark resynchronization scheme aimed to alleviate the effects of random bending attacks in this scheme a deformable mesh is used to correct the distortion caused by the attack the watermark is then extracted from the corrected image in contrast to the first scheme the latter is suitable for private watermarking applications where the original image is necessary for watermark detection in both schemes we employ a direct sequence code division multiple access approach to embed a multibit watermark in the discrete cosine transform domain of the image numerical experiments demonstrate that the proposed watermarking schemes are robust to a wide range of geometric attacks

ssc a classifier combination method based on signal strength we propose a new classifier combination method the signal strength based combining ssc approach to combine the outputs of multiple classifiers to support the decision making process in classification tasks as ensemble learning methods have attracted growing attention from both academia and industry recently it is critical to understand the fundamental issues of the combining rule motivated by the signal strength concept our proposed ssc algorithm can effectively integrate the individual vote from different classifiers in an ensemble learning system comparative studies of our method with nine major existing combining rules namely geometric average rule arithmetic average rule median value rule majority voting rule borda count max and min rule weighted average and weighted majority voting rules is presented furthermore we also discuss the relationship of the proposed method with respect to margin based classifiers including the boosting method adaboost m1 and adaboost m2 and support vector machines by margin analysis detailed analyses of margin distribution graphs are presented to discuss the characteristics of the proposed method simulation results for various real world datasets illustrate the effectiveness of the proposed method

constructing ensembles of classifiers by means of weighted instance selection in this paper we approach the problem of constructing ensembles of classifiers from the point of view of instance selection instance selection is aimed at obtaining a subset of the instances available for training capable of achieving at least the same performance as the whole training set in this way instance selection algorithms try to keep the performance of the classifiers while reducing the number of instances in the training set meanwhile boosting methods construct an ensemble of classifiers iteratively focusing each new member on the most difficult instances by means of a biased distribution of the training instances in this work we show how these two methodologies can be combined advantageously we can use instance selection algorithms for boosting using as objective to optimize the training error weighted by the biased distribution of the instances given by the boosting method our method can be considered as boosting by instance selection instance selection has mostly been developed and used for k nearest neighbor k nn classifiers so as a first step our methodology is suited to construct ensembles of k nn classifiers constructing ensembles of classifiers by means of instance selection has the important feature of reducing the space complexity of the final ensemble as only a subset of the instances is selected for each classifier however the methodology is not restricted to k nn classifier other classifiers such as decision trees and support vector machines svms may also benefit from a smaller training set as they produce simpler classifiers if an instance selection algorithm is performed before training in the experimental section we show that the proposed approach is able to produce better and simpler ensembles than random subspace method rsm method for k nn and standard ensemble methods for c4 5 and svms

solving the multiple instance problem with axis parallel rectangles the multiple instance problem arises in tasks where the training examples are ambiguous a single example object may have many alternative feature vectors instances that describe it and yet only one of those feature vectors may be responsible for the observed classification of the object this paper describes and compares three kinds of algorithms that learn axis parallel rectangles to solve the multiple instance problem algorithms that ignore the multiple instance problem perform very poorly an algorithm that directly confronts the multiple instance problem by attempting to identify which feature vectors are responsible for the observed classifications performs best giving 89 correct predictions on a musk odor prediction task the paper also illustrates the use of artificial data to debug and compare these algorithms

efficient k nn based hep 2 cells classifier human epithelial hep 2 cells are commonly used in the indirect immunofluorescence iif tests to detect autoimmune diseases the diagnosis consists of searching and classification to specific patterns created by anti nuclear antibodies anas in the patient serum evaluation of the iif test is mostly done by humans which means that it is highly dependent on the experience and expertise of the physician therefore a significant amount of research has been focused on the development of computer aided diagnostic systems which could help with the analysis of images from microscopes this work deals with the design and development of hep 2 cells classifier the classifier is able to categorize pre segmented images of hep 2 cells into 6 classes the core of this engine consists of the following image descriptors haralick features local binary patterns sift surface description and a granulometry based descriptor these descriptors produce vectors that form metric spaces k nn classification is based on aggregated distance function which combines several features together an extensive set of evaluations was performed on the publicly available mivia hep 2 images dataset which allows a direct comparison of our approach with other solutions the results show that our approach is one of the leading classifiers when comparing with other participants in the hep 2 cells classification contest

elite ensemble of optimal input pruned neural networks using trust tech the ensemble of optimal input pruned neural networks using trust tech elite method for constructing high quality ensemble through an optimal linear combination of accurate and diverse neural networks is developed the optimization problems in the proposed methodology are solved by a global optimization a global optimization method called transformation under stability retraining equilibrium characterization trust tech whose main features include its capability in identifying multiple local optimal solutions in a deterministic systematic and tier by tier manner elite creates a diverse population via a feature selection procedure of different local optimal neural networks obtained using tier 1 trust tech search in addition the capability of each input pruned network is fully exploited through a trust tech based optimal training finally finding the optimal linear combination weights for an ensemble is modeled as a nonlinear programming problem and solved using trust tech and the interior point method where the issue of non convexity can be effectively handled extensive numerical experiments have been carried out for pattern classification on the synthetic and benchmark datasets numerical results show that elite consistently outperforms existing methods on the benchmark datasets the results show that elite can be very promising for constructing high quality neural network ensembles

prototype selection for dissimilarity based classifiers a conventional way to discriminate between objects represented by dissimilarities is the nearest neighbor method a more efficient and sometimes a more accurate solution is offered by other dissimilarity based classifiers they construct a decision rule based on the entire training set but they need just a small set of prototypes the so called representation set as a reference for classifying new objects such alternative approaches may be especially advantageous for non euclidean or even non metric dissimilarities the choice of a proper representation set for dissimilarity based classifiers is not yet fully investigated it appears that a random selection may work well in this paper a number of experiments has been conducted on various metric and non metric dissimilarity representations and prototype selection methods several procedures like traditional feature selection methods here effectively searching for prototypes mode seeking and linear programming are compared to the random selection in general we find out that systematic approaches lead to better results than the random selection especially for a small number of prototypes although there is no single winner as it depends on data characteristics the k centres works well in general for two class problems an important observation is that our dissimilarity based discrimination functions relying on significantly reduced prototype sets 3 10 of the training objects offer a similar or much better classification accuracy than the best k nn rule on the entire training set this may be reached for multi class data as well however such problems are more difficult

on fusers that perform better than best sensor in a multiple sensor system sensor s sub i i 1 2 n outputs y sup i spl isin 0 1 according to an unknown probability distribution p y sup i x in response to input x spl isin 0 1 we choose a fuser that combines the outputs of sensors from a function class f f 0 1 sup n spl rarr 0 1 by minimizing empirical error based on an i i d sample if f satisfies the isolation property we show that the fuser performs at least as well as the best sensor in a probably approximately correct sense several well known fusers such as linear combinations special potential functions and certain feedforward networks satisfy the isolation property

sum versus vote fusion in multiple classifier systems amidst the conflicting experimental evidence of superiority of one over the other we investigate the sum and majority vote combining rules in a two class case under the assumption of experts being of equal strength and estimation errors conditionally independent and identically distributed we show analytically that for gaussian estimation error distributions sum always outperforms vote for heavy tail distributions we demonstrate by simulation that vote may outperform sum results on synthetic data confirm the theoretical predictions experiments on real data support the general findings but also show the effect of the usual assumptions of conditional independence identical error distributions and common target outputs of the experts not being fully satisfied

conjunctive and disjunctive combination of belief functions induced by nondistinct bodies of evidence dempster s rule plays a central role in the theory of belief functions however it assumes the combined bodies of evidence to be distinct an assumption which is not always verified in practice in this paper a new operator the cautious rule of combination is introduced this operator is commutative associative and idempotent this latter property makes it suitable to combine belief functions induced by reliable but possibly overlapping bodies of evidence a dual operator the bold disjunctive rule is also introduced this operator is also commutative associative and idempotent and can be used to combine belief functions issues from possibly overlapping and unreliable sources finally the cautious and bold rules are shown to be particular members of infinite families of conjunctive and disjunctive combination rules based on triangular norms and conorms

rotation scale and translation resilient watermarking for images many electronic watermarks for still images and video content are sensitive to geometric distortions for example simple rotation scaling and or translation rst of an image can prevent blind detection of a public watermark in this paper we propose a watermarking algorithm that is robust to rst distortions the watermark is embedded into a one dimensional 1 d signal obtained by taking the fourier transform of the image resampling the fourier magnitudes into log polar coordinates and then summing a function of those magnitudes along the log radius axis rotation of the image results in a cyclical shift of the extracted signal scaling of the image results in amplification of the extracted signal and translation of the image has no effect on the extracted signal we can therefore compensate for rotation with a simple search and compensate for scaling by using the correlation coefficient as the detection measure false positive results on a database of 10 000 images are reported robustness results on a database of 2000 images are described it is shown that the watermark is robust to rotation scale and translation in addition we describe tests examining the watermarks resistance to cropping and jpeg compression

robust and blind image watermarking in dct domain using inter block coefficient correlation abstract this study presents a robust and transparent watermarking method that exploits block based discrete cosine transform dct coefficient modification the difference in the dct coefficients of two blocks is calculated and modified based on the watermark bit to adjust this difference to a predefined range the first coefficient in the upper left corner of the array basis function is known as the direct current dc coefficient whereas the remainder includes the alternating current ac coefficients the extent of the dct coefficient modifications depends on the dc coefficient and median of the ac coefficients ordered by a zig zag sequence the robustness of the proposed method against various attacks was evaluated experimentally and experimental results demonstrate that the proposed method possesses great robustness against various single and combined attacks

fast computation of krawtchouk moments the paper describes the calculation of the krawtchouk moments kms from an image which is a computationally demanding task we present two original methods that use the outputs of cascaded digital filters in deriving kms the first approach uses the digital filter outputs to form geometric moments gms and the kms are obtained via gms the second method uses a direct relationship to obtain kms from the digital filter outputs this is possible thanks to the formulation of krawtchouk polynomials in terms of binomial functions which are equivalent to the digital filter outputs in this study the performance of the proposed techniques is compared with other existing methods of kms calculation the experimental study shows that the first and the second proposed techniques perform 57 and 87 faster than the recurrence method for a real image of a size 128 128 pixels which performs a significant improvement

an improved method to construct basic probability assignment based on the confusion matrix for classification problem the determination of basic probability assignment bpa is a crucial issue in the application of dempster shafer evidence theory classification is a process of determining the class label that a sample belongs to in classification problem the construction of bpa based on the confusion matrix has been studied however the existing methods do not make full use of the available information provided by the confusion matrix in this paper an improved method to construct the bpa is proposed based on the confusion matrix the proposed method takes into account both the precision rate and the recall rate of each class an illustrative case regarding the prediction of transmembrane protein topology is given to demonstrate the effectiveness of the proposed method

a minimax theorem for functions taking values in a riesz space on demontre des theoremes de minimax pour des fonctions prenant des valeurs dans un espace de riesz on donne une bonne extension de la version en espace de riesz du theoreme de hahn banach classique

geometrically invariant image watermarking using polar harmonic transforms this paper presents an invariant image watermarking scheme by introducing the polar harmonic transform pht which is a recently developed orthogonal moment method similar to zernike moment zm and pseudo zernike moment pzm approaches pht is defined on a circular domain the magnitudes of phts are invariant to image rotation and scaling furthermore the phts are free of numerical instability so they are more suitable for watermarking in this paper the invariant properties of phts are investigated during embedding a subset of the accurate phts are modified according to the binary watermark sequence then a compensation image is formatted by reconstructing the modified pht vector the final watermarked image is obtained by adding the compensation image to the original image in the decoder the watermark can be retrieved from the magnitudes of the phts directly experimental results illustrate that the proposed scheme outperforms zm pzm based schemes in terms of embedding capacity and watermark robustness and is also robust to both geometric and signal processing based attacks

an improved aqim watermarking method with minimum distortion angle quantization and amplitude projection strategy about ten years ago angle qim aqim was proposed by ourique et al for the purpose of resisting gain attacks later to improve aqim nezhadarya et al proposed absolute aqim aaqim and further applied aaqim to their gradient based watermarking framework both of aqim and aaqim have the ability in principle to resist gain attacks but neither of them can achieve optimal performance in terms of angular distortion in this paper we propose an improved aqim watermarking method iaqim with minimum distortion angle quantization and amplitude projection strategy iaqim can not only obtain an optimal performance in terms of angular distortion by using the minimum distortion angle quantization but also further reduce the embedding distortion of host signals with the amplitude projection strategy a theoretical analysis is presented for the performance of the proposed method experiments have been conducted to validate the theoretical analysis and to demonstrate the superiority of the proposed iaqim watermarking method over the existing gain invariant watermarking methods including aqim and aaqim

experimental study for the comparison of classifier combination methods in this paper we compare the performances of classifier combination methods bagging modified random subspace method classifier selection parametric fusion to logistic regression in consideration of various characteristics of input data four factors used to simulate the logistic model are a combination function among input variables b correlation between input variables c variance of observation and d training data set size in view of typically unknown combination function among input variables we use a taguchi design to improve the practicality of our study results by letting it as an uncontrollable factor our experimental study results indicate the following when training set size is large performances of logistic regression and bagging are not significantly different however when training set size is small the performance of logistic regression is worse than bagging when training data set size is small and correlation is strong both modified random subspace method and bagging perform better than the other three methods when correlation is weak and variance is small both parametric fusion and classifier selection algorithm appear to be the worst at our disappointment

from cluster ensemble to structure ensemble this paper investigates the problem of integrating multiple structures which are extracted from different sets of data points into a single unified structure we first propose a new generalized concept called structure ensemble for the fusion of multiple structures unlike traditional cluster ensemble approaches the main objective of which is to align individual labels obtained from different clustering solutions the structure ensemble approach focuses on how to unify the structures obtained from different data sources based on this framework a new structure ensemble approach called the probabilistic bagging based structure ensemble approach bsea is designed which integrates the bagging technique the force based self organizing map fbsom and the normalized cut algorithm into the proposed framework bsea views structures obtained from different datasets generated by the bagging technique as nodes in a graph and adopts graph theory to find the most representative structure in addition the force based self organizing map fbsom which is a generalized form of som is proposed to serve as the basic clustering algorithm in the structure ensemble framework finally a new external index called correlation index ci which considers the correlation relationship of both the similarity and dissimilarity between the predicted solution and the true solution is proposed to evaluate the performance of bsea the experiments show that i the performance of bsea outperforms most of the state of the art clustering approaches and ii bsea performs well on datasets from the uci repository and real cancer gene expression profiles

robust watermarking against geometric attacks using partial calculation of radial moments and interval phase modulation this paper presents a new image watermarking scheme which is blind and robust against geometric attacks the proposed algorithm is based on the features of radial moments in this algorithm the radial moments were first computed and then the watermark was embedded into the differential phase of various blocks of the original image using a special type of psk modulation called interval phase modulation ipm in order to embed the watermark into the phase of moments a particular method namely the partial calculation of moments was utilized for computation the implementation results show the improvement in the robustness and quality of the watermarked images in comparison with other methods

combining feature spaces for classification in this paper we offer a variational bayes approximation to the multinomial probit model for basis expansion and kernel combination our model is well founded within a hierarchical bayesian framework and is able to instructively combine available sources of information for multinomial classification the proposed framework enables informative integration of possibly heterogeneous sources in a multitude of ways from the simple summation of feature expansions to weighted product of kernels and it is shown to match and in certain cases outperform the well known ensemble learning approaches of combining individual classifiers at the same time the approximation reduces considerably the cpu time and resources required with respect to both the ensemble learning methods and the full markov chain monte carlo metropolis hastings within gibbs solution of our model we present our proposed framework together with extensive experimental studies on synthetic and benchmark datasets and also for the first time report a comparison between summation and product of individual kernels as possible different methods for constructing the composite kernel matrix

the random subspace method for constructing decision forests much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes the dilemma between overfitting and achieving maximum accuracy is seldom resolved a method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity the classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector that is trees constructed in randomly chosen subspaces the subspace method is compared to single tree classifiers and other forest construction methods by experiments on publicly available datasets where the method s superiority is demonstrated we also discuss independence between trees in a forest and relate that to the combined classification accuracy

averaging regularized estimators we compare the performance of averaged regularized estimators we show that the improvement in performance that can be achieved by averaging depends critically on the degree of regularization which is used in training the individual estimators we compare four different averaging approaches simple averaging bagging variance based weighting and variance based bagging in any of the averaging methods the greatest degree of improvement if compared to the individual estimators is achieved if no or only a small degree of regularization is used here variance based weighting and variance based bagging are superior to simple averaging or bagging our experiments indicate that better performance for both individual estimators and for averaging is achieved in combination with regularization with increasing degrees of regularization the two bagging based approaches bagging and variance based bagging outperform the individual estimators simple averaging and variance based weighting bagging and variance

incremental construction of classifier and discriminant ensembles we discuss approaches to incrementally construct an ensemble the first constructs an ensemble of classifiers choosing a subset from a larger set and the second constructs an ensemble of discriminants where a classifier is used for some classes only we investigate criteria including accuracy significant improvement diversity correlation and the role of search direction for discriminant ensembles we test subset selection and trees fusion is by voting or by a linear model using 14 classifiers on 38 data sets incremental search finds small accurate ensembles in polynomial time the discriminant ensemble uses a subset of discriminants and is simpler interpretable and accurate we see that an incremental ensemble has higher accuracy than bagging and random subspace method and it has a comparable accuracy to adaboost but fewer classifiers

efficient sparse generalized multiple kernel learning kernel methods have been successfully applied in various applications to succeed in these applications it is crucial to learn a good kernel representation whose objective is to reveal the data similarity precisely in this paper we address the problem of multiple kernel learning mkl searching for the optimal kernel combination weights through maximizing a generalized performance measure most mkl methods employ the norm simplex constraints on the kernel combination weights which therefore involve a sparse but non smooth solution for the kernel weights despite the success of their efficiency they tend to discard informative complementary or orthogonal base kernels and yield degenerated generalization performance alternatively imposing the norm constraint on the kernel weights will keep all the information in the base kernels this leads to non sparse solutions and brings the risk of being sensitive to noise and incorporating redundant information to tackle these problems we propose a generalized mkl gmkl model by introducing an elastic net type constraint on the kernel weights more specifically it is an mkl model with a constraint on a linear combination of the norm and the squared norm on the kernel weights to seek the optimal kernel combination weights therefore previous mkl problems based on the norm or the norm constraints can be regarded as special cases furthermore our gmkl enjoys the favorable sparsity property on the solution and also facilitates the grouping effect moreover the optimization of our gmkl is a convex optimization problem where a local solution is the global optimal solution we further derive a level method to efficiently solve the optimization problem a series of experiments on both synthetic and real world datasets have been conducted to show the effectiveness and efficiency of our gmkl