discriminative face alignment this paper proposes a discriminative framework for efficiently aligning images although conventional active appearance models aams based approaches have achieved some success they suffer from the generalization problem i e how to align any image with a generic model we treat the iterative image alignment problem as a process of maximizing the score of a trained two class classifier that is able to distinguish correct alignment positive class from incorrect alignment negative class during the modeling stage given a set of images with ground truth landmarks we train a conventional point distribution model pdm and a boosting based classifier which acts as an appearance model when tested on an image with the initial landmark locations the proposed algorithm iteratively updates the shape parameters of the pdm via the gradient ascent method such that the classification score of the warped image is maximized we use the term boosted appearance models bams to refer to the learned shape and appearance models as well as our specific alignment method the proposed framework is applied to the face alignment problem using extensive experimentation we show that compared to the aam based approach this framework greatly improves the robustness accuracy and efficiency of face alignment by a large margin especially for unseen data

complexity in the case against accuracy estimation some authors have repeatedly pointed out that the use of the accuracy in particular for comparing classifiers is not adequate the main argument concerns some assumptions of seldom validity or correctness underlying the use of this criterion in this paper we study the computational burden of the accuracy s replacement for building and comparing classifiers using the framework of inductive logic programming replacement is investigated in three ways completion of the accuracy with an additional requirement replacement of the accuracy with a bi criterion recently introduced from statistical decision theory the receiver operating characteristic analysis and replacement of the accuracy by a single criterion we prove very hard results for most of the possible replacements a first result shows that allowing the arbitrary multiplication of clauses appears to be totally useless arbitrary is to be taken in its broadest meaning in particular exponential the second point is the sudden appearance of the negative result which is not a function of the criteria s demands the third point is the equivalence in difficulty of all these different criteria in contrast the single accuracy s optimization appears to be tractable in this framework

two stage orthogonal least squares methods for neural network construction a number of neural networks can be formulated as the linear in the parameters models training such networks can be transformed to a model selection problem where a compact model is selected from all the candidates using subset selection algorithms forward selection methods are popular fast subset selection approaches however they may only produce suboptimal models and can be trapped into a local minimum more recently a two stage fast recursive algorithm tsfra combining forward selection and backward model refinement has been proposed to improve the compactness and generalization performance of the model this paper proposes unified two stage orthogonal least squares methods instead of the fast recursive based methods in contrast to the tsfra this paper derives a new simplified relationship between the forward and the backward stages to avoid repetitive computations using the inherent orthogonal properties of the least squares methods furthermore a new term exchanging scheme for backward model refinement is introduced to reduce computational demand finally given the error reduction ratio criterion effective and efficient forward and backward subset selection procedures are proposed extensive examples are presented to demonstrate the improved model compactness constructed by the proposed technique in comparison with some popular methods

polygonal approximation using genetic algorithms a polygon approximation method based on genetic algorithms is proposed in this paper in the method a chromosome is used to represent a polygon and is represented by a binary string each bit called a gene represents a point on the object curve the objective function is defined as the integral square error between the given curve and the approximated polygon three genetic operators namely selection crossover and mutation have been constructed for this specific problem the proposed method when compared with three existing methods can obtain superior approximation results with less error norm with respect to the original curves

polygonal approximation of closed discrete curves optimal polygonal approximation of closed curves differs from the case of open curve in the sense that the location of the starting point must also be determined straightforward exhaustive search would take n times more time than the corresponding optimal algorithm for an open curve because there are n possible points to be considered as the starting point faster sub optimal solution can be found by iterating the search and heuristically selecting different starting point at each iteration in this paper we propose to find the optimal approximation of a cyclically extended closed curve of double size and to select the best possible starting point by search in the extended search space for the curve the proposed approach provides solution very close to the optimal one using at most twice as much time as required by the optimal algorithm for the corresponding open curve

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

a discrete geometry approach for dominant point detection we propose two fast methods for dominant point detection and polygonal representation of noisy and possibly disconnected curves based on a study of the decomposition of the curve into the sequence of maximal blurred segments 2 starting from results of discrete geometry 3 4 the notion of maximal blurred segment of width n 2 has been proposed well adapted to possibly noisy curves the first method uses a fixed parameter that is the width of considered maximal blurred segments the second method is deduced from the first one based on a multi width approach to obtain a non parametric method that uses no threshold for working with noisy curves comparisons with other methods in the literature prove the efficiency of our approach thanks to a recent result 5 concerning the construction of the sequence of maximal blurred segments the complexity of the proposed methods is o nlogn an application of vectorization is also given in this paper

hubs in space popular nearest neighbors in high dimensional data different aspects of the curse of dimensionality are known to present serious challenges to various machine learning methods and tasks this paper explores a new aspect of the dimensionality curse referred to as hubness that affects the distribution of k occurrences the number of times a point appears among the k nearest neighbors of other points in a data set through theoretical and empirical analysis involving synthetic and real data sets we show that under commonly used assumptions this distribution becomes considerably skewed as dimensionality increases causing the emergence of hubs that is points with very high k occurrences which effectively represent popular nearest neighbors we examine the origins of this phenomenon showing that it is an inherent property of data distributions in high dimensional vector space discuss its interaction with dimensionality reduction and explore its influence on a wide range of machine learning tasks directly or indirectly based on measuring distances belonging to supervised semi supervised and unsupervised learning families

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

robust estimation for the varying coefficient partially nonlinear models abstract in this paper we propose a robust estimation procedure based on the exponential squared loss esl function for the varying coefficient partially nonlinear model under some conditions the asymptotic properties of proposed estimators are established furthermore we develop a new minorization maximization mm algorithm to calculate the estimates for both non parametric and parametric parts and introduce a data driven procedure to select the tuning parameters simulation studies illustrate that the proposed method is more robust and efficient than the classical least squares technique when there are outliers in the dataset finally we apply the proposed methodology to analyze a real dataset the results reveal that the proposed has better the predictive ability

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

logistic evolutionary product unit neural networks innovation capacity of poor guatemalan households a new logistic regression algorithm based on evolutionary product unit pu neural networks is used in this paper to determine the assets that influence the decision of poor households with respect to the cultivation of non traditional crops ntc in the guatemalan highlands in order to evaluate high order covariate interactions pus were considered to be independent variables in product unit neural networks punn analysing two different models either including the initial covariates logistic regression by the product unit and initial covariate model or not logistic regression by the product unit model our results were compared with those obtained using a standard logistic regression model and allow us to interpret the most relevant household assets and their complex interactions when adopting ntc in order to aid in the design of rural policies

polygonal approximation of digital planar curves through vertex betweenness contour polygonal approximation is usually defined as a set of selected points which describes a polygon and best represents the original contour this paper presents a novel graph based approach to compute a polygonal approximation of a shape contour in a graph such points correspond to a high transitivity region of the graph we use the vertex betweenness to measure the importance of each vertice in a graph according to the number of shortest paths where each vertice occurs by selecting the vertices with higher vertex betweenness a polygon which retains the main characteristics of the contour is achieved by using benchmark curves a comparative experiment with other commonly used algorithms is presented results indicate that the proposed approach produced efficient and effective polygonal approximations for digital planar curves

a boosting approach for supervised mahalanobis distance metric learning determining a proper distance metric is often a crucial step for machine learning in this paper a boosting algorithm is proposed to learn a mahalanobis distance metric similar to most boosting algorithms the proposed algorithm improves a loss function iteratively in particular the loss function is defined in terms of hypothesis margins and a metric matrix base learner specific to the boosting framework is also proposed experimental results show that the proposed approach can yield effective mahalanobis distance metrics for a variety of data sets and demonstrate the feasibility of the proposed approach

piecewise linear approximation of plane curves by perimeter optimization abstract a method for the piecewise linear approximation of a plane curve is described an approximate curve is obtained by choosing a certain number of points from a set of sampled points of the original curve a point choice function that represents the relation between the original points and the chosen points is formulated the approximation is performed by the dynamic programming principle searching for the optimal point choice function that attains the minimal error about arc length of the curve evaluation of the global approximate error provides efficient curve approximations irrespective of shape complexity number of sampled points and irregularity of sampling interval

a novel training algorithm for rbf neural network using a hybrid fuzzy clustering approach this paper introduces a novel clustering based algorithm to train gaussian type radial basis function neural networks in contrast to existing approaches we develop a specialized learning strategy that combines the merits of fuzzy and crisp clustering crisp clustering is a fast process yet very sensitive to initialization on the other hand fuzzy clustering reduces the dependency on initialization however it constitutes a slow learning process the proposed strategy aims to search for a trade off among these two potentially different effects the produced clusters possess fuzzy and crisp areas and therefore the final result is a hybrid partition where the fuzzy and crisp conditions coexist the hybrid clusters are directly involved in the estimation process of the neural network s parameters specifically the center elements of the basis functions coincide with cluster centers while the respective widths are calculated by taking into account the topology of the hybrid clusters to this end the network s design becomes a fast and efficient procedure the proposed method is successfully applied to a number of experimental cases where the produced networks prove to be highly accurate and compact in size

a hybrid learning algorithm combined with generalized rls approach for radial basis function neural networks in this paper a new hybrid learning method for radial basis function neural networks based on generalized recursive least square algorithm is proposed firstly the generalized recursive least square grls model including a general quadratic weight decay term in the energy function for the training of rbf neural networks is described then combined with the grls approach a new hybrid learning method is proposed to meet the design goals improving the generalization ability of the trained network finally experimental results demonstrate that our approach can achieve a significantly improved generalization performance of the rbf networks

a fast nonlinear model identification method the identification of nonlinear dynamic systems using linear in the parameters models is studied a fast recursive algorithm fra is proposed to select both the model structure and to estimate the model parameters unlike orthogonal least squares ols method fra solves the least squares problem recursively over the model order without requiring matrix decomposition the computational complexity of both algorithms is analyzed along with their numerical stability the new method is shown to require much less computational effort and is also numerically more stable than ols

a constructive hybrid structure optimization methodology for radial basis probabilistic neural networks in this paper a novel heuristic structure optimization methodology for radial basis probabilistic neural networks rbpnns is proposed first a minimum volume covering hyperspheres mvch algorithm is proposed to select the initial hidden layer centers of the rbpnn and then the recursive orthogonal least square algorithm rolsa combined with the particle swarm optimization pso algorithm is adopted to further optimize the initial structure of the rbpnn the proposed algorithms are evaluated through eight benchmark classification problems and two real world application problems a plant species identification task involving 50 plant species and a palmprint recognition task experimental results show that our proposed algorithm is feasible and efficient for the structure optimization of the rbpnn the rbpnn achieves higher recognition rates and better classification efficiency than multilayer perceptron networks mlpns and radial basis function neural networks rbfnns in both tasks moreover the experimental results illustrated that the generalization performance of the optimized rbpnn in the plant species identification task was markedly better than that of the optimized rbfnn

on detecting dominant points abstract detecting dominant points is a crucial preprocessing step for shape recognition and point based motion estimation polygonal approximation has been a commonly used approach in detecting dominant points this paper presents two alternatives which detect stable dominant points in the first method we find a set of positive maximum and negative minimum curvature points along the gaussian smoothed boundary followed by a split and merge polygonal approximation algorithm the resulting break points vertices of the approximated polygon are the dominant points experimental results show that dominant points obtained by this method are less sensitive to the orientation of the boundary than other polygonal approximation algorithms in the sense that the number and the location of the dominant points along the contour remain relatively unchanged in the second method we smooth a boundary by a gaussian filter using various widths until the extreme curvature points remain relatively unchanged for a range of filter widths the resulting extreme curvature points which are stable to orientation and a reasonable range of scaling are the dominant points

ant colony search algorithms for optimal polygonal approximation of plane curves this paper presents a new polygonal approximation method using ant colony search algorithm the problem is represented by a directed graph such that the objective of the original problem becomes to find the shortest closed circuit on the graph under the problem specific constraints a number of artificial ants are distributed on the graph and communicate with one another through the pheromone trails which are a form of the long term memory guiding the future exploration of the graph the important properties of the proposed method are thoroughly investigated the performance of the proposed method as compared to those of the genetic based and the tabu search based approaches is very promising

self improved gaps almost everywhere for the agnostic approximation of monomials given a learning sample we focus on the hardness of finding monomials having low error inside the interval bounded below by the smallest error achieved by a monomial the best rule and bounded above by the error of the default class the poorest rule it is well known that when its lower bound is zero it is an easy task to find in linear time a monomial with zero error what we prove is that when this bound is not zero regardless of the location of the default class in 0 1 2 it becomes a huge complexity burden to beat significantly the default class in fact under some complexity theoretical assumptions it may already be hard to beat the trivial approximation ratios even when relaxing the time complexity constraint to be quasi polynomial or sub exponential our results also hold with uniform weights over the examples

margin based ranking and an equivalence between adaboost and rankboost we study boosting algorithms for learning to rank we give a general margin based bound for ranking based on covering numbers for the hypothesis space our bound suggests that algorithms that maximize the ranking margin will generalize well we then describe a new algorithm smooth margin ranking that precisely converges to a maximum ranking margin solution the algorithm is a modification of rankboost analogous to approximate coordinate ascent boosting finally we prove that adaboost and rankboost are equally good for the problems of bipartite ranking and classification in terms of their asymptotic behavior on the training set under natural conditions adaboost achieves an area under the roc curve that is equally as good as rankboost s furthermore rankboost when given a specific intercept achieves a misclassification error that is as good as adaboost s this may help to explain the empirical observations made by cortes and mohri and caruana and niculescu mizil about the excellent performance of adaboost as a bipartite ranking algorithm as measured by the area under the roc curve

boosted classification trees and class probability quantile estimation the standard by which binary classifiers are usually judged misclassification error assumes equal costs of misclassifying the two classes or equivalently classifying at the 1 2 quantile of the conditional class probability function p y 1 x boosted classification trees are known to perform quite well for such problems in this article we consider the use of standard off the shelf boosting for two more general problems 1 classification with unequal costs or equivalently classification at quantiles other than 1 2 and 2 estimation of the conditional class probability function p y 1 x we first examine whether the latter problem estimation of p y 1 x can be solved with logitboost and with adaboost when combined with a natural link function the answer is negative both approaches are often ineffective because they overfit p y 1 x even though they perform well as classifiers a major negative point of the present article is the disconnect between class probability estimation and classification next we consider the practice of over under sampling of the two classes we present an algorithm that uses adaboost in conjunction with over under sampling and jittering of the data jous boost this algorithm is simple yet successful and it preserves the advantage of relative protection against overfitting but for arbitrary misclassification costs and equivalently arbitrary quantile boundaries we then use collections of classifiers obtained from a grid of quantiles to form estimators of class probabilities the estimates of the class probabilities compare favorably to those obtained by a variety of methods across both simulated and real data sets

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

on equivalence relationships between classification and ranking algorithms we demonstrate that there are machine learning algorithms that can achieve success for two separate tasks simultaneously namely the tasks of classification and bipartite ranking this means that advantages gained from solving one task can be carried over to the other task such as the ability to obtain conditional density estimates and an order of magnitude reduction in computational time for training the algorithm it also means that some algorithms are robust to the choice of evaluation metric used they can theoretically perform well when performance is measured either by a misclassification error or by a statistic of the roc curve such as the area under the curve specifically we provide such an equivalence relationship between a generalization of freund et al s rankboost algorithm called the p norm push and a particular cost sensitive classification algorithm that generalizes adaboost which we call p classification we discuss and validate the potential benefits of this equivalence relationship and perform controlled experiments to understand p classification s empirical performance there is no established equivalence relationship for logistic regression and its ranking counterpart so we introduce a logistic regression style algorithm that aims in between classification and ranking and has promising experimental performance with respect to both tasks

determination of optimal polygon from digital curve using l1 norm abstract a technique to determine optimal polygons of digital curves is proposed it determines the longest possible line segments with the minimum possible error the l 1 norm is used to measure the closeness of a polygon to a digital curve the algorithm runs in linear time the procedure is applied to several digital curves and promising results are obtained

a unified classification model based on robust optimization a wide variety of machine learning algorithms such as the support vector machine svm minimax probability machine mpm and fisher discriminant analysis fda exist for binary classification the purpose of this letter is to provide a unified classification model that includes these models through a robust optimization approach this unified model has several benefits one is that the extensions and improvements intended for svms become applicable to mpm and fda and vice versa for example we can obtain nonconvex variants of mpm and fda by mimicking perez cruz weston hermann and scholkopf s 2003 extension from convex i svm to nonconvex ei svm another benefit is to provide theoretical results concerning these learning methods at once by dealing with the unified model we give a statistical interpretation of the unified classification model and prove that the model is a good approximation for the worst case minimization of an expected loss with respect to the uncertain probability distribution we also propose a nonconvex optimization algorithm that can be applied to nonconvex variants of existing learning methods and show promising numerical results

a convergent incremental gradient method with a constant step size an incremental aggregated gradient method for minimizing a sum of continuously differentiable functions is presented the method requires a single gradient evaluation per iteration and uses a constant step size for the case that the gradient is bounded and lipschitz continuous we show that the method visits infinitely often regions in which the gradient is small under certain unimodality assumptions global convergence is established in the quadratic case a global linear rate of convergence is shown the method is applied to distributed optimization problems arising in wireless sensor networks and numerical experiments compare the new method with other incremental gradient methods

on the detection of dominant points on digital curves a parallel algorithm is presented for detecting dominant points on a digital closed curve the procedure requires no input parameter and remains reliable even when features of multiple sizes are present on the digital curve the procedure first determines the region of support for each point based on its local properties then computes measures of relative significance e g curvature of each point and finally detects dominant points by a process of nonmaximum suppression this procedure leads to the observation that the performance of dominant points detection depends not only on the accuracy of the measure of significance but also on the precise determination of the region of support this solves the fundamental problem of scale factor selection encountered in various dominant point detection algorithms the inherent nature of scale space filtering in the procedure is addressed and the performance of the procedure is compared to those of several other dominant point detection algorithms using a number of examples

fast polygonal approximation of digital curves using relaxed straightness properties several existing digital straight line segment dss recognition algorithms can be used to determine the digital straightness of a given one pixel thick digital curve because of the inherent geometric constraints of digital straightness these algorithms often produce a large number of segments to cover a given digital curve representing a real life object image thus a curve segment which is not exactly digitally straight but appears to be visually straight is fragmented into multiple dss when these algorithms are run in this paper a new concept of approximate straightness is introduced by relaxing certain conditions of dss and an algorithm is described to extract those segments from a digital curve the number of such segments required to cover the curve is found to be significantly fewer than that of the exact dss cover as a result the data set required for representing a curve also reduces to a large extent the extracted set of segments can further be combined to determine a compact polygonal approximation of a digital curve based on certain approximation criteria and a specified error tolerance the proposed algorithm involves only primitive integer operations and thus runs very fast compared to those based on exact dss the overall time complexity becomes linear in the number of points present in the representative set experimental results on several digital curves demonstrate the speed elegance and efficacy of the proposed method

grafting fast incremental feature selection by gradient descent in function space we present a novel and flexible approach to the problem of feature selection called grafting rather than considering feature selection as separate from learning grafting treats the selection of suitable features as an integral part of learning a predictor in a regularized learning framework to make this regularized learning process sufficiently fast for large scale problems grafting operates in an incremental iterative fashion gradually building up a feature set while training a predictor model using gradient descent at each iteration a fast gradient based heuristic is used to quickly assess which feature is most likely to improve the existing model that feature is then added to the model and the model is incrementally optimized using gradient descent the algorithm scales linearly with the number of data points and at most quadratically with the number of features grafting can be used with a variety of predictor model classes both linear and non linear and can be used for both classification and regression experiments are reported here on a variant of grafting for classification using both linear and non linear models and using a logistic regression inspired loss function results on a variety of synthetic and real world data sets are presented finally the relationship between grafting stagewise additive modelling and boosting is explored

probability estimation for multi class classification using adaboost abstract it is a general viewpoint that adaboost classifier has excellent performance on classification problems but could not produce good probability estimations in this paper we put forward a theoretical analysis of probability estimation model and present some verification experiments which indicate that adaboost can be used for probability estimation with the theory we suggest some useful measures for using adaboost algorithms properly and then we deduce a probability estimation model for multi class classification by pairwise coupling unlike previous approximate methods we provide an analytical solution instead of a special iterative procedure moreover a new problem that how to get a robust prediction with classifier scores is proposed experiments show that the traditional predict framework which chooses one with the highest score from all classes as the prediction is not always good while our model performs well

bipartite ranking a risk theoretic perspective we present a systematic study of the bipartite ranking problem with the aim of explicating its connections to the class probability estimation problem our study focuses on the properties of the statistical risk for bipartite ranking with general losses which is closely related to a generalised notion of the area under the roc curve we establish alternate representations of this risk relate the bayes optimal risk to a class of probability divergences and characterise the set of bayes optimal scorers for the risk we further study properties of a generalised class of bipartite risks based on the p norm push of rudin 2009 our analysis is based on the rich framework of proper losses which are the central tool in the study of class probability estimation we show how this analytic tool makes transparent the generalisations of several existing results such as the equivalence of the minimisers for four seemingly disparate risks from bipartite ranking and class probability estimation a novel practical implication of our analysis is the design of new families of losses for scenarios where accuracy at the head of ranked list is paramount with comparable empirical performance to the p norm push

optimum uniform piecewise linear approximation of planar curves two dimensional digital curves are often uniformly approximated by polygons or piecewise linear curves several algorithms have been proposed in the literature to find such curves we present an algorithm that finds a piecewise linear curve with the minimal number of segments required to approximate a curve within a uniform error with fixed initial and final points we compare our optimal algorithm to several suboptimal algorithms with respect to the number of linear segments required in the approximation and the execution time of the algorithm

data reduction of large vector graphics fast algorithm for joint near optimal approximation of multiple polygonal curves is proposed it is based on iterative reduced search dynamic programming introduced earlier for the min eproblem of a single polygonal curve the proposed algorithm jointly optimizes the number of line segments allocated to the different individual curves and the approximation of the curves by the given number of segments trade off between time and optimality is controlled by the breadth of the search and by the numbers of iterations applied

geometry based ensembles toward a structural characterization of the classification boundary this article introduces a novel binary discriminative learning technique based on the approximation of the non linear decision boundary by a piece wise linear smooth additive model the decision border is geometrically defined by means of the characterizing boundary points points that belong to the optimal boundary under a certain notion of robustness based on these points a set of locally robust linear classifiers is defined and assembled by means of a tikhonov regularized optimization procedure in an additive model to create a final lambda smooth decision rule as a result a very simple and robust classifier with a strong geometrical meaning and non linear behavior is obtained the simplicity of the method allows its extension to cope with some of nowadays machine learning challenges such as online learning large scale learning or parallelization with linear computational complexity we validate our approach on the uci database finally we apply our technique in online and large scale scenarios and in six real life computer vision and pattern recognition problems gender recognition intravascular ultrasound tissue classification speed traffic sign detection chagas disease severity detection clef classification and action recognition using a 3d accelerometer data the results are promising and this paper opens a line of research that deserves further attention

the p norm push a simple convex ranking algorithm that concentrates at the top of the list we are interested in supervised ranking algorithms that perform especially well near the top of the ranked list and are only required to perform sufficiently well on the rest of the list in this work we provide a general form of convex objective that gives high scoring examples more importance this push near the top of the list can be chosen arbitrarily large or small based on the preference of the user we choose lp norms to provide a specific type of push if the user sets p larger the objective concentrates harder on the top of the list we derive a generalization bound based on the p norm objective working around the natural asymmetry of the problem we then derive a boosting style algorithm for the problem of ranking with a push at the top the usefulness of the algorithm is illustrated through experiments on repository data we prove that the minimizer of the algorithm s objective is unique in a specific sense furthermore we illustrate how our objective is related to quality measurements for information retrieval

a new image thresholding method based on gaussian mixture model in this paper an efficient approach to search for the global threshold of image using gaussian mixture model is proposed firstly a gray level histogram of an image is represented as a function of the frequencies of gray level then to fit the gaussian mixtures to the histogram of image the expectation maximization em algorithm is developed to estimate the number of gaussian mixture of such histograms and their corresponding parameterization finally the optimal threshold which is the average of these gaussian mixture means is chosen and the experimental results show that the new algorithm performs better

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

a fast genetic method for inducting descriptive fuzzy models under certain inference mechanisms fuzzy rule bases can be regarded as extended additive models this relationship can be applied to extend some statistical techniques to learn fuzzy models from data the interest in this parallelism is twofold theoretical and practical first extended additive models can be estimated by means of the matching pursuit algorithm which has been related to support vector machines boosting and radial basis neural networks learning this connection can be exploited to better understand the learning of fuzzy models in particular the technique we propose here can be regarded as the counterpart to boosting fuzzy classifiers in the field of fuzzy modeling second since matching pursuit is very efficient in time we can expect to obtain faster algorithms to learn fuzzy rules from data we show that the combination of a genetic algorithm and the backfitting process learns faster than ad hoc methods in certain datasets

on robustness properties of convex risk minimization methods for pattern recognition the paper brings together methods from two disciplines machine learning theory and robust statistics we argue that robustness is an important aspect and we show that many existing machine learning methods based on the convex risk minimization principle have besides other good properties also the advantage of being robust robustness properties of machine learning methods based on convex risk minimization are investigated for the problem of pattern recognition assumptions are given for the existence of the influence function of the classifiers and for bounds on the influence function kernel logistic regression support vector machines least squares and the adaboost loss function are treated as special cases some results on the robustness of such methods are also obtained for the sensitivity curve and the maxbias which are two other robustness criteria a sensitivity analysis of the support vector machine is given

robust variable selection with exponential squared loss robust variable selection procedures through penalized regression have been gaining increased attention in the literature they can be used to perform variable selection and are expected to yield robust estimates however to the best of our knowledge the robustness of those penalized regression procedures has not been well characterized in this article we propose a class of penalized robust regression estimators based on exponential squared loss the motivation for this new procedure is that it enables us to characterize its robustness in a way that has not been done for the existing procedures while its performance is near optimal and superior to some recently developed methods specifically under defined regularity conditions our estimators are consistent and possess the oracle property importantly we show that our estimators can achieve the highest asymptotic breakdown point of 1 2 and that their influence functions are bounded with respect to the outliers in either the response or the covari

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

functional gradient ascent for probit regression this paper proposes two gradient based methods to fit a probit regression model by maximizing the sample log likelihood function using the property of the hessian of the objective function the first method performs weighted least square regression in each iteration of the newton raphson framework resulting in probitboost a boosting like algorithm motivated by the gradient boosting algorithm 10 the second proposed approach maximizes the sample log likelihood function by updating the fitted function a small step in the gradient direction performing gradient ascent in functional space resulting in gradient probitboost we also generalize the algorithms to multi class problems by two strategies one of which is to use the gradient ascent to maximize the multi class sample log likelihood function for fitting all the classifiers simultaneously and the second approach uses the one versus all scheme to reduce the multi class problem to a series of binary classification problems the proposed algorithms are tested on typical classification problems including face detection cancer classification and handwritten digit recognition the results show that compared to the alternative methods the proposed algorithms perform similar or better in terms of testing error rates

orthogonal least squares learning algorithm for radial basis function networks the radial basis function network offers a viable alternative to the two layer neural network in many applications of signal processing a common learning algorithm for radial basis function networks is based on first choosing randomly some data points as radial basis function centers and then using singular value decomposition to solve for the weights of the network such a procedure has several drawbacks and in particular an arbitrary selection of centers is clearly unsatisfactory the authors propose an alternative learning procedure based on the orthogonal least squares method the procedure chooses radial basis function centers one by one in a rational way until an adequate network has been constructed in the algorithm each selected center maximizes the increment to the explained variance or energy of the desired output and does not suffer numerical ill conditioning problems the orthogonal least squares learning strategy provides a simple and efficient means for fitting radial basis function networks this is illustrated using examples taken from two different signal processing applications

classification based on hybridization of parametric and nonparametric classifiers parametric methods of classification assume specific parametric models for competing population densities e g gaussian population densities can lead to linear and quadratic discriminant analysis and they work well when these model assumptions are valid violation in one or more of these parametric model assumptions often leads to a poor classifier on the other hand nonparametric classifiers e g nearest neighbor and kernel based classifiers are more flexible and free from parametric model assumptions but the statistical instability of these classifiers may lead to poor performance when we have small numbers of training sample observations nonparametric methods however do not use any parametric structure of population densities therefore even when one has some additional information about population densities that important information is not used to modify the nonparametric classification rule this paper makes an attempt to overcome these limitations of parametric and nonparametric approaches and combines their strengths to develop some hybrid classification methods we use some simulated examples and benchmark data sets to examine the performance of these hybrid discriminant analysis tools asymptotic results on their misclassification rates have been derived under appropriate regularity conditions

gender discriminating models from facial surface normals in this paper we show how to use facial shape information to construct discriminating models for gender classification we represent facial shapes using 2 5d fields of facial surface normals and investigate three different methods to improve the gender discriminating capacity of the model constructed using the standard eigenspace method the three methods are novel variants of principal geodesic analysis pga namely a weighted pga b supervised weighted pga and c supervised pga our starting point is to define a weight map over the facial surface that indicates the importance of different locations in discriminating gender we show how to compute the relevant weights and how to incorporate the weights into the 2 5d model construction we evaluate the performance of the alternative methods using facial surface normals extracted from 3d range images or recovered from brightness images experimental results demonstrate the effectiveness of our methods moreover the classification accuracy which is as high as 97 demonstrates the effectiveness of using facial shape information for gender classification

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

gradient tree boosting for training conditional random fields conditional random fields crfs provide a flexible and powerful model for sequence labeling problems however existing learning algorithms are slow particularly in problems with large numbers of potential input features and feature combinations this paper describes a new algorithm for training crfs via gradient tree boosting in tree boosting the crf potential functions are represented as weighted sums of regression trees which provide compact representations of feature interactions so the algorithm does not explicitly consider the potentially large parameter space as a result gradient tree boosting scales linearly in the order of the markov model and in the order of the feature interactions rather than exponentially as in previous algorithms based on iterative scaling and gradient descent gradient tree boosting also makes it possible to use instance weighting as in c4 5 and surrogate splitting as in cart to handle missing values experimental studies of the effectiveness of these two methods as well as standard imputation and indicator feature methods show that instance weighting is the best method in most cases when feature values are missing at random

tree based ranking methods this paper investigates how recursive partitioning methods can be adapted to the bipartite ranking problem in ranking the pursued goal is global based on past data define an order on the whole input space x so that positive instances take up the top ranks with maximum probability the most natural way to order all instances consists of projecting the input data onto the real line through a real valued scoring function s and use the natural order on r the accuracy of the ordering induced by a candidate s is classically measured in terms of the roc curve or the auc here we discuss the design of tree structured scoring functions obtained by recursively maximizing the auc criterion the connection with recursive piecewise linear approximation of the optimal roc curve both in the l1 sense and in the linfin sense is highlighted a novel tree based algorithm for ranking called treerank is proposed consistency results and generalization bounds of functional nature are established for this ranking method when considering either the l1 or linfin distance we also describe committee based learning procedures using treerank as a ldquobase ranker rdquo in order to overcome obvious drawbacks of such a top down partitioning technique simulation results on artificial data are also displayed

polygonal approximation using integer particle swarm optimization polygonal approximation is an effective yet challenging digital curve representation for image analysis pattern recognition and computer vision this paper proposes a novel approach integer particle swarm optimization ipso for polygonal approximation when compared to the traditional binary version of particle swarm optimization bpso the new ipso directly uses an integer vector to represent the candidate solution and provides a more efficient and convenient means for solution processing the velocity and position updating mechanisms in ipso not only have clear physical meaning but also guarantee the optimality of the solutions the method is suitable for polygonal approximation which could otherwise be an intractable optimization problem the proposed method has been tested on commonly used synthesized shapes and lake contours extracted from the maps of four famous lakes in the world the experimental results show that the proposed ipso has better solution quality and computational efficiency than the bpso based methods and better solution quality than the other state of the art methods

determining the centers of radial basis probabilistic neural networks by recursive orthogonal least square algorithms in this paper we adopt a recursive orthogonal least squares algorithm rolsa to train radial basis probabilistic neural networks rbpnn and select the corresponding hidden centers from the training samples the rolsa is first used to recursively find the weights between the second hidden layer and the output layer of the rbpnn then the basic principle to select the hidden centers from the training set and a detailed selection procedure are presented the solution to orthogonal decomposition terms under the condition of varying hidden centers is obtained theoretically finally the effectiveness and efficiency of our proposed approach are demonstrated by two examples

logistic regression using covariates obtained by product unit neural network models we propose a logistic regression method based on the hybridation of a linear model and product unit neural network models for binary classification in a first step we use an evolutionary algorithm to determine the basic structure of the product unit model and afterwards we apply logistic regression in the new space of the derived features this hybrid model has been applied to seven benchmark data sets and a new microbiological problem the hybrid model outperforms the linear part and the nonlinear part obtaining a good compromise between them and they perform well compared to several other learning classification techniques we obtain a binary classifier with very promising results in terms of classification accuracy and the complexity of the classifier