on chaotic simulated annealing chen and aihara 1995 proposed a chaotic simulated annealing approach to solving optimization problems by adding a negative self coupling to a network model proposed earlier by aihara et al and gradually removing this negative self coupling they used the transient chaos for searching and self organizing thereby achieving great improvement over other neural network approaches to optimization problems with or without simulated annealing in this paper we suggest a new approach to chaotic simulated annealing with guaranteed convergence and minimization of the energy function by gradually reducing the time step in the euler approximation of the differential equations that describe the continuous hopfield neural network this approach eliminates the need to carefully select other system parameters we also generalize the convergence theorems of chen and aihara to arbitrarily increasing neuronal input output functions and to less restrictive and yet more compact forms

approximately optimizing ndcg using pair wise loss abstract the normalized discounted cumulative gain ndcg is used to measure the performance of ranking algorithms much of the work on learning to rank by optimizing ndcg directly or indirectly is based on list wise approaches in our work we approximately optimize a variant of ndcg called ndcg using pair wise approaches ndcg utilizes the linear discounting function we first prove that the dcg error of ndcg is equal to the weighted pair wise loss then on that basis rankboostndcg and ranksvmndcg are proposed to optimize the upper bound of the pair wise 0 1 loss function the experimental results from applying our approaches and ten other state of the art methods to five public datasets show the superiority of the proposed methods especially ranksvmndcg in addition rankboostndcg are less influenced by the initial weight distribution

multitask quantile regression under the transnormal model abstractwe consider estimating multitask quantile regression under the transnormal model with focus on high dimensional setting we derive a surprisingly simple closed form solution through rank based covariance regularization in particular we propose the rank based l1 penalization with positive definite constraints for estimating sparse covariance matrices and the rank based banded cholesky decomposition regularization for estimating banded precision matrices by taking advantage of the alternating direction method of multipliers nearest correlation matrix projection is introduced that inherits sampling properties of the unprojected one our work combines strengths of quantile regression and rank based covariance regularization to simultaneously deal with nonlinearity and nonnormality for high dimensional regression furthermore the proposed method strikes a good balance between robustness and efficiency achieves the oracle like convergence rate and provides the provable prediction interval und

prediction and classification in nonlinear data analysis something old something new something borrowed something blue prediction and classification are two very active areas in modern data analysis in this paper prediction with nonlinear optimal scaling transformations of the variables is reviewed and extended to the use of multiple additive components much in the spirit of statistical learning techniques that are currently popular among other areas in data mining also a classification clustering method is described that is particularly suitable for analyzing attribute value data from systems biology genomics proteomics and metabolomics and which is able to detect groups of objects that have similar values on small subsets of the attributes

on the characterization of a class of fisher consistent loss functions and its application to boosting accurate classification of categorical outcomes is essential in a wide range of applications due to computational issues with minimizing the empirical 0 1 loss fisher consistent losses have been proposed as viable proxies however even with smooth losses direct minimization remains a daunting task to approximate such a minimizer various boosting algorithms have been suggested for example with exponential loss the adaboost algorithm freund and schapire 1995 is widely used for two class problems and has been extended to the multi class setting zhu et al 2009 alternative loss functions such as the logistic and the hinge losses and their corresponding boosting algorithms have also been proposed zou et al 2008 wang 2012 in this paper we demonstrate that a broad class of losses including non convex functions achieve fisher consistency and in addition can be used for explicit estimation of the conditional class probabilities furthermore we provide a generic boosting algorithm that is not loss specific extensive simulation results suggest that the proposed boosting algorithms could outperform existing methods with properly chosen losses and bags of weak learners

minimizing interference in satellite communications using transiently chaotic neural networks the frequency assignment problem fap in satellite communications is solved with transiently chaotic neural networks tcnn the objective of this optimization problem is to minimize cochannel interference between two satellite systems by rearranging the frequency assignments for an n carrier m segment fap problem we construct a tcnn consisting of nxm neurons the performance of the tcnn is demonstrated through solving a set of benchmark problems where the tcnn finds comparative if not better solutions as compared to the existing algorithms

identifying risk factors for severe childhood malnutrition by boosting additive quantile regression we investigated the risk factors for childhood malnutrition in india based on the 2005 2006 demographic and health survey by applying a novel estimation technique for additive quantile regression ordinary linear and generalized linear regression models relate the mean of a response variable to a linear combination of covariate effects and as a consequence focus on average properties of the response the use of such a regression model for analyzing childhood malnutrition in developing or transition countries implies that the estimated effects describe the average nutritional status however it is of even greater interest to analyze quantiles of the response distribution such as the 5 or 10 quantile which relate to the risk of extreme malnutrition our investigation is based on a semiparametric extension of quantile regression models where different types of nonlinear effects are included in the model equation leading to additive quantile regression we addressed the variable selection and model c

propagation separation approach for local likelihood estimation the paper presents a unified approach to local likelihood estimation for a broad class of nonparametric models including e g the regression density poisson and binary response model the method extends the adaptive weights smoothing aws procedure introduced in polzehl and spokoiny 2000 in context of image denoising the main idea of the method is to describe a greatest possible local neighborhood of every design point x i in which the local parametric assumption is justified by the data the method is especially powerful for model functions having large homogeneous regions and sharp discontinuities the performance of the proposed procedure is illustrated by numerical examples for density estimation and classification we also establish some remarkable theoretical nonasymptotic results on properties of the new algorithm this includes the propagation property which particularly yields the root n consistency of the resulting estimate in the homogeneous case we also state an oracle result which implies rate optimality of the estimate under usual smoothness conditions and a separation result which explains the sensitivity of the method to structural changes

causal inference by using invariant prediction identification and confidence intervals summary what is the difference between a prediction that is made with a causal model and that with a non causal model suppose that we intervene on the predictor variables or change the whole environment the predictions from a causal model will in general work as well under interventions as for observational data in contrast predictions from a non causal model can potentially be very wrong if we actively intervene on variables here we propose to exploit this invariance of a prediction under a causal model for causal inference given different experimental settings e g various interventions we collect all models that do show invariance in their predictive accuracy across settings and interventions the causal model will be a member of this set of models with high probability this approach yields valid confidence intervals for the causal relationships in quite general scenarios we examine the example of structural equation models in more detail and provide sufficient assumptions under which the set of causal predictors becomes identifiable we further investigate robustness properties of our approach under model misspecification and discuss possible extensions the empirical properties are studied for various data sets including large scale gene perturbation experiments

a multiplier adjustment approach for the set partitioning problem we introduce an effective branch and bound algorithm for solving the set partitioning problem the new algorithm employs a new multiplier adjustment based bounding procedure and a complementary branching strategy which results in relatively small search trees computational results based on 20 moderately sized crew scheduling problems indicate that our new algorithm is on average 16 6 times faster than the popular code setpar the improvements are mainly due to the bounding procedure which is fast easy to use and provides tight lower bounds on average the bounds are 97 6 of the optimal objective value of the linear programming relaxation after only five iterations and 98 5 after ten iterations moreover the lower bounds are observed to be monotonically nondecreasing we also apply the technique of variable elimination which is very effective in reducing the size of the problems on average 89 of the variables are eliminated from the problem at the root node

adaptive annealing for chaotic optimization the chaotic simulated annealing algorithm for combinatorial optimization problems is examined in the light of the global bifurcation structure of the chaotic neural networks we show that the result of the chaotic simulated annealing algorithm is primarily dependent upon the global bifurcation structure of the chaotic neural networks and unlike the stochastic simulated annealing infinitely slow chaotic annealing does not necessarily provide an optimum result as an improved algorithm the adaptive chaotic simulated annealing algorithm is introduced using several instances of 20 and 40 city traveling salesman problems efficiency of the adaptive algorithm is demonstrated s1063 651x 98 15510 1

applications of integer programming in radio frequency management the radio frequency communication environment for example in large vessels aircraft space vehicles and satellites has become increasingly complex as more and more transmitters receivers and corresponding communication nets are being added to perform multiple functions this paper discusses some of the problems faced by a decision maker in managing a communication network a particular and important problem worth investigating occurs when different frequencies are simultaneously active in a network this may cause various types of interference creating a disturbance in communication one interesting and important type of interference which may be expressed mathematically is intermodulation a typical problem scenario is thus to assign frequencies to the different nets of the network so that intermodulation interference is within desirable limits a second problem is to add a net to an existing network with previously assigned frequencies so that the new net produces an acceptable level of inter

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

statistical analysis of bayes optimal subset ranking the ranking problem has become increasingly important in modern applications of statistical methods in automated decision making systems in particular we consider a formulation of the statistical ranking problem which we call subset ranking and focus on the discounted cumulated gain dcg criterion that measures the quality of items near the top of the rank list similar to error minimization for binary classification direct optimization of natural ranking criteria such as dcg leads to a nonconvex optimization problems that can be np hard therefore a computationally more tractable approach is needed we present bounds that relate the approximate optimization of dcg to the approximate minimization of certain regression errors these bounds justify the use of convex learning formulations for solving the subset ranking problem the resulting estimation methods are not conventional in that we focus on the estimation quality in the top portion of the rank list we further investigate the asymptotic statistical behavior of these formulations under appropriate conditions the consistency of the estimation schemes with respect to the dcg metric can be derived

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

contrast trees and distribution boosting a method for decision tree induction is presented given a set of predictor variables formula see text and two outcome variables y and z associated with each x the goal is to identify those values of x for which the respective distributions of formula see text and formula see text or selected properties of those distributions such as means or quantiles are most different contrast trees provide a lack of fit measure for statistical models of such statistics or for the complete conditional distribution formula see text as a function of x they are easily interpreted and can be used as diagnostic tools to reveal and then understand the inaccuracies of models produced by any learning method a corresponding contrast boosting strategy is described for remedying any uncovered errors thereby producing potentially more accurate predictions this leads to a distribution boosting strategy for directly estimating the full conditional distribution of y at each x under no assumptions concerning its shape form or parametric representation

credit spread approximation and improvement using random forest regression abstract credit default swap cds levels provide a market appreciation of companies default risk these derivatives are not always available creating a need for cds approximations this paper offers a simple global and transparent cds structural approximation which contrasts with more complex and proprietary approximations currently in use this equity to credit formula e2c inspired by creditgrades obtains better cds approximations according to empirical analyses based on a large sample spanning 2016 2018 a random forest regression run with this e2c formula and selected additional financial data results in an 87 3 out of sample accuracy in cds approximations the transparency property of this algorithm confirms the predominance of the e2c estimate and the impact of companies debt rating and size in predicting their cds

the rate of convergence of adaboost the adaboost algorithm was designed to combine many weak hypotheses that perform slightly better than random guessing into a strong hypothesis that has very low error we study the rate at which adaboost iteratively converges to the minimum of the exponential loss unlike previous work our proofs do not require a weak learning assumption nor do they require that minimizers of the exponential loss are finite our first result shows that the exponential loss of adaboost s computed parameter vector will be at most e more than that of any parameter vector of l1 norm bounded by b in a number of rounds that is at most a polynomial in b and 1 e we also provide lower bounds showing that a polynomial dependence is necessary our second result is that within c e iterations adaboost achieves a value of the exponential loss that is at most e more than the best possible value where c depends on the data set we show that this dependence of the rate on e is optimal up to constant factors that is at least 1 e rounds are necessary to achieve within e of the optimal exponential loss

training hard margin support vector machines using greedy stagewise algorithm hard margin support vector machines hm svms suffer from getting overfitting in the presence of noise soft margin svms deal with this problem by introducing a regularization term and obtain a state of the art performance however this disposal leads to a relatively high computational cost in this paper an alternative method greedy stagewise algorithm for svms named gs svms is presented to cope with the overfitting of hm svms without employing the regularization term the most attractive property of gs svms is that its computational complexity in the worst case only scales quadratically with the size of training samples experiments on the large data sets with up to 400 000 training samples demonstrate that gs svms can be faster than libsvm 2 83 without sacrificing the accuracy finally we employ statistical learning theory to analyze the empirical results which shows that the success of gs svms lies in that its early stopping rule can act as an implicit regularization term

cal u boost boosting with the universum it has been shown that the universum data which do not belong to either class of the classification problem of interest may contain useful prior domain knowledge for training a classifier 1 2 in this work we design a novel boosting algorithm that takes advantage of the available universum data hence the name uboost uboost is a boosting implementation of vapnik s alternative capacity concept to the large margin approach in addition to the standard regularization term uboost also controls the learned model s capacity by maximizing the number of observed contradictions our experiments demonstrate that uboost can deliver improved classification accuracy over standard boosting algorithms that use labeled data alone

a nonparallel support vector machine for a classification problem with universum learning universum samples defined as samples not belonging to any class for a classification problem of interest have been useful in supervised learning here we design a new nonparallel support vector machine u nsvm that can exploit prior knowledge embedded in the universum to construct a more robust classifier for training to this end u nsvm maximizes the two margins associated with the two closest neighboring classes which is combined by two nonparallel hyperplanes therefore u nsvm has better flexibility and can yield a more reasonable classifier in most cases in addition our method includes fewer parameters than u svm so is easier to implement experiments demonstrate that u nsvm outperforms the traditional svm and u svm

the adaboost flow we introduce a dynamical system that we call the adaboost flow the flow is defined by a system of odes with control we show that three algorithms of the adaboost family i the adaboost algorithm of schapire and freund ii the arc gv algorithm of breiman and iii the confidence rated prediction of schapire and singer can be embedded in the adaboost flow the nontrivial part of the adaboost flow equations coincides with the equations of dynamics of the nonperiodic toda system written in terms of spectral variables we provide a novel invariant geometrical description of the adaboost algorithm as a gradient flow on a foliation defined by level sets of the potential function we propose a new approach for constructing boosting algorithms as a continuous time gradient flow on measures defined by various metrics and potential functions finally we explain the similarity of the adaboost algorithm with the perelman s construction for the ricci flow 2015 wiley periodicals inc

an efficient modified boosting method for solving classification problems based on the adaboost algorithm a modified boosting method is proposed in this paper for solving classification problems this method predicts the class label of an example as the weighted majority voting of an ensemble of classifiers each classifier is obtained by applying a given weak learner to a subsample with size smaller than that of the original training set which is drawn from the original training set according to the probability distribution maintained over the training set a parameter is introduced into the reweighted scheme proposed in adaboost to update the probabilities assigned to training examples so that the algorithm can be more accurate than adaboost the experimental results on synthetic and several real world data sets available from the uci repository show that the proposed method improves the prediction accuracy the execution speed as well as the robustness to classification noise of adaboost furthermore the diversity accuracy patterns of the ensemble classifiers are investigated by kappa error diagrams

building sparse twin support vector machine classifiers in primal space abstract twin support vector machines tsvm obtain faster training speeds than classical support vector machines svm however tsvm augmented vectors lose sparsity in this paper a rapid sparse twin support vector machine stsvm classifier in primal space is proposed to improve the sparsity and robustness of tsvm based on a simple back fitting strategy the stsvm iteratively builds each nonparallel hyperplanes by adding one support vector sv from the corresponding class at one time this process is terminated using an adaptive and stable stopping criterion stsvm learning is implemented by linear equation computing systems through introducing a quadratic function to approximate the empirical risk the computational results on several synthetic and benchmark datasets indicate that the stsvm obtains a sparse separating hyperplane at a low cost without sacrificing its generalization performance

an l_ 2 boosting algorithm for estimation of a regression function an l 2 boosting algorithm for estimation of a regression function from random design is presented which consists of fitting repeatedly a function from a fixed nonlinear function space to the residuals of the data by least squares and by defining the estimate as a linear combination of the resulting least squares estimates splitting of the sample is used to decide after how many iterations of smoothing of the residuals the algorithm terminates the rate of convergence of the algorithm is analyzed in case of an unbounded response variable the method is used to fit a sum of maxima of minima of linear functions to a given data set and is compared with other nonparametric regression estimates using simulated data

on weak base hypotheses and their implications for boosting regression and classification when studying the training error and the prediction error for boosting it is often assumed that the hypotheses returned by the base learner are weakly accurate or are able to beat a random guesser by a certain amount of difference it has been an open question how much this difference can be whether it will eventually disappear in the boosting process or be bounded by a positive amount this question is crucial for the behavior of both the training error and the prediction error in this paper we study this problem and show affirmatively that the amount of improvement over the random guesser will be at least a positive amount for almost all possible sample realizations and for most of the commonly used base hypotheses this has a number of implications for the prediction error including for example that boosting forever may not be good and regularization may be necessary the problem is studied by first considering an analog of adaboost in regression where we study similar properties and find that for good performance one cannot hope to avoid regularization by just adopting the boosting device to regression

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

the reconstruction approach from interpolation to regression this article introduces an interpolation based method called the reconstruction approach for nonparametric regression based on the fact that interpolation usually has negligible errors compared

boosting as a regularized path to a maximum margin classifier in this paper we study boosting methods from a new perspective we build on recent work by efron et al to show that boosting approximately and in some cases exactly minimizes its loss criterion with an l1 constraint on the coefficient vector this helps understand the success of boosting with early stopping as regularized fitting of the loss criterion for the two most commonly used criteria exponential and binomial log likelihood we further show that as the constraint is relaxed or equivalently as the boosting iterations proceed the solution converges in the separable case to an l1 optimal separating hyper plane we prove that this l1 optimal separating hyper plane has the property of maximizing the minimal l1 margin of the training data as defined in the boosting literature an interesting fundamental similarity between boosting and kernel support vector machines emerges as both can be described as methods for regularized optimization in high dimensional predictor space using a computational trick to make the calculation practical and converging to margin maximizing solutions while this statement describes svms exactly it applies to boosting only approximately

boosting algorithms for detector cascade learning the problem of learning classifier cascades is considered a new cascade boosting algorithm fast cascade boosting fcboost is proposed fcboost is shown to have a number of interesting properties namely that it 1 minimizes a lagrangian risk that jointly accounts for classification accuracy and speed 2 generalizes adaboost 3 can be made cost sensitive to support the design of high detection rate cascades and 4 is compatible with many predictor structures suitable for sequential decision making it is shown that a rich family of such structures can be derived recursively from cascade predictors of two stages denoted cascade generators generators are then proposed for two new cascade families last stage and multiplicative cascades that generalize the two most popular cascade architectures in the literature the concept of neutral predictors is finally introduced enabling fcboost to automatically determine the cascade configuration i e number of stages and number of weak learners per stage for the learned cascades experiments on face and pedestrian detection show that the resulting cascades outperform current state of the art methods in both detection accuracy and speed

on the dual formulation of boosting algorithms we study boosting algorithms from a new perspective we show that the lagrange dual problems of l1 norm regularized adaboost logitboost and soft margin lpboost with generalized hinge loss are all entropy maximization problems by looking at the dual problems of these boosting algorithms we show that the success of boosting algorithms can be understood in terms of maintaining a better margin distribution by maximizing margins and at the same time controlling the margin variance we also theoretically prove that approximately l1 norm regularized adaboost maximizes the average margin instead of the minimum margin the duality formulation also enables us to develop column generation based optimization algorithms which are totally corrective we show that they exhibit almost identical classification results to that of standard stagewise additive boosting algorithms but with much faster convergence rates therefore fewer weak classifiers are needed to build the ensemble using our proposed optimization technique

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

functional gradient ascent for probit regression this paper proposes two gradient based methods to fit a probit regression model by maximizing the sample log likelihood function using the property of the hessian of the objective function the first method performs weighted least square regression in each iteration of the newton raphson framework resulting in probitboost a boosting like algorithm motivated by the gradient boosting algorithm 10 the second proposed approach maximizes the sample log likelihood function by updating the fitted function a small step in the gradient direction performing gradient ascent in functional space resulting in gradient probitboost we also generalize the algorithms to multi class problems by two strategies one of which is to use the gradient ascent to maximize the multi class sample log likelihood function for fitting all the classifiers simultaneously and the second approach uses the one versus all scheme to reduce the multi class problem to a series of binary classification problems the proposed algorithms are tested on typical classification problems including face detection cancer classification and handwritten digit recognition the results show that compared to the alternative methods the proposed algorithms perform similar or better in terms of testing error rates

a tutorial on propensity score estimation for multiple treatments using generalized boosted models the use of propensity scores to control for pretreatment imbalances on observed variables in non randomized or observational studies examining the causal effects of treatments or interventions has become widespread over the past decade for settings with two conditions of interest such as a treatment and a control inverse probability of treatment weighted estimation with propensity scores estimated via boosted models has been shown in simulation studies to yield causal effect estimates with desirable properties there are tools e g the twang package in r and guidance for implementing this method with two treatments however there is not such guidance for analyses of three or more treatments the goals of this paper are twofold 1 to provide step by step guidance for researchers who want to implement propensity score weighting for multiple treatments and 2 to propose the use of generalized boosted models gbm for estimation of the necessary propensity score weights we define the causal quantities that may be of interest to studies of multiple treatments and derive weighted estimators of those quantities we present a detailed plan for using gbm to estimate propensity scores and using those scores to estimate weights and causal effects we also provide tools for assessing balance and overlap of pretreatment variables among treatment groups in the context of multiple treatments a case study examining the effects of three treatment programs for adolescent substance abuse demonstrates the methods

solution of the optimization problem using the neural network model as a globally coupled map abstract in solving optimization problems by the neural network models the hopfield model and the boltzmann machine etc a serious problem is that the existence of a huge number of local minimum solutions makes the search efficiency very bad in this report the hopfield model is modified in the framework of the globally coupled map gcm to overcome the problem and the traveling salesman problem tsp is solved by the gcm model it is demonstrated that the gcm model is able to search for the optimal solutions of the tsp based on the chaotic itinerancy and the chaotic search by the gcm model is more efficient than the probabilistic search of other neural network models for solution of the tsp

a gradual neural network approach for frequency assignment in satellite communication systems a novel neural network approach called gradual neural network gnn is presented for a class of combinatorial optimization problems of requiring the constraint satisfaction and the goal function optimization simultaneously the frequency assignment problem in the satellite communication system is efficiently solved by gnn as the typical problem of this class the goal of this np complete problem is to minimize the cochannel interference between satellite communication systems by rearranging the frequency assignment so that they can accommodate the increasing demands the gnn consists of n spl times m binary neurons for the n carrier m segment system with the gradual expansion scheme of activated neurons the binary neural network achieves the constrain satisfaction with the help of heuristic methods whereas the gradual expansion scheme seeks the cost optimization the capability of gnn is demonstrated through solving 15 instances in practical size systems where gnn can find far better solutions than the existing algorithm

frequency planning as a set partitioning problem abstract some aspects of a new computerized method for automatic generation of frequency plans for radio communication systems are presented the emphasis is on problem formulation where the frequency planning problem is recast as a set partitioning problem the objective is minimization of total system interference a solution algorithm that has been found useful in practical applications is presented an alternative algorithm is also demonstrated

gradient tree boosting for training conditional random fields conditional random fields crfs provide a flexible and powerful model for sequence labeling problems however existing learning algorithms are slow particularly in problems with large numbers of potential input features and feature combinations this paper describes a new algorithm for training crfs via gradient tree boosting in tree boosting the crf potential functions are represented as weighted sums of regression trees which provide compact representations of feature interactions so the algorithm does not explicitly consider the potentially large parameter space as a result gradient tree boosting scales linearly in the order of the markov model and in the order of the feature interactions rather than exponentially as in previous algorithms based on iterative scaling and gradient descent gradient tree boosting also makes it possible to use instance weighting as in c4 5 and surrogate splitting as in cart to handle missing values experimental studies of the effectiveness of these two methods as well as standard imputation and indicator feature methods show that instance weighting is the best method in most cases when feature values are missing at random

building support vector machines with reduced classifier complexity support vector machines svms though accurate are not preferred in applications requiring great classification speed due to the number of support vectors being large to overcome this problem we devise a primal method with the following properties 1 it decouples the idea of basis functions from the concept of support vectors 2 it greedily finds a set of kernel basis functions of a specified maximum size dmax to approximate the svm primal cost function well 3 it is efficient and roughly scales as o ndmax2 where n is the number of training examples and 4 the number of basis functions it requires to achieve an accuracy close to the svm accuracy is usually far less than the number of svm support vectors

accurately estimating rigid transformations in registration using a boosting inspired mechanism feature extraction and matching provide the basis of many methods for object registration modeling retrieval and recognition however this approach typically introduces false matches due to lack of features noise occlusion and cluttered backgrounds in registration these false matches lead to inaccurate estimation of the underlying transformation that brings the overlapping shapes into best possible alignment in this paper we propose a novel boosting inspired method to tackle this challenging task it includes three key steps i underlying transformation estimation in the weighted least squares sense ii boosting parameter estimation and regularization via tsallis entropy and iii weight re estimation and regularization via shannon entropy and update with a maximum fusion rule the process is iterated the final optimal underlying transformation is estimated as a weighted average of the transformations estimated from the latest iterations with weights given by the boosting parameters a comparative study based on real shape data shows that the proposed method outperforms four other state of the art methods for evaluating the established point matches enabling more accurate and stable estimation of the underlying transformation highlightswe provide a novel method for evaluating point matches established by typical methods real data based experimental results show that the method outperforms the state of the art under certain conditions the finally estimated underlying transformation is optimal the transformation can be recovered with an error as small as 5 from typical point matches

functional dissipation microarrays for classification in this article we describe a new method of extracting information from signals called functional dissipation that proves to be very effective for enhancing classification of high resolution texture rich data our algorithm bypasses to some extent the need to have very specialized feature extraction techniques and can potentially be used as an intermediate feature enhancement step in any classification scheme functional dissipation is based on signal transforms but uses the transforms recursively to uncover new features we generate a variety of masking functions and extract features with several generalized matching pursuit iterations in each iteration the recursive process modifies several coefficients of the transformed signal with the largest absolute values according to the specific masking function in this way the greedy pursuit is turned into a slow controlled dissipation of the structure of the signal that for some masking functions enhances separation among classes our case study in this paper is the classification of crystallization patterns of amino acids solutions affected by the addition of small quantities of proteins

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n