a real generalization of discrete adaboost scaling discrete adaboost to handle real valued weak hypotheses has often been done under the auspices of convex optimization but little is generally known from the original boosting model standpoint we introduce a novel generalization of discrete adaboost which departs from this mainstream of algorithms from the theoretical standpoint it formally displays the original boosting property as it brings fast improvements of the accuracy of a weak learner up to arbitrary high levels furthermore it brings interesting computational and numerical improvements that make it significantly easier to handle as is conceptually speaking it provides a new and appealing scaling to r of some well known facts about discrete ada boosting perhaps the most popular is an iterative weight modification mechanism according to which examples have their weights decreased iff they receive the right class by the current discrete weak hypothesis in our generalization this property does not hold anymore as examples that receive the right class can still be reweighted higher with real valued weak hypotheses from the experimental standpoint our generalization displays the ability to produce low error formulas with particular cumulative margin distribution graphs and it provides a nice handling of those noisy domains that represent achilles heel for common adaptive boosting algorithms

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

shrinkage degree in l_ 2 rescale boosting for regression l 2 rescale boosting l 2 rboosting is a variant of l 2 boosting which can essentially improve the generalization performance of l 2 boosting the key feature of l 2 rboosting lies in introducing a shrinkage degree to rescale the ensemble estimate in each iteration thus the shrinkage degree determines the performance of l 2 rboosting the aim of this paper is to develop a concrete analysis concerning how to determine the shrinkage degree in l 2 rboosting we propose two feasible ways to select the shrinkage degree the first one is to parameterize the shrinkage degree and the other one is to develop a data driven approach after rigorously analyzing the importance of the shrinkage degree in l 2 rboosting we compare the pros and cons of the proposed methods we find that although these approaches can reach the same learning rates the structure of the final estimator of the parameterized approach is better which sometimes yields a better generalization capability when the number of sample is finite with this we recommend to parameterize the shrinkage degree of l 2 rboosting we also present an adaptive parameter selection strategy for shrinkage degree and verify its feasibility through both theoretical analysis and numerical verification the obtained results enhance the understanding of l 2 rboosting and give guidance on how to use it for regression tasks

identifying risk factors for severe childhood malnutrition by boosting additive quantile regression we investigated the risk factors for childhood malnutrition in india based on the 2005 2006 demographic and health survey by applying a novel estimation technique for additive quantile regression ordinary linear and generalized linear regression models relate the mean of a response variable to a linear combination of covariate effects and as a consequence focus on average properties of the response the use of such a regression model for analyzing childhood malnutrition in developing or transition countries implies that the estimated effects describe the average nutritional status however it is of even greater interest to analyze quantiles of the response distribution such as the 5 or 10 quantile which relate to the risk of extreme malnutrition our investigation is based on a semiparametric extension of quantile regression models where different types of nonlinear effects are included in the model equation leading to additive quantile regression we addressed the variable selection and model c

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

causal inference by using invariant prediction identification and confidence intervals summary what is the difference between a prediction that is made with a causal model and that with a non causal model suppose that we intervene on the predictor variables or change the whole environment the predictions from a causal model will in general work as well under interventions as for observational data in contrast predictions from a non causal model can potentially be very wrong if we actively intervene on variables here we propose to exploit this invariance of a prediction under a causal model for causal inference given different experimental settings e g various interventions we collect all models that do show invariance in their predictive accuracy across settings and interventions the causal model will be a member of this set of models with high probability this approach yields valid confidence intervals for the causal relationships in quite general scenarios we examine the example of structural equation models in more detail and provide sufficient assumptions under which the set of causal predictors becomes identifiable we further investigate robustness properties of our approach under model misspecification and discuss possible extensions the empirical properties are studied for various data sets including large scale gene perturbation experiments

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

on the consistency of multi label learning multi label learning has attracted much attention during the past few years many multi label approaches have been developed mostly working with surrogate loss functions because multi label loss functions are usually difficult to optimize directly owing to their non convexity and discontinuity these approaches are effective empirically however little effort has been devoted to the understanding of their consistency i e the convergence of the risk of learned functions to the bayes risk in this paper we present a theoretical analysis on this important issue we first prove a necessary and sufficient condition for the consistency of multi label learning based on surrogate loss functions then we study the consistency of two well known multi label loss functions i e ranking loss and hamming loss for ranking loss our results disclose that surprisingly none of convex surrogate loss is consistent we present the partial ranking loss with which some surrogate losses are proven to be consistent we also discuss on the consistency of univariate surrogate losses for hamming loss we show that two multi label learning methods i e one vs all and pairwise comparison which can be regarded as direct extensions from multi class learning are inconsistent in general cases yet consistent under the dominating setting and similar results also hold for some recent multi label approaches that are variations of one vs all in addition we discuss on the consistency of learning approaches that address multi label learning by decomposing into a set of binary classification problems

a boosting approach to flexible semiparametric mixed models in linear mixed models the influence of covariates is restricted to a strictly parametric form with the rise of semi and non parametric regression also the mixed model has been expanded to allow for additive predictors the common approach uses the representation of additive models as mixed models an alternative approach that is proposed in the present paper is likelihood based boosting boosting originates in the machine learning community where it has been proposed as a technique to improve classification procedures by combining estimates with reweighted observations likelihood based boosting is a general method which may be seen as an extension of l2 boost in additive mixed models the advantage of boosting techniques in the form of componentwise boosting is that it is suitable for high dimensional settings where many explanatory variables are present it allows to fit additive models for many covariates with implicit selection of relevant variables and automatic selection of smoothing parameters moreover boosting techniques may be used to incorporate the subject specific variation of smooth influence functions by specifying random slopes on smooth effects this results in flexible semiparametric mixed models which are appropriate in cases where a simple random intercept is unable to capture the variation of effects across subjects

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

variable selection and model choice in geoadditive regression models model choice and variable selection are issues of major concern in practical regression analyses we propose a boosting procedure that facilitates both tasks in a class of complex geoadditive regression models comprising spatial effects nonparametric effects of continuous covariates interaction surfaces random effects and varying coefficient terms the major modelling component are penalized splines and their bivariate tensor product extensions all smooth model terms are represented as the sum of a parametric component and a remaining smooth component with one degree of freedom to obtain a fair comparison between all model terms a generic representation of the geoadditive model allows to devise a general boosting algorithm that implements automatic model choice and variable selection we demonstrate the versatility of our approach with two examples a geoadditive poisson regression model for species counts in habitat suitability analyses and a geoadditive logit model for the analysis of forest health

on the doubt about margin explanation of boosting margin theory provides one of the most popular explanations to the success of adaboost where the central point lies in the recognition that margin is the key for characterizing the performance of adaboost this theory has been very influential e g it has been used to argue that adaboost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero previously the minimum margin bound was established for adaboost however breiman 1999 9 pointed out that maximizing the minimum margin does not necessarily lead to a better generalization later reyzin and schapire 2006 37 emphasized that the margin distribution rather than minimum margin is crucial to the performance of adaboost in this paper we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and emargin bound then we improve the previous empirical bernstein bounds audibert et al 2009 maurer and pontil 2009 2 30 and based on such findings we defend the margin based explanation against breiman s doubts by proving a new generalization error bound that considers exactly the same factors as schapire et al 1998 39 but is sharper than breiman s 1999 9 minimum margin bound by incorporating factors such as average margin and variance we present a generalization error bound that is heavily related to the whole margin distribution we also provide margin distribution bounds for generalization error of voting classifiers in finite vc dimension space

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

eeboost a general method for prediction and variable selection based on estimating equations the modern statistical literature is replete with methods for performing variable selection and prediction in standard regression problems however simple models may misspecify or fail to capture important aspects of the data generating process such as missingness correlation and over underdispersion in this article we describe eeboost a strategy for variable selection and prediction which can be applied in high dimensional settings where inference for low dimensional parameters would typically be based on estimating equations the method is simple flexible and easily implemented using existing software the eeboost algorithm is obtained as a modification of the standard boosting or functional gradient descent technique we show that eeboost is closely related to a class of l1 constrained projected likelihood ratio minimizations and therefore produces similar variable selection paths to penalized methods without the need to apply constrained optimization algorithms the flexibility of eeboost is

minimax nonparametric classification i rates of convergence this paper studies minimax aspects of nonparametric classification we first study minimax estimation of the conditional probability of a class label given the feature variable this function say f is assumed to be in a general nonparametric class we show the minimax rate of convergence under square l sub 2 loss is determined by the massiveness of the class as measured by metric entropy the second part of the paper studies minimax classification the loss of interest is the difference between the probability of misclassification of a classifier and that of the bayes decision as is well known an upper bound on risk for estimating f gives an upper bound on the risk for classification but the rate is known to be suboptimal for the class of monotone functions this suggests that one does not have to estimate f well in order to classify well however we show that the two problems are in fact of the same difficulty in terms of rates of convergence under a sufficient condition which is satisfied by many function classes including besov sobolev lipschitz and bounded variation this is somewhat surprising in view of a result of devroye gorfi and lugosi see a probabilistic theory of pattern recognition new york springer verlag 1996

a bayesian reassessment of nearest neighbor classification the k nearest neighbor knn procedure is a well known deterministic method used in supervised classification this article proposes a reassessment of this approach as a statistical technique derived from a proper probabilistic model in particular we modify the assessment found in holmes and adams and evaluated by manocha and girolami where the underlying probabilistic model is not completely well defined once provided with a clear probabilistic basis for the knn procedure we derive computational tools for bayesian inference on the parameters of the corresponding model in particular we assess the difficulties inherent to both pseudo likelihood and path sampling approximations of an intractable normalizing constant we implement a correct mcmc sampler based on perfect sampling when perfect sampling is not available we use instead a gibbs sampling approximation illustrations of the performance of the corresponding bayesian classifier are provided for benchmark datasets demonstrating in particula

estimating a change point in a sequence of very high dimensional covariance matrices this article considers the problem of estimating a change point in the covariance matrix in a sequence of high dimensional vectors where the dimension is substantially larger than the sample size

reducing multiclass to binary a unifying approach for margin classifiers we present a unifying framework for studying the solution of multiclass categorization problems by reducing them to multiple binary problems that are then solved using a margin based binary learning algorithm the proposed framework unifies some of the most popular approaches in which each class is compared against all others or in which all pairs of classes are compared to each other or in which output codes with error correcting properties are used we propose a general method for combining the classifiers generated on the binary problems and we prove a general empirical multiclass loss bound given the empirical loss of the individual binary learning algorithms the scheme and the corresponding bounds apply to many popular classification learning algorithms including support vector machines adaboost regression logistic regression and decision tree algorithms we also give a multiclass generalization error analysis for general output codes with adaboost as the binary learner experimental results with svm and adaboost show that our scheme provides a viable alternative to the most commonly used multiclass algorithms

nonparametric rotations for sphere sphere regression regression of data represented as points on a hypersphere has traditionally been treated using parametric families of transformations that include the simple rigid rotation as an important special

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

high dimensional classification using features annealed independence rules classification using high dimensional features arises frequently in many contemporary statistical studies such as tumor classification using microarray or other high throughput data the impact of dimensionality on classifications is largely poorly understood in a seminal paper bickel and levina 2004 show that the fisher discriminant performs poorly due to diverging spectra and they propose to use the independence rule to overcome the problem we first demonstrate that even for the independence classification rule classification using all the features can be as bad as the random guessing due to noise accumulation in estimating population centroids in high dimensional feature space in fact we demonstrate further that almost all linear discriminants can perform as bad as the random guessing thus it is paramountly important to select a subset of important features for high dimensional classification resulting in features annealed independence rules fair the conditions under which all the important features can be selected by the two sample t statistic are established the choice of the optimal number of features or equivalently the threshold value of the test statistics are proposed based on an upper bound of the classification error simulation studies and real data analysis support our theoretical results and demonstrate convincingly the advantage of our new classification procedure

generalized additive modeling with implicit variable selection by likelihood based boosting the use of generalized additive models in statistical data analysis suffers from the restriction to few explanatory variables and the problems of selection of smoothing parameters generalized additive model boosting circumvents these problems by means of stagewise fitting of weak learners a fitting procedure is derived which works for all simple exponential family distributions including binomial poisson and normal response variables the procedure combines the selection of variables and the determination of the appropriate amount of smoothing penalized regression splines and the newly introduced penalized stumps are considered as weak learners estimates of standard deviations and stopping criteria which are notorious problems in iterative procedures are based on an approximate hat matrix the method is shown to be a strong competitor to common procedures for the fitting of generalized additive models in particular in high dimensional settings with many nuisance predictor variables it performs very well

spectral bounds and oscillations of a differential difference system of neutral type abstract this paper is concerned with the oscillatory behavior of a differential difference system of neutral type spectral bounds are used in order to obtain sufficient conditions such that all their solutions are oscillatory

arcing classifier with discussion and a rejoinder by the author recent work has shown that combining multiple versions of unstable classifiers such as trees or neural nets results in reduced test set error one of the more effective is bagging here modified training sets are formed by resampling from the original training set classifiers constructed using these training sets and then combined by voting freund and schapire propose an algorithm the basis of which is to adaptively resample and combine hence the acronym arcing so that the weights in the resampling are increased for those cases most often misclassified and the combining is done by weighted voting arcing is more successful than bagging in test set error reduction we explore two arcing algorithms compare them to each other and to bagging and try to understand how arcing works we introduce the definitions of bias and variance for a classifier as components of the test set error unstable classifiers can have low bias on a large range of data sets their problem is high variance combining multiple versions either through bagging or arcing reduces variance significantly

conditional transformation models the ultimate goal of regression analysis is to obtain information about the conditional distribution of a response given a set of explanatory variables this goal is however seldom achieved because most established regression models only estimate the conditional mean as a function of the explanatory variables and assume that higher moments are not aected by the regressors the underlying reason for such a restriction is the assumption of additivity of signal and noise we propose to relax this common assumption in the framework of transformation models the novel class of semiparametric regression models proposed herein allows transformation functions to depend on explanatory variables these transformation functions are estimated by regularised optimisation of scoring rules for probabilistic forecasts e g the continuous ranked probability score the corresponding estimated conditional distribution functions are consistent based on applications from dierent domains we show that these conditional transformation models are useful for describing possible heteroscedasticity comparing spatially varying distributions identifying extreme events deriving prediction intervals and selecting variables beyond mean regression eects an empirical investigation based on a heteroscedastic varying coecient simulation model demonstrates that semiparametric estimation of conditional distribution functions can be more benecial than kernel based non parametric approaches or parametric generalised additive models for location scale and shape

natural spline functions their associated eigenvalue problem we consider the problem of studying the behaviour of the eigenvalues associated with spline functions with equally spaced knots we show that they are o left frac i 2m n right i 1 ldots n m wherem is the order of the spline andn the number of knots this result is of particular interest to prove optimality properties of the generalized cross validation method and had been conjectured by craven and wahba in a recent paper

sparse boosting for high dimensional survival data with varying coefficients motivated by high throughput profiling studies in biomedical research variable selection methods have been a focus for biostatisticians in this paper we consider semiparametric varying coefficient accelerated failure time models for right censored survival data with high dimensional covariates instead of adopting the traditional regularization approaches we offer a novel sparse boosting sparsel2 boosting algorithm to conduct model based prediction and variable selection one main advantage of this new method is that we do not need to perform the time consuming selection of tuning parameters extensive simulations are conducted to examine the performance of our sparse boosting feature selection techniques we further illustrate our methods using a lung cancer data analysis

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

boosting iterative stochastic ensemble method for nonlinear calibration of subsurface flow models abstract a novel parameter estimation algorithm is proposed the inverse problem is formulated as a sequential data integration problem in which gaussian process regression gpr is used to integrate the prior knowledge static data the search space is further parameterized using karhunen loeve expansion to build a set of basis functions that spans the search space optimal weights of the reduced basis functions are estimated by an iterative stochastic ensemble method isem isem employs directional derivatives within a gauss newton iteration for efficient gradient estimation the resulting update equation relies on the inverse of the output covariance matrix which is rank deficient in the proposed algorithm we use an iterative regularization based on the l2 boosting algorithm l2 boosting iteratively fits the residual and the amount of regularization is controlled by the number of iterations a termination criteria based on akaike information criterion aic is utilized this regularization method is very attractive in terms of performance and simplicity of implementation the proposed algorithm combining isem and l2 boosting is evaluated on several nonlinear subsurface flow parameter estimation problems the efficiency of the proposed algorithm is demonstrated by the small size of utilized ensembles and in terms of error convergence rates

functional gradient ascent for probit regression this paper proposes two gradient based methods to fit a probit regression model by maximizing the sample log likelihood function using the property of the hessian of the objective function the first method performs weighted least square regression in each iteration of the newton raphson framework resulting in probitboost a boosting like algorithm motivated by the gradient boosting algorithm 10 the second proposed approach maximizes the sample log likelihood function by updating the fitted function a small step in the gradient direction performing gradient ascent in functional space resulting in gradient probitboost we also generalize the algorithms to multi class problems by two strategies one of which is to use the gradient ascent to maximize the multi class sample log likelihood function for fitting all the classifiers simultaneously and the second approach uses the one versus all scheme to reduce the multi class problem to a series of binary classification problems the proposed algorithms are tested on typical classification problems including face detection cancer classification and handwritten digit recognition the results show that compared to the alternative methods the proposed algorithms perform similar or better in terms of testing error rates

spectral algorithms for supervised learning we discuss how a large class of regularization methods collectively known as spectral regularization and originally designed for solving ill posed inverse problems gives rise to regularized learning algorithms all of these algorithms are consistent kernel methods that can be easily implemented the intuition behind their derivation is that the same principle allowing for the numerical stabilization of a matrix inversion problem is crucial to avoid overfitting the various methods have a common derivation but different computational and theoretical properties we describe examples of such algorithms analyze their classification performance on several data sets and discuss their applicability to real world problems

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

a geometrical approach to iterative isotone regression in the present paper we propose and analyze a novel method for estimating a univariate regression function of bounded variation the underpinning idea is to combine two classical tools in nonparametric statistics namely isotonic regression and the estimation of additive models a geometrical interpretation enables us to link this iterative method with von neumann s algorithm moreover making a connection with the general property of isotonicity of projection onto convex cones we derive another equivalent algorithm and go further in the analysis

sparse hp filter finding kinks in the covid 19 contact rate in this paper we estimate the time varying covid 19 contact rate of a susceptible infected recovered sir model our measurement of the contact rate is constr

optimal rates of convergence to bayes risk in nonparametric discrimination consider the multiclassification discrimination problem with known prior probabilities and a multi dimensional vector of observations assume the underlying densities corresponding to the various classes are unknown but a training sample of size n is available from each class rates of convergence to bayes risk are investigated under smoothness conditions on the underlying densities of the type often seen in nonparametric density estimation these rates can be drastically affected by a small change in the prior probabilities so the error criterion used here is bayes risk averaged uniformly over all prior probabilities then it is shown that a certain rate n r is optimal in the sense that no rule can do better uniformly over the class of smooth densities and a rule is exhibited which does that well the optimal value of r depends on the smoothness of the distributions and the dimensionality of the observations in the same way as for nonparametric density estimation with integrated square error loss

cam causal additive models high dimensional order search and penalized regression we develop estimation for potentially high dimensional additive structural equation models a key component of our approach is to decouple order search among the variables from feature or edge selection in a directed acyclic graph encoding the causal structure we show that the former can be done with nonregularized restricted maximum likelihood estimation while the latter can be efficiently addressed using sparse regression techniques thus we substantially simplify the problem of structure search and estimation for an important class of causal models we establish consistency of the restricted maximum likelihood estimator for low and high dimensional scenarios and we also allow for misspecification of the error distribution furthermore we develop an efficient computational algorithm which can deal with many variables and the new method s accuracy and performance is illustrated on simulated and real data

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

sparse representation for coarse and fine object recognition this paper offers a sparse multiscale representation of objects it captures the object appearance by selection from a very large dictionary of gaussian differential basis functions the learning procedure results from the matching pursuit algorithm while the recognition is based on polynomial approximation to the bases turning image matching into a problem of polynomial evaluation the method is suited for coarse recognition between objects and by adding more bases also for fine recognition of the object pose the advantages over the common representation using pca include storing sampled points for recognition is not required adding new objects to an existing data set is trivial because retraining other object models is not needed and significantly in the important case where one has to scan an image over multiple locations in search for an object the new representation is readily available as opposed to pca projection at each location the experimental result on the coil 100 data set demonstrates high recognition accuracy with real time performance