prediction and classification in nonlinear data analysis something old something new something borrowed something blue prediction and classification are two very active areas in modern data analysis in this paper prediction with nonlinear optimal scaling transformations of the variables is reviewed and extended to the use of multiple additive components much in the spirit of statistical learning techniques that are currently popular among other areas in data mining also a classification clustering method is described that is particularly suitable for analyzing attribute value data from systems biology genomics proteomics and metabolomics and which is able to detect groups of objects that have similar values on small subsets of the attributes

approximately optimizing ndcg using pair wise loss abstract the normalized discounted cumulative gain ndcg is used to measure the performance of ranking algorithms much of the work on learning to rank by optimizing ndcg directly or indirectly is based on list wise approaches in our work we approximately optimize a variant of ndcg called ndcg using pair wise approaches ndcg utilizes the linear discounting function we first prove that the dcg error of ndcg is equal to the weighted pair wise loss then on that basis rankboostndcg and ranksvmndcg are proposed to optimize the upper bound of the pair wise 0 1 loss function the experimental results from applying our approaches and ten other state of the art methods to five public datasets show the superiority of the proposed methods especially ranksvmndcg in addition rankboostndcg are less influenced by the initial weight distribution

multitask quantile regression under the transnormal model abstractwe consider estimating multitask quantile regression under the transnormal model with focus on high dimensional setting we derive a surprisingly simple closed form solution through rank based covariance regularization in particular we propose the rank based l1 penalization with positive definite constraints for estimating sparse covariance matrices and the rank based banded cholesky decomposition regularization for estimating banded precision matrices by taking advantage of the alternating direction method of multipliers nearest correlation matrix projection is introduced that inherits sampling properties of the unprojected one our work combines strengths of quantile regression and rank based covariance regularization to simultaneously deal with nonlinearity and nonnormality for high dimensional regression furthermore the proposed method strikes a good balance between robustness and efficiency achieves the oracle like convergence rate and provides the provable prediction interval und

ridge regression biased estimation for nonorthogonal problems in multiple regression it is shown that parameter estimates based on minimum residual sum of squares have a high probability of being unsatisfactory if not incorrect if the prediction vectors are not orthogonal proposed is an estimation procedure based on adding small positive quantities to the diagonal of x x introduced is the ridge trace a method for showing in two dimensions the effects of nonorthogonality it is then shown how to augment x x to obtain biased estimates with smaller mean square error

on the characterization of a class of fisher consistent loss functions and its application to boosting accurate classification of categorical outcomes is essential in a wide range of applications due to computational issues with minimizing the empirical 0 1 loss fisher consistent losses have been proposed as viable proxies however even with smooth losses direct minimization remains a daunting task to approximate such a minimizer various boosting algorithms have been suggested for example with exponential loss the adaboost algorithm freund and schapire 1995 is widely used for two class problems and has been extended to the multi class setting zhu et al 2009 alternative loss functions such as the logistic and the hinge losses and their corresponding boosting algorithms have also been proposed zou et al 2008 wang 2012 in this paper we demonstrate that a broad class of losses including non convex functions achieve fisher consistency and in addition can be used for explicit estimation of the conditional class probabilities furthermore we provide a generic boosting algorithm that is not loss specific extensive simulation results suggest that the proposed boosting algorithms could outperform existing methods with properly chosen losses and bags of weak learners

parallel tree contraction part 2 further applications this paper applies the parallel tree contraction techniques developed in miller and reif s paper randomness and computation vol 5 s micali ed jai press 1989 pp 47 72 to a number of fundamental graph problems the paper presents an o log n time and n log n processor a 0 sided randomized algorithm for testing the isomorphism of trees and an o log n time n algorithm for maximal subtree isomorphism and for common subexpression elimination an o log n time n processor algorithm for computing the canonical forms of trees and subtrees is given an olog n time algorithm for computing the tree of 3 connected components of a graph an o log 2 n time algorithm for computing an explicit planar embedding of a planar graph and an o log 3 n time algorithm for computing a canonical form for a planar graph are also given all these latter algorithms use only n o 1 processors on a parallel random access machine pram model with concurrent writes and concurrent reads

identifying risk factors for severe childhood malnutrition by boosting additive quantile regression we investigated the risk factors for childhood malnutrition in india based on the 2005 2006 demographic and health survey by applying a novel estimation technique for additive quantile regression ordinary linear and generalized linear regression models relate the mean of a response variable to a linear combination of covariate effects and as a consequence focus on average properties of the response the use of such a regression model for analyzing childhood malnutrition in developing or transition countries implies that the estimated effects describe the average nutritional status however it is of even greater interest to analyze quantiles of the response distribution such as the 5 or 10 quantile which relate to the risk of extreme malnutrition our investigation is based on a semiparametric extension of quantile regression models where different types of nonlinear effects are included in the model equation leading to additive quantile regression we addressed the variable selection and model c

on total variation minimization and surface evolution using parametric maximum flows in a recent paper boykov et al lncs vol 3953 pp 409 422 2006 propose an approach for computing curve and surface evolution using a variational approach and the geo cuts method of boykov and kolmogorov international conference on computer vision pp 26 33 2003 we recall in this paper how this is related to well known approaches for mean curvature motion introduced by almgren et al siam journal on control and optimization 31 2 387 438 1993 and luckhaus and sturzenhecker calculus of variations and partial differential equations 3 2 253 271 1995 and show how the corresponding problems can be solved with sub pixel accuracy using parametric maximum flow techniques this provides interesting algorithms for computing crystalline curvature motion possibly with a forcing term

propagation separation approach for local likelihood estimation the paper presents a unified approach to local likelihood estimation for a broad class of nonparametric models including e g the regression density poisson and binary response model the method extends the adaptive weights smoothing aws procedure introduced in polzehl and spokoiny 2000 in context of image denoising the main idea of the method is to describe a greatest possible local neighborhood of every design point x i in which the local parametric assumption is justified by the data the method is especially powerful for model functions having large homogeneous regions and sharp discontinuities the performance of the proposed procedure is illustrated by numerical examples for density estimation and classification we also establish some remarkable theoretical nonasymptotic results on properties of the new algorithm this includes the propagation property which particularly yields the root n consistency of the resulting estimate in the homogeneous case we also state an oracle result which implies rate optimality of the estimate under usual smoothness conditions and a separation result which explains the sensitivity of the method to structural changes

causal inference by using invariant prediction identification and confidence intervals summary what is the difference between a prediction that is made with a causal model and that with a non causal model suppose that we intervene on the predictor variables or change the whole environment the predictions from a causal model will in general work as well under interventions as for observational data in contrast predictions from a non causal model can potentially be very wrong if we actively intervene on variables here we propose to exploit this invariance of a prediction under a causal model for causal inference given different experimental settings e g various interventions we collect all models that do show invariance in their predictive accuracy across settings and interventions the causal model will be a member of this set of models with high probability this approach yields valid confidence intervals for the causal relationships in quite general scenarios we examine the example of structural equation models in more detail and provide sufficient assumptions under which the set of causal predictors becomes identifiable we further investigate robustness properties of our approach under model misspecification and discuss possible extensions the empirical properties are studied for various data sets including large scale gene perturbation experiments

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

spectral regularization algorithms for learning large incomplete matrices we use convex relaxation techniques to provide a sequence of regularized low rank solutions for large scale matrix completion problems using the nuclear norm as a regularizer we provide a simple and very efficient convex algorithm for minimizing the reconstruction error subject to a bound on the nuclear norm our algorithm soft impute iteratively replaces the missing elements with those obtained from a soft thresholded svd with warm starts this allows us to efficiently compute an entire regularization path of solutions on a grid of values of the regularization parameter the computationally intensive part of our algorithm is in computing a low rank svd of a dense matrix exploiting the problem structure we show that the task can be performed with a complexity of order linear in the matrix dimensions our semidefinite programming algorithm is readily scalable to large matrices for example soft impute takes a few hours to compute low rank approximations of a 106 x 106 incomplete matrix with 107 observed entries and fits a rank 95 approximation to the full netflix training set in 3 3 hours our methods achieve good training and test errors and exhibit superior timings when compared to other competitive state of the art techniques

the solution path of the generalized lasso we present a path algorithm for the generalized lasso problem this problem penalizes the l1 norm of a matrix d times the coefficient vector and has a wide range of applications dictated by the choice of d our algorithm is based on solving the dual of the generalized lasso which greatly facilitates computation of the path for d i the usual lasso we draw a connection between our approach and the well known lars algorithm for an arbitrary d we derive an unbiased estimate of the degrees of freedom of the generalized lasso fit this estimate turns out to be quite intuitive in many applications

a view of margin losses as regularizers of probability estimates regularization is commonly used in classifier design to assure good generalization classical regularization enforces a cost on classifier complexity by constraining parameters this is usually combined with a margin loss which favors large margin decision rules a novel and unified view of this architecture is proposed by showing that margin losses act as regularizers of posterior class probabilities in a way that amplifies classical parameter regularization the problem of controlling the regularization strength of a margin loss is considered using a decomposition of the loss in terms of a link and a binding function the link function is shown to be responsible for the regularization strength of the loss while the binding function determines its outlier robustness a large class of losses is then categorized into equivalence classes of identical regularization strength or outlier robustness it is shown that losses in the same regularization class can be parameterized so as to have tunable regularization strength this parameterization is finally used to derive boosting algorithms with loss regularization boostlr three classes of tunable regularization losses are considered in detail canonical losses can implement all regularization behaviors but have no flexibility in terms of outlier modeling shrinkage losses support equally parameterized link and binding functions leading to boosting algorithms that implement the popular shrinkage procedure this offers a new explanation for shrinkage as a special case of loss based regularization finally tunable losses enable the independent parameterization of link and binding functions leading to boosting algorithms of great exibility this is illustrated by the derivation of an algorithm that generalizes both adaboost and logitboost behaving as either one when that best suits the data to classify various experiments provide evidence of the benefits of probability regularization for both classification and estimation of posterior class probabilities

a boosting approach to flexible semiparametric mixed models in linear mixed models the influence of covariates is restricted to a strictly parametric form with the rise of semi and non parametric regression also the mixed model has been expanded to allow for additive predictors the common approach uses the representation of additive models as mixed models an alternative approach that is proposed in the present paper is likelihood based boosting boosting originates in the machine learning community where it has been proposed as a technique to improve classification procedures by combining estimates with reweighted observations likelihood based boosting is a general method which may be seen as an extension of l2 boost in additive mixed models the advantage of boosting techniques in the form of componentwise boosting is that it is suitable for high dimensional settings where many explanatory variables are present it allows to fit additive models for many covariates with implicit selection of relevant variables and automatic selection of smoothing parameters moreover boosting techniques may be used to incorporate the subject specific variation of smooth influence functions by specifying random slopes on smooth effects this results in flexible semiparametric mixed models which are appropriate in cases where a simple random intercept is unable to capture the variation of effects across subjects

statistical analysis of bayes optimal subset ranking the ranking problem has become increasingly important in modern applications of statistical methods in automated decision making systems in particular we consider a formulation of the statistical ranking problem which we call subset ranking and focus on the discounted cumulated gain dcg criterion that measures the quality of items near the top of the rank list similar to error minimization for binary classification direct optimization of natural ranking criteria such as dcg leads to a nonconvex optimization problems that can be np hard therefore a computationally more tractable approach is needed we present bounds that relate the approximate optimization of dcg to the approximate minimization of certain regression errors these bounds justify the use of convex learning formulations for solving the subset ranking problem the resulting estimation methods are not conventional in that we focus on the estimation quality in the top portion of the rank list we further investigate the asymptotic statistical behavior of these formulations under appropriate conditions the consistency of the estimation schemes with respect to the dcg metric can be derived

boosting with the l2 loss this article investigates a computationally simple variant of boosting l2boost which is constructed from a functional gradient descent algorithm with the l2 loss function like other boosting algorithms l2boost uses many times in an iterative fashion a prechosen fitting method called the learner based on the explicit expression of refitting of residuals of l2boost the case with symmetric linear learners is studied in detail in both regression and classification in particular with the boosting iteration m working as the smoothing or regularization parameter a new exponential bias variance trade off is found with the variance complexity term increasing very slowly as m tends to infinity when the learner is a smoothing spline an optimal rate of convergence result holds for both regression and classification and the boosted smoothing spline even adapts to higher order unknown smoothness moreover a simple expansion of a smoothed 0 1 loss function is derived to reveal the importance of the d

deciding context unification in first order term unification variables represent well formed terms over a given signature and we are to solve equations built using function symbols from the signature and such variables this problem is well known to be decidable in linear time in second order term unification the variables take arguments i e other terms and a substitution uses those arguments an arbitrary number of times for instance an equation f x c x c e x f c c has a solution x e b where b is a special symbol denoting the place in which the argument is substituted under this substitution both sides evaluate to f c c there are other solutions for instance x e f b b which evaluates both sides tof f c c f c c in general a solution that evaluates both sides to full binary tree of arbitrary height is easy to construct second order unification is in general undecidable context unification is a natural problem in between first and second order unification we deal with equations over terms the variables take arguments but we restrict the set of solutions the argument is used exactly once formally contexts are terms with exactly one occurrence of the special symbol b and in context unification we are given an equation over terms with variables representing contexts and ask about the satisfiability of this equation for instance when the aforementioned equation f x c x c e x f c c is treated as a context unification problem then it has exactly one solution x e b other substitutions that are solutions of it as an instance of the second unification problem say x e f b b are not valid as b is used more than once context unification also generalizes satisfiability of word equations which is decidable in pspace the decidability status of context unification remained unknown for almost two decades in this article we show that context unification is in pspace in exptime when tree regular constraints are also allowed those results are obtained by extending the recently developed recompression technique which was previously defined for strings and used to obtain a new pspace algorithm for satisfiability of word equations in this article the technique is generalized to trees and the corresponding algorithm is generalized from word equations to context unification the idea of recompression is to apply simple compression rules replacing pairs of neighboring function symbols to the solution of the context equation to this end we appropriately modify the equation without the knowledge of the actual solution so compressing the solution can be simulated by compressing parts of the equation it is shown that if the compression operations are appropriately chosen then the size of the instance is polynomial during the whole algorithm thus giving a pspace upper bound

area stationary surfaces in the heisenberg group h1 we consider area stationary surfaces perhaps with a volume constraint in the heisenberg group h1 endowed with its carnot caratheodory distance by analyzing the first variation of area we characterize c2 area stationary surfaces as those with mean curvature zero or constant if a volume preserving condition is assumed and such that the characteristic curves meet orthogonally the singular curves moreover a minkowski type formula relating the area the mean curvature and the volume is obtained for volume preserving area stationary surfaces enclosing a given region as a consequence of the characterization of area stationary surfaces we refine the bernstein type theorem given in jih hsin cheng jenn fang hwang andrea malchiodi paul yang minimal surfaces in pseudohermitian geometry ann sc norm super pisa cl sci 5 4 1 2005 129 177 mr 2165405 and nicola garofalo scott pauls the bernstein problem in the heisenberg group arxiv math dg 0209065 to describe all c2 entire area stationary graphs over the xy plane in a calibration argument shows that these graphs are globally area minimizing finally by using the description of the singular set in jih hsin cheng jenn fang hwang andrea malchiodi paul yang minimal surfaces in pseudohermitian geometry ann sc norm super pisa cl sci 5 4 1 2005 129 177 mr 2165405 the characterization of area stationary surfaces and the ruling property of constant mean curvature surfaces we prove our main results where we classify area stationary surfaces in with or without a volume constraint and non empty singular set in particular we deduce the following counterpart to alexandrov uniqueness theorem in euclidean space any compact connected c2 surface in area stationary under a volume constraint must be congruent to a rotationally symmetric sphere obtained as the union of all the geodesics of the same curvature joining two points as a consequence we solve the isoperimetric problem in assuming c2 smoothness of the solutions 13 completely reducible hypersurfaces in a pencil pages 672 688 j v pereira s yuzvinsky preview pdf 324 k related articles 14 modules over motivic cohomology pages 689 727 oliver rondigs paul arne ostvaer preview pdf 303 k related articles 15 the fundamental group of period domains over finite fields pages 728 741 sascha orlik preview pdf 154 k related articles articles 1 15

contrast trees and distribution boosting a method for decision tree induction is presented given a set of predictor variables formula see text and two outcome variables y and z associated with each x the goal is to identify those values of x for which the respective distributions of formula see text and formula see text or selected properties of those distributions such as means or quantiles are most different contrast trees provide a lack of fit measure for statistical models of such statistics or for the complete conditional distribution formula see text as a function of x they are easily interpreted and can be used as diagnostic tools to reveal and then understand the inaccuracies of models produced by any learning method a corresponding contrast boosting strategy is described for remedying any uncovered errors thereby producing potentially more accurate predictions this leads to a distribution boosting strategy for directly estimating the full conditional distribution of y at each x under no assumptions concerning its shape form or parametric representation

credit spread approximation and improvement using random forest regression abstract credit default swap cds levels provide a market appreciation of companies default risk these derivatives are not always available creating a need for cds approximations this paper offers a simple global and transparent cds structural approximation which contrasts with more complex and proprietary approximations currently in use this equity to credit formula e2c inspired by creditgrades obtains better cds approximations according to empirical analyses based on a large sample spanning 2016 2018 a random forest regression run with this e2c formula and selected additional financial data results in an 87 3 out of sample accuracy in cds approximations the transparency property of this algorithm confirms the predominance of the e2c estimate and the impact of companies debt rating and size in predicting their cds

l1 regularization path algorithm for generalized linear models summary we introduce a path following algorithm for l1 regularized generalized linear models the l1 regularization procedure is useful especially because it in effect selects variables according to the amount of penalization on the l1 norm of the coefficients in a manner that is less greedy than forward selection backward deletion the generalized linear model path algorithm efficiently computes solutions along the entire regularization path by using the predictor corrector method of convex optimization selecting the step length of the regularization parameter is critical in controlling the overall accuracy of the paths we suggest intuitive and flexible strategies for choosing appropriate values we demonstrate the implementation with several simulated and real data sets

the rate of convergence of adaboost the adaboost algorithm was designed to combine many weak hypotheses that perform slightly better than random guessing into a strong hypothesis that has very low error we study the rate at which adaboost iteratively converges to the minimum of the exponential loss unlike previous work our proofs do not require a weak learning assumption nor do they require that minimizers of the exponential loss are finite our first result shows that the exponential loss of adaboost s computed parameter vector will be at most e more than that of any parameter vector of l1 norm bounded by b in a number of rounds that is at most a polynomial in b and 1 e we also provide lower bounds showing that a polynomial dependence is necessary our second result is that within c e iterations adaboost achieves a value of the exponential loss that is at most e more than the best possible value where c depends on the data set we show that this dependence of the rate on e is optimal up to constant factors that is at least 1 e rounds are necessary to achieve within e of the optimal exponential loss

a general framework for fast stagewise algorithms forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates it starts with all coefficients equal to zero and iteratively updates the coefficient by a small amount e of the variable that achieves the maximal absolute inner product with the current residual this procedure has an interesting connection to the lasso under some conditions it is known that the sequence of forward stagewise estimates exactly coincides with the lasso path as the step size e goes to zero furthermore essentially the same equivalence holds outside of least squares regression with the minimization of a differentiable convex loss function subject to an l1 norm constraint the stagewise algorithm now updates the coefficient corresponding to the maximal absolute component of the gradient even when they do not match their l1 constrained analogues stagewise estimates provide a useful approximation and are computationally appealing their success in sparse modeling motivates the question can a simple effective strategy like forward stagewise be applied more broadly in other regularization settings beyond the l1 norm and sparsity the current paper is an attempt to do just this we present a general framework for stagewise estimation which yields fast algorithms for problems such as group structured learning matrix completion image denoising and more

feature selection with ensembles artificial variables and redundancy elimination predictive models benefit from a compact non redundant subset of features that improves interpretability and generalization modern data sets are wide dirty mixed with both numerical and categorical predictors and may contain interactive effects that require complex models this is a challenge for filters wrappers and embedded feature selection methods we describe details of an algorithm using tree based ensembles to generate a compact subset of non redundant features parallel and serial ensembles of trees are combined into a mixed method that can uncover masking and detect features of secondary effect simulated and actual examples illustrate the effectiveness of the approach

training hard margin support vector machines using greedy stagewise algorithm hard margin support vector machines hm svms suffer from getting overfitting in the presence of noise soft margin svms deal with this problem by introducing a regularization term and obtain a state of the art performance however this disposal leads to a relatively high computational cost in this paper an alternative method greedy stagewise algorithm for svms named gs svms is presented to cope with the overfitting of hm svms without employing the regularization term the most attractive property of gs svms is that its computational complexity in the worst case only scales quadratically with the size of training samples experiments on the large data sets with up to 400 000 training samples demonstrate that gs svms can be faster than libsvm 2 83 without sacrificing the accuracy finally we employ statistical learning theory to analyze the empirical results which shows that the success of gs svms lies in that its early stopping rule can act as an implicit regularization term

bundle methods for regularized risk minimization a wide variety of machine learning problems can be described as minimizing a regularized risk functional with different algorithms using different notions of risk and different regularizers examples include linear support vector machines svms gaussian processes logistic regression conditional random fields crfs and lasso amongst others this paper describes the theory and implementation of a scalable and modular convex solver which solves all these estimation problems it can be parallelized on a cluster of workstations allows for data locality and can deal with regularizers such as l1 and l2 penalties in addition to the unified framework we present tight convergence bounds which show that our algorithm converges in o 1 e steps to e precision for general convex problems and in o log 1 e steps for continuously differentiable problems we demonstrate the performance of our general purpose solver on a variety of publicly available data sets

the adaboost flow we introduce a dynamical system that we call the adaboost flow the flow is defined by a system of odes with control we show that three algorithms of the adaboost family i the adaboost algorithm of schapire and freund ii the arc gv algorithm of breiman and iii the confidence rated prediction of schapire and singer can be embedded in the adaboost flow the nontrivial part of the adaboost flow equations coincides with the equations of dynamics of the nonperiodic toda system written in terms of spectral variables we provide a novel invariant geometrical description of the adaboost algorithm as a gradient flow on a foliation defined by level sets of the potential function we propose a new approach for constructing boosting algorithms as a continuous time gradient flow on measures defined by various metrics and potential functions finally we explain the similarity of the adaboost algorithm with the perelman s construction for the ricci flow 2015 wiley periodicals inc

an efficient modified boosting method for solving classification problems based on the adaboost algorithm a modified boosting method is proposed in this paper for solving classification problems this method predicts the class label of an example as the weighted majority voting of an ensemble of classifiers each classifier is obtained by applying a given weak learner to a subsample with size smaller than that of the original training set which is drawn from the original training set according to the probability distribution maintained over the training set a parameter is introduced into the reweighted scheme proposed in adaboost to update the probabilities assigned to training examples so that the algorithm can be more accurate than adaboost the experimental results on synthetic and several real world data sets available from the uci repository show that the proposed method improves the prediction accuracy the execution speed as well as the robustness to classification noise of adaboost furthermore the diversity accuracy patterns of the ensemble classifiers are investigated by kappa error diagrams

genericity results for singular curves let m be a smooth manifold and cal d m m geq 2 be the set of rank m distributions on m endowed with the whitney c infty topology we show the existence of an open set o m dense in cal d m so that every nontrivial singular curve of a distribution d of o m is of minimal order and of corank one in particular for m geq 3 every distribution of o m does not admit nontrivial rigid curves as a consequence for generic sub riemannian structures of rank greater than or equal to three there does not exist nontrivial minimizing singular curves

isoholonomic problems and some applications we study the problem of finding the shortest loops with a given holonomy we show that the solutions are the trajectories of particles in yang mills potentials theorem 4 or equivalently the projections of kaluza klein geodesics theorem 2 applications to quantum mechanics berry s phase sect 3 and the optimal control of deformable bodies sect 6 are touched upon

building sparse twin support vector machine classifiers in primal space abstract twin support vector machines tsvm obtain faster training speeds than classical support vector machines svm however tsvm augmented vectors lose sparsity in this paper a rapid sparse twin support vector machine stsvm classifier in primal space is proposed to improve the sparsity and robustness of tsvm based on a simple back fitting strategy the stsvm iteratively builds each nonparallel hyperplanes by adding one support vector sv from the corresponding class at one time this process is terminated using an adaptive and stable stopping criterion stsvm learning is implemented by linear equation computing systems through introducing a quadratic function to approximate the empirical risk the computational results on several synthetic and benchmark datasets indicate that the stsvm obtains a sparse separating hyperplane at a low cost without sacrificing its generalization performance

non minimality of corners in subriemannian geometry we give a short solution to one of the main open problems in subriemannian geometry namely we prove that length minimizers do not have corner type singularities with this result we solve problem ii of agrachev s list and provide the first general result toward the 30 year old open problem of regularity of subriemannian geodesics

on weak base hypotheses and their implications for boosting regression and classification when studying the training error and the prediction error for boosting it is often assumed that the hypotheses returned by the base learner are weakly accurate or are able to beat a random guesser by a certain amount of difference it has been an open question how much this difference can be whether it will eventually disappear in the boosting process or be bounded by a positive amount this question is crucial for the behavior of both the training error and the prediction error in this paper we study this problem and show affirmatively that the amount of improvement over the random guesser will be at least a positive amount for almost all possible sample realizations and for most of the commonly used base hypotheses this has a number of implications for the prediction error including for example that boosting forever may not be good and regularization may be necessary the problem is studied by first considering an analog of adaboost in regression where we study similar properties and find that for good performance one cannot hope to avoid regularization by just adopting the boosting device to regression

an l_ 2 boosting algorithm for estimation of a regression function an l 2 boosting algorithm for estimation of a regression function from random design is presented which consists of fitting repeatedly a function from a fixed nonlinear function space to the residuals of the data by least squares and by defining the estimate as a linear combination of the resulting least squares estimates splitting of the sample is used to decide after how many iterations of smoothing of the residuals the algorithm terminates the rate of convergence of the algorithm is analyzed in case of an unbounded response variable the method is used to fit a sum of maxima of minima of linear functions to a given data set and is compared with other nonparametric regression estimates using simulated data

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

adaptive piecewise polynomial estimation via trend filtering we study trend filtering a recently proposed tool of kim et al siam rev 51 2009 339 360 for nonparametric regression the trend filtering estimate is defined as the minimizer of a penalized least squares criterion in which the penalty term sums the absolute k th order discrete derivatives over the input points perhaps not surprisingly trend filtering estimates appear to have the structure of k th degree spline functions with adaptively chosen knot points we say appear here as trend filtering estimates are not really functions over continuous domains and are only defined over the discrete set of inputs this brings to mind comparisons to other nonparametric regression tools that also produce adaptive splines in particular we compare trend filtering to smoothing splines which penalize the sum of squared derivatives across input points and to locally adaptive regression splines ann statist 25 1997 387 413 which penalize the total variation of the k th derivative empirically we discover that trend filtering estimates adapt to the local level of smoothness much better than smoothing splines and further they exhibit a remarkable similarity to locally adaptive regression splines we also provide theoretical support for these empirical findings most notably we prove that with the right choice of tuning parameter the trend filtering estimate converges to the true underlying function at the minimax rate for functions whose k th derivative is of bounded variation this is done via an asymptotic pairing of trend filtering and locally adaptive regression splines which have already been shown to converge at the minimax rate ann statist 25 1997 387 413 at the core of this argument is a new result tying together the fitted values of two lasso problems that share the same outcome vector but have different predictor matrices

the reconstruction approach from interpolation to regression this article introduces an interpolation based method called the reconstruction approach for nonparametric regression based on the fact that interpolation usually has negligible errors compared

complexity of makanin s algorithm the exponent of periodicity is an important factor in estimates of complexity of word unification algorithms we prove that the exponent of periodicity of a minimal solution of a word equation is of order 2 1 07d where d is the length of the equation we also give a lower bound 2 0 29d so our upper bound is almost optimal and exponentially better than the original bound 6d 22d4 2 consequently our result implies an exponential improvement of known upper bounds on complexity of word unification algorithms

smoothness of subriemannian isometries we show that the group of isometries i e distance preserving homeomorphisms of an equiregular subriemannian manifold is a finite dimensional lie group of smooth transformations the proof is based on a new pde argument in the spirit of harmonic coordinates establishing that in an arbitrary subriemannian manifold there exists an open dense subset where all isometries are smooth

boosting as a regularized path to a maximum margin classifier in this paper we study boosting methods from a new perspective we build on recent work by efron et al to show that boosting approximately and in some cases exactly minimizes its loss criterion with an l1 constraint on the coefficient vector this helps understand the success of boosting with early stopping as regularized fitting of the loss criterion for the two most commonly used criteria exponential and binomial log likelihood we further show that as the constraint is relaxed or equivalently as the boosting iterations proceed the solution converges in the separable case to an l1 optimal separating hyper plane we prove that this l1 optimal separating hyper plane has the property of maximizing the minimal l1 margin of the training data as defined in the boosting literature an interesting fundamental similarity between boosting and kernel support vector machines emerges as both can be described as methods for regularized optimization in high dimensional predictor space using a computational trick to make the calculation practical and converging to margin maximizing solutions while this statement describes svms exactly it applies to boosting only approximately

approximation of grammar based compression via recompression in this paper we present a simple linear time algorithm constructing a context free grammar of size o g log n g for the input string where n is the size of the input string and g the size of the optimal grammar generating this string the algorithm works for arbitrary size alphabets but the running time is linear assuming that the alphabet of the input string can be identified with numbers from 1 n c for some constant c otherwise additional cost of o n log is needed algorithms with such an approximation guarantee and running time are known the novelty of this paper is a particular simplicity of the algorithm as well as the analysis of the algorithm which uses a general technique of recompression recently introduced by the author furthermore contrary to the previous results this work does not use the lz representation of the input string in the construction nor in the analysis

boosting algorithms for detector cascade learning the problem of learning classifier cascades is considered a new cascade boosting algorithm fast cascade boosting fcboost is proposed fcboost is shown to have a number of interesting properties namely that it 1 minimizes a lagrangian risk that jointly accounts for classification accuracy and speed 2 generalizes adaboost 3 can be made cost sensitive to support the design of high detection rate cascades and 4 is compatible with many predictor structures suitable for sequential decision making it is shown that a rich family of such structures can be derived recursively from cascade predictors of two stages denoted cascade generators generators are then proposed for two new cascade families last stage and multiplicative cascades that generalize the two most popular cascade architectures in the literature the concept of neutral predictors is finally introduced enabling fcboost to automatically determine the cascade configuration i e number of stages and number of weak learners per stage for the learned cascades experiments on face and pedestrian detection show that the resulting cascades outperform current state of the art methods in both detection accuracy and speed

the undecidability of the second order unification problem abstract it is shown that there is no effective procedure for determining whether or not two terms of the language of second order logic have a common instance

random forests and adaptive nearest neighbors in this article we study random forests through their connection with a new framework of adaptive nearest neighbor methods we introduce a concept of potential nearest neighbors k pnns and show that random forests can be viewed as adaptively weighted k pnn methods various aspects of random forests can be studied from this perspective we study the effect of terminal node sizes on the prediction accuracy of random forests we further show that random forests with adaptive splitting schemes assign weights to k pnns in a desirable way for the estimation at a given target point these random forests assign voting weights to the k pnns of the target point according to the local importance of different input variables we propose a new simple splitting scheme that achieves desirable adaptivity in a straightforward fashion this simple scheme can be combined with existing algorithms the resulting algorithm is computationally faster and gives comparable results other possible aspects of random forests such

approximation of smallest linear tree grammar a simple linear time algorithm for constructing a linear context free tree grammar of size o r g r g log n r g for a given input tree t of size n is presented where g is the size of a minimal linear context free tree grammar for t and r is the maximal rank of symbols in t which is a constant in many applications this is the first example of a grammar based tree compression algorithm with a good i e logarithmic in terms of the size of the input tree approximation ratio the analysis of the algorithm uses an extension of the recompression technique from strings to trees

finding all solutions of equations in free groups and monoids with involution this paper presents a pspace algorithm which yields a finite graph of exponential size that describes the set of all solutions of equations in free groups as well as the set of all solutions of equations with rational constraints in free monoids this became possible due to the recent recompression technique while this technique was successfully applied for pure word equations without involution or rational constraints it could not be used as a black box for free groups actually the presence of an involution and rational constraints complicates the situation and some additional analysis is necessary still the technique is general enough to accommodate both extensions in the end it simplifies proofs that satisfiability of word equations is in pspace and the corresponding result for equations in free groups with rational constraints as a byproduct we can decide in pspace whether the solution set is finite

the power of convex relaxation near optimal matrix completion this paper is concerned with the problem of recovering an unknown matrix from a small fraction of its entries this is known as the matrix completion problem and comes up in a great number of applications including the famous netflix prize and other similar questions in collaborative filtering in general accurate recovery of a matrix from a small number of entries is impossible but the knowledge that the unknown matrix has low rank radically changes this premise making the search for solutions meaningful this paper presents optimality results quantifying the minimum number of entries needed to recover a matrix of rank r exactly by any method whatsoever the information theoretic limit more importantly the paper shows that under certain incoherence assumptions on the singular vectors of the matrix recovery is possible by solving a convenient convex program as soon as the number of entries is on the order of the information theoretic limit up to logarithmic factors this convex program simply finds among all matrices consistent with the observed entries that with minimum nuclear norm as an example we show that on the order of nr log n samples are needed to recover a random n x n matrix of rank r by any method and to be sure nuclear norm minimization succeeds as soon as the number of entries is of the form nr polylog n

smooth e insensitive regression by loss symmetrization we describe new loss functions for regression problems along with an accompanying algorithmic framework which utilizes these functions these loss functions are derived by symmetrization of margin based losses commonly used in boosting algorithms namely the logistic loss and the exponential loss the resulting symmetric logistic loss can be viewed as a smooth approximation to the e insensitive hinge loss used in support vector regression we describe and analyze two parametric families of batch learning algorithms for minimizing these symmetric losses the first family employs an iterative log additive update which can be viewed as a regression counterpart to recent boosting algorithms the second family utilizes an iterative additive update step we also describe and analyze online gradient descent gd and exponentiated gradient eg algorithms for the symmetric logistic loss a byproduct of our work is a new simple form of regularization for boosting based classification and regression algorithms our regression framework also has implications on classification algorithms namely a new additive update boosting algorithm for classification we demonstrate the merits of our algorithms in a series of experiments

nonparametric regression and generalized linear models a roughness penalty approach preface introduction approaches to regression roughness penalties extensions of the roughness penalty approach computing the estimates further reading interpolating and smoothing splines one dimensional case further topics partial splines generalized linear models extending the model thin plate splines available software reference author index subject index

decidability of bounded second order unification it is well known that first order unification is decidable whereas second order and higher order unification is undecidable bounded second order unification is second order unification under the restriction that only a bounded number of bound variables in the instantiating terms for second order variables is permitted however the size of the instantiation is not restricted in this paper a decision algorithm for bounded second order unification is described this is the first nontrivial decidability result for second order unification where the signature is not restricted and there are no restrictions on the occurrences of variables this supports the claim that bounded second order unification is easier than context unification whose decidability is currently an open question

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

boosting iterative stochastic ensemble method for nonlinear calibration of subsurface flow models abstract a novel parameter estimation algorithm is proposed the inverse problem is formulated as a sequential data integration problem in which gaussian process regression gpr is used to integrate the prior knowledge static data the search space is further parameterized using karhunen loeve expansion to build a set of basis functions that spans the search space optimal weights of the reduced basis functions are estimated by an iterative stochastic ensemble method isem isem employs directional derivatives within a gauss newton iteration for efficient gradient estimation the resulting update equation relies on the inverse of the output covariance matrix which is rank deficient in the proposed algorithm we use an iterative regularization based on the l2 boosting algorithm l2 boosting iteratively fits the residual and the amount of regularization is controlled by the number of iterations a termination criteria based on akaike information criterion aic is utilized this regularization method is very attractive in terms of performance and simplicity of implementation the proposed algorithm combining isem and l2 boosting is evaluated on several nonlinear subsurface flow parameter estimation problems the efficiency of the proposed algorithm is demonstrated by the small size of utilized ensembles and in terms of error convergence rates

functional gradient ascent for probit regression this paper proposes two gradient based methods to fit a probit regression model by maximizing the sample log likelihood function using the property of the hessian of the objective function the first method performs weighted least square regression in each iteration of the newton raphson framework resulting in probitboost a boosting like algorithm motivated by the gradient boosting algorithm 10 the second proposed approach maximizes the sample log likelihood function by updating the fitted function a small step in the gradient direction performing gradient ascent in functional space resulting in gradient probitboost we also generalize the algorithms to multi class problems by two strategies one of which is to use the gradient ascent to maximize the multi class sample log likelihood function for fitting all the classifiers simultaneously and the second approach uses the one versus all scheme to reduce the multi class problem to a series of binary classification problems the proposed algorithms are tested on typical classification problems including face detection cancer classification and handwritten digit recognition the results show that compared to the alternative methods the proposed algorithms perform similar or better in terms of testing error rates

simultaneous variable selection we propose a new method for selecting a common subset of explanatory variables where the aim is to model several response variables the idea is a natural extension of the lasso technique proposed by tibshirani 1996 and is based on the joint residual sum of squares while constraining the parameter estimates to lie within a suitable polyhedral region the properties of the resulting convex programming problem are analyzed for the special case of an orthonormal design for the general case we develop an efficient interior point algorithm the method is illustrated on a dataset with infrared spectrometry measurements on 14 qualitatively different but correlated responses using 770 wavelengths the aim is to select a subset of the wavelengths suitable for use as predictors for as many of the responses as possible

a tutorial on propensity score estimation for multiple treatments using generalized boosted models the use of propensity scores to control for pretreatment imbalances on observed variables in non randomized or observational studies examining the causal effects of treatments or interventions has become widespread over the past decade for settings with two conditions of interest such as a treatment and a control inverse probability of treatment weighted estimation with propensity scores estimated via boosted models has been shown in simulation studies to yield causal effect estimates with desirable properties there are tools e g the twang package in r and guidance for implementing this method with two treatments however there is not such guidance for analyses of three or more treatments the goals of this paper are twofold 1 to provide step by step guidance for researchers who want to implement propensity score weighting for multiple treatments and 2 to propose the use of generalized boosted models gbm for estimation of the necessary propensity score weights we define the causal quantities that may be of interest to studies of multiple treatments and derive weighted estimators of those quantities we present a detailed plan for using gbm to estimate propensity scores and using those scores to estimate weights and causal effects we also provide tools for assessing balance and overlap of pretreatment variables among treatment groups in the context of multiple treatments a case study examining the effects of three treatment programs for adolescent substance abuse demonstrates the methods

the complexity of monadic second order unification monadic second order unification is second order unification where all function constants occurring in the equations are unary here we prove that the problem of deciding whether a set of monadic equations has a unifier is np complete where we use the technique of compressing solutions using singleton context free grammars we prove that monadic second order matching is also np complete

building support vector machines with reduced classifier complexity support vector machines svms though accurate are not preferred in applications requiring great classification speed due to the number of support vectors being large to overcome this problem we devise a primal method with the following properties 1 it decouples the idea of basis functions from the concept of support vectors 2 it greedily finds a set of kernel basis functions of a specified maximum size dmax to approximate the svm primal cost function well 3 it is efficient and roughly scales as o ndmax2 where n is the number of training examples and 4 the number of basis functions it requires to achieve an accuracy close to the svm accuracy is usually far less than the number of svm support vectors

gradient tree boosting for training conditional random fields conditional random fields crfs provide a flexible and powerful model for sequence labeling problems however existing learning algorithms are slow particularly in problems with large numbers of potential input features and feature combinations this paper describes a new algorithm for training crfs via gradient tree boosting in tree boosting the crf potential functions are represented as weighted sums of regression trees which provide compact representations of feature interactions so the algorithm does not explicitly consider the potentially large parameter space as a result gradient tree boosting scales linearly in the order of the markov model and in the order of the feature interactions rather than exponentially as in previous algorithms based on iterative scaling and gradient descent gradient tree boosting also makes it possible to use instance weighting as in c4 5 and surrogate splitting as in cart to handle missing values experimental studies of the effectiveness of these two methods as well as standard imputation and indicator feature methods show that instance weighting is the best method in most cases when feature values are missing at random

accurately estimating rigid transformations in registration using a boosting inspired mechanism feature extraction and matching provide the basis of many methods for object registration modeling retrieval and recognition however this approach typically introduces false matches due to lack of features noise occlusion and cluttered backgrounds in registration these false matches lead to inaccurate estimation of the underlying transformation that brings the overlapping shapes into best possible alignment in this paper we propose a novel boosting inspired method to tackle this challenging task it includes three key steps i underlying transformation estimation in the weighted least squares sense ii boosting parameter estimation and regularization via tsallis entropy and iii weight re estimation and regularization via shannon entropy and update with a maximum fusion rule the process is iterated the final optimal underlying transformation is estimated as a weighted average of the transformations estimated from the latest iterations with weights given by the boosting parameters a comparative study based on real shape data shows that the proposed method outperforms four other state of the art methods for evaluating the established point matches enabling more accurate and stable estimation of the underlying transformation highlightswe provide a novel method for evaluating point matches established by typical methods real data based experimental results show that the method outperforms the state of the art under certain conditions the finally estimated underlying transformation is optimal the transformation can be recovered with an error as small as 5 from typical point matches

functional dissipation microarrays for classification in this article we describe a new method of extracting information from signals called functional dissipation that proves to be very effective for enhancing classification of high resolution texture rich data our algorithm bypasses to some extent the need to have very specialized feature extraction techniques and can potentially be used as an intermediate feature enhancement step in any classification scheme functional dissipation is based on signal transforms but uses the transforms recursively to uncover new features we generate a variety of masking functions and extract features with several generalized matching pursuit iterations in each iteration the recursive process modifies several coefficients of the transformed signal with the largest absolute values according to the specific masking function in this way the greedy pursuit is turned into a slow controlled dissipation of the structure of the signal that for some masking functions enhances separation among classes our case study in this paper is the classification of crystallization patterns of amino acids solutions affected by the addition of small quantities of proteins

recompression a simple and powerful technique for word equations in this article we present an application of a simple technique of local recompression previously developed by the author in the context algorithms for compressed strings jez 2014a 2015b 2015a to word equations the technique is based on local modification of variables replacing x by ax or xa and iterative replacement of pairs of letters occurring in the equation by a fresh letter which can be seen as a bottom up compression of the solution of the given word equation or to be more specific building a straight line programme for the solution of the word equation using this technique we give new independent and self contained proofs of many known results for word equations to be more specific the presented nondeterministic algorithm runs in o n log n space and in time polynomial in n and log n where n is the size of the input equation and n the size of the length minimal solution of the word equation furthermore for o 1 variables the bound on the space consumption is in fact linear that is o m where m is the size of the space used by the input this yields that for each k the set of satisfiable word equations with k variables is context sensitive the presented algorithm can be easily generalised to a generator of all solutions of the given word equation without increasing the space usage furthermore a further analysis of the algorithm yields an independent proof of doubly exponential upper bound on the size of the length minimal solution the presented algorithm does not use exponential bound on the exponent of periodicity conversely the analysis of the algorithm yields an independent proof of the exponential bound on exponent of periodicity

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

nonconvex regularizations for feature selection in ranking with sparse svm feature selection in learning to rank has recently emerged as a crucial issue whereas several preprocessing approaches have been proposed only a few have focused on integrating feature selection into the learning process in this paper we propose a general framework for feature selection in learning to rank using support vector machines with a sparse regularization term we investigate both classical convex regularizations such as ell 1 or weighted ell 1 and nonconvex regularization terms such as log penalty minimax concave penalty or ell p pseudo norm with p two algorithms are proposed the first an accelerated proximal approach for solving the convex problems and the second a reweighted ell 1 scheme to address nonconvex regularizations we conduct intensive experiments on nine datasets from letor 3 0 and letor 4 0 corpora numerical results show that the use of nonconvex regularizations we propose leads to more sparsity in the resulting models while preserving the prediction performance the number of features is decreased by up to a factor of 6 compared to the ell 1 regularization in addition the software is publicly available on the web 1 available at http remi flamary com soft soft ranksvm nc html