estimation of gradients and coordinate covariation in classification we introduce an algorithm that simultaneously estimates a classification function as well as its gradient in the supervised learning framework the motivation for the algorithm is to find salient variables and estimate how they covary an efficient implementation with respect to both memory and time is given the utility of the algorithm is illustrated on simulated data as well as a gene expression data set an error analysis is given for the convergence of the estimate of the classification function and its gradient to the true classification function and true gradient

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

determinant bundles manifolds with boundary and surgery we define determinant bundles associated to the following data i a family of generalized dirac operators on even dimensional manifolds with boundary ii the choice of a spectral section for the family of dirac operators induced on the boundary under the assumption that the operators of the boundary family have null spaces of constant dimension we define through the notion ofb zeta function a quillen metric we also introduce the analogue of the bismut freed connection we prove that the curvature of a natural perturbation of the bismut freed connection equals the 2 form piece in the right hand side of the family index formula thus extending to manifolds with boundary results of quillen bismut and freed given a closed fibration we investigate the behaviour of the quillen metric and of the bismut freed connection under the operation of surgery along a fibering hypersurface we prove in particular additivity formulae for the curvature and for the logarithm of the holonomy

on the non negative garrotte estimator summary we study the non negative garrotte estimator from three different aspects consistency computation and flexibility we argue that the non negative garrotte is a general procedure that can be used in combination with estimators other than the original least squares estimator as in its original form in particular we consider using the lasso the elastic net and ridge regression along with ordinary least squares as the initial estimate in the non negative garrotte we prove that the non negative garrotte has the nice property that with probability tending to 1 the solution path contains an estimate that correctly identifies the set of important variables and is consistent for the coefficients of the important variables whereas such a property may not be valid for the initial estimators in general we show that the non negative garrotte can turn a consistent estimate into an estimate that is not only consistent in terms of estimation but also in terms of variable selection we also show that the non negative garrotte has a piecewise linear solution path using this fact we propose an efficient algorithm for computing the whole solution path for the non negative garrotte simulations and a real example demonstrate that the non negative garrotte is very effective in improving on the initial estimator in terms of variable selection and estimation accuracy

moyal quantization of 2 1 dimensional galilean systems stratonovich weyl kernels are constructed for some of the coadjoint orbits of the two dimensional extended galilean group g 2 1 as an intermediate step the unitary irreducible representations associated with a given group orbit are obtained by using the kirillov mackey theory star products are defined in the sense of moyal for functions on each of these orbits the central extension of g 2 1 with parameter k is also analyzed which results from the commutator between the generators of boosts to conclude that it originates a sort of nonrelativistic remainder of the thomas precession

learning coordinate covariances via gradients we introduce an algorithm that learns gradients from samples in the supervised learning framework an error analysis is given for the convergence of the gradient estimated by the algorithm to the true gradient the utility of the algorithm for the problem of variable selection as well as determining variable covariance is illustrated on simulated data as well as two gene expression data sets for square loss we provide a very efficient implementation with respect to both memory and time

functorial qft gauge anomalies and the dirac determinant bundle using properties of the determinant line bundle for a family of elliptic boundary value problems we explain how the fock space functor defines an axiomatic quantum field theory which formally models the fermionic path integral the sewing axiom of the theory arises as an algebraic pasting law for the determinant of the dirac operator we show how representations of the boundary gauge group fit into this description and that this leads to a fock functor description of certain gauge anomalies

kernel methods in machine learning we review machine learning methods employing positive definite kernels these methods formulate learning and estimation problems in a reproducing kernel hilbert space rkhs of functions defined on the data domain expanded in terms of a kernel working in linear spaces of function has the benefit of facilitating the construction and analysis of learning algorithms while at the same time allowing large classes of functions the latter include nonlinear functions as well as functions defined on nonvectorial data we cover a wide range of methods ranging from binary classifiers to sophisticated methods for estimation with structured data

multicategory support vector machines two category support vector machines svm have been very popular in the machine learning community for classification problems solving multicategory problems by a series of binary classifiers is quite common in the svm paradigm however this approach may fail under various circumstances we propose the multicategory support vector machine msvm which extends the binary svm to the multicategory case and has good theoretical properties the proposed method provides a unifying framework when there are either equal or unequal misclassification costs as a tuning criterion for the msvm an approximate leave one out cross validation function called generalized approximate cross validation is derived analogous to the binary case the effectiveness of the msvm is demonstrated through the applications to cancer classification using microarray data and cloud classification with satellite radiance profiles

local representation groups group cohomology techniques are used to derive conditions for a group g to be a local splitting group for g as a first step the exactness of the inflation restriction sequence gives a characterisation of factor systems arising in locally operating representations of transitive transformation groups some examples and applications of the theory are also given

hermite learning with gradient data the problem of learning from data involving function values and gradients is considered in a framework of least square regularized regression in reproducing kernel hilbert spaces the algorithm is implemented by a linear system with the coefficient matrix involving both block matrices for generating graph laplacians and hessians the additional data for function gradients improve learning performance of the algorithm error analysis is done by means of sampling operators for sample error and integral operators in sobolev spaces for approximation error

convex calibration dimension for multiclass loss matrices we study consistency properties of surrogate loss functions for general multiclass learning problems defined by a general multiclass loss matrix we extend the notion of classification calibration which has been studied for binary and multiclass 0 1 classification problems and for certain other specific learning problems to the general multiclass setting and derive necessary and sufficient conditions for a surrogate loss to be calibrated with respect to a loss matrix in this setting we then introduce the notion of convex calibration dimension of a multiclass loss matrix which measures the smallest size of a prediction space in which it is possible to design a convex surrogate that is calibrated with respect to the loss matrix we derive both upper and lower bounds on this quantity and use these results to analyze various loss matrices in particular we apply our framework to study various subset ranking losses and use the convex calibration dimension as a tool to show both the existence and non existence of various types of convex calibrated surrogates for these losses our results strengthen recent results of duchi et al 2010 and calauzenes et al 2012 on the non existence of certain types of convex calibrated surrogates in subset ranking we anticipate the convex calibration dimension may prove to be a useful tool in the study and design of surrogate losses for general multiclass learning problems

locally operating realizations of transformation lie groups using the mackey theory of induced representations a systematic study of the locally operating multiplier realizations of a connected lie group g that acts transitively on a space time manifold is presented we obtain a mathematical characterization of the locally operating multiplier realizations and a reduction of the problem of multiplier locally operating realizations to linear ones via a splitting group g m for g in this way the locally operating multiplier realizations are obtained by induction from finite dimensional linear representations of a well determined subgroup of g some examples such as the two dimensional euclidean group the galilei group and the one dimensional newton hooke group are given

improving the sample complexity using global data we study the sample complexity of proper and improper learning problems with respect to different q loss functions we improve the known estimates for classes which have relatively small covering numbers in empirical l sub 2 spaces e g log covering numbers which are polynomial with exponent p 2 we present several examples of relevant classes which have a small fat shattering dimension and hence fit our setup the most important of which are kernel machines

multicategory ψ learning in binary classification margin based techniques usually deliver high performance as a result a multicategory problem is often treated as a sequence of binary classifications in the absence of a dominating class this treatment may be suboptimal and may yield poor performance such as for support vector machines svms we propose a novel multicategory generalization of learning that treats all classes simultaneously the new generalization eliminates this potential problem while at the same time retaining the desirable properties of its binary counterpart we develop a statistical learning theory for the proposed methodology and obtain fast convergence rates for both linear and nonlinear learning examples we demonstrate the operational characteristics of this method through a simulation our results indicate that the proposed methodology can deliver accurate class prediction and is more robust against extreme observations than its svm counterpart

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

smooth discrimination analysis ties f and g can be based on the estimation of the set g x fx gx we consider applications where it is appropriate to assume that the region g has a smooth boundary or belongs to another nonparametric class of sets in particular this assumption makes sense if discrimination is used as a data analytic tool decision rules based on minimization of empirical risk over the whole class of sets and over sieves are considered their rates of convergence are obtained we show that these rules achieve optimal rates for estimation of g and optimal rates of convergence for bayes risks an interesting conclusion is that the optimal rates for bayes risks can be very fast in particular faster than the parametric root n rate these fast rates cannot be guaranteed for plug in rules

on the local equivalence in a unidimensional world the physical meaning of some semiunitary irreducible realizations of the unidimensional poincare and galilei groups including time inversion which are characteristic of a unidimensional world is investigated it is shown that in order to achieve this purpose the local equivalence is the most appropriate and it does not coincide with an equivalence of the global kind in this study use has been made of the theory of locally operating realizations developed by hoogland while dealing with the nontrivial factor systems the representation group technics were used

on the bayes risk consistency of regularized boosting methods the probability of error of classification methods based on convex combinations of simple base classifiers by boosting algorithms is investigated the main result of the paper is that certain regularized boosting algorithms provide bayes risk consistent classifiers under the sole assumption that the bayes classifier may be approximated by a convex combination of the base classifiers nonasymptotic distribution free bounds are also developed which offer interesting new insight into how boosting works and help explain its success in practical classification problems

statistical behavior and consistency of classification methods based on convex risk minimization we study how closely the optimal bayes error rate can be approximately reached using a classification algorithm that computes a classifier by minimizing a convex upper bound of the classification error function the measurement of closeness is characterized by the loss function used in the estimation we show that such a classification scheme can be generally regarded as a nonmaximum likelihood conditional in class probability estimate and we use this analysis to compare various convex loss functions that have appeared in the literature furthermore the theoretical insight allows us to design good loss functions with desirable properties another aspect of our analysis is to demonstrate the consistency of certain classification methods using convex risk minimization this study sheds light on the good performance of some recently proposed linear classification methods including boosting and support vector machines it also shows their limitations and suggests possible improvements

online learning with samples drawn from non identical distributions learning algorithms are based on samples which are often drawn independently from an identical distribution i i d in this paper we consider a different setting with samples drawn according to a non identical sequence of probability distributions each time a sample is drawn from a different distribution in this setting we investigate a fully online learning algorithm associated with a general convex loss function and a reproducing kernel hilbert space rkhs error analysis is conducted under the assumption that the sequence of marginal distributions converges polynomially in the dual of a holder space for regression with least square or insensitive loss learning rates are given in both the rkhs norm and the l2 norm for classification with hinge loss and support vector machine q norm loss rates are explicitly stated with respect to the excess misclassification error

index theory gerbes and hamiltonian quantization we give an atiyah patodi singer index theory construction of the bundle of fermionic fock spaces parametrized by vector potentials in odd space dimensions and prove that this leads in a simple manner to the known schwinger terms faddeev mickelsson cocycle for the gauge group action we relate the aps construction to the bundle gerbe approach discussed recently by carey and murray including an explicit computation of the dixmier douady class an advantage of our method is that it can be applied whenever one has a form of the aps theorem at hand as in the case of fermions in an external gravitational field

learning gradients by a gradient descent algorithm we propose a stochastic gradient descent algorithm for learning the gradient of a regression function from random samples of function values this is a learning algorithm involving mercer kernels by a detailed analysis in reproducing kernel hilbert spaces we provide some error bounds to show that the gradient estimated by the algorithm converges to the true gradient under some natural conditions on the regression function and suitable choices of the step size and regularization parameters

svm soft margin classifiers linear programming versus quadratic programming support vector machine svm soft margin classifiers are important learning algorithms for classification problems they can be stated as convex optimization problems and are suitable for a large data setting linear programming svm classifiers are especially efficient for very large size samples but little is known about their convergence compared with the well understood quadratic programming svm classifier in this article we point out the difficulty and provide an error analysis our analysis shows that the convergence behavior of the linear programming svm is almost the same as that of the quadratic programming svm this is implemented by setting a stepping stone between the linear programming svm and the classical 1 norm soft margin classifier an upper bound for the misclassification error is presented for general probability distributions explicit learning rates are derived for deterministic and weakly separable distributions and for distributions satisfying some tsybakov noise condition

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

local realizations of kinematical groups with a constant electromagnetic field i the relativistic case this paper is devoted to the study of the description of elementary physical systems interacting with an external constant electromagnetic field and the construction of their differential wave equations from a group theoretical point of view in this context certain local realizations of the poincare group are studied the linearization of this problem is carried out by building the associated representation group that turns out to be the well known maxwell group in this way the usual method concerning local realizations that has been employed in studying free systems to the interacting case is extended

convexity classification and risk bounds many of the classification algorithms developed in the machine learning literature including the support vector machine and boosting can be viewed as minimum contrast methods that minimize a convex surrogate of the 0 1 loss function the convexity makes these algorithms computationally efficient the use of a surrogate however has statistical consequences that must be balanced against the computational virtues of convexity to study these issues we provide a general quantitative relationship between the risk as assessed using the 0 1 loss and the risk as assessed using any nonnegative surrogate loss function we show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function that it satisfies a pointwise form of fisher consistency for classification the relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications we also present a refined version of this result in the

vector bundle representations of groups in quantum physics the theory of vector bundle representations of lie groups g is developed induced and locally operating linear representations of g are shown to be generated by vector bundle representations and gauge equivalence of locally operating linear representations is shown to correspond to equivalence of vector bundle representations it is pointed out that every vector bundle representation is equivalent to an induced one some applications to the quantum systems associated with monopole and instanton gauge field configurations are also discussed

prediction games and arcing algorithms the theory behind the success of adaptive reweighting and combining algorithms arcing such as adaboost freund schapire 1996a 1997 and others in reducing generalization error has not been well understood by formulating prediction as a game where one player makes a selection from instances in the training set and the other a convex linear combination of predictors from a finite set existing arcing algorithms are shown to be algorithms for finding good game strategies the minimax theorem is an essential ingredient of the convergence proofs an arcing algorithm is described that converges to the optimal strategy a bound on the generalization error for the combined predictors in terms of their maximum error is proven that is sharper than bounds to date schapire freund bartlett and lee 1997 offered an explanation of why adaboost works in terms of its ability to produce generally high margins the empirical comparison of adaboost to the optimal arcing algorithm shows that their explanation is n

classification using geometric level sets a variational level set method is developed for the supervised classification problem nonlinear classifier decision boundaries are obtained by minimizing an energy functional that is composed of an empirical risk term with a margin based loss and a geometric regularization term new to machine learning the surface area of the decision boundary this geometric level set classifier is analyzed in terms of consistency and complexity through the calculation of its e entropy for multicategory classification an efficient scheme is developed using a logarithmic number of decision functions in the number of classes rather than the typical linear number of decision functions geometric level set classification yields performance results on benchmark data sets that are competitive with well established methods

robust gradient learning with applications this paper addresses the robust gradient learning rgl problem gradient learning models aim at learning the gradient vector of some target functions in supervised learning problems which can be further used to applications such as variable selection coordinate covariance estimation and supervised dimension reduction however existing gl models are not robust to outliers or heavy tailed noise this paper provides an rgl framework to address this problem in both regression and classification this is achieved by introducing a robust regression loss function and proposing a robust classification loss moreover our rgl algorithm works in an instance based kernelized dictionary instead of some fixed reproducing kernel hilbert space which may provide more flexibility to solve the proposed nonconvex model a simple computational algorithm based on gradient descent is provided and the convergence of the proposed method is also analyzed we then apply the proposed rgl model to applications such as nonlinear variable selection and coordinate covariance estimation the efficiency of our proposed model is verified on both synthetic and real data sets

process consistency for adaboost recent experiments and theoretical studies show that adaboost can overfit in the limit of large time if running the algorithm forever is suboptimal a natural question is how low can the prediction error be during the process of adaboost we show under general regularity conditions that during the process of adaboost a consistent prediction is generated which has the prediction error approximating the optimal bayes error as the sample size increases this result suggests that while running the algorithm forever can be suboptimal it is reasonable to expect that some regularization method via truncation of the process may lead to a near optimal performance for sufficiently large sample size

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

higher degree analogs of the determinant line bundle in the first part of this paper given a smooth family of dirac type operators on an odd dimensional closed manifold we construct an abelian gerbe with connection whose curvature is the three form component of the atiyah singer families index the orem in the second part of the paper given a smooth family of dirac type operators whose index lies in the subspace b of the reduced k theory of the parametrizing space we construct a set of deligne cohomology classes of degree i whose curvatures are the i form component of the atiyah singer families index theorem

refined generalization bounds of gradient learning over reproducing kernel hilbert spaces gradient learning gl initially proposed by mukherjee and zhou 2006 has been proved to be a powerful tool for conducting variable selection and dimensional reduction simultaneously this approach presents a nonparametric version of a gradient estimator with positive definite kernels without estimating the true function itself so that the proposed version has wide applicability and allows for complex effects between predictors in terms of theory however existing generalization bounds for gl depend on capacity independent techniques and the capacity of kernel classes cannot be characterized completely thus this letter considers gl estimators that minimize the empirical convex risk we prove generalization bounds for such estimators with rates that are faster than previous results moreover we provide a novel upper bound for rademacher chaos complexity of order two which also plays an important role in general pairwise type estimations including ranking and score problems

locally operating realizations of nonconnected transformation lie groups a systematic study of locally operating multiplier realizations of nonconnected lie groups of transformations is presented that generalizes previous results on connected groups the semilinear locally operating multiplier realizations of a nonconnected group g are those obtained through an induction process from the finite dimensional semilinear representations of a given subgroup of a representation group g for g

consistency of support vector machines and other regularized kernel classifiers it is shown that various classifiers that are based on minimization of a regularized risk are universally consistent i e they can asymptotically learn in every classification task the role of the loss functions used in these algorithms is considered in detail as an application of our general framework several types of support vector machines svms as well as regularization networks are treated our methods combine techniques from stochastics approximation theory and functional analysis

of quantiles and expectiles consistent scoring functions choquet representations and forecast rankings summary in the practice of point prediction it is desirable that forecasters receive a directive in the form of a statistical functional for example forecasters might be asked to report the mean or a quantile of their predictive distributions when evaluating and comparing competing forecasts it is then critical that the scoring function used for these purposes be consistent for the functional at hand in the sense that the expected score is minimized when following the directive we show that any scoring function that is consistent for a quantile or an expectile functional can be represented as a mixture of elementary or extremal scoring functions that form a linearly parameterized family scoring functions for the mean value and probability forecasts of binary events constitute important examples the extremal scoring functions admit appealing economic interpretations of quantiles and expectiles in the context of betting and investment problems the choquet type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function in empirical settings it suffices to compare the average scores for only a finite number of extremal elements plots of the average scores with respect to the extremal scoring functions which we call murphy diagrams permit detailed comparisons of the relative merits of competing forecasts

distance weighted discrimination high dimension low sample size statistical analysis is becoming increasingly important in a wide range of applied contexts in such situations the popular support vector machine suffers from data piling at the margin which can diminish generalizability this leads naturally to the development of distance weighted discrimination which is based on second order cone programming a modern computationally intensive optimization method

universally consistent vertex classification for latent positions graphs in this work we show that using the eigen decomposition of the adjacency matrix we can consistently estimate feature maps for latent position graphs with positive definite link function provided that the latent positions are i i d from some distribution f we then consider the exploitation task of vertex classification where the link

online pairwise learning algorithms with convex loss functions online pairwise learning algorithms with general convex loss functions without regularization in a reproducing kernel hilbert space rkhs are investigated under mild conditions on loss functions and the rkhs upper bounds for the expected excess generalization error are derived in terms of the approximation error when the stepsize sequence decays polynomially in particular for lipschitz loss functions such as the hinge loss the logistic loss and the absolute value loss the bounds can be of order o t13logt after t iterations while for the least squares loss the bounds can be of order o t14logt in comparison with previous works for these algorithms a broader family of convex loss functions is studied here and refined upper bounds are obtained

generalization performance of bipartite ranking algorithms with convex losses abstract previous works describing the generalization performance of bipartite ranking algorithms are usually based on the assumption of 0 1 loss or the area under the receiver operating characteristic roc curve in this paper we go far beyond this classical framework by investigating the generalization performance of bipartite ranking algorithms with convex losses over reproducing kernel hilbert spaces based on the mcdiarmid inequality and rademacher complexity we establish the upper bound on the generalization error for a bipartite ranking algorithm the theoretical analysis is different from the previous results on error analysis and shows the attractive uniform convergence property of regularized bipartite ranking algorithms

mickelsson rajeev cocycle corresponding to dimension five we construct a mickelsson rajeev type cocycle corresponding to five space dimensions in the three dimensional case this cocycle is shown to be equivalent to the original cocycle proposed by mickelsson and rajeev furthermore we construct a local representative for this cocycle and evaluate it explicitly on a five dimensional torus

derivative reproducing properties for kernel methods in learning theory the regularity of functions from reproducing kernel hilbert spaces rkhss is studied in the setting of learning theory we provide a reproducing property for partial derivatives up to order s when the mercer kernel is c 2 s for such a kernel on a general domain we show that the rkhs can be embedded into the function space c s these observations yield a representer theorem for regularized learning algorithms involving data for function values and gradients examples of hermite learning and semi supervised learning penalized by gradients on data are considered

gravitational anomalies gerbes and hamiltonian quantization in ref 1 schwinger terms in hamiltonian quantization of chiral fermions coupled to vector potentials were computed using some ideas from the theory of gerbes with the help of the family index theorem for a manifold with boundary here we generalize this method to include gravitational schwinger terms

refinement of operator valued reproducing kernels this paper studies the construction of a refinement kernel for a given operator valued reproducing kernel such that the vector valued reproducing kernel hilbert space of the refinement kernel contains that of the given kernel as a subspace the study is motivated from the need of updating the current operator valued reproducing kernel in multi task learning when underfitting or overfitting occurs numerical simulations confirm that the established refinement kernel method is able to meet this need various characterizations are provided based on feature maps and vector valued integral representations of operator valued reproducing kernels concrete examples of refining translation invariant and finite hilbert schmidt operator valued reproducing kernels are provided other examples include refinement of hessian of scalar valued translation invariant kernels and transformation kernels existence and properties of operator valued reproducing kernels preserved during the refinement process are also investigated

boosting with early stopping convergence and consistency boosting is one of the most significant advances in machine learning for classification and regression in its original and computationally flexible version boosting seeks to minimize empirically a loss function in a greedy fashion the resulting estimator takes an additive function form and is built iteratively by applying a base estimator or learner to updated samples depending on the previous iterations an unusual regularization technique early stopping is employed based on cv or a test set this paper studies numerical convergence consistency and statistical rates of convergence of boosting with early stopping when it is carried out over the linear span of a family of basis functions for general loss functions we prove the convergence of boosting s greedy optimization to the infinimum of the loss function over the linear span using the numerical convergence result we find early stopping strategies under which boosting is shown to be consistent based on i i d samples and we obtain bounds on the rates of convergence for boosting estimators simulation studies are also presented to illustrate the relevance of our theoretical results for providing insights to practical aspects of boosting as a side product these results also reveal the importance of restricting the greedy search step sizes as known in practice through the work of friedman and others moreover our results lead to a rigorous proof that for a linearly separable problem adaboost with e 0 step size becomes an l 1 margin maximizer when left to run to convergence

from tracial anomalies to anomalies in quantum field theory regularized traces resp super traces are defined on a classical pseudo differential operator a by tr q a f p tr a q z vert z 0 quad resp quad str q a f p str a q z vert z 0 where f p refers to the finite part and q is an invertible and admissible elliptic reference operator with positive order they are commonly used in quantum field theory in spite of the fact that unlike ordinary traces on matrices they are neither cyclic nor do they commute with exterior differentiation thus giving rise to tracial anomalies the purpose of this article is to show on two examples how tracial anomalies can lead to anomalous phenomena in quantum field theory

online pairwise learning algorithms pairwise learning usually refers to a learning task that involves a loss function depending on pairs of examples among which the most notable ones are bipartite ranking metric learning and auc maximization in this letter we study an online algorithm for pairwise learning with a least square loss function in an unconstrained setting of a reproducing kernel hilbert space rkhs that we refer to as the online pairwise learning algorithm opera in contrast to existing works kar sriperumbudur jain karnick 2013 wang khardon pechyony jones 2012 which require that the iterates are restricted to a bounded domain or the loss function is strongly convex opera is associated with a non strongly convex objective function and learns the target function in an unconstrained rkhs specifically we establish a general theorem that guarantees the almost sure convergence for the last iterate of opera without any assumptions on the underlying distribution explicit convergence rates are derived under the condition of polynomially decaying step sizes we also establish an interesting property for a family of widely used kernels in the setting of pairwise learning and illustrate the convergence results using such kernels our methodology mainly depends on the characterization of rkhss using its associated integral operators and probability inequalities for random variables with values in a hilbert space

learning gradients predictive models that infer geometry and statistical dependence the problems of dimension reduction and inference of statistical dependence are addressed by the modeling framework of learning gradients the models we propose hold for euclidean spaces as well as the manifold setting the central quantity in this approach is an estimate of the gradient of the regression or classification function two quadratic forms are constructed from gradient estimates the gradient outer product and gradient based diffusion maps the first quantity can be used for supervised dimension reduction on manifolds as well as inference of a graphical model encoding dependencies that are predictive of a response variable the second quantity can be used for nonlinear projections that incorporate both the geometric structure of the manifold as well as variation of the response variable on the manifold we relate the gradient outer product to standard statistical quantities such as covariances and provide a simple and precise comparison of a variety of supervised dimensionality reduction methods we provide rates of convergence for both inference of informative directions as well as inference of a graphical model of variable dependencies

population theory for boosting ensembles tree ensembles are looked at in distribution space that is the limit case of infinite sample size it is shown that the simplest kind of trees is complete in d dimensional l 2 p space if the number of terminal nodes t is greater than d for such trees we show that the adaboost algorithm gives an ensemble converging to the bayes risk

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

on learning vector valued functions in this letter we provide a study of learning in a hilbert space of vectorvalued functions we motivate the need for extending learning theory of scalar valued functions by practical considerations and establish some basic results for learning vector valued functions that should prove useful in applications specifically we allow an output space y to be a hilbert space and we consider a reproducing kernel hilbert space of functions whose values lie in y in this setting we derive the form of the minimal norm interpolant to a finite set of data and apply it to study some regularization functionals that are important in learning theory we consider specific examples of such functionals corresponding to multiple output regularization networks and support vector machines for both regression and classification finally we provide classes of operator valued kernels of the dot product and translation invariant type

inegalites de concentration pour les processus empiriques de classes de parties we propose new concentration inequalities for maxima of set indexed empirical processes our approach is based either on entropy inequalities or on martingale methods the improvements we get concern the rate function which is exactly the large deviations rate function of a binomial law in most of the cases

on ψ learning the concept of large margins have been recognized as an important principle in analyzing learning methodologies including boosting neural networks and support vector machines svms however this concept alone is not adequate for learning in nonseparable cases we propose a learning methodology called learning that is derived from a direct consideration of generalization errors we provide a theory for learning and show that it essentially attains the optimal rates of convergence in two learning examples finally results from simulation studies and from breast cancer classification confirm the ability of learning to outperform svm in generalization

local rademacher complexities we propose new bounds on the error of learning algorithms in terms of a data dependent notion of complexity the estimates we establish give optimal rates and are based on a local and empirical version of rademacher averages in the sense that the rademacher averages are computed from the data on a subset of functions with small empirical error we present some applications to classification and prediction with convex function classes and with kernel classes in particular

complexity regularization via localized random penalties in this article model selection via penalized empirical loss minimization in nonparametric classification problems is studied data dependent penalties are constructed which are based on estimates of the complexity of a small subclass of each model class containing only those functions with small empirical loss the penalties are novel since those considered in the literature are typically based on the entire model class oracle inequalities using these penalties are established and the advantage of the new penalties over those based on the complexity of the whole model class is demonstrated

on bregman distances and divergences of probability measures this paper introduces scaled bregman distances of probability distributions which admit nonuniform contributions of observed events they are introduced in a general form covering not only the distances of discrete and continuous stochastic observations but also the distances of random processes and signals it is shown that the scaled bregman distances extend not only the classical ones studied in the previous literature but also the information divergence and the related wider class of convex divergences of probability measures an information processing theorem is established too but only in the sense of invariance w r t statistically sufficient transformations and not in the sense of universal monotonicity pathological situations where coding can increase the classical bregman distance are illustrated by a concrete example in addition to the classical areas of application of the bregman distances and convex divergences such as recognition classification learning and evaluation of proximity of various features and signals the paper mentions a new application in 3 d exploratory data analysis explicit expressions for the scaled bregman distances are obtained in general exponential families with concrete applications in the binomial poisson and rayleigh families and in the families of exponential processes such as the poisson and diffusion processes including the classical examples of the wiener process and geometric brownian motion

locally operating realizations of groups and superequivalence of factor systems the unitary multiplier locally operating realizations of a transitive group of transformations g are identified with those obtained by an induction process from linear finite dimensional representations of a group related with the isotopy group of the given action in this induction process each pseudoequivalence class of linear representations of splits into several gauge equivalence classes of unitary multiplier locally operating realizations of g the concept of superequivalence of factor systems appears here in a natural way