on the spaces lp x ω and wm p x ω in this paper we present some basic results on the generalized lebesgue spaces l p x omega and generalized lebesgue sobolev spaces w m w p x omega these results provide the necessary framework for the study of variational problems and elliptic equations with non standard p x growth conditions c 2001 academic press

boosting the margin a new explanation for the effectiveness of voting methods one of the surprising recurring phenomena observed in experiments with boosting is that the test error of the generated classifier usually does not increase as its size becomes very large and often is observed to decrease even after the training error reaches zero in this paper we show that this phenomenon is related to the distribution of margins of the training examples with respect to the generated voting classification rule where the margin of an example is simply the difference between the number of correct votes and the maximum number of votes received by any incorrect label we show that techniques used in the analysis of vapnik s support vector classifiers and of neural networks with small weights can be applied to voting methods to relate the margin distribution to the test error we also show theoretically and experimentally that boosting is especially effective at increasing the margins of the training examples finally we compare our explanation to those based on the bias variance decomposition

on variational inequalities with maximal monotone operators and multivalued perturbing terms in sobolev spaces with variable exponents abstract we are concerned in this paper with variational inequalities of the form a u v u f u v u l v u v k u k where a is a maximal monotone operator f is an integral multivalued lower order term and k is a closed convex set in a sobolev space of variable exponent we study both coercive and noncoercive inequalities in the noncoercive case a sub supersolution approach is followed to obtain the existence and some other qualitative properties of solutions between sub and supersolutions

a clustering technique for the identification of piecewise affine systems we propose a new technique for the identification of discrete time hybrid systems in the piecewise affine pwa form this problem can be formulated as the reconstruction of a possibly discontinuous pwa map with a multi dimensional domain in order to achieve our goal we provide an algorithm that exploits the combined use of clustering linear identification and pattern recognition techniques this allows to identify both the affine submodels and the polyhedral partition of the domain on which each submodel is valid avoiding gridding procedures moreover the clustering step used for classifying the datapoints is performed in a suitably defined feature space which allows also to reconstruct different submodels that share the same coefficients but are defined on different regions measures of confidence on the samples are introduced and exploited in order to improve the performance of both the clustering and the final linear regression procedure

greedy algorithms for classification consistency convergence rates and adaptivity many regression and classification algorithms proposed over the years can be described as greedy procedures for the stagewise minimization of an appropriate cost function some examples include additive models matching pursuit and boosting in this work we focus on the classification problem for which many recent algorithms have been proposed and applied successfully for a specific regularized form of greedy stagewise optimization we prove consistency of the approach under rather general conditions focusing on specific classes of problems we provide conditions under which our greedy procedure achieves the nearly minimax rate of convergence implying that the procedure cannot be improved in a worst case setting we also construct a fully adaptive procedure which without knowing the smoothness parameter of the decision boundary converges at the same rate as if the smoothness parameter were known

on the generalization ability of on line learning algorithms in this paper it is shown how to extract a hypothesis with small risk from the ensemble of hypotheses generated by an arbitrary on line learning algorithm run on an independent and identically distributed i i d sample of data using a simple large deviation argument we prove tight data dependent bounds for the risk of this hypothesis in terms of an easily computable statistic m sub n associated with the on line performance of the ensemble via sharp pointwise bounds on m sub n we then obtain risk tail bounds for kernel perceptron algorithms in terms of the spectrum of the empirical kernel matrix these bounds reveal that the linear hypotheses found via our approach achieve optimal tradeoffs between hinge loss and margin size over the class of all linear functions an issue that was left open by previous results a distinctive feature of our approach is that the key tools for our analysis come from the model of prediction of individual sequences i e a model making no probabilistic assumptions on the source generating the data in fact these tools turn out to be so powerful that we only need very elementary statistical facts to obtain our final risk bounds

existence and multiplicity of solutions for quasilinear nonhomogeneous problems an orlicz sobolev space setting abstract we study the boundary value problem div log 1 u q u p 2 u f u in u 0 on where is a bounded domain in r n with smooth boundary we distinguish the cases where either f u u p 2 u u r 2 u or f u u p 2 u u r 2 u with p q 1 p q min n r and r n p n p n p in the first case we show the existence of infinitely many weak solutions for any 0 in the second case we prove the existence of a nontrivial weak solution if is sufficiently large our approach relies on adequate variational methods in orlicz sobolev spaces

on a sub supersolution method for variational inequalities with leray lions operators in variable exponent spaces abstract in this paper we consider a sub supersolution method for variational inequalities with leray lions operators in sobolev spaces with variable exponents existence and qualitative properties of solutions between sub and supersolutions are established

multivariate adaptive regression splines a new method is presented for flexible regression modeling of high dimensional data the model takes the form of an expansion in product spline basis functions where the number of basis functions as well as the parameters associated with each one product degree and knot locations are automatically determined by the data this procedure is motivated by the recursive partitioning approach to regression and shares its attractive properties unlike recursive partitioning however this method produces continuous models with continuous derivatives it has more power and flexibility to model relationships that are nearly additive or involve interactions in at most a few variables in addition the model can be represented in a form that separately identifies the additive contributions and those associated with the different multivariable interactions

a multiplicity result for a nonlinear degenerate problem arising in the theory of electrorheological fluids the goal of this correction is to correct a mistake that appears in the proof of both lemma 3 9 and 3 10 in the article mih ilescu r dulescu 2006 1 the mistake in the proof of lemma 3 9 and 3 10 is mainly owing to the fact that we wrongly computed the expression of function g x t

elliptic inequalities with multi valued operators existence comparison and related variational hemivariational type inequalities abstract we study multi valued elliptic variational inclusions in a bounded domain r n of the form u k 0 a u i k u f u f u where a is a second order quasilinear elliptic operator of leray lions type k is a closed convex subset of some sobolev space i k is the indicator function related to k and i k denoting its subdifferential the lower order multi valued operators f and f are generated by multi valued upper semicontinuous functions f r 2 r 0 and f r 2 r 0 respectively with our main goals are as follows first we provide an existence theory for the above multi valued variational inequalities second we establish an enclosure and comparison principle based on appropriately defined sub supersolutions and prove the existence of extremal solutions third by means of the sub supersolution method provided here we are going to show that rather general classes of variational hemivariational type inequalities turn out to be only subclasses of the above general multi valued elliptic variational inequalities which in a way fills a gap in the current literature where these kind of problems are studied independently finally the existence of extremal solutions will allow us to deal with classes of multi valued function f and f that are neither lower nor upper semicontinuous which in turn will provide a tool to obtain existence results for variational hemivariational type inequalities whose clarke s generalized directional derivative may in addition discontinuously depend on the function we are looking for this paper though of surveying nature provides an analytical framework that allows to present in a unifying way and to extend a number of recent results due to the authors

a systematic approach to the soliton equations of a discrete eigenvalue problem the ablowitz ladik al problem is a linear vector difference equation whose isospectral flow equations include several important soliton equations e g the discrete nonlinear schrodinger equation iqn qn 1 2qn qn 1 qn 2 qn 1 qn 1 there is an established procedure for describing the soliton hierarchy of the more familiar akns ablowitz kaup newell and segur problem it is based on the notion of a generator for the hierarchy in this paper the soliton equations of the al hierarchy are described and characterized by a generator pair a new continuous spectral problem is introduced and the akns hierarchy is embedded in its hierarchy as a specialization

rademacher penalties and structural risk minimization we suggest a penalty function to be used in various problems of structural risk minimization this penalty is data dependent and is based on the sup norm of the so called rademacher process indexed by the underlying class of functions sets the standard complexity penalties used in learning problems and based on the vc dimensions of the classes are conservative upper bounds in a probabilistic sense uniformly over the set of all underlying distributions for the penalty we suggest thus for a particular distribution of training examples one can expect better performance of learning algorithms with the data driven rademacher penalties we obtain oracle inequalities for the theoretical risk of estimators obtained by structural minimization of the empirical risk with rademacher penalties the inequalities imply some form of optimality of the empirical risk minimizers we also suggest an iterative approach to structural risk minimization with rademacher penalties in which the hierarchy of classes is not given in advance but is determined in the data driven iterative process of risk minimization we prove probabilistic oracle inequalities for the theoretical risk of the estimators based on this approach as well

pac bayesian generalisation error bounds for gaussian process classification approximate bayesian gaussian process gp classification techniques are powerful non parametric learning methods similar in appearance and performance to support vector machines based on simple probabilistic models they render interpretable results and can be embedded in bayesian frameworks for model selection feature selection etc in this paper by applying the pac bayesian theorem of mcallester 1999a we prove distribution free generalisation error bounds for a wide range of approximate bayesian gp classification techniques we also provide a new and much simplified proof for this powerful theorem making use of the concept of convex duality which is a backbone of many machine learning techniques we instantiate and test our bounds for two particular gpc techniques including a recent sparse method which circumvents the unfavourable scaling of standard gp algorithms as is shown in experiments on a real world task the bounds can be very tight for moderate training sample sizes to the best of our knowledge these results provide the tightest known distribution free error bounds for approximate bayesian gpc methods giving a strong learning theoretical justification for the use of these techniques

improved risk tail bounds for on line algorithms tight bounds are derived on the risk of models in the ensemble generated by incremental training of an arbitrary learning algorithm the result is based on proof techniques that are remarkably different from the standard risk analysis based on uniform convergence arguments and improves on previous bounds published by the same authors

nonlinearization of the lax pairs for discrete ablowitz ladik hierarchy abstract the discrete ablowitz ladik hierarchy with four potentials and the hamiltonian structures are derived under a constraint between the potentials and eigenfunctions the nonlinearization of the lax pairs associated with the discrete ablowitz ladik hierarchy leads to a new symplectic map and a class of finite dimensional hamiltonian systems the generating function of the integrals of motion is presented by which the symplectic map and these finite dimensional hamiltonian systems are further proved to be completely integrable in the liouville sense each member in the discrete ablowitz ladik hierarchy is decomposed into a hamiltonian system of ordinary differential equations plus the discrete flow generated by the symplectic map

eigenvalues of the p x laplacian steklov problem abstract consider steklov eigenvalue problem involving the p x laplacian on a bounded domain the open subset of r n with n 2 as follows p x u u p x 2 u in u p x 2 u u p x 2 u on where p x constant we prove that the existence of infinitely many eigenvalue sequences unlike the p laplacian case for a variable exponent p x constant there does not exist a principal eigenvalue and the set of all eigenvalues is not closed under some assumptions finally we present some sufficient conditions for the infimum of all eigenvalues is zero and positive respectively

remarks on ricceri s variational principle and applications to the p x laplacian equations abstract in this paper we give some remarks on a variational principle of ricceri in the case of sequentially weakly lower semi continuous functionals defined on a reflexive real banach space in particular we introduce the notions of ricceri block and ricceri box which are more convenient in some applications than the weakly connected components using the variational principle of ricceri and a local mountain pass lemma we study the multiplicity of solutions of the p x laplacian equations with neumann dirichlet or no flux boundary condition and under appropriate hypotheses in which the integral functionals need not satisfy the ps condition on the global space we prove that the problem has at least seven solutions

variable exponent linear growth functionals in image restoration we study a functional with variable exponent 1 leq p x leq 2 which provides a model for image denoising enhancement and restoration the diffusion resulting from the proposed model is a combination of total variation tv based regularization and gaussian smoothing the existence uniqueness and long time behavior of the proposed model are established experimental results illustrate the effectiveness of the model in image restoration

on the convergence of solutions of inclusions containing maximal monotone and generalized pseudomonotone mappings abstract we are concerned in this paper with the existence boundedness and the convergence of solutions to a sequence of inclusions a k u b k u l k where a k is a maximal monotone mapping b k is a generalized pseudomonotone mapping defined on a reflexive banach space x and l k x we study appropriate kinds of convergence for a k and b k such that a limit of a sequence of solutions of these inclusions is also a solution of the limit inclusion

optimal aggregation of classifiers in statistical learning classification can be considered as nonparametric estimation of sets where the risk is defined by means of a specific distance between sets associated with misclassification error it is shown that the rates of convergence of classifiers depend on two parameters the complexity of the class of candidate sets and the margin parameter the dependence is explicitly given indicating that optimal fast rates approaching o n 1 can be attained where n is the sample size and that the proposed classifiers have the property of robustness to the margin the main result of the paper concerns optimal aggregation of classifiers we suggest a classifier that automatically adapts both to the complexity and to the margin and attains the optimal fast rates up to a logarithmic factor

data dependent margin based generalization bounds for classification we derive new margin based inequalities for the probability of error of classifiers the main feature of these bounds is that they can be calculated using the training data and therefore may be effectively used for model selection purposes in particular the bounds involve empirical complexities measured on the training data such as the empirical fat shattering dimension as opposed to their worst case counterparts traditionally used in such analyses also our bounds appear to be sharper and more general than recent results involving empirical complexity measures in addition we develop an alternative data based bound for the generalization error of classes of convex combinations of classifiers involving an empirical complexity measure that is easier to compute than the empirical covering number or fat shattering dimension we also show examples of efficient computation of the new bounds

nonparametric decentralized detection using kernel methods we consider the problem of decentralized detection under constraints on the number of bits that can be transmitted by each sensor in contrast to most previous work in which the joint distribution of sensor observations is assumed to be known we address the problem when only a set of empirical samples is available we propose a novel algorithm using the framework of empirical risk minimization and marginalized kernels and analyze its computational and statistical properties both theoretically and empirically we provide an efficient implementation of the algorithm and demonstrate its performance on both simulated and real data sets

on the doubt about margin explanation of boosting margin theory provides one of the most popular explanations to the success of adaboost where the central point lies in the recognition that margin is the key for characterizing the performance of adaboost this theory has been very influential e g it has been used to argue that adaboost usually does not overfit since it tends to enlarge the margin even after the training error reaches zero previously the minimum margin bound was established for adaboost however breiman 1999 9 pointed out that maximizing the minimum margin does not necessarily lead to a better generalization later reyzin and schapire 2006 37 emphasized that the margin distribution rather than minimum margin is crucial to the performance of adaboost in this paper we first present the kth margin bound and further study on its relationship to previous work such as the minimum margin bound and emargin bound then we improve the previous empirical bernstein bounds audibert et al 2009 maurer and pontil 2009 2 30 and based on such findings we defend the margin based explanation against breiman s doubts by proving a new generalization error bound that considers exactly the same factors as schapire et al 1998 39 but is sharper than breiman s 1999 9 minimum margin bound by incorporating factors such as average margin and variance we present a generalization error bound that is heavily related to the whole margin distribution we also provide margin distribution bounds for generalization error of voting classifiers in finite vc dimension space

error bounds for piecewise smooth and switching regression this paper deals with regression problems in which the nonsmooth target is assumed to switch between different operating modes specifically piecewise smooth pws regression considers target functions switching deterministically via a partition of the input space while switching regression considers arbitrary switching laws this paper derives generalization error bounds in these two settings by following the approach based on rademacher complexities for pws regression our derivation involves a chaining argument and a decomposition of the covering numbers of pws classes in terms of the ones of their component function classes and the capacity of the classifier partitioning the input space this yields error bounds with a radical dependence on the number of modes for switching regression the decomposition can be performed directly at the level of the rademacher complexities which yields bounds with a linear dependence on the number of modes by using once more chaining and decomposition at the level of covering numbers we show how to recover a radical dependence examples of applications are given in particular for pws and swichting regression with linear and kernel based component functions

a bounded error approach to piecewise affine system identification this paper proposes a three stage procedure for parametric identification of piecewise affine autoregressive exogenous pwarx models the first stage simultaneously classifies the data points and estimates the number of submodels and the corresponding parameters by solving the partition into a minimum number of feasible subsystems min pfs problem for a suitable set of linear complementary inequalities derived from data second a refinement procedure reduces misclassifications and improves parameter estimates the third stage determines a polyhedral partition of the regressor set via two class or multiclass linear separation techniques as a main feature the algorithm imposes that the identification error is bounded by a quantity spl delta such a bound is a useful tuning parameter to trade off between quality of fit and model complexity the performance of the proposed pwa system identification procedure is demonstrated via numerical examples and on experimental data from an electronic component placement process in a pick and place machine

a trace identity and its applications to the theory of discrete integrable systems a general theory for studying discrete integrable systems is developed based on a trace identity that the author previously proposed a scheme for generating hierarchies of discrete integrable systems is presented under broad assumptions the resulting hierarchies are shown to consist of liouville integrable hamiltonian systems

eigenvalue problems for anisotropic quasilinear elliptic equations with variable exponent in this paper we are concerned with a new class of anisotropic quasilinear elliptic equations with a power like variable reaction term one of the main features of our work is that the differential operator involves partial derivatives with different variable exponents so that the functional analytic framework relies upon anisotropic sobolev and lebesgue spaces existence and nonexistence results are deeply influenced by the competition between the growth rates of the anisotropic coefficients our main results point out some striking phenomena related to the existence of a continuous spectrum in several distinct situations

new concentration inequalities in product spaces we introduce three new ways to measure the distance from a point to a subset of a product space and we prove corresponding concentration inequalities each of them allows to control the fluctuation of a new class of random variables

empirical margin distributions and bounding the generalization error of combined classifiers we prove new probabilistic upper bounds on generalization error of complex classifiers that are combinations of simple classifiers such combinations could be implemented by neural networks or by voting methods of combining the classifiers such as boosting and bagging the bounds are in terms of the empirical distribution of the margin of the combined classifier they are based on the methods of the theory of gaussian and empirical processes comparison inequalities symmetrization method concentration inequalities and they improve previous results of bartlett 1998 on bounding the generalization error of neural networks in terms of ell 1 norms of the weights of neurons and of schapire freund bartlett and lee 1998 on bounding the generalization error of boosting we also obtain rates of convergence in levy distance of empirical margin distribution to the true margin distribution uniformly over the classes of classifiers and prove the optimality of these rates

nonlinear normal modes for the toda chain abstract the toda chain is a nonlinear mass spring chain which can in principle be integrated analytically by the action angle theory of classical mechanics we show that the transformation from physical variables to action variables can be implemented very simply on a computer we study the correspondence between action variables and the motions of the chain and find the role of the action variables to be very similar to the role of normal mode amplitudes in the harmonic chain this similarity leads to a partly rigorous partly heuristic normal mode analysis for the toda chain new results are obtained when this normal mode analysis is applied to certain perturbations of the toda chain such as the fermi pasta ulam chain or the toda chain with a mass impurity

error bounds for functional approximation and estimation using mixtures of experts we examine some mathematical aspects of learning unknown mappings with the mixture of experts model mem specifically we observe that the mem is at least as powerful as a class of neural networks in a sense that will be made precise upper bounds on the approximation error are established for a wide class of target functions the general theorem states that spl par f f sub n spl par sub p spl les c n sup r d for f spl isin w sub p sup r l a sobolev class over 1 1 sup d and f sub n belongs to an n dimensional manifold of normalized ridge functions the same bound holds for the mem as a special case of the above the stochastic error in the context of learning from independent and identically distributed i i d examples is also examined an asymptotic analysis establishes the limiting behavior of this error in terms of certain pseudo information matrices these results substantiate the intuition behind the mem and motivate applications

restricted flows of the ablowitz ladik hierarchy and their continuous limits it is studied here how the ablowitz ladik al hierarchy and their restricted flows are related to the akns hierarchy and to their restricted flows in the continuous limit it is shown that restricted flows of the al hierarchy yield discrete maps approximating restricted flows of the akns hierarchy integrals of motion and lax representation for restricted flows of the al hierarchy as well as a discrete zero curvature representation for the al hierarchy with sources are given

data dependent generalization bounds for multi class classification in this paper we study data dependent generalization error bounds that exhibit a mild dependency on the number of classes making them suitable for multi class learning with a large number of label classes the bounds generally hold for empirical multi class risk minimization algorithms using an arbitrary norm as the regularizer key to our analysis is new structural results for multi class gaussian complexities and empirical ell infty norm covering numbers which exploit the lipschitz continuity of the loss function with respect to the ell 2 and ell infty norm respectively we establish data dependent error bounds in terms of the complexities of a linear function class defined on a finite set induced by training examples for which we show tight lower and upper bounds we apply the results to several prominent multi class learning machines and show a tighter dependency on the number of classes than the state of the art for instance for the multi class support vector machine of crammer and singer 2002 we obtain a data dependent bound with a logarithmic dependency which is a significant improvement of the previous square root dependency the experimental results are reported to verify the effectiveness of our theoretical findings

a difference of convex functions algorithm for switched linear regression this paper deals with switched linear system identification and more particularly aims at solving switched linear regression problems in a large scale setting with both numerous data and many parameters to learn we consider the recent minimum of error framework with a quadratic loss function in which an objective function based on a sum of minimum errors with respect to multiple submodels is to be minimized the paper proposes a new approach to the optimization of this nonsmooth and nonconvex objective function which relies on difference of convex dc functions programming in particular we formulate a proper dc decomposition of the objective function which allows us to derive a computationally efficient dc algorithm numerical experiments show that the method can efficiently and accurately learn switching models in large dimensions and from many data points

rademacher averages and phase transitions in glivenko cantelli classes we introduce a new parameter which may replace the fat shattering dimension using this parameter we are able to provide improved complexity estimates for the agnostic learning problem with respect to any l sub p norm moreover we show that if fat sub spl epsi f o spl epsi sup p then f displays a clear phase transition which occurs at p 2 the phase transition appears in the sample complexity estimates covering numbers estimates and in the growth rate of the rademacher averages associated with the class as a part of our discussion we prove the best known estimates on the covering numbers of a class when considered as a subset of l sub p spaces we also estimate the fat shattering dimension of the convex hull of a given class both these estimates are given in terms of the fat shattering dimension of the original class

risk bounds for model selection via penalization performance bounds for criteria for model selection are devel oped using recent theory for sieves the model selection criteria are based on an empirical loss or contrast function with an added penalty term moti vated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of ob servations most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function we show that the quadratic risk of the minimum penal ized empirical contrast estimatoris bounded by an index of the accuracy of the sieve this accuracy index quanties the trade off among the candidate models between the approximation error and parameter dimension relative to sample size if we choose a list of models which exhibit good approximation prop erties with respect to different classes of smoothness the estimator can be simultaneously minimax rate optimal in each of those classes this is what is usually called adaptation the type of classes of smoothness in which one gets adaptation depends heavily on the list of models if too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously it may happen that the estimator is only approx

complexities of convex combinations and bounding the generalization error in classification we introduce and study several measures of complexity of functions from the convex hull of a given base class these complexity measures take into account the sparsity of the weights of a convex combination as well as certain clustering properties of the base functions involved in it we prove new upper confidence bounds on the generalization error of ensemble voting classification algorithms that utilize the new complexity measures along with the empirical distributions of classification margins providing a better explanation of generalization performance of large margin classification methods

on the sub supersolution method for p x laplacian equations abstract this paper deals with the sub supersolution method for the p x laplacian equations a sub supersolution principle for the dirichlet problems involving the p x laplacian is established it is proved that the local minimizers in the c 1 topology are also local minimizers in the w 1 p x topology for given energy functionals a strong comparison theorem for the p x laplacian equations is presented some applications of the abstract theorems obtained in this paper to the eigenvalue problems for the p x laplacian equations are given

existence of solutions for p x laplacian dirichlet problem this paper presents several sufficient conditions for the existence of solutions for the dirichlet problem of p x laplacian div u p x 2 u f x u x u 0 x especially an existence criterion for infinite many pairs of solutions for the problem is obtained the discussion is based on the theory of the spaces lp x and w01 p x

entropy and the combinatorial dimension we solve talagrand s entropy problem the l2 covering numbers of every uniformly bounded class of functions are exponential in its shattering dimension this extends dudley s theorem on classes of 0 1 valued functions for which the shattering dimension is the vapnik chervonenkis dimension in convex geometry the solution means that the entropy of a convex body k is controlled by the maximal dimension of a cube of a fixed side contained in the coordinate projections of k this has a number of consequences including the optimal elton s theorem and estimates on the uniform central limit theorem in the real valued case

additive logistic regression a statistical view of boosting boosting is one of the most important recent developments in classification methodology boosting works by sequentially applying a classification algorithm to reweighted versions of the training data and then taking a weighted majority vote of the sequence of classifiers thus produced for many classification algorithms this simple strategy results in dramatic improvements in performance we show that this seemingly mysterious phenomenon can be understood in terms of well known statistical principles namely additive modeling and maximum likelihood for the two class problem boosting can be viewed as an approximation to additive modeling on the logistic scale using maximum bernoulli likelihood as a criterion we develop more direct approximations and show that they exhibit nearly identical results to boosting direct multiclass generalizations based on multinomial likelihood are derived that exhibit performance comparable to other recently proposed multiclass generalizations of boosting in most situations and far superior in some we suggest a minor modification to boosting that can reduce computation often by factors of 10 to 50 finally we apply these insights to produce an alternative formulation of boosting decision trees this approach based on best first truncated tree induction often leads to better performance and can provide interpretable descriptions of the aggregate decision rule it is also much faster computationally making it more suitable to large scale data mining applications

a deterministic annealing approach for parsimonious design of piecewise regression models a new learning algorithm is proposed for piecewise regression modeling it employs the technique of deterministic annealing to design space partition regression functions while the performance of traditional space partition regression functions such as cart and mars is limited by a simple tree structured partition and by a hierarchical approach for design the deterministic annealing algorithm enables the joint optimization of a more powerful piecewise structure based on a voronoi partition the new method is demonstrated to achieve consistent performance improvements over regular cart as well as over its extension to allow arbitrary hyperplane boundaries comparison tests on several benchmark data sets from the regression literature are provided

on the additive properties of the fat shattering dimension the properties of the vc dimension under various compositions are well understood but this is much less the case for classes of continuous functions in this brief we show that a commonly used scale sensitive dimension v y is much less well behaved under minkowski summation than its vc cousin while the fat shattering dimension retains some compositional similarity to the vc dimension as an application we analyze the fat shattering dimension of trigonometric functions and series

learning theory estimates with observations from general stationary stochastic processes this letter investigates the supervised learning problem with observations drawn from certain general stationary stochastic processes here by general we mean that many stationary stochastic processes can be included we show that when the stochastic processes satisfy a generalized bernstein type inequality a unified treatment on analyzing the learning schemes with various mixing processes can be conducted and a sharp oracle inequality for generic regularized empirical risk minimization schemes can be established the obtained oracle inequality is then applied to derive convergence rates for several learning schemes such as empirical risk minimization erm least squares support vector machines ls svms using given generic kernels and svms using gaussian kernels for both least squares and quantile regression it turns out that for independent and identically distributed i i d processes our learning rates for erm recover the optimal rates for non i i d processes including geometrically mixing markov processes geometrically mixing processes with restricted decay mixing processes and time reversed geometrically mixing processes our learning rates for svms with gaussian kernels match up to some arbitrarily small extra term in the exponent the optimal rates for the remaining cases our rates are at least close to the optimal rates as a by product the assumed generalized bernstein type inequality also provides an interpretation of the so called effective number of observations for various mixing processes

a new integrable hierarchy of lattice equations a new type of discrete spectral problem is introduced and studied the related hierarchy of isospectral flows is tri hamiltonian the miura maps and modifications are presented

vote counting measures for ensemble classifiers various measures such as margin and bias variance have been proposed with the aim of gaining a better understanding of why multiple classifier systems mcs perform as well as they do while these measures provide different perspectives for mcs analysis it is not clear how to use them for mcs design in this paper a different measure based on a spectral representation is proposed for two class problems it incorporates terms representing positive and negative correlation of pairs of training patterns with respect to class labels experiments employing mlp base classifiers in which parameters are fixed but systematically varied demonstrate the sensitivity of the proposed measure to base classifier complexity

structural multiple empirical kernel learning multiple kernel learning mkl can boost classification performance through using multiple kernels rather than a single fixed one unlike the traditional mkl with the implicit kernels multiple empirical kernel learning mekl explicitly maps input data into multiple feature spaces this paper focuses on mekl and proposes an effective threefold structural mekl tsmekl the first fold structure is the space structural information between different mapped feature spaces the second one is the class discriminant information within each mapped feature space the third one is the cluster structural information of samples in each mapped feature space the classical mekl mainly pays attention to the first two structures but neglects the last one the proposed tsmekl introduces the cluster structural information into mekl doing so can simultaneously utilize the space the class and the cluster information in the way from globality to locality therefore tsmekl utilizes threefold structural information to result in the improvement of classification performance to the best of our knowledge it is the first time to introduce the cluster information into the mekl framework the main advantage of the developed tsmekl is considering different folds of data information to improve classification performance the experimental results validate the feasibility and effectiveness of tsmekl moreover we discuss the theoretical and experimental generalization risk bound of the proposed algorithm in terms of the rademacher complexity

identification of piecewise affine systems via mixed integer programming this paper addresses the problem of identification of hybrid dynamical systems by focusing the attention on hinging hyperplanes and wiener piecewise affine autoregressive exogenous models in which the regressor space is partitioned into polyhedra with affine submodels for each polyhedron in particular we provide algorithms based on mixed integer linear or quadratic programming which are guaranteed to converge to a global optimum for the special case where the estimation data only seldom switches between the different submodels we also suggest a way of trading off between optimality and complexity by using a change detection approach

rigorous results for the hopfield model with many patterns we perform a thorough investigation of the main aspects of the hopfield model with many patterns advances are made toward the validity of the replica symmetric solution strong evidence of the validity of this solution is given over the entire domain where this validity is conjectured complete proof is given in a subregion that contains strictly the ergodic region

a new look at independence the concentration of measure phenomenon in product spaces is a farreaching abstract generalization of the classical exponential inequalities for sums of independent random variables we attempt to explain in the simplest possible terms the basic concepts underlying this phenomenon the basic method to prove concentration inequalities and the meaning of several of the most useful inequalities

the dynamics of adaboost cyclic behavior and convergence of margins in order to study the convergence properties of the adaboost algorithm we reduce adaboost to a nonlinear iterated map and study the evolution of its weight vectors this dynamical systems approach allows us to understand adaboost s convergence properties completely in certain cases for these cases we find stable cycles allowing us to explicitly solve for adaboost s output using this unusual technique we are able to show that adaboost does not always converge to a maximum margin combined classifier answering an open question in addition we show that non optimal adaboost where the weak learning algorithm does not necessarily choose the best weak classifier at each iteration may fail to converge to a maximum margin classifier even if optimal adaboost produces a maximum margin also we show that if adaboost cycles it cycles among support vectors i e examples that achieve the same smallest margin

toda flows with infinitely many variables abstract the toda flow and related flows extend naturally to operators in hilbert space and the purpose of this paper is to describe these flows and to analyse some of their special properties

on the complexity of piecewise affine system identification the paper provides results regarding the computational complexity of hybrid system identification more precisely we focus on the estimation of piecewise affine pwa maps from input output data and analyze the complexity of computing a global minimizer of the error previous work showed that a global solution could be obtained for continuous pwa maps with a worst case complexity exponential in the number of data in this paper we show how global optimality can be reached for a slightly more general class of possibly discontinuous pwa maps with a complexity only polynomial in the number of data however with an exponential complexity with respect to the data dimension this result is obtained via an analysis of the intrinsic classification subproblem of associating the data points to the different modes in addition we prove that the problem is np hard and thus that the exponential complexity in the dimension is a natural expectation for any exact algorithm

greedy function approximation a gradient boosting machine function estimation approximation is viewed from the perspective of numerical optimization in function space rather than parameter space a connection is made between stagewise additive expansions and steepest descent minimization a general gradient descent boosting paradigm is developed for additive expansions based on any fitting criterion specific algorithms are presented for least squares least absolute deviation and huber m loss functions for regression and multiclass logistic likelihood for classification special enhancements are derived for the particular case where the individual additive components are regression trees and tools for interpreting such treeboost models are presented gradient boosting of regression trees produces competitive highly robust interpretable procedures for both regression and classification especially appropriate for mining less than clean data connections between this approach and the boosting methods of freund and shapire and friedman hastie and tibshirani are discussed

brief paper a continuous optimization framework for hybrid system identification we propose a new framework for hybrid system identification which relies on continuous optimization this framework is based on the minimization of a cost function that can be chosen as either the minimum or the product of loss functions the former is inspired by traditional estimation methods while the latter is inspired by recent algebraic and support vector regression approaches to hybrid system identification in both cases the identification problem is recast as a continuous optimization program involving only the real parameters of the model as variables thus avoiding the use of discrete optimization this program can be solved efficiently by using standard optimization methods even for very large data sets in addition the proposed framework easily incorporates robustness to different kinds of outliers through the choice of the loss function

a new hierarchy of integrable differential difference equations and darboux transformation a new discrete isospectral problem and the corresponding hierarchy of nonlinear differential difference equations are proposed it is shown that the hierarchy of differential difference equations possesses the hamiltonian structures a darboux transformation for the discrete spectral problem is found as an application two soliton solutions for the first system of differential difference equations in the hierarchy are given

generalization bounds and complexities based on sparsity and clustering for convex combinations of functions from random classes a unified approach is taken for deriving new generalization data dependent bounds for several classes of algorithms explored in the existing literature by different approaches this unified approach is based on an extension of vapnik s inequality for vc classes of sets to random classes of sets that is classes depending on the random data invariant under permutation of the data and possessing the increasing property generalization bounds are derived for convex combinations of functions from random classes with certain properties algorithms such as svms support vector machines boosting with decision stumps radial basis function networks some hierarchies of kernel machines or convex combinations of indicator functions over sets with finite vc dimension generate classifier functions that fall into the above category we also explore the individual complexities of the classifiers such as sparsity of weights and weighted variance over clusters from the convex combination introduced by koltchinskii and panchenko 2004 and show sparsity type and cluster variance type generalization bounds for random classes

boosting method for local learning in statistical pattern recognition we propose a local boosting method in classification problems borrowing from an idea of the local likelihood method our proposal local boosting includes a simple device for localization for computational feasibility we proved the bayes risk consistency of the local boosting in the framework of probably approximately correct learning inspection of the proof provides a useful viewpoint for comparing ordinary boosting and local boosting with respect to the estimation error and the approximation error both boosting methods have the bayes risk consistency if their approximation errors decrease to zero compared to ordinary boosting local boosting may perform better by controlling the trade off between the estimation error and the approximation error ordinary boosting with complicated base classifiers or other strong classification methods including kernel machines may have classification performance comparable to local boosting with simple base classifiers for example decision stumps local boosting however has an advantage with respect to interpretability local boosting with simple base classifiers offers a simple way to specify which features are informative and how their values contribute to a classification rule even though locally several numerical studies on real data sets confirm these advantages of local boosting

reduced size kernel models for nonlinear hybrid system identification this brief paper focuses on the identification of nonlinear hybrid dynamical systems i e systems switching between multiple nonlinear dynamical behaviors thus the aim is to learn an ensemble of submodels from a single set of input output data in a regression setting with no prior knowledge on the grouping of the data points into similar behaviors to be able to approximate arbitrary nonlinearities kernel submodels are considered however in order to maintain efficiency when applying the method to large data sets a preprocessing step is required in order to fix the submodel sizes and limit the number of optimization variables this brief paper proposes four approaches respectively inspired by the fixed size least squares support vector machines the feature vector selection method the kernel principal component regression and a modification of the latter in order to deal with this issue and build sparse kernel submodels these are compared in numerical experiments which show that the proposed approach achieves the simultaneous classification of data points and approximation of the nonlinear behaviors in an efficient and accurate manner

kac moody lie algebras and soliton equations ii lax equations associated with a1 1 the soliton equations associated with sl 2 eigenvalue problems polynomial in the eigenvalue parameter are given a unified treatment they are shown to be generated by a single family of commuting hamiltonians on a subalgebra of the loop algebra of sl 2 the conserved densities and fluxes of the usual anks hierarchy are identified with conserved densities and fluxes for the polynomial eigenvalue problems the hamiltonian structures of the soliton equations associated with the polynomial eigenvalue problems are given a unified treatment

learning the kernel matrix with semidefinite programming kernel based learning algorithms work by embedding the data into a euclidean space and then searching for linear relations among the embedded data points the embedding is performed implicitly by specifying the inner products between each pair of points in the embedding space this information is contained in the so called kernel matrix a symmetric and positive semidefinite matrix that encodes the relative positions of all points specifying this matrix amounts to specifying the geometry of the embedding space and inducing a notion of similarity in the input space classical model selection problems in machine learning in this paper we show how the kernel matrix can be learned from data via semidefinite programming sdp techniques when applied to a kernel matrix associated with both training and test data this gives a powerful transductive algorithm using the labeled part of the data one can learn an embedding also for the unlabeled part the similarity between test points is inferred from training points and their labels importantly these learning problems are convex so we obtain a method for learning both the model class and the function without local minima furthermore this approach leads directly to a convex method for learning the 2 norm soft margin parameter in support vector machines solving an important open problem

ranking and empirical minimization of u statistics the problem of ranking ordering instances instead of simply classifying them has recently gained much attention in machine learning in this paper we formulate the ranking problem in a rigorous statistical framework the goal is to learn a ranking rule for deciding among two instances which one is better with minimum ranking risk since the natural estimates of the risk are of the form of a u statistic results of the theory of u processes are required for investigating the consistency of empirical risk minimizers we establish in particular a tail inequality for degenerate u processes and apply it for showing that fast rates of convergence may be achieved under specific noise assumptions just like in classification convex risk minimization methods are also studied

sup norm approximation bounds for networks through probabilistic methods we consider the problem of approximating a smooth target function and its derivatives by networks involving superpositions and translations of a fixed activation function the approximation is with respect to the sup norm and the rate is shown to be of order o n sup 1 2 that is the rate is independent of the dimension d the results apply to neural and wavelet networks and extend the work of barren see proc 7th yale workshop on adaptive and learning systems may 1992 and ibid vol 39 p 930 1993 the approach involves probabilistic methods based on central limit theorems for empirical processes indexed by classes of functions

existence of solutions for p x laplacian problems on a bounded domain in this paper we study the following p x laplacian problem div a x vd uv p x 2 d u b x v uv p x 2 u f x u x e u u 0 on p u where 1 p 1 l p x l p 2 n u s r n is a bounded domain and applying the mountain pass theorem we obtain the existence of solutions in w 0 1 p x for the p x laplacian problems in the superlinear and sublinear cases 2004 elsevier inc all rights reserved

bounds for linear multi task learning we give dimension free and data dependent bounds for linear multi task learning where a common linear operator is chosen to preprocess data for a vector of task specific linear thresholding classifiers the complexity penalty of multi task learning is bounded by a simple expression involving the margins of the task specific classifiers the hilbert schmidt norm of the selected preprocessor and the hilbert schmidt norm of the covariance operator for the total mixture of all task distributions or alternatively the frobenius norm of the total gramian matrix for the data dependent version the results can be compared to state of the art results on linear single task learning

covering number bounds of certain regularized linear function classes recently sample complexity bounds have been derived for problems involving linear functions such as neural networks and support vector machines in many of these theoretical studies the concept of covering numbers played an important role it is thus useful to study covering numbers for linear function classes in this paper we investigate two closely related methods to derive upper bounds on these covering numbers the first method already employed in some earlier studies relies on the so called maurey s lemma the second method uses techniques from the mistake bound framework in online learning we compare results from these two methods as well as their consequences in some learning formulations

on quantile regression in reproducing kernel hilbert spaces with the data sparsity constraint for spline regressions it is well known that the choice of knots is crucial for the performance of the estimator as a general learning framework covering the smoothing splines learning in a reproducing kernel hilbert space rkhs has a similar issue however the selection of training data points for kernel functions in the rkhs representation has not been carefully studied in the literature in this paper we study quantile regression as an example of learning in a rkhs in this case the regular squared norm penalty does not perform training data selection we propose a data sparsity constraint that imposes thresholding on the kernel function coefficients to achieve a sparse kernel function representation we demonstrate that the proposed data sparsity method can have competitive prediction performance for certain situations and have comparable performance in other cases compared to that of the traditional squared norm penalty therefore the data sparsity method can serve as a competitive alternative to the squared norm penalty method some theoretical properties of our proposed method using the data sparsity constraint are obtained both simulated and real data sets are used to demonstrate the usefulness of our data sparsity constraint