on line learning of smooth functions of a single variable abstract we study the on line learning of classes of functions of a single real variable formed through bounds on various norms of functions derivatives we determine the best bounds obtainable on the worst case sum of squared errors also absolute errors for several such classes we prove upper bounds for these classes of smooth functions for other loss functions and prove upper and lower bounds in terms of the number of trials

using vapnik chervonenkis dimension to analyze the testing complexity of program segments we examine the complexity of testing different program constructs we do this by defining a measure of testing complexity known as vcp dimension which is similar to the vapnik chervonenkis dimension and applying it to classes of programs where all programs in a class share the same syntactic structure vcp dimension gives bounds on the number of test points needed to determine that a program is approximately correct so by studying it for a class of programs we gain insight into the difficulty of testing the program construct represented by the class we investigate the vcp dimension of straight line code if then else statements and for loops we also compare the vcp dimension of nested and sequential if then else statements as well as that of two types of for loops with embedded if then else statements finally we perform an empirical study to estimate the expected complexity of straight line code

tight bounds on l1 approximation and learning of self bounding functions abstract we study the complexity of learning and approximation of self bounding functions over the uniform distribution on the boolean hypercube 0 1 n informally a function f 0 1 n r is self bounding if for every x 0 1 n f x upper bounds the sum of all the n marginal decreases in the value of the function at x self bounding functions include such well known classes of functions as submodular and fractionally subadditive xos functions they were introduced by boucheron et al 2010 in the context of concentration of measure inequalities our main result is a nearly tight l 1 approximation of self bounding functions by low degree juntas specifically all self bounding functions can be approximated in l 1 by a polynomial of degree o 1 over 2 o 1 variables we show that both the degree and junta size are optimal up to logarithmic terms previous techniques considered stronger l 2 approximation and proved nearly tight bounds of 1 2 on the degree and 2 1 2 on the number of variables our bounds rely on the analysis of noise stability of self bounding functions together with a stronger connection between noise stability and l 1 approximation by low degree polynomials this technique can also be used to get tighter bounds on l 1 approximation by low degree polynomials and a faster learning algorithm for halfspaces these results lead to improved and in several cases almost tight bounds for pac and agnostic learning of self bounding functions relative to the uniform distribution in particular assuming hardness of learning juntas we show that pac and agnostic learning of self bounding functions have complexity of n 1

test error fluctuations in finite linear perceptrons we examine the fluctuations in the test error induced by random finite training and test sets for the linear perceptron of input dimension n with a spherically constrained weight vector this variance enables us to address such issues as the partitioning of a data set into a test and training set we find that the optimal assignment of the test set size scales with n2 3

on learning multicategory classification with sample queries consider the pattern recognition problem of learning multicategory classification from a labeled sample for instance the problem of learning character recognition where a category corresponds to an alphanumeric letter the classical theory of pattern recognition assumes labeled examples appear according to the unknown underlying pattern class conditional probability distributions where the pattern classes are picked randomly according to their a priori probabilities in this paper we pose the following question can the learning accuracy be improved if labeled examples are independently randomly drawn according to the underlying class conditional probability distributions but the pattern classes are chosen not necessarily according to their a priori probabilities we answer this in the affirmative by showing that there exists a tuning of the sub sample proportions which minimizes a loss criterion the tuning is relative to the intrinsic complexity of the bayes classifier as this complexity depends on the underlying probability distributions which are assumed to be unknown we provide an algorithm which learns the proportions in an on line manner utilizing sample querying which asymptotically minimizes the criterion in practice this algorithm may be used to boost the performance of existing learning classification algorithms by apportioning better sub sample proportions

learning a decision boundary from stochastic examples incremental algorithms with and without queries even if it is not possible to reproduce a target input output relation a learning machine should be able to minimize the probability of making errors a practical learning algorithm should also be simple enough to go without memorizing example data if possible incremental algorithms such as error backpropagation satisfy this requirement we propose incremental algorithms that provide fast convergence of the machine parameter to its optimal choice o with respect to the number of examples t we will consider the binary choice model whose target relation has a blurred boundary and the machine whose parameter specifies a decision boundary to make the output prediction the question we wish to address here is how fast can approach o depending upon whether in the learning stage the machine can specify inputs as queries to the target relation or the inputs are drawn from a certain distribution if queries are permitted the machine can achieve the fastest convergence o 2 o t 1 if not o t 1 convergence is generally not attainable for learning without queries we showed in a previous paper that the error minimum algorithm exhibits a slow convergence o 2 o t 2 3 we propose here a practical algorithm that provides a rather fast convergence o t 4 5 it is possible to further accelerate the convergence by using more elaborate algorithms the fastest convergence turned out to be o lnt 2t 1 this scaling is considered optimal among possible algorithms and is not due to the incremental nature of our algorithm

randomized algorithms for robust controller synthesis using statistical learning theory by now it is known that several problems in the robustness analysis and synthesis of control systems are np complete or np hard these negative results force us to modify our notion of solving a given problem an approach that is recently gaining popularity is that of using randomized algorithms which can be used to solve a problem approximately most of the time we begin with the premise that many problems in robustness analysis and synthesis can be formulated as the minimization of an objective function with respect to the controller parameters it is argued that in order to assess the performance of a controller as the plant varies over a prespecified family it is better to use the average performance of the controller as the objective function to be minimized rather than its worst case performance as the worst case objective function usually leads to rather conservative designs then it is shown that a property from statistical learning theory known as uniform convergence of empirical means ucem plays an important role in allowing us to construct efficient randomized algorithms for a wide variety of controller synthesis problems in particular whenever the ucem property holds there exists an efficient i e polynomial time randomized algorithm using very recent results in statistical learning theory it is shown that the ucem property holds in any problem in which the satisfaction of a performance constraint can be expressed in terms of a finite number of polynomial inequalities in particular several problems such as robust stabilization and weighted h 2 h norm minimization are amenable to the randomized approach

concentration inequalities large and moderate deviations for self normalized empirical processes we consider the supremum w n of self normalized empirical processes indexed by unbounded classes of functions f such variables are of interest in various statistical applications for example the likelihood ratio tests of contamination using the herbst method we prove an exponential concentration inequality for w n under a second moment assumption on the envelope function of f this inequality is applied to obtain moderate deviations for w n we also provide large deviations results for some unbounded parametric classes f

generalization performance of radial basis function networks this paper studies the generalization performance of radial basis function rbf networks using local rademacher complexities we propose a general result on controlling local rademacher complexities with the l 1 metric capacity we then apply this result to estimate the rbf networks complexities based on which a novel estimation error bound is obtained an effective approximation error bound is also derived by carefully investigating the holder continuity of the l p loss function s derivative furthermore it is demonstrated that the rbf network minimizing an appropriately constructed structural risk admits a significantly better learning rate when compared with the existing results an empirical study is also performed to justify the application of our structural risk in model selection

on the power of membership queries in agnostic learning we study the properties of the agnostic learning framework of haussler 1992 and kearns schapire and sellie 1994 in particular we address the question is there any situation in which membership queries are useful in agnostic learning our results show that the answer is negative for distribution independent agnostic learning and positive for agnostic learning with respect to a specific marginal distribution namely we give a simple proof that any concept class learnable agnostically by a distribution independent algorithm with access to membership queries is also learnable agnostically without membership queries this resolves an open problem posed by kearns et al 1994 for agnostic learning with respect to the uniform distribution over 0 1 n we show a concept class that is learnable with membership queries but computationally hard to learn from random examples alone assuming that one way functions exist

hardness results for neural network approximation problems we consider the problem of efficiently learning in two layer neural networks we investigate the computational complexity of agnostically learning with simple families of neural networks as the hypothesis classes we show that it is np hard to find a linear threshold network of a fixed size that approximately minimizes the proportion of misclassified examples in a training set even if there is a network that correctly classifies all of the training examples in particular for a training set that is correctly classified by some two layer linear threshold network with k hidden units it is np hard to find such a network that makes mistakes on a proportion smaller than c k2 of the examples for some constant c we prove a similar result for the problem of approximately minimizing the quadratic loss of a two layer network with a sigmoid output unit

a learning result for continuous time recurrent neural networks abstract the following learning problem is considered for continuous time recurrent neural networks having sigmoidal activation functions given a black box representing an unknown system measurements of output derivatives are collected for a set of randomly generated inputs and a network is used to approximate the observed behavior it is shown that the number of inputs needed for reliable generalization the sample complexity of the learning problem is upper bounded by an expression that grows polynomially with the dimension of the network and logarithmically with the number of output derivatives being matched

covering numbers for real valued function classes we find tight upper and lower bounds on the growth rate for the covering numbers of functions of bounded variation in the spl lscr sub 1 metric in terms of all the relevant constants we also find upper and lower bounds on covering numbers for general function classes over the family of spl lscr sub 1 dp metrics in terms of a scale sensitive combinatorial dimension of the function class

efficient agnostic learning of neural networks with bounded fan in we show that the class of two layer neural networks with bounded fan in is efficiently learnable in a realistic extension to the probably approximately correct pac learning model in this model a joint probability distribution is assumed to exist on the observations and the learner is required to approximate the neural network which minimizes the expected quadratic error as special cases the model allows learning real valued functions with bounded noise learning probabilistic concepts and learning the best approximation to a target function that cannot be well approximated by the neural network the networks we consider have real valued inputs and outputs an unlimited number of threshold hidden units with bounded fan in and a bound on the sum of the absolute values of the output weights the number of computation steps of the learning algorithm is bounded by a polynomial in 1 spl epsiv 1 spl delta n and b where spl epsiv is the desired accuracy spl delta is the probability that the algorithm fails n is the input dimension and b is the bound on both the absolute value of the target which may be a random variable and the sum of the absolute values of the output weights in obtaining the result we also extended some results on iterative approximation of functions in the closure of the convex hull of a function class and on the sample complexity of agnostic learning with the quadratic loss function

product units a computationally powerful and biologically plausible extension to backpropagation networks we introduce a new form of computational unit for feedforward learning networks of the backpropagation type instead of calculating a weighted sum this unit calculates a weighted product where each input is raised to a power determined by a variable weight such a unit can learn an arbitrary polynomial term which would then feed into higher level standard summing units we show how learning operates with product units provide examples to show their efficiency for various types of problems and argue that they naturally extend the family of theoretical feedforward net structures there is a plausible neurobiological interpretation for one interesting configuration of product and summing units

complexity of hyperconcepts in machine learning maximizing the sample margin can reduce the learning generalization error samples on which the target function has a large margin convey more information since they yield more accurate hypotheses let x be a finite domain and s denote the set of all samples s x of fixed cardinality m let h be a class of hypotheses h on x a hyperconcept h is defined as an indicator function for a set a s of all samples on which the corresponding hypothesis h has a margin of at least an estimate on the complexity of the class h of hyperconcepts h is obtained with explicit dependence on the pseudodimension of h and m

a metric entropy bound is not sufficient for learnability the authors prove by means of a counterexample that it is not sufficient for probably approximately correct pac learning under a class of distributions to have a uniform bound on the metric entropy of the class of concepts to be learned this settles a conjecture of benedek and itai 1991

the complexity of model classes and smoothing noisy data abstract we consider the problem of smoothing a sequence of noisy observations using a fixed class of models via a deterministic analysis we obtain necessary and sufficient conditions on the noise sequence and model class that ensure that a class of natural estimators gives near optimal smoothing in the case of i i d random noise we show that the accuracy of these estimators depends on a measure of complexity of the model class involving covering numbers our formulation and results are quite general and are related to a number of problems in learning prediction and estimation as a special case we consider an application to output smoothing for certain classes of linear and nonlinear systems the performance of output smoothing is given in terms of natural complexity parameters of the model class such as bounds on the order of linear systems the l1 norm of the impulse response of stable linear systems or the memory of a lipschitz nonlinear system satisfying a fading memory condition

finite sample properties of linear model identification in this paper we consider the finite sample properties of prediction error methods using a quadratic criterion function for system identification the problem we pose is how many data points are required to guarantee with high probability that the expected value of the quadratic identification criterion is close to its empirical mean value the sample sizes are obtained using risk minimization theory which provides uniform probabilistic bounds on the difference between the expected value of the squared prediction error and its empirical mean evaluated on a finite number of data points the bounds are very general no assumption is made about the true system belonging to the model class and the noise sequence is not assumed to be uniformly bounded further analysis shows that in order to maintain a given bound on the deviation the number of data points needed grows no faster than quadratically with the number of parameters for fir and arx models

learning from multiple sources we consider the problem of learning accurate models from multiple sources of nearby data given distinct samples from multiple data sources and estimates of the dissimilarities between these sources we provide a general theory of which samples should be used to learn models for each source this theory is applicable in a broad decision theoretic learning framework and yields general results for classification and regression a key component of our approach is the development of approximate triangle inequalities for expected loss which may be of independent interest we discuss the related problem of learning parameters of a distribution from multiple data sources finally we illustrate our theory through a series of synthetic simulations

bounds for the computational power and learning complexity of analog neural nets it is shown that high order feedforward neural nets of constant depth with piecewise polynomial activation functions and arbitrary real weights can be simulated for boolean inputs and outputs by neural nets of a somewhat larger size and depth with heaviside gates and weights from 1 0 1 this provides the first known upper bound for the computational power of the former type of neural nets it is also shown that in the case of first order nets with piecewise linear activation functions one can replace arbitrary real weights by rational numbers with polynomially many bits without changing the boolean function that is computed by the neural net in order to prove these results we introduce two new methods for reducing nonlinear problems about weights in multilayer neural nets to linear problems for a transformed set of parameters these transformed parameters can be interpreted as weights in a somewhat larger neural net as another application of our new proof technique we show that neural nets with piecewise polynomial activation functions and a constant number of analog inputs are probably approximately correct pac learnable in valiant s model for pac learning comm assoc comput mach 27 1984 pp 1134 1142

stochastic complexity and modeling on demontre un theoreme fondamental qui donne une borne inferieure pour la longueur de code et donc pour les erreurs de prediction on definit les notions d information a priori et d information utile dans les donnees

information fusion methods based on physical laws we consider systems whose parameters satisfy certain easily computable physical laws each parameter is directly measured by a number of sensors or estimated using measurements or both the measurement process may introduce both systematic and random errors which may then propagate into the estimates furthermore the actual parameter values are not known since every parameter is measured or estimated which makes the existing sample based fusion methods inapplicable we propose a fusion method for combining the measurements and estimators based on the least violation of physical laws that relate the parameters under fairly general smoothness and nonsmoothness conditions on the physical laws we show the asymptotic convergence of our method and also derive distribution free performance bounds based on finite samples for suitable choices of the fuser classes we show that for each parameter the fused estimate is probabilistically at least as good as its best measurement as well as best estimate we illustrate the effectiveness of this method for a practical problem of fusing well log data in methane hydrate exploration

probably approximate learning over classes of distributions this paper presents algorithms that construct approximations to sets and functions on the reals from randomly chosen sample points the model that is analyzed is a generalization of the paradigm of probably approximate learning proposed by valiant previously necessary and sufficient conditions for learning sets were established for the case when the class of sampling distributions is finite and for the case when the class of sampling distributions is the set of all possible distributions here sufficient conditions are obtained for learning sets and functions over general classes of sampling distributions the results for functions are with respect to general metrics for measuring the distance between two functions

learning curves for error minimum and maximum likelihood algorithms for the problem of dividing the space originally partitioned by a blurred boundary every learning algorithm can make the probability of incorrect prediction of an individual example decrease with the number of training examples t we address here the question of how the asymptotic form of t as well as its limit of convergence reflect the choice of learning algorithms the error minimum algorithm is found to exhibit rather slow convergence of t to its lower bound 0 t 0o t 2 3 even for the purpose of minimizing prediction error the maximum likelihood algorithm can be utilized as an alternative if the true probability distribution happens to be contained in the family of hypothetical functions then the boundary estimated from the hypothetical distribution function eventually converges to the best choice convergence of the prediction error is then t 0o t 1 if the true distribution is not available from the algorithm however the boundary generally does not converge to the best choice but instead t 1o t 1 2 where 1 0 0

on the complexity of learning from drifting distributions abstract we consider two models of on line learning of binary valued functions from drifting distributions due to bartlett we show that if each example is drawn from a joint distribution which changes in total variation distance by at mosto e3 d log 1 e between trials then an algorithm can achieve a probability of a mistake at mosteworse than the best function in a class of vc dimensiond we prove a corresponding necessary condition ofo e3 d finally in the case that a fixed function is to be learned from noise free examples we show that if the distributions on the domain generating the examples change by at mosto e2 d log 1 e then any consistent algorithm learns to within accuracye

scalar curvature equation on sn part ii analytic characterizations this is the second part of a series devoting to the study of the prescribing scalar curvature problem on the standard sphere of any dimension by studying topological degrees for certain abstract maps we will give explicit analytic conditions on the scalar curvature function which verify the topological degree conditions given in the first part of the series to ensure the solvability of the problem general existence results for the prescribing scalar curvature equation will be given on both h symmetric and sub h symmetric solutions corresponding to h symmetric scalar curvature functions as well as on non symmetric solutions corresponding to symmetric like scalar curvature functions special axisymmetric and axisymmetric like cases will be also considered our analysis will be based on a general approach of dimension reductions and degree calculations by taking advantage of symmetries and symmetric like properties

nonparametric estimation and classification using radial basis function nets and empirical risk minimization studies convergence properties of radial basis function rbf networks for a large class of basis functions and reviews the methods and results related to this topic the authors obtain the network parameters through empirical risk minimization the authors show the optimal nets to be consistent in the problem of nonlinear function approximation and in nonparametric classification for the classification problem the authors consider two approaches the selection of the rbf classifier via nonlinear function estimation and the direct method of minimizing the empirical error probability the tools used in the analysis include distribution free nonasymptotic probability inequalities and covering numbers for classes of functions

neural networks with local receptive fields and superlinear vc dimension local receptive field neurons comprise such well known and widely used unit types as radial basis function rbf neurons and neurons with center surround receptive field we study the vapnik chervonenkis vc dimension of feedforward neural networks with one hidden layer of these units for several variants of local receptive field neurons we show that the vc dimension of these networks is superlinear in particular we establish the bound w log k for any reasonably sized network with w parameters and k hidden nodes this bound is shown to hold for discrete center surround receptive field neurons which are physiologically relevant models of cells in the mammalian visual system for neurons computing a difference of gaussians which are popular in computational vision and for standard rbf neurons a major alternative to sigmoidal neurons in artificial neural networks the resutlt for rbf neural networks is of particular interest since it answers a question that has been open for several years the results also give rise to lower bounds for networks with fixed input dimension regarding constants all bounds are larger than those known thus far for similar architectures with sigmoidal neurons the superlinear lower bounds contrast with linear upper bounds for single local receptive field neurons also derived here

stochastic choice of basis functions in adaptive function approximation and the functional link net a theoretical justification for the random vector version of the functional link rvfl net is presented in this paper based on a general approach to adaptive function approximation the approach consists of formulating a limit integral representation of the function to be approximated and subsequently evaluating that integral with the monte carlo method two main results are 1 the rvfl is a universal approximator for continuous functions on bounded finite dimensional sets and 2 the rvfl is an efficient universal approximator with the rate of approximation error convergence to zero of order o c spl radic n where n is number of basis functions and with c independent of n similar results are also obtained for neural nets with hidden nodes implemented as products of univariate functions or radial basis functions some possible ways of enhancing the accuracy of multivariate function approximations are discussed

approximate dynamic programming via a smoothed linear program we present a novel linear program for the approximation of the dynamic programming cost to go function in high dimensional stochastic control problems lp approaches to approximate dp have typically relied on a natural projection of a well studied linear program for exact dynamic programming such programs restrict attention to approximations that are lower bounds to the optimal cost to go function our program the smoothed approximate linear program is distinct from such approaches and relaxes the restriction to lower bounding approximations in an appropriate fashion while remaining computationally tractable doing so appears to have several advantages first we demonstrate bounds on the quality of approximation to the optimal cost to go function afforded by our approach these bounds are in general no worse than those available for extant lp approaches and for specific problem instances can be shown to be arbitrarily stronger second experiments with our approach on a pair of challenging problems the game of tetris and a queueing network control problem show that the approach outperforms the existing lp approach which has previously been shown to be competitive with several adp algorithms by a substantial margin

a probabilistic analysis of em for mixtures of separated spherical gaussians we show that given data from a mixture of k well separated spherical gaussians in d a simple two round variant of em will with high probability learn the parameters of the gaussians to near optimal precision if the dimension is high d ln k we relate this to previous theoretical and empirical work on the em algorithm

a general lower bound on the number of examples needed for learning abstract we prove a lower bound of 1 ln 1 vcdim c on the number of random examples required for distribution free learning of a concept class c where vcdim c is the vapnik chervonenkis dimension and and are the accuracy and confidence parameters this improves the previous best lower bound of 1 ln 1 vcdim c and comes close to the known general upper bound of o 1 ln 1 vcdim c ln 1 for consistent algorithms we show that for many interesting concept classes including k cnf and k dnf our bound is actually tight to within a constant factor

on convergence proofs in system identification a general principle using ideas from learning theory it is shown that learning theory offers convergence analysis tools that are useful in system identification problems they allow analysis in a parameter free context which elevates the analysis from parameter sets to model sets and from parameter identification to model identification when a parameterization is eventually introduced this leads to alternative assumptions on the parameterization and parameter set moreover structural identification can be analyzed within the same framework another advantage is that the proofs are technically and conceptually simple

the dimension of chaotic attractors dimension is perhaps the most basic property of an attractor in this paper we discuss a variety of different definitions of dimension compute their values for a typical example and review previous work on the dimension of chaotic attractors the relevant definitions of dimension are of two general types those that depend only on metric properties and those that depend on the frequency with which a typical trajectory visits different regions of the attractor both our example and the previous work that we review support the conclusion that all of the frequency dependent dimensions take on the same value which we call the dimension of the natural measure and all of the metric dimensions take on a common value which we call the fractal dimension furthermore the dimension of the natural measure is typically equal to the lyapunov dimension which is defined in terms of lyapunov numbers and thus is usually far easier to calculate than any other definition because it is computable and more physically relevant we feel that the dimension of the natural measure is more important than the fractal dimension

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

a vector quantization approach to universal noiseless coding and quantization a two stage code is a block code in which each block of data is coded in two stages the first stage codes the identity of a block code among a collection of codes and the second stage codes the data using the identified code the collection of codes may be noiseless codes fixed rate quantizers or variable rate quantizers we take a vector quantization approach to two stage coding in which the first stage code can be regarded as a vector quantizer that quantizes the input data of length n to one of a fixed collection of block codes we apply the generalized lloyd algorithm to the first stage quantizer using induced measures of rate and distortion to design locally optimal two stage codes on a source of medical images two stage variable rate vector quantizers designed in this way outperform standard one stage fixed rate vector quantizers by over 9 db the tail of the operational distortion rate function of the first stage quantizer determines the optimal rate of convergence of the redundancy of a universal sequence of two stage codes we show that there exist two stage universal noiseless codes fixed rate quantizers and variable rate quantizers whose per letter rate and distortion redundancies converge to zero as k 2 n sup 1 log n when the universe of sources has finite dimension k this extends the achievability part of rissanen s theorem from universal noiseless codes to universal quantizers further we show that the redundancies converge as o n sup 1 when the universe of sources is countable and as o n sup 1 spl epsiv when the universe of sources is infinite dimensional under appropriate conditions

bregman divergence bounds and universality properties of the logarithmic loss a loss function measures the discrepancy between the true values and their estimated fits for a given instance of data in classification problems a loss function is said to be proper if a minimizer of the expected loss is the true underlying probability we show that for binary classification the divergence associated with smooth proper and convex loss functions is upper bounded by the kullback leibler kl divergence to within a normalization constant this implies that by minimizing the logarithmic loss associated with the kl divergence we minimize an upper bound to any choice of loss from this set as such the logarithmic loss is universal in the sense of providing performance guarantees with respect to a broad class of accuracy measures importantly this notion of universality is not problem specific enabling its use in diverse applications including predictive modeling data clustering and sample complexity analysis generalizations to arbitary finite alphabets are also developed the derived inequalities extend several well known f divergence results

on weak base hypotheses and their implications for boosting regression and classification when studying the training error and the prediction error for boosting it is often assumed that the hypotheses returned by the base learner are weakly accurate or are able to beat a random guesser by a certain amount of difference it has been an open question how much this difference can be whether it will eventually disappear in the boosting process or be bounded by a positive amount this question is crucial for the behavior of both the training error and the prediction error in this paper we study this problem and show affirmatively that the amount of improvement over the random guesser will be at least a positive amount for almost all possible sample realizations and for most of the commonly used base hypotheses this has a number of implications for the prediction error including for example that boosting forever may not be good and regularization may be necessary the problem is studied by first considering an analog of adaboost in regression where we study similar properties and find that for good performance one cannot hope to avoid regularization by just adopting the boosting device to regression

automatic pattern recognition a study of the probability of error a test sequence is used to select the best rule from a class of discrimination rules defined in terms of the training sequence the vapnik chervonenkis and related inequalities are used to obtain distribution free bounds on the difference between the probability of error of the selected rule and the probability of error of the best rule in the given class the bounds are used to prove the consistency and asymptotic optimality for several popular classes including linear discriminators nearest neighbor rules kernel based rules histogram rules binary tree classifiers and fourier series classifiers in particular the method can be used to choose the smoothing parameter in kernel based rules to choose k in the k nearest neighbor rule and to choose between parametric and nonparametric rules

universal donsker classes and metric entropy when mathfrak f is a universal donsker class then for independent indetically distributed i i d observation mathbf x 1 ldots mathbf x n with an unknown law p for any mathfrak f i in mathfrak f i 1 ldots m quad n 1 2 left mathfrak f 1 left mathbf x 1 right ldots mathfrak f i left mathbf x n right right 1 leq i leq m is asymptotically normal with mean vector n 1 2 left int mathfrak f i left mathbf x n right d mathbf p left x right right 1 leq i leq m and covariance matrix int mathfrak f i mathfrak f j d mathbf p int mathfrak f id mathbf p int mathfrak f jd mathbf p uniformly for mathfrak f i in mathfrak f then for certain statistics formed frome the mathfrak f i left mathbf x k right even where mathfrak f i may be chosen depending on the mathbf x k there will be asymptotic distribution as n rightarrow infty for example for mathbf x 2 statistics where f i are indicators of disjoint intervals depending suitably on mathbf x 1 ldots mathbf x n whose union is the real line mathbf x 2 quadratic forms have limiting distributions roy 1956 and watson 1958 which may however not be mathbf x 2 distributions and may depend on p chernoff and lehmann 1954 universal donsker classes of sets are up to mild measurability conditions just classes satisfying the vapnik cervonenkis comdinatorial conditions defined later in this section donsker the vapnik cervonenkis combinatorial conditions defined later in this section durst and dudley 1981 and dudley 1984 chapter 11 the use of such classes allows a variety of extensions of the roy watson results to general multidimensional sample spaces pollard 1979 and moore and subblebine 1981 vapnik and cervonenkis 1974 indicated application of their families of sets to classification pattern recognition problems more recently the classes have been applied to tree structured classifiacation breiman friedman olshen and stone 1984 chapter 12

stability and generalization we define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave one out error the methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real valued function we study the stability properties of large classes of learning algorithms such as regularization based algorithms in particular we focus on hilbert space regularization and kullback leibler regularization we demonstrate how to apply the results to svm for regression and classification

vapnik chervonenkis generalization bounds for real valued neural networks we show how lower bounds on the generalization ability of feedforward neural nets with real outputs can be derived within a formalism based directly on the concept of vc dimension and vapnik s theorem on uniform convergence of estimated probabilities

on the complexity of learning for spiking neurons with temporal coding abstract spiking neurons are models for the computational units in biological neural systems where information is considered to be encoded mainly in the temporal patterns of their activity in a network of spiking neurons a new set of parameters becomes relevant which has no counterpart in traditional neural network models the time that a pulse needs to travel through a connection between two neurons also known as delay of a connection it is known that these delays are tuned in biological neural systems through a variety of mechanisms in this article we consider the arguably most simple model for a spiking neuron which can also easily be implemented in pulsed vlsi we investigate the vapnik chervonenkis vc dimension of networks of spiking neurons where the delays are viewed as programmable parameters and we prove tight bounds for this vc dimension thus we get quantitative estimates for the diversity of functions that a network with fixed architecture can compute with different settings of its delays in particular it turns out that a network of spiking neurons with k adjustable delays is able to compute a much richer class of functions than a threshold circuit with k adjustable weights the results also yield bounds for the number of training examples that an algorithm needs for tuning the delays of a network of spiking neurons results about the computational complexity of such algorithms are also given

efficient algorithms for learning functions with bounded variation we show that the class fbv of 0 1 valued functions with total variation at most 1 can be agnostically learned with respect to the absolute loss in polynomial time from o 1 e2 log 1 examples matching a known lower bound to within a constant factor we establish a bound of o 1 m on the expected error of a polynomial time algorithm for learning fbv in the prediction model also matching a known lower bound to within a constant factor applying a known algorithm transformation to our prediction algorithm we obtain a polynomial time pac learning algorithm for fbv with a sample complexity bound of o 1 e log 1 this also matches a known lower bound to within a constant factor

nonparametric estimation via empirical risk minimization a general notion of universal consistency of nonparametric estimators is introduced that applies to regression estimation conditional median estimation curve fitting pattern recognition and learning concepts general methods for proving consistency of estimators based on minimizing the empirical error are shown in particular distribution free almost sure consistency of neural network estimates and generalized linear estimators is established

noise injection theoretical prospects noise injection consists of adding noise to the inputs during neural network training experimental results suggest that it might improve the generalization ability of the resulting neural network a justification of this improvement remains elusive describing analytically the average perturbed cost function is difficult and controlling the fluctuations of the random perturbed cost function is hard hence recent papers suggest replacing the random perturbed cost by a deterministic taylor approximation of the average perturbed cost function this article takes a different stance when the injected noise is gaussian noise injection is naturally connected to the action of the heat kernel this provides indications on the relevance domain of traditional taylor expansions and shows the dependence of the quality of taylor approximations on global smoothness properties of neural networks under consideration the connection between noise injection and heat kernel also enables controlling the fluctuations of

equivalence of models for polynomial learnability abstract in this paper we consider several variants of valiant s learnability model that have appeared in the literature we give conditions under which these models are equivalent in terms of the polynomially learnable concept classes they define these equivalences allow comparisons of most of the existing theorems in valiant style learnability and show that several simplifying assumptions on polynomial learning algorithms can be made without loss of generality we also give a useful reduction of learning problems to the problem of finding consistent hypotheses and give comparisons and equivalences between valiant s model and the prediction learning models of haussler littlestone and warmuth in 29th annual ieee symposium on foundations of computer science 1988

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

some contributions to fixed distribution learning theory we consider some problems in learning with respect to a fixed distribution we introduce two new notions of learnability these are probably uniformly approximately correct puac learnability which is a stronger requirement than the widely studied pac learnability and minimal empirical risk mer learnability which is a stronger requirement than the previously defined notions of solid or potential learnability it is shown that although the motivations for defining these two notions of learnability are entirely different these two notions are in fact equivalent to each other and in turn equivalent to a property introduced here referred to as the shrinking width property it is further shown that if the function class to be learned has the property that empirical means converge uniformly to their true values then all of these learnability properties hold in the course of proving conditions for these forms of learnability we also obtain a new estimate for the vc dimension of a collection of sets obtained by performing boolean operations on a given collection this result is of independent interest we consider both the case in which there is an underlying target function as well as the case of model free or agnostic learning finally we consider the issue of representation of a collection of sets by its subcollection of equivalence classes it is shown by example that by suitably choosing representatives of each equivalence class it is possible to affect the property of uniform convergence of empirical probabilities

u processes rates of convergence on introduit un nouveau processus stochastique une collection de statistiques u indicees par une famille de noyaux symetriques on obtient des conditions pour la convergence presque sure uniforme d une suite de tels processus

covering numbers for bounded variation functions abstract in this paper we provide upper and lower estimates for the minimal number of functions needed to represent a bounded variation function with an accuracy of epsilon with respect to l 1 distance

the vc dimension for mixtures of binary classifiers the mixtures of experts me methodology provides a tool of classification when experts of logistic regression models or bernoulli models are mixed according to a set of local weights we show that the vapnik chervonenkis dimension of the me architecture is bounded below by the number of experts m and above by o m4s2 where s is the dimension of the input for mixtures of bernoulli experts with a scalar input we show that the lower bound m is attained in which case we obtain the exact result that the vc dimension is equal to the number of experts

algorithmic stability and sanity check bounds for leave one out cross validation in this article we prove sanity check bounds for the error of the leave one out cross validation estimate of the generalization error that is bounds showing that the worst case error of this estimate is not much worse than that of the training error estimate the name sanity check refers to the fact that although we often expect the leave one out estimate to perform considerably better than the training error estimate we are here only seeking assurance that its performance will not be considerably worse perhaps surprisingly such assurance has been given only for limited cases in the prior literature on cross validation any nontrivial bound on the error of leave one out must rely on some notion of algorithmic stability previous bounds relied on the rather strong notion of hypothesis stability whose application was primarily limited to nearest neighbor and other local algorithms here we introduce the new and weaker notion of error stability and apply it to obtain sanity check bounds for leave one out for other classes of learning algorithms including training error minimization procedures and bayesian algorithms we also provide lower bounds demonstrating the necessity of some form of error stability for proving bounds on the error of the leave one out estimate and the fact that for training error minimization algorithms in the worst case such bounds must still depend on the vapnik chervonenkis dimension of the hypothesis class

minimum complexity density estimation the authors introduce an index of resolvability that is proved to bound the rate of convergence of minimum complexity density estimators as well as the information theoretic redundancy of the corresponding total description length the results on the index of resolvability demonstrate the statistical effectiveness of the minimum description length principle as a method of inference the minimum complexity estimator converges to true density nearly as fast as an estimator based on prior knowledge of the true subclass of densities interpretations and basic properties of minimum complexity estimators are discussed some regression and classification problems that can be examined from the minimum description length framework are considered

quantifying inductive bias ai learning algorithms and valiant s learning framework abstract we show that the notion of inductive bias in concept learning can be quantified in a way that directly relates to learning performance in the framework recently introduced by valiant our measure of bias is based on the growth function introduced by vapnik and chervonenkis and on the vapnik chervonenkis dimension we measure some common language biases including restriction to conjunctive concepts conjunctive concepts with internal disjunction k dnf and k cnf concepts we also measure certain types of bias that result from a preference for simpler hypotheses using these bias measurements we analyze the performance of the classical learning algorithm for conjunctive concepts from the perspective of valiant s learning framework we then augment this algorithm with a hypothesis simplification routine that uses a greedy heuristic and show how this improves learning performance on simpler target concepts improved learning algorithms are also developed for conjunctive concepts with internal disjunction k dnf and k cnf concepts we show that all our algorithms are within a logarithmic factor of optimal in terms of the number of examples they require to achieve a given level of learning performance in the valiant framework our results hold for arbitrary attribute based instance spaces defined by either tree structured or linear attributes

rates of growth and sample moduli for weighted empirical processes indexed by sets probability inequalities are obtained for the supremum of a weighted empirical process indexed by a vapnik cervonenkis class c of sets these inequalities are particularly useful under the assumption p c c p c t 0 as t 0 they are used to obtain almost sure bounds on the rate of growth of the process as the sample size approaches infinity to find an asymptotic sample modulus for the unweighted empirical process and to study the ratio pn p of the empirical measure to the actual measure

sample complexity for learning recurrent perceptron mappings recurrent perceptron classifiers generalize the usual perceptron model they correspond to linear transformations of input vectors obtained by means of autoregressive moving average schemes or infinite impulse response filters and take into account those correlations and dependences among input coordinates which arise from linear digital filtering this paper provides tight bounds on the sample complexity associated to the fitting of such models to experimental data the results are expressed in the context of the theory of probably approximately correct pac learning

on the relationship between generalization error hypothesis complexity and sample complexity for radial basis functions feedforward networks together with their training algorithms are a class of regression techniques that can be used to learn to perform some task from a set of examples the question of generalization of network performance from a finite training set to unseen data is clearly of crucial importance in this article we first show that the generalization error can be decomposed into two terms the approximation error due to the insufficient representational capacity of a finite sized network and the estimation error due to insufficient information about the target function because of the finite number of samples we then consider the problem of learning functions belonging to certain sobolev spaces with gaussian radial basis functions using the above mentioned decomposition we bound the generalization error in terms of the number of basis functions and number of examples while the bound that we derive is specific for radial basis functions a number of observations deriving from it apply to any approximation technique our result also sheds light on ways to choose an appropriate network architecture for a particular problem and the kinds of problems that can be effectively solved with finite resources i e with a finite number of parameters and finite amounts of data

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

generalization performance of bayes optimal classification algorithm for learning a perceptron the generalization error of the bayes optimal classification algorithm when learning a perceptron from noise free random training examples is calculated exactly using methods of statistical mechanics it is shown that if an assumption of replica symmetry is made then in the thermodynamic limit the error of the bayes optimal algorithm is less than the error of a canonical stochastic learning algorithm by a factor approaching ensuremath surd 2 as the ratio of the number of training examples to perceptron weights grows in addition it is shown that approximations to the generalization error of the bayes optimal algorithm can be achieved by learning algorithms that use a two layer neutral net to learn a perceptron

adaptive model selection using empirical complexities given n independent replicates of a jointly distributed pair x y rd r we wish to select from a fixed sequence of model classes f1 f2 a deterministic prediction rule f rd r whose risk is small we investigate the possibility of empirically assessing the complexity of each model class that is the actual difficulty of the estimation problem within each class the estimated complexities are in turn used to define an adaptive model selection procedure which is based on complexity penalized empirical risk the available data are divided into two parts the first is used to form an empirical cover of each model class and the second is used to select a candidate rule from each cover based on empirical risk the covering radii are determined empirically to optimize a tight upper bound on the estimation error an estimate is chosen from the list of candidates in order to minimize the sum of class complexity and empirical risk a distinguishing feature of the approach is that the complexity of each model class is assessed empirically based on the size of its empirical cover finite sample performance bounds are established for the estimates and these bounds are applied to several non parametric estimation problems the estimates are shown to achieve a favorable tradeoff between approximation and estimation error and to perform as well as if the distribution dependent complexities of the model classes were known beforehand in addition it is shown that the estimate can be consistent and even possess near optimal rates of convergence when each model class has an infinite vc or pseudo dimension for regression estimation with squared loss we modify our estimate to achieve a faster rate of convergence appears in annals of statistics vol 27 1830 1864 ams 1991 subject classifications primary 62g07 62g20 secondary 62h30

characterization of experimental time series from taylor couette flow abstract rotational taylor couette flow shows a rich variety of well defined and controllable scenarios even for short cylinders where the number of possible flow states is found to be very small period doubling cascades intermittencies homoclinic orbits and n tori can be found depending on boundary conditions we characterize these flow states by fractal dimensions lyapunov spectra and entropies in the experiments the axial velocity component was measured using laser doppler anemometry and the phase space was reconstructed by using delay time coordinates a simple noise reduction algorithm was applied to the noisy data set to improve the accuracy of the estimation of the dynamical variables

learning dynamical systems in a stationary environment abstract we consider the problem of learning the input output relation of a dynamical system from noisy data our method rests on the use of a smooth simultaneous estimator which generalizes the standard empirical estimator in a stationary environment our algorithm is shown to select a model which exhibits the probably approximately correct pac property under very mild conditions this contribution should be thought of as a first attempt to extend concepts developed in learning theory to the field of system identification where due to the presence of the system dynamics the typical i i d assumption on the data made in learning theory is not satisfied

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

information theoretic asymptotics of bayes methods in the absence of knowledge of the true density function bayesian models take the joint density function for a sequence of n random variables to be an average of densities with respect to a prior the authors examine the relative entropy distance d sub n between the true density and the bayesian density and show that the asymptotic distance is d 2 log n c where d is the dimension of the parameter vector therefore the relative entropy rate d sub n n converges to zero at rate log n n the constant c which the authors explicitly identify depends only on the prior density function and the fisher information matrix evaluated at the true parameter value consequences are given for density estimation universal data compression composite hypothesis testing and stock market portfolio selection