generalization and pac learning some new results for the class of generalized single layer networks the ability of connectionist networks to generalize is often cited as one of their most important properties we analyze the generalization ability of the class of generalized single layer networks gslns which includes volterra networks radial basis function networks regularization networks and the modified kanerva model using techniques based on the theory of probably approximately correct pac learning which have previously been used to analyze the generalization ability of feedforward networks of linear threshold elements ltes an introduction to the relevant computational learning theory is included we derive necessary and sufficient conditions on the number of training examples required by a gsln to guarantee a particular generalization performance we compare our results to those given previously for feedforward networks of ltes and show that on the basis of the currently available bounds the sufficient number of training examples for gslns is typically considerably less than for feedforward networks of ltes with the same number of weights we show that the use of self structuring techniques for gslns may reduce the number of training examples sufficient to guarantee good generalization performance and we provide an explanation for the fact that gslns can require a relatively large number of weights

refined error bounds for several learning algorithms this article studies the achievable guarantees on the error rates of certain learning algorithms with particular focus on refining logarithmic factors many of the results are based on a general technique for obtaining bounds on the error rates of sample consistent classifiers with monotonic error regions in the realizable case we prove bounds of this type expressed in terms of either the vc dimension or the sample compression size this general technique also enables us to derive several new bounds on the error rates of general sample consistent learning algorithms as well as refined bounds on the label complexity of the cal active learning algorithm additionally we establish a simple necessary and sufficient condition for the existence of a distribution free bound on the error rates of all sample consistent learning rules converging at a rate inversely proportional to the sample size we also study learning in the presence of classification noise deriving a new excess error rate guarantee for general vc classes under tsybakov s noise condition and establishing a simple and general necessary and sufficient condition for the minimax excess risk under bounded noise to converge at a rate inversely proportional to the sample size

generalization error bounds for threshold decision lists in this paper we consider the generalization accuracy of classification methods based on the iterative use of linear classifiers the resulting classifiers which we call threshold decision lists act as follows some points of the data set to be classified are given a particular classification according to a linear threshold function or hyperplane these are then removed from consideration and the procedure is iterated until all points are classified geometrically we can imagine that at each stage points of the same classification are successively chopped off from the data set by a hyperplane we analyse theoretically the generalization properties of data classification techniques that are based on the use of threshold decision lists and on the special subclass of multilevel threshold functions we present bounds on the generalization error in a standard probabilistic learning framework the primary focus in this paper is on obtaining generalization error bounds that depend on the levels of separation or margins achieved by the successive linear classifiers we also improve and extend previously published theoretical bounds on the generalization ability of perceptron decision trees

computational complexity of art gallery problems we study the computational complexity of the art gallery problem originally posed by klee and its variations specifically the problem of determining the minimum number of vertex guards that can see an n wall simply connected art gallery is shown to be np hard the proof can be modified to show that the problems of determining the minimum number of edge guards and the minimum number of point guards in a simply connected polygonal region are also np hard as a byproduct the problem of decomposing a simple polygon into a minimum number of star shaped polygons such that their union is the original polygon is also shown to be np hard

linear programming in linear time when the dimension is fixed it is demonstrated that the linear programming problem in d variables and n constraints can be solved in o n time when d is fixed this bound follows from a multidimensional search technique which is applicable for quadratic programming as well there is also developed an algorithm that is polynomial in both n and d provided d is bounded by a certain slowly growing function of n

information theoretic lower bounds on the oracle complexity of stochastic convex optimization relative to the large literature on upper bounds on complexity of convex optimization lesser attention has been paid to the fundamental hardn4516420ess of these problems given the extensive use of convex optimization in machine learning and statistics gaining an understanding of these complexity theoretic issues is important in this paper we study the complexity of stochastic convex optimization in an oracle model of computation we introduce a new notion of discrepancy between functions and use it to reduce problems of stochastic convex optimization to statistical parameter estimation which can be lower bounded using information theoretic methods using this approach we improve upon known results and obtain tight minimax complexity estimates for various function classes

learnability with respect to fixed distributions consider a first order typed language with semantics s for expressions and types adding subtyping means that a partial order on types is defined and that the typing rules are extended to the effect that expression e has type t whenever e has type s and s t we show how to adapt the semantics s in a simple set theoretic way obtaining a semantics s that satisfies in addition to some obvious requirements also the property that s s is included in s t whenever s t

learning by distances a model of learning by distances is presented in this model a concept is a point in a metric space at each step of the learning process the student guesses a hypothesis and receives from the teacher an approximation of its distance to the target a notion of a distance measuring the proximity of a hypothesis to the correct answer is common to many models of learnability by focusing on this fundamental aspect we discover some general and simple tools for the analysis of learnability tasks as a corollary we present new learning algorithms for valiant s pac scenario with any given distribution these algorithms can learn any pac learnable class and in some cases settle for significantly less information than the usual labeled examples insight gained by the new model is applied to show that every class of subsets c that has a finite vc dimension is pac learnable with respect to any fixed distribution previously known results of this nature were subject to complicated measurability constraints

bounds on the minimax rate for estimating a prior over a vc class from independent learning tasks abstract we study the optimal rates of convergence for estimating a prior distribution over a vc class from a sequence of independent data sets respectively labeled by independent target functions sampled from the prior we specifically derive upper and lower bounds on the optimal rates under a smoothness condition on the correct prior with the number of samples per data set equal the vc dimension these results have implications for the improvements achievable via transfer learning we additionally extend this setting to real valued function where we establish consistency of an estimator for the prior and discuss an additional application to a preference elicitation problem in algorithmic economics

machine learning parameter regions abstract how many random points from an identified set a confidence set or a highest posterior density set suffice to describe them this paper argues that taking random draws from a parameter region in order to approximate its shape is a supervised learning problem analogous to sampling pixels of an image to recognize it misclassification error a common criterion in machine learning provides an off the shelf tool to assess the quality of a given approximation we say a parameter region can be learned if there is an algorithm that yields a misclassification error of at most with probability at least 1 regardless of the sampling distribution we show that learning a parameter region is possible if and only if its potential shapes are not too complex moreover the tightest band that contains a d dimensional parameter region is always learnable from the inside in a sense we make precise with at least max 1 ln 1 3 16 d draws but at most min 2 d ln 2 d exp 1 2 d ln 1 these bounds grow linearly in the dimension of the parameter region and are uniform with respect to its true shape we illustrate the usefulness of our results using structural vector autoregressions we show how many orthogonal matrices are necessary sufficient to evaluate the impulse responses identified set and how many shotgun plots to report when conducting joint inference on impulse responses

results on learnability and the vapnik chervonenkis dimension abstract we consider the problem of learning a concept from examples in the distribution free model by valiant an essentially equivalent model if one ignores issues of computational difficulty was studied by vapnik and chervonenkis we introduce the notion of dynamic sampling wherein the number of examples examined may increase with the complexity of the target concept this method is used to establish the learnability of various concept classes with an infinite vapnik chervonenkis dimension we also discuss an important variation on the problem of learning from examples called approximating from examples here we do not assume that the target concept t is a member of the concept class c from which approximations are chosen this problem takes on particular interest when the vc dimension of c is infinite finally we discuss the problem of computing the vc dimension of a finite concept set defined on a finite domain and consider the structure of classes of a fixed small dimension

convexity and logical analysis of data a boolean function is called k convex if for any pair x y of its true points at hamming distance at most k every point between x and y is also true given a set of true points and a set of false points the central question of logical analysis of data is the study of those boolean functions whose values agree with those of the given points in this paper we examine data sets which admit k convex boolean extensions we provide polynomial algorithms for finding a k convex extension if any and for finding the maximum k for which a k convex extension exists we study the problem of uniqueness and provide a polynomial algorithm for checking whether all k convex extensions agree in a point outside the given data set we estimate the number of k convex boolean functions and show that for small k this number is doubly exponential on the other hand we also show that for large k the class of k convex boolean functions is pac learnable

distribution dependent sample complexity of large margin learning we obtain a tight distribution specific characterization of the sample complexity of large margin classification with l2 regularization we introduce the margin adapted dimension which is a simple function of the second order statistics of the data distribution and show distribution specific upper and lower bounds on the sample complexity both governed by the margin adapted dimension of the data distribution the upper bounds are universal and the lower bounds hold for the rich family of sub gaussian distributions with independent features we conclude that this new quantity tightly characterizes the true sample complexity of large margin classification to prove the lower bound we develop several new tools of independent interest these include new connections between shattering and hardness of learning new properties of shattering with linear classifiers and a new lower bound on the smallest eigenvalue of a random gram matrix generated by sub gaussian variables our results can be used to quantitatively compare large margin learning to other learning rules and to improve the effectiveness of methods that use sample complexity bounds such as active learning

nearly optimal classification for semimetrics we initiate the rigorous study of classification in semimetric spaces which are point sets with a distance function that is non negative and symmetric but need not satisfy the triangle inequality we define the density dimension dens and discover that it plays a central role in the statistical and algorithmic feasibility of learning in semimetric spaces we compute this quantity for several widely used semimetrics and present nearly optimal sample compression algorithms which are then used to obtain generalization guarantees including fast rates our claim of near optimality holds in both computational and statistical senses when the sample has radius r and margin we show that it can be compressed down to roughly d r dens points and further that finding a significantly better compression is algorithmically intractable unless p np this compression implies generalization via standard occam type arguments to which we provide a nearly matching lower bound

noise tolerant parallel learning of geometric concepts we present several efficient parallel algorithms for pac learning geometric concepts in a constant dimensional space the algorithms are robust even against malicious classification noise of any rate less than 1 2 we first give an efficient noise tolerant parallel algorithm to pac learn the class of geometric concepts defined by a polynomial number of d 1 dimensional hyperplanes against an arbitrary distribution where each hyperplane has a slope from a set of known slopes we then describe how boosting techniques can be used so that our algorithms dependence on and does not depend ond next we give an efficient noise tolerant parallel algorithm to pac learn the class of geometric concepts defined by a polynomial number of d 1 dimensional hyperplanes of unrestricted slopes against a uniform distribution we then show how to extend our algorithm to learn this class against any unknown product distribution next we define a complexity measure of any setsof d 1 dimensional surfaces that we call thevariantofsand prove that the class of geometric concepts defined by surfaces of polynomial variant can be efficiently learned in parallel under a product distribution even under malicious classification noise furthermore we show that the vc dimension of the class of geometric concepts defined by a single surface of variant one is finally we give an efficient parallel noise tolerant algorithm to pac learn any 2 dimensional geometric concept defined by a setsof 1 dimensional surfaces of polynomial length under a uniform distribution

minimax analysis of active learning this work establishes distribution free upper and lower bounds on the minimax label complexity of active learning with general hypothesis classes under various noise models the results reveal a number of surprising facts in particular under the noise model of tsybakov 2004 the minimax label complexity of active learning with a vc class is always asymptotically smaller than that of passive learning and is typically signi cantly smaller than the best previously published upper bounds in the active learning literature in high noise regimes it turns out that all active learning problems of a given vc dimension have roughly the same minimax label complexity which contrasts with well known results for bounded noise in low noise regimes we find that the label complexity is well characterized by a simple combinatorial complexity measure we call the star number interestingly we find that almost all of the complexity measures previously explored in the active learning literature have worst case values exactly equal to the star number we also propose new active learning strategies that nearly achieve these minimax label complexities

learning with prior information a notion of learnability is introduced referred to as learnability with prior information w p i this notion is weaker than the standard notion of probably approximately correct pac learnability a property called dispersability is introduced and it is shown that dispersability plays a key role in the study of learnability w p i specifically dispersability of a function class is always a sufficient condition for the function class to be learnable moreover in the case of concept classes dispersability is also a necessary condition for learnability w p i thus in the case of learnability w p i the dispersability property plays a role similar to the finite metric entropy condition in the case of pac learnability with a fixed distribution next the notion of learnability w p i is extended to the distribution free d f situation and it is shown that a property called d f dispersability is always a sufficient condition for d f learnability w p i and is also a necessary condition for d f learnability in the case of concept classes the approach to learning introduced in the paper is believed to be significant in all problems where a nonlinear system has to be designed based on data this includes direct inverse control and system identification

supervised learning and co training co training under the conditional independence assumption is among the models which demonstrate how radically the need for labeled data can be reduced if a huge amount of unlabeled data is available in this paper we explore how much credit for this saving must be assigned solely to the extra assumptions underlying the co training model to this end we compute general almost tight upper and lower bounds on the sample size needed to achieve the success criterion of pac learning in the realizable case within the model of co training under the conditional independence assumption in a purely supervised setting the upper bounds lie significantly below the lower bounds for pac learning without co training thus co training saves labeled data even when not combined with unlabeled data on the other hand the saving is much less radical than the known savings in the semi supervised setting

how to construct random functions a constructive theory of randomness for functions based on computational complexity is developed and a pseudorandom function generator is presented this generator is a deterministic polynomial time algorithm that transforms pairs g r where g is any one way function and r is a random k bit string to polynomial time computable functions r 1 2 k 1 2 k these r s cannot be distinguished from random functions by any probabilistic polynomial time algorithm that asks and receives the value of a function at arguments of its choice the result has applications in cryptography random constructions and complexity theory

identification criteria and lower bounds for perceptron like learning rules perceptron like learning rules are known to require exponentially many correction steps in order to identify boolean threshold functions exactly we introduce criteria that are weaker than exact identification and investigate whether learning becomes significantly faster if exact identification is replaced by one of these criteria probably approximately correct pac identification order identification and sign identification pac identification is based on the learning paradigm introduced by valiant 1984 and is known to be easier than exact identification order identification uses the fact that each threshold function induces an ordering relation on the input variables that can be represented by weights of linear size sign identification is based on a property of threshold functions known as unateness and requires only weights of constant size we show that perceptron like learning rules cannot satisfy these criteria when the number of correction steps is to be bounded by a polynomial we also pre

the learnability of quantum states traditional quantum state tomography requires a number of measurements that grows exponentially with the number of qubits n but using ideas from computational learning theory we show that one can do exponentially better in a statistical setting in particular to predict the outcomes of most measurements drawn from an arbitrary probability distribution one needs only a number of sample measurements that grows linearly with n this theorem has the conceptual implication that quantum states despite being exponentially long vectors are nevertheless reasonable in a learning theory sense the theorem also has two applications to quantum computing first a new simulation of quantum one way communication protocols and second the use of trusted classical advice to verify untrusted quantum advice

optimality of svm novel proofs and tighter bounds abstract we provide a new proof that the expected error rate of consistent support vector machines matches the minimax rate up to a constant factor in its dependence on the sample size and margin the upper bound was originally established by 1 while the lower bound follows from an argument of 2 together with reasoning about the vc dimension of large margin classifiers our proof differs from the original in that many of our steps concern reasoning about the primal space while the original carried out these steps by reasoning about the dual space our approach provides a unified framework for analyzing both the homogeneous and non homogeneous cases with slightly better results for the former the fact that our analysis explicitly handles the non homogeneous case offers significant improvements in the bounds compared to the usual textbook approach of reducing to the homogeneous case we also extend our proof to provide a new upper bound on the error rate of transductive svm which yields an improved constant factor compared to inductive svm in addition to these bounds on the expected error rate we also provide a simple proof of a margin based pac style bound for support vector machines and an extension of the agnostic pac analysis that explicitly handles the non homogeneous case

pac learning of concept classes through the boundaries of their items abstract we present a new perspective for investigating the probably approximate correct pac learnability of classes of concepts we focus on special sets of points for characterizing the concepts within their class this gives rise to a general notion of boundary of a concept which holds even in discrete spaces and to a special probability measuring technique this technique is applied i to narrow the gap between the minimum and maximum sample sizes necessary to learn even under a more stringent learnability definition and ii to get self explanatory indices of the complexity of the learning task these indices can be roughly estimated during the learning process and appear very useful in the treatment of nonsymbolic procedures e g in the context of neutral networks

bounds on the number of examples needed for learning functions we prove general lower bounds on the number of examples needed for learning function classes within different natural learning models which are related to pac learning and coincide with the pac learning model of valiant in the case of 0 1 valued functions the lower bounds are obtained by showing that all nontrivial function classes contain a hard binary valued subproblem although at first glance it seems to be likely that real valued function classes are much harder to learn than their hardest binary valued subproblem we show that these general lower bounds cannot be improved by more than a logarithmic factor this is done by discussing some natural function classes like nondecreasing functions or piecewise smooth functions the function classes that were discussed in m j kearns and r e schapire proc 31st annual symposium on the foundations of computer science ieee computer society press los alamitos ca 1990 pp 382 392 full version j comput system sci 48 1994 pp 464 497 d kimber and p m long proc 5th annual workshop on computational learning theory acm new york 1992 pp 153 160 with certain restrictions concerning their slope

gaining degrees of freedom in subsymbolic learning we provide some theoretical results on sample complexity of pac learning when the hypotheses are given by subsymbolical devices such as neural networks in this framework we give new foundations to the notion of degrees of freedom of a statistic and relate it to the complexity of a concept class thus for a given concept class and a given sample size we discuss the efficiency of subsymbolical learning algorithms in terms of degrees of freedom of the computed statistic in this setting we appraise the sample complexity overhead coming from relying on approximate hypotheses and display an increase in the degrees of freedom yield by embedding available formal knowledge into the algorithm for known sample distribution these quantities are related to the learning approximation goal and a special production prize is shown finally we prove that testing the approximation capability of a neural network generally demands smaller sample size than training it

exact learning of discretized geometric concepts we first present an algorithm that uses membership and equivalence queries to exactly identify a discretized geometric concept defined by the union of m axis parallel boxes in d dimensional discretized euclidean space where each coordinate can have n discrete values this algorithm receives at most md counterexamples and uses time and membership queries polynomial in m and log n for any constant d furthermore all equivalence queries can be formulated as the union of o md log m axis parallel boxes next we show how to extend our algorithm to efficiently learn from only equivalence queries any discretized geometric concept generated from any number of halfspaces with any number of known to the learner slopes in a constant dimensional space in particular our algorithm exactly learns from equivalence queries only unions of discretized axis parallel boxes in constant dimensional space in polynomial time furthermore this equivalence query only algorithm can be modified to handle a polynomial number of lies in the counterexamples provided by the environment finally we introduce a new complexity measure that better captures the complexity of the union of m boxes than simply the number of boxes and the dimension our new measure sigma is the number of segments in the target where a segment is a maximum portion of one of the sides of the target that lies entirely inside or entirely outside each of the other halfspaces defining the target we present a modification of our first algorithm that uses time and queries polynomial in sigma and log n in fact the time and queries both membership and equivalence used by this single algorithm are polynomial for either m or d constant

on the necessity of occam algorithms abstract the distribution independent model of concept learning from examples pac learning due to valiant 1984 is investigated it has been shown that the existence of an occam algorithm for a class of concepts is a sufficient condition for the pac learnability of that class blumer 1987 1989 an occam algorithm is a randomized polynomial time algorithm that when given as input a sample of strings of some unknown concept to be learned outputs a small description of a concept that is consistent with the sample in this paper it is shown for many natural concept classes that the pac learnability of the class implies the existence of an occam algorithm for the class more generally the property of closure under exception lists is defined and it is shown that for any concept class with this property pac learnability of the class is equivalent to the existence of an occam algorithm for the class an interpretation of these results is that for many classes pac learnability is equivalent to data compression

a metric entropy bound is not sufficient for learnability the authors prove by means of a counterexample that it is not sufficient for probably approximately correct pac learning under a class of distributions to have a uniform bound on the metric entropy of the class of concepts to be learned this settles a conjecture of benedek and itai 1991

bandwidth choice for nonparametric classification it is shown that for kernel based classification with univariate distributions and two populations optimal bandwidth choice has a dichotomous character if the two densities cross at just one point where their curvatures have the same signs then minimum bayes risk is achieved using bandwidths which are an order of magnitude larger than those which minimize pointwise estimation error on the other hand if the curvature signs are different or if there are multiple crossing points then bandwidths of conventional size are generally appropriate the range of different modes of behavior is narrower in multivariate settings there the optimal size of bandwidth is generally the same as that which is appropriate for pointwise density estimation these properties motivate empirical rules for bandwidth choice

probably approximate learning over classes of distributions this paper presents algorithms that construct approximations to sets and functions on the reals from randomly chosen sample points the model that is analyzed is a generalization of the paradigm of probably approximate learning proposed by valiant previously necessary and sufficient conditions for learning sets were established for the case when the class of sampling distributions is finite and for the case when the class of sampling distributions is the set of all possible distributions here sufficient conditions are obtained for learning sets and functions over general classes of sampling distributions the results for functions are with respect to general metrics for measuring the distance between two functions

on the complexity of learning from drifting distributions abstract we consider two models of on line learning of binary valued functions from drifting distributions due to bartlett we show that if each example is drawn from a joint distribution which changes in total variation distance by at mosto e3 d log 1 e between trials then an algorithm can achieve a probability of a mistake at mosteworse than the best function in a class of vc dimensiond we prove a corresponding necessary condition ofo e3 d finally in the case that a fixed function is to be learned from noise free examples we show that if the distributions on the domain generating the examples change by at mosto e2 d log 1 e then any consistent algorithm learns to within accuracye

non redundant precoding based blind and semi blind channel estimation for mimo block transmission with a cyclic prefix by combining multiple input multiple output mimo communication with block transmission using a cyclic prefix cp mimo block transmission systems can not only achieve high data rates but also simplify channel estimation and equalization in this paper to provide a bandwidth efficient solution for channel estimation in mimo block transmission systems with a cp we present a framework for blind channel estimation based on a general non redundant precoding using this framework we propose a blind estimator exploiting a simplified non redundant precoding that is robust against overestimates of a true mimo channel order furthermore in the case with the number of transmit antennas greater than that of receive antennas we show under specific system conditions that the proposed blind algorithms can be used for estimation of the mimo channel without oversampling the received signals with the simplified precoding conditions established in this paper the proposed method does not impose the strict identifiability conditions on the mimo channel which are required for the existing methods furthermore the proposed estimator achieves accurate channel estimation with a small number of symbol blocks in addition we derive a simplified precoder that is optimized in the sense of minimizing the impact of unknown additive noise we discuss the tradeoff between channel estimation accuracy and bit error rate performance associated with the simplified precoder finally we develop a technique for resolving the channel ambiguity in the proposed blind estimator to create a semi blind channel estimator relying on only a few known symbols

pac learning with generalized samples and an applicaiton to stochastic geometry an extension of the standard probably approximately correct pac learning model that allows the use of generalized samples is introduced a generalized sample is viewed as a pair consisting of a functional on the concept class together with the value obtained by the functional operating on the unknown concept it appears that this model can be applied to a number of problems in signal processing and geometric reconstruction to provide sample size bounds under a pac criterion a specific application of the generalized model to a problem of curve reconstruction is considered and some connections with a result from stochastic geometry are discussed

learning by canonical smooth estimation ii learning and choice of model complexity this paper analyzes the properties of a procedure for learning from examples this canonical learner is based on a canonical error estimator developed in part i in learning problems one can observe data that consists of labeled sample points and the goal is to find a model or hypothesis from a set of candidates that will accurately predict the labels of new sample points the expected mismatch between a hypothesis prediction and the actual label of a new sample point is called the hypothesis generalization error we compare the canonical learner with the traditional technique of finding hypotheses that minimize the relative frequency based empirical error estimate it is shown that for a broad class of learning problems the set of cases for which such empirical error minimization works is a proper subset of the cases for which the canonical learner works we derive bounds to show that the number of samples required by these two methods is comparable we also address the issue of how to determine the appropriate complexity for the class of candidate hypotheses

a general lower bound on the number of examples needed for learning abstract we prove a lower bound of 1 ln 1 vcdim c on the number of random examples required for distribution free learning of a concept class c where vcdim c is the vapnik chervonenkis dimension and and are the accuracy and confidence parameters this improves the previous best lower bound of 1 ln 1 vcdim c and comes close to the known general upper bound of o 1 ln 1 vcdim c ln 1 for consistent algorithms we show that for many interesting concept classes including k cnf and k dnf our bound is actually tight to within a constant factor

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

sample efficient strategies for learning in the presence of noise in this paper we prove various results about pac learning in the presence of malicious noise our main interest is the sample size behavior of learning algorithms we prove the first nontrivial sample complexity lower bound in this model by showing that order of e 2 d up to logarithmic factors examples are necessary for pac learning any target class of 0 1 valued functions of vc dimension d where e is the desired accuracy and e e 1 e the malicious noise rate it is well known that any nontrivial target class cannot be pac learned with accuracy e and malicious noise rate e e 1 e this irrespective to sample complexity we also show that this result cannot be significantly improved in general by presenting efficient learning algorithms for the class of all subsets of d elements and the class of unions of at most d intervals on the real line this is especialy interesting as we can also show that the popular minimum disagreement strategy needs samples of size d e 2 hence is not optimal with respect to sample size we then discuss the use of randomized hypotheses for these the bound e 1 e on the noise rate is no longer true and is replaced by 2e 1 2e in fact we present a generic algorithm using randomized hypotheses that can tolerate noise rates slightly larger than e 1 e while using samples of size d e as in the noise free case again one observes a quadratic powerlaw in this case d e 2 2e 1 2e e as goes to zero we show upper and lower bounds of this order

modified chernoff bounds for pam systems with noise and interference corresp by using principles of analytic continuation upper and lower bounds on the error probability of a canonical binary system corrupted by additive interference and independent zero mean gaussian noise are derived the bounds which are simple functions require only the evaluation or bounding of the moment generating function of the interference for large signal to noise ratios snr and even for moderately large interference the bounds are shown to be tight and useful

localization of vc classes beyond local rademacher complexities abstract in statistical learning the excess risk of empirical risk minimization erm is controlled by comp n f n where n is a size of a learning sample comp n f is a complexity term associated with a given class f and 1 2 1 interpolates between slow and fast learning rates in this paper we introduce an alternative localization approach for binary classification that leads to a novel complexity measure fixed points of the local empirical entropy we show that this complexity measure gives a tight control over comp n f in the upper bounds under bounded noise our results are accompanied by a minimax lower bound that involves the same quantity in particular we practically answer the question of optimality of erm under bounded noise for general vc classes

optimization approaches to supervised classification abstract the supervised classification problem one of the oldest and most recurrent problems in applied data analysis has always been analyzed from many different perspectives when the emphasis is placed on its overall goal of developing classification rules with minimal classification cost supervised classification can be understood as an optimization problem on the other hand when the focus is in modeling the uncertainty involved in the classification of future unknown entities it can be formulated as a statistical problem other perspectives that pay particular attention to pattern recognition and machine learning aspects of supervised classification have also a long history that has lead to influential insights and different methodologies in this review two approaches to supervised classification strongly related to optimization theory will be discussed and compared in particular we will review methodologies based on mathematical programming models that optimize observable criteria linked to the true objective of misclassification error or cost minimization and approaches derived from the minimization of known bounds on the true misclassification error the former approach is known as the mathematical programming approach to supervised classification while the latter is in the origin of the well known classification support vector machines throughout the review two group as well as general multi group problems will be considered and the review will conclude with a discussion of the most promising research directions in this area

improved lower bounds for learning from noisy examples this paper presents a general information theoretic approach for obtaining lower bounds on the number of examples required for probably approximately correct pac learning in the presence of noise this approach deals directly with the fundamental information quantities avoiding a bayesian analysis the technique is applied to several different models illustrating its generality and power the resulting bounds add logarithmic factors to or improve the constants in previously known lower bounds

optimum placement of guards abstract we consider the problem of placing guards in a polygon so that 1 a the area or 2 b the portion of the boundary visible to the guards is maximized we show that finding optimum placements for k guards is np hard if k is a variable we reduce the problem of optimally placing one guard to solving a high order equation and give a polynomial time approximation scheme for placing one guard in a simple polygon

automatic pattern recognition a study of the probability of error a test sequence is used to select the best rule from a class of discrimination rules defined in terms of the training sequence the vapnik chervonenkis and related inequalities are used to obtain distribution free bounds on the difference between the probability of error of the selected rule and the probability of error of the best rule in the given class the bounds are used to prove the consistency and asymptotic optimality for several popular classes including linear discriminators nearest neighbor rules kernel based rules histogram rules binary tree classifiers and fourier series classifiers in particular the method can be used to choose the smoothing parameter in kernel based rules to choose k in the k nearest neighbor rule and to choose between parametric and nonparametric rules

smart pac learners the pac learning model is distribution independent in the sense that the learner must reach a learning goal with a limited number of labeled random examples without any prior knowledge of the underlying domain distribution in order to achieve this one needs generalization error bounds that are valid uniformly for every domain distribution these bounds are almost tight in the sense that there is a domain distribution which does not admit a generalization error being significantly smaller than the general bound note however that this leaves open the possibility to achieve the learning goal faster if the underlying distribution is simple informally speaking we say a pac learner l is smart if for a vast majority of domain distributions d l does not require significantly more examples to reach the learning goal than the best learner whose strategy is specialized to d in this paper focusing on sample complexity and ignoring computational issues we show that smart learners do exist this implies at least from an information theoretical perspective that full prior knowledge of the domain distribution or access to a huge collection of unlabeled examples does for a vast majority of domain distributions not significantly reduce the number of labeled examples required to achieve the learning goal

universal donsker classes and metric entropy when mathfrak f is a universal donsker class then for independent indetically distributed i i d observation mathbf x 1 ldots mathbf x n with an unknown law p for any mathfrak f i in mathfrak f i 1 ldots m quad n 1 2 left mathfrak f 1 left mathbf x 1 right ldots mathfrak f i left mathbf x n right right 1 leq i leq m is asymptotically normal with mean vector n 1 2 left int mathfrak f i left mathbf x n right d mathbf p left x right right 1 leq i leq m and covariance matrix int mathfrak f i mathfrak f j d mathbf p int mathfrak f id mathbf p int mathfrak f jd mathbf p uniformly for mathfrak f i in mathfrak f then for certain statistics formed frome the mathfrak f i left mathbf x k right even where mathfrak f i may be chosen depending on the mathbf x k there will be asymptotic distribution as n rightarrow infty for example for mathbf x 2 statistics where f i are indicators of disjoint intervals depending suitably on mathbf x 1 ldots mathbf x n whose union is the real line mathbf x 2 quadratic forms have limiting distributions roy 1956 and watson 1958 which may however not be mathbf x 2 distributions and may depend on p chernoff and lehmann 1954 universal donsker classes of sets are up to mild measurability conditions just classes satisfying the vapnik cervonenkis comdinatorial conditions defined later in this section donsker the vapnik cervonenkis combinatorial conditions defined later in this section durst and dudley 1981 and dudley 1984 chapter 11 the use of such classes allows a variety of extensions of the roy watson results to general multidimensional sample spaces pollard 1979 and moore and subblebine 1981 vapnik and cervonenkis 1974 indicated application of their families of sets to classification pattern recognition problems more recently the classes have been applied to tree structured classifiacation breiman friedman olshen and stone 1984 chapter 12

stability and generalization we define notions of stability for learning algorithms and show how to use these notions to derive generalization error bounds based on the empirical error and the leave one out error the methods we use can be applied in the regression framework as well as in the classification one when the classifier is obtained by thresholding a real valued function we study the stability properties of large classes of learning algorithms such as regularization based algorithms in particular we focus on hilbert space regularization and kullback leibler regularization we demonstrate how to apply the results to svm for regression and classification

single carrier frequency domain equalization with decision feedback processing for time reversal space time block coded systems in this letter we propose to introduce frequency domain equalization with decision feedback processing fd dfe for time reversal space time block coded tr stbc systems with multiple transmit antennas to effectively achieve both spatial diversity and multipath diversity over frequency selective channels this letter will also present a theoretical performance analysis result that relates the minimum mean squared error mmse to a new tight upper bound for the probability of error it is shown that the new bound is not only valid for the tr stbc systems but also valid for single input single output single input multiple output and multiple input multiple output systems with mmse equalization

some contributions to fixed distribution learning theory we consider some problems in learning with respect to a fixed distribution we introduce two new notions of learnability these are probably uniformly approximately correct puac learnability which is a stronger requirement than the widely studied pac learnability and minimal empirical risk mer learnability which is a stronger requirement than the previously defined notions of solid or potential learnability it is shown that although the motivations for defining these two notions of learnability are entirely different these two notions are in fact equivalent to each other and in turn equivalent to a property introduced here referred to as the shrinking width property it is further shown that if the function class to be learned has the property that empirical means converge uniformly to their true values then all of these learnability properties hold in the course of proving conditions for these forms of learnability we also obtain a new estimate for the vc dimension of a collection of sets obtained by performing boolean operations on a given collection this result is of independent interest we consider both the case in which there is an underlying target function as well as the case of model free or agnostic learning finally we consider the issue of representation of a collection of sets by its subcollection of equivalence classes it is shown by example that by suitably choosing representatives of each equivalence class it is possible to affect the property of uniform convergence of empirical probabilities

sample size lower bounds in pac learning by algorithmic complexity theory this paper focuses on a general setup for obtaining sample size lower bounds for learning concept classes under fixed distribution laws in an extended pac learning framework these bounds do not depend on the running time of learning procedures and are information theoretic in nature they are based on incompressibility methods drawn from kolmogorov complexity and algorithmic probability theories

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

integrated common sense learning and planning in pomdps we formulate a new variant of the problem of planning in an unknown environment for which we can provide algorithms with reasonable theoretical guarantees in spite of large state spaces and time horizons partial observability and complex dynamics in this variant an agent is given a collection of example traces produced by a reference policy which may for example capture the agent s past behavior the agent is only asked to find policies that are supported by regularities in the dynamics that are observable on these example traces we describe an efficient algorithm that uses such common sense knowledge reflected in the example traces to construct decision tree policies for goal oriented factored pomdps more precisely our algorithm provably succeeds at finding policy for a given input goal when 1 there is a cnf that is almost always observed satisfied on the traces of the pomdp capturing a sufficient approximation of its dynamics and 2 for a decision tree policy of bounded complexity there exist small space resolution proofs that the goal is achieved on each branch using the aforementioned cnf capturing the common sense rules such a cnf always exists for noisy strips domains for example our results thus essentially establish that the possession of a suitable exploration policy for collecting the necessary examples is the fundamental obstacle to learning to act in such environments

the optimal sample complexity of pac learning this work establishes a new upper bound on the number of samples sufficient for pac learning in the realizable case the bound matches known lower bounds up to numerical constant factors this solves a long standing open problem on the sample complexity of pac learning the technique and analysis build on a recent breakthrough by hans simon

the one inclusion graph algorithm is near optimal for the prediction model of learning haussler littlestone and warmuth 1994 described a general purpose algorithm for learning according to the prediction model and proved an upper bound on the probability that their algorithm makes a mistake in terms of the number of examples seen and the vapnik chervonenkis vc dimension of the concept class being learned we show that their bound is within a factor of 1 o 1 of the best possible such bound for any algorithm

quantifying inductive bias ai learning algorithms and valiant s learning framework abstract we show that the notion of inductive bias in concept learning can be quantified in a way that directly relates to learning performance in the framework recently introduced by valiant our measure of bias is based on the growth function introduced by vapnik and chervonenkis and on the vapnik chervonenkis dimension we measure some common language biases including restriction to conjunctive concepts conjunctive concepts with internal disjunction k dnf and k cnf concepts we also measure certain types of bias that result from a preference for simpler hypotheses using these bias measurements we analyze the performance of the classical learning algorithm for conjunctive concepts from the perspective of valiant s learning framework we then augment this algorithm with a hypothesis simplification routine that uses a greedy heuristic and show how this improves learning performance on simpler target concepts improved learning algorithms are also developed for conjunctive concepts with internal disjunction k dnf and k cnf concepts we show that all our algorithms are within a logarithmic factor of optimal in terms of the number of examples they require to achieve a given level of learning performance in the valiant framework our results hold for arbitrary attribute based instance spaces defined by either tree structured or linear attributes

on the sample complexity of pac learning half spaces against the uniform distribution we prove an spl omega d spl epsiv 1 spl epsiv log1 spl delta lower bound on the pac probably approximately correct learning sample complexity of learning half spaces against the uniform distribution on the unit ball in r sup d

neural nets with superlinear vc dimension it has been known for quite a while that the vapnik chervonenkis dimension vc dimension of a feedforward neural net with linear threshold gates is at most o w log w where w is the total number of weights in the neural net we show in this paper that this bound is in fact asymptotically optimal more precisely we exhibit for any depth d 3 a large class of feedforward neural nets of depth d with w weights that have vc dimension w log w this lower bound holds even if the inputs are restricted to boolean values the proof of this result relies on a new method that allows us to encode more program bits in the weights of a neural net than previously thought possible

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

learning regular sets from queries and counterexamples the problem of identifying an unknown regular set from examples of its members and nonmembers is addressed it is assumed that the regular set is presented by a minimamy adequate teacher which can answer membership queries about the set and can also test a conjecture and indicate whether it is equal to the unknown set and provide a counterexample if not a counterexample is a string in the symmetric difference of the correct set and the conjectured set a learning algorithm l is described that correctly learns any regular set from any minimally adequate teacher in time polynomial in the number of states of the minimum dfa for the set and the maximum length of any counterexample provided by the teacher it is shown that in a stochastic setting the ability of the teacher to test conjectures may be replaced by a random sampling oracle ex a polynomial time learning algorithm is shown for a particular problem of context free language identification

on the capabilities of higher order neurons a radial basis function approach higher order neurons with k monomials in n variables are shown to have vapnik chervonenkis vc dimension at least nk 1 this result supersedes the previously known lower bound obtained via k term monotone disjunctive normal form dnf formulas moreover it implies that the vc dimension of higher order neurons with k monomials is strictly larger than the vc dimension of k term monotone dnf the result is achieved by introducing an exponential approach that employs gaussian radial basis function neural networks for obtaining classifications of points in terms of higher order neurons

inferring decision trees using the minimum description length principle abstract we explore the use of rissanen s minimum description length principle for the construction of decision trees empirical results comparing this approach to other methods are given

channel estimation for ofdm transmission in multipath fading channels based on parametric channel modeling we present an improved channel estimation algorithm for orthogonal frequency division multiplexing mobile communication systems using pilot subcarriers this algorithm is based on a parametric channel model where the channel frequency response is estimated using an l path channel model in the algorithm we employ the esprit estimation of signal parameters by rotational invariance techniques method to do the initial multipath time delays acquisition and propose an interpath interference cancellation delay locked loop to track the channel multipath time delays with the multipath time delays information a minimum mean square error estimator is derived to estimate the channel frequency response it is demonstrated that the use of the parametric channel model can effectively reduce the signal subspace dimension of the channel correlation matrix for the sparse multipath fading channels and consequently improve the channel estimation performance

minimum polygonal separation abstract in this paper we study the problem of polygonal separation in the plane i e finding a convex polygon with minimum number k of sides separating two given finite point sets k separator if it exists we show that for k n n log n is a lower bound to the running time of any algorithm for this problem and exhibit two algorithms of distinctly different flavors the first relies on an o n log n time preprocessing task which constructs the convex hull of the internal set and a nested star shaped polygon determined by the external set the k separator is contained in the annulus between the boundaries of these two polygons and is constructed in additional linear time the second algorithm adapts the prune and search approach and constructs in each iteration one side of the separator its running time is o kn but the separator may have one more side than the minimum

lower bounds in pattern recognition and learning lower bounds are derived for the performance of any pattern recognition algorithm which using training data selects a discrimination rule from a certain class of rules the bounds involve the vapnik chervonenkis dimension of the class and l the minimal error probability within the class we provide lower bounds when l 0 the usual assumption in valiant s theory of learning and l 0

on the necessity of irrelevant variables this work explores the effects of relevant and irrelevant boolean variables on the accuracy of classifiers the analysis uses the assumption that the variables are conditionally independent given the class and focuses on a natural family of learning algorithms for such sources when the relevant variables have a small advantage over random guessing the main result is that algorithms relying predominately on irrelevant variables have error probabilities that quickly go to 0 in situations where algorithms that limit the use of irrelevant variables have errors bounded below by a positive constant we also show that accurate learning is possible even when there are so few examples that one cannot determine with high confidence whether or not any individual variable is relevant

sequential and parallel complexity of approximate evaluation of polynomial zeros our new sequential and parallel algorithms establish new record upper bounds on both arithmetic and boolean complexity of approximating to complex polynomial zeros o n2 log b log n arithmetic operations or o n log n log bn parallel steps and n log b log bn processors suffice in order to approximate with absolute errors 2m b to all the complex zeros of an nth degree polynomial p x whose coefficients have mod 2m if we only need such an approximation to a single zero of p x then o n log b log n arithmetic operations or o log2 n log bn steps and n log n log b log bn processors suffice which places the latter problem in nc that is in the class of problems that can be solved using polylogarithmic parallel time and a polynomial number of processors those estimates are reached in computations with o bn binary bits where the polynomial has integer coefficients we also reach the sequential boolean time bounds o bn3 log bn log log bn for approximating to all the zeros very minor improvement of the bound announced in 1982 by schonhage and o bn2log log n log bn log log bn for approximating to a single zero among further implications are the improvements of the known algorithms and complexity estimates for computing matrix eigenvalues for polynomial factorization over the field of complex numbers and for solving systems of polynomial equations the computations rely on recursive application of turan s proximity test of 1968 on its more recent extensions to root radii computations on contour integration via fast fourier transform fft within geometric constructions for search and exclusion and for the final minor improvements of the complexity bounds on the recursive factorization of p x over discs on the complex plane via numerical integration and newton s iterations

computational limitations on learning from examples the computational complexity of learning boolean concepts from examples is investigated it is shown for various classes of concept representations that these cannot be learned feasibly in a distribution free sense unless r np these classes include a disjunctions of two monomials b boolean threshold functions and c boolean formulas in which each variable occurs at most once relationships between learning of heuristics and finding approximate solutions to np hard optimization problems are given

the importance of convexity in learning with squared loss we show that if the closure of a function class f under the metric induced by some probability distribution is not convex then the sample complexity for agnostically learning f with squared loss using only hypotheses in f is spl omega ln 1 spl delta spl epsiv sup 2 where 1 spl delta is the probability of success and spl epsiv is the required accuracy in comparison if the class f is convex and has finite pseudodimension then the sample complexity is o 1 spl epsiv ln 1 spl epsiv ln 1 b if a nonconvex class f has finite pseudodimension then the sample complexity for agnostically learning the closure of the convex hull of f is o 1 spl epsiv 1 spl epsiv ln 1 spl epsiv ln 1 spl delta hence for agnostic learning learning the convex hull provides better approximation capabilities with little sample complexity penalty