generalization and pac learning some new results for the class of generalized single layer networks the ability of connectionist networks to generalize is often cited as one of their most important properties we analyze the generalization ability of the class of generalized single layer networks gslns which includes volterra networks radial basis function networks regularization networks and the modified kanerva model using techniques based on the theory of probably approximately correct pac learning which have previously been used to analyze the generalization ability of feedforward networks of linear threshold elements ltes an introduction to the relevant computational learning theory is included we derive necessary and sufficient conditions on the number of training examples required by a gsln to guarantee a particular generalization performance we compare our results to those given previously for feedforward networks of ltes and show that on the basis of the currently available bounds the sufficient number of training examples for gslns is typically considerably less than for feedforward networks of ltes with the same number of weights we show that the use of self structuring techniques for gslns may reduce the number of training examples sufficient to guarantee good generalization performance and we provide an explanation for the fact that gslns can require a relatively large number of weights

refined error bounds for several learning algorithms this article studies the achievable guarantees on the error rates of certain learning algorithms with particular focus on refining logarithmic factors many of the results are based on a general technique for obtaining bounds on the error rates of sample consistent classifiers with monotonic error regions in the realizable case we prove bounds of this type expressed in terms of either the vc dimension or the sample compression size this general technique also enables us to derive several new bounds on the error rates of general sample consistent learning algorithms as well as refined bounds on the label complexity of the cal active learning algorithm additionally we establish a simple necessary and sufficient condition for the existence of a distribution free bound on the error rates of all sample consistent learning rules converging at a rate inversely proportional to the sample size we also study learning in the presence of classification noise deriving a new excess error rate guarantee for general vc classes under tsybakov s noise condition and establishing a simple and general necessary and sufficient condition for the minimax excess risk under bounded noise to converge at a rate inversely proportional to the sample size

generalization error bounds for threshold decision lists in this paper we consider the generalization accuracy of classification methods based on the iterative use of linear classifiers the resulting classifiers which we call threshold decision lists act as follows some points of the data set to be classified are given a particular classification according to a linear threshold function or hyperplane these are then removed from consideration and the procedure is iterated until all points are classified geometrically we can imagine that at each stage points of the same classification are successively chopped off from the data set by a hyperplane we analyse theoretically the generalization properties of data classification techniques that are based on the use of threshold decision lists and on the special subclass of multilevel threshold functions we present bounds on the generalization error in a standard probabilistic learning framework the primary focus in this paper is on obtaining generalization error bounds that depend on the levels of separation or margins achieved by the successive linear classifiers we also improve and extend previously published theoretical bounds on the generalization ability of perceptron decision trees

the decomposition of lie powers let g be a group f a field of prime characteristic p and v a finite dimensional fg module let l v denote the free lie algebra on v regarded as an fg submodule of the free associative algebra or tensor algebra t v for each positive integer r let l r v and t r v be the r th homogeneous components of l v and t v respectively here l r v is called the r th lie power of v our main result is that there are submodules b 1 b 2 of l v such that for all r b r is a direct summand of t r v and whenever m geqslant 0 and k is not divisible by p the module l p mk v is the direct sum of l p m b k l p m 1 b pk l 1 b p mk thus every lie power is a direct sum of lie powers of p power degree the approach builds on an analysis of t r v as a bimodule for g and the solomon descent algebra

information theoretic lower bounds on the oracle complexity of stochastic convex optimization relative to the large literature on upper bounds on complexity of convex optimization lesser attention has been paid to the fundamental hardn4516420ess of these problems given the extensive use of convex optimization in machine learning and statistics gaining an understanding of these complexity theoretic issues is important in this paper we study the complexity of stochastic convex optimization in an oracle model of computation we introduce a new notion of discrepancy between functions and use it to reduce problems of stochastic convex optimization to statistical parameter estimation which can be lower bounded using information theoretic methods using this approach we improve upon known results and obtain tight minimax complexity estimates for various function classes

on lie powers of the augmentation ideal in characteristic 2 abstract let g be a finite group k a field and v a finite dimensional kg module write l v for the free lie algebra on v the action of g on v extends naturally to l v so it becomes a kg module which is direct sum of finite dimensional submodules for a finite dimensional kg module v let i v denote the set of isomorphism classes of indecomposable direct summands of v and i l v n 1 i l n v it is natural to ask whether i l v is finite of course this is not a question unless there exist infinitely many isomorphism classes of indecomposable kg modules that is k has positive characteristic p and the sylow p subgroups of g are non cyclic and dim v 2 let k be a field of characteristic 2 and g the klein four group we write d k g for the augmentation ideal of kg for the dual of and v 1 v 2 v 3 for the three 2 dimensional induced modules from the three cyclic subgroups of g in this article we concentrate on lie powers l n of and describe the module decomposition of l 16 a module of dimension 2690010 in particular i l 16 d v 1 v 2 v 3

learnability with respect to fixed distributions consider a first order typed language with semantics s for expressions and types adding subtyping means that a partial order on types is defined and that the typing rules are extended to the effect that expression e has type t whenever e has type s and s t we show how to adapt the semantics s in a simple set theoretic way obtaining a semantics s that satisfies in addition to some obvious requirements also the property that s s is included in s t whenever s t

noise tolerant parallel learning of geometric concepts we present several efficient parallel algorithms for pac learning geometric concepts in a constant dimensional space the algorithms are robust even against malicious classification noise of any rate less than 1 2 we first give an efficient noise tolerant parallel algorithm to pac learn the class of geometric concepts defined by a polynomial number of d 1 dimensional hyperplanes against an arbitrary distribution where each hyperplane has a slope from a set of known slopes we then describe how boosting techniques can be used so that our algorithms dependence on and does not depend ond next we give an efficient noise tolerant parallel algorithm to pac learn the class of geometric concepts defined by a polynomial number of d 1 dimensional hyperplanes of unrestricted slopes against a uniform distribution we then show how to extend our algorithm to learn this class against any unknown product distribution next we define a complexity measure of any setsof d 1 dimensional surfaces that we call thevariantofsand prove that the class of geometric concepts defined by surfaces of polynomial variant can be efficiently learned in parallel under a product distribution even under malicious classification noise furthermore we show that the vc dimension of the class of geometric concepts defined by a single surface of variant one is finally we give an efficient parallel noise tolerant algorithm to pac learn any 2 dimensional geometric concept defined by a setsof 1 dimensional surfaces of polynomial length under a uniform distribution

machine learning parameter regions abstract how many random points from an identified set a confidence set or a highest posterior density set suffice to describe them this paper argues that taking random draws from a parameter region in order to approximate its shape is a supervised learning problem analogous to sampling pixels of an image to recognize it misclassification error a common criterion in machine learning provides an off the shelf tool to assess the quality of a given approximation we say a parameter region can be learned if there is an algorithm that yields a misclassification error of at most with probability at least 1 regardless of the sampling distribution we show that learning a parameter region is possible if and only if its potential shapes are not too complex moreover the tightest band that contains a d dimensional parameter region is always learnable from the inside in a sense we make precise with at least max 1 ln 1 3 16 d draws but at most min 2 d ln 2 d exp 1 2 d ln 1 these bounds grow linearly in the dimension of the parameter region and are uniform with respect to its true shape we illustrate the usefulness of our results using structural vector autoregressions we show how many orthogonal matrices are necessary sufficient to evaluate the impulse responses identified set and how many shotgun plots to report when conducting joint inference on impulse responses

results on learnability and the vapnik chervonenkis dimension abstract we consider the problem of learning a concept from examples in the distribution free model by valiant an essentially equivalent model if one ignores issues of computational difficulty was studied by vapnik and chervonenkis we introduce the notion of dynamic sampling wherein the number of examples examined may increase with the complexity of the target concept this method is used to establish the learnability of various concept classes with an infinite vapnik chervonenkis dimension we also discuss an important variation on the problem of learning from examples called approximating from examples here we do not assume that the target concept t is a member of the concept class c from which approximations are chosen this problem takes on particular interest when the vc dimension of c is infinite finally we discuss the problem of computing the vc dimension of a finite concept set defined on a finite domain and consider the structure of classes of a fixed small dimension

convexity and logical analysis of data a boolean function is called k convex if for any pair x y of its true points at hamming distance at most k every point between x and y is also true given a set of true points and a set of false points the central question of logical analysis of data is the study of those boolean functions whose values agree with those of the given points in this paper we examine data sets which admit k convex boolean extensions we provide polynomial algorithms for finding a k convex extension if any and for finding the maximum k for which a k convex extension exists we study the problem of uniqueness and provide a polynomial algorithm for checking whether all k convex extensions agree in a point outside the given data set we estimate the number of k convex boolean functions and show that for small k this number is doubly exponential on the other hand we also show that for large k the class of k convex boolean functions is pac learnable

modular lie representations of groups of prime order let k be a field of prime characteristic p and let g be a group of order p for any finite dimensional kg module v and any positive integer n let ln v denote the nth homogeneous component of the free lie k algebra generated by a basis of v then ln v can be considered as a kg module called the nth lie power of v the main result of the paper is a formula which describes the module structure of ln v up to isomorphism

lie powers of modules for groups of prime order let l v be the free lie algebra on a finite dimensional vector space v over a field k with homogeneous components l n v for n geq 1 if g is a group and v is a kg module the action of g extends naturally to l v and the l n v become finite dimensional kg modules called the lie powers of v in the decomposition problem the aim is to identify the isomorphism types of indecomposable kg modules with their multiplicities in unrefinable direct decompositions of the lie powers this paper is concerned with the case where g has prime order p and k has characteristic p as is well known there are p indecomposables denoted here by j 1 dots j p where j r has dimension r a theory is developed which provides information about the overall module structure of l v and gives a recursive method for finding the multiplicities of j 1 dots j p in the lie powers l n v for example the theory yields decompositions of l v as a direct sum of modules isomorphic either to j 1 or to an infinite sum of the form j r oplus j p 1 oplus j p 1 oplus ldots with r geq 2 closed formulae are obtained for the multiplicities of j 1 dots j p in l n j p and l n j p 1 for r p 1 the indecomposables which occur with non zero multiplicity in l n j r are identified for all sufficiently large n

time to depth conversion and velocity estimation by image wavefront propagation abstracta new strategy for time to depth conversion and interval velocity estimation is based entirely on image wavefront propagation without the need to follow individual image rays the procedure has three main features 1 it computes the velocity field and the traveltime directly allowing us to dispense with dynamic ray tracing 2 it requires only the knowledge of the image wavefront at the previous time step and 3 it inherently smooths the image wavefront inhibiting the formation of caustics as a consequence the method tends to be faster than the usual techniques and does not carry the constraints and limitations inherent to common ray tracing strategies synthetic tests using a gaussian velocity anomaly as well as the marmousi velocity model and two smoothed versions of it show the feasibility of the method a field data example demonstrates the use of different numerical procedures our results indicate that the present strategy can be used to construct reasonable depth velocity models t

minimax analysis of active learning this work establishes distribution free upper and lower bounds on the minimax label complexity of active learning with general hypothesis classes under various noise models the results reveal a number of surprising facts in particular under the noise model of tsybakov 2004 the minimax label complexity of active learning with a vc class is always asymptotically smaller than that of passive learning and is typically signi cantly smaller than the best previously published upper bounds in the active learning literature in high noise regimes it turns out that all active learning problems of a given vc dimension have roughly the same minimax label complexity which contrasts with well known results for bounded noise in low noise regimes we find that the label complexity is well characterized by a simple combinatorial complexity measure we call the star number interestingly we find that almost all of the complexity measures previously explored in the active learning literature have worst case values exactly equal to the star number we also propose new active learning strategies that nearly achieve these minimax label complexities

analysis and algorithms for a regularized cauchy problem arising from a non linear elliptic pde for seismic velocity estimation in the present work we derive and study a non linear elliptic pde coming from the problem of estimation of sound speed inside the earth the physical setting of the pde allows us to pose only a cauchy problem and hence is ill posed however we are still able to solve it numerically on a long enough time interval to be of practical use we used two approaches the first approach is a finite difference time marching numerical scheme inspired by the lax friedrichs method the key features of this scheme is the lax friedrichs averaging and the wide stencil in space the second approach is a spectral chebyshev method with truncated series we show that our schemes work because of i the special input corresponding to a positive finite seismic velocity ii special initial conditions corresponding to the image rays iii the fact that our finite difference scheme contains small error terms which damp the high harmonics truncation of the chebyshev series and iv the need to compute the solution only for a short interval of time we test our numerical schemes on a collection of analytic examples and demonstrate a dramatic improvement in accuracy in the estimation of the sound speed inside the earth in comparison with the conventional dix inversion our test on the marmousi example confirms the effectiveness of the proposed approach

supervised learning and co training co training under the conditional independence assumption is among the models which demonstrate how radically the need for labeled data can be reduced if a huge amount of unlabeled data is available in this paper we explore how much credit for this saving must be assigned solely to the extra assumptions underlying the co training model to this end we compute general almost tight upper and lower bounds on the sample size needed to achieve the success criterion of pac learning in the realizable case within the model of co training under the conditional independence assumption in a purely supervised setting the upper bounds lie significantly below the lower bounds for pac learning without co training thus co training saves labeled data even when not combined with unlabeled data on the other hand the saving is much less radical than the known savings in the semi supervised setting

learning with prior information a notion of learnability is introduced referred to as learnability with prior information w p i this notion is weaker than the standard notion of probably approximately correct pac learnability a property called dispersability is introduced and it is shown that dispersability plays a key role in the study of learnability w p i specifically dispersability of a function class is always a sufficient condition for the function class to be learnable moreover in the case of concept classes dispersability is also a necessary condition for learnability w p i thus in the case of learnability w p i the dispersability property plays a role similar to the finite metric entropy condition in the case of pac learnability with a fixed distribution next the notion of learnability w p i is extended to the distribution free d f situation and it is shown that a property called d f dispersability is always a sufficient condition for d f learnability w p i and is also a necessary condition for d f learnability in the case of concept classes the approach to learning introduced in the paper is believed to be significant in all problems where a nonlinear system has to be designed based on data this includes direct inverse control and system identification

optimality of svm novel proofs and tighter bounds abstract we provide a new proof that the expected error rate of consistent support vector machines matches the minimax rate up to a constant factor in its dependence on the sample size and margin the upper bound was originally established by 1 while the lower bound follows from an argument of 2 together with reasoning about the vc dimension of large margin classifiers our proof differs from the original in that many of our steps concern reasoning about the primal space while the original carried out these steps by reasoning about the dual space our approach provides a unified framework for analyzing both the homogeneous and non homogeneous cases with slightly better results for the former the fact that our analysis explicitly handles the non homogeneous case offers significant improvements in the bounds compared to the usual textbook approach of reducing to the homogeneous case we also extend our proof to provide a new upper bound on the error rate of transductive svm which yields an improved constant factor compared to inductive svm in addition to these bounds on the expected error rate we also provide a simple proof of a margin based pac style bound for support vector machines and an extension of the agnostic pac analysis that explicitly handles the non homogeneous case

the learnability of quantum states traditional quantum state tomography requires a number of measurements that grows exponentially with the number of qubits n but using ideas from computational learning theory we show that one can do exponentially better in a statistical setting in particular to predict the outcomes of most measurements drawn from an arbitrary probability distribution one needs only a number of sample measurements that grows linearly with n this theorem has the conceptual implication that quantum states despite being exponentially long vectors are nevertheless reasonable in a learning theory sense the theorem also has two applications to quantum computing first a new simulation of quantum one way communication protocols and second the use of trusted classical advice to verify untrusted quantum advice

velocity independent marchenko focusing in time and depth imaging domains for media with mild lateral heterogeneity abstractthe marchenko method retrieves green s functions between the acquisition surface and any arbitrary point in the medium the process generally involves solving an inversion starting with an initial focusing function e g a direct wave green s function from the desired subsurface position typically obtained using an approximate velocity model we have formulated the marchenko method in the time imaging domain in that domain we recognize that the traveltime of the direct wave green s function is related to the cheop s traveltime pyramid commonly used in time domain processing which in turn can be readily obtained from the local slopes of the common midpoint gathers this observation allows us to substitute the velocity model based initial focusing operator with that from a data driven slope estimation process moreover we found that working in the time imaging domain allows for the specification of the desired subsurface position in terms of vertical time which is connected to the cartesian depth position via the time to depth conversion our results suggest that the prior velocity model is only required when specifying the position in depth but this requirement can be circumvented by making use of the time imaging domain within its usual assumptions e g mild lateral heterogeneity provided that those assumptions are satisfied the estimated green s functions from the proposed method have comparable quality to those obtained with the knowledge of a prior velocity model

bounds on the number of examples needed for learning functions we prove general lower bounds on the number of examples needed for learning function classes within different natural learning models which are related to pac learning and coincide with the pac learning model of valiant in the case of 0 1 valued functions the lower bounds are obtained by showing that all nontrivial function classes contain a hard binary valued subproblem although at first glance it seems to be likely that real valued function classes are much harder to learn than their hardest binary valued subproblem we show that these general lower bounds cannot be improved by more than a logarithmic factor this is done by discussing some natural function classes like nondecreasing functions or piecewise smooth functions the function classes that were discussed in m j kearns and r e schapire proc 31st annual symposium on the foundations of computer science ieee computer society press los alamitos ca 1990 pp 382 392 full version j comput system sci 48 1994 pp 464 497 d kimber and p m long proc 5th annual workshop on computational learning theory acm new york 1992 pp 153 160 with certain restrictions concerning their slope

pac learning of concept classes through the boundaries of their items abstract we present a new perspective for investigating the probably approximate correct pac learnability of classes of concepts we focus on special sets of points for characterizing the concepts within their class this gives rise to a general notion of boundary of a concept which holds even in discrete spaces and to a special probability measuring technique this technique is applied i to narrow the gap between the minimum and maximum sample sizes necessary to learn even under a more stringent learnability definition and ii to get self explanatory indices of the complexity of the learning task these indices can be roughly estimated during the learning process and appear very useful in the treatment of nonsymbolic procedures e g in the context of neutral networks

bandwidth choice for nonparametric classification it is shown that for kernel based classification with univariate distributions and two populations optimal bandwidth choice has a dichotomous character if the two densities cross at just one point where their curvatures have the same signs then minimum bayes risk is achieved using bandwidths which are an order of magnitude larger than those which minimize pointwise estimation error on the other hand if the curvature signs are different or if there are multiple crossing points then bandwidths of conventional size are generally appropriate the range of different modes of behavior is narrower in multivariate settings there the optimal size of bandwidth is generally the same as that which is appropriate for pointwise density estimation these properties motivate empirical rules for bandwidth choice

on the complexity of learning from drifting distributions abstract we consider two models of on line learning of binary valued functions from drifting distributions due to bartlett we show that if each example is drawn from a joint distribution which changes in total variation distance by at mosto e3 d log 1 e between trials then an algorithm can achieve a probability of a mistake at mosteworse than the best function in a class of vc dimensiond we prove a corresponding necessary condition ofo e3 d finally in the case that a fixed function is to be learned from noise free examples we show that if the distributions on the domain generating the examples change by at mosto e2 d log 1 e then any consistent algorithm learns to within accuracye

learning by canonical smooth estimation ii learning and choice of model complexity this paper analyzes the properties of a procedure for learning from examples this canonical learner is based on a canonical error estimator developed in part i in learning problems one can observe data that consists of labeled sample points and the goal is to find a model or hypothesis from a set of candidates that will accurately predict the labels of new sample points the expected mismatch between a hypothesis prediction and the actual label of a new sample point is called the hypothesis generalization error we compare the canonical learner with the traditional technique of finding hypotheses that minimize the relative frequency based empirical error estimate it is shown that for a broad class of learning problems the set of cases for which such empirical error minimization works is a proper subset of the cases for which the canonical learner works we derive bounds to show that the number of samples required by these two methods is comparable we also address the issue of how to determine the appropriate complexity for the class of candidate hypotheses

a general lower bound on the number of examples needed for learning abstract we prove a lower bound of 1 ln 1 vcdim c on the number of random examples required for distribution free learning of a concept class c where vcdim c is the vapnik chervonenkis dimension and and are the accuracy and confidence parameters this improves the previous best lower bound of 1 ln 1 vcdim c and comes close to the known general upper bound of o 1 ln 1 vcdim c ln 1 for consistent algorithms we show that for many interesting concept classes including k cnf and k dnf our bound is actually tight to within a constant factor

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

sample efficient strategies for learning in the presence of noise in this paper we prove various results about pac learning in the presence of malicious noise our main interest is the sample size behavior of learning algorithms we prove the first nontrivial sample complexity lower bound in this model by showing that order of e 2 d up to logarithmic factors examples are necessary for pac learning any target class of 0 1 valued functions of vc dimension d where e is the desired accuracy and e e 1 e the malicious noise rate it is well known that any nontrivial target class cannot be pac learned with accuracy e and malicious noise rate e e 1 e this irrespective to sample complexity we also show that this result cannot be significantly improved in general by presenting efficient learning algorithms for the class of all subsets of d elements and the class of unions of at most d intervals on the real line this is especialy interesting as we can also show that the popular minimum disagreement strategy needs samples of size d e 2 hence is not optimal with respect to sample size we then discuss the use of randomized hypotheses for these the bound e 1 e on the noise rate is no longer true and is replaced by 2e 1 2e in fact we present a generic algorithm using randomized hypotheses that can tolerate noise rates slightly larger than e 1 e while using samples of size d e as in the noise free case again one observes a quadratic powerlaw in this case d e 2 2e 1 2e e as goes to zero we show upper and lower bounds of this order

improved lower bounds for learning from noisy examples this paper presents a general information theoretic approach for obtaining lower bounds on the number of examples required for probably approximately correct pac learning in the presence of noise this approach deals directly with the fundamental information quantities avoiding a bayesian analysis the technique is applied to several different models illustrating its generality and power the resulting bounds add logarithmic factors to or improve the constants in previously known lower bounds

time to depth conversion and seismic velocity estimation using time migration velocity the objective was to build an efficient algorithm 1 to estimate seismic velocity from time migration velocity and 2 to convert time migrated images to depth we established theoretical relations between the time migration velocity and seismic velocity in two and three dimensions using paraxial ray tracing theory the relation in two dimensions implies that the conventional dix velocity is the ratio of the interval seismic velocity and the geometric spreading of image rays we formulated an inverse problem of finding seismic velocity from the dix velocity and developed a numerical procedure for solving it the procedure consists of two steps 1 computation of the geometric spreading of image rays and the true seismic velocity in time domain coordinates from the dix velocity 2 conversion of the true seismic velocity from the time domain to the depth domain and computation of the transition matrices from time domain coordinates todepth for step 1 we derived a partial differential equation pde i

hinging hyperplanes for regression classification and function approximation a hinge function y h x consists of two hyperplanes continuously joined together at a hinge in regression prediction classification pattern recognition and noiseless function approximation use of sums of hinge functions gives a powerful and efficient alternative to neural networks with computation times several orders of magnitude less than is obtained by fitting neural networks with a comparable number of parameters a simple and effective method for finding good hinges is presented

the optimal sample complexity of pac learning this work establishes a new upper bound on the number of samples sufficient for pac learning in the realizable case the bound matches known lower bounds up to numerical constant factors this solves a long standing open problem on the sample complexity of pac learning the technique and analysis build on a recent breakthrough by hans simon

the one inclusion graph algorithm is near optimal for the prediction model of learning haussler littlestone and warmuth 1994 described a general purpose algorithm for learning according to the prediction model and proved an upper bound on the probability that their algorithm makes a mistake in terms of the number of examples seen and the vapnik chervonenkis vc dimension of the concept class being learned we show that their bound is within a factor of 1 o 1 of the best possible such bound for any algorithm

velocity continuation and the anatomy of residual prestack time migration velocity continuation is an imaginary continuous process of seismic image transformation in the postmigration domain it generalizes the concepts of residual and cascaded migrations understanding the laws of velocity continuation is crucially important for a successful application of time migration velocity analysis these laws predict the changes in the geometry and intensity of reflection events on migrated images with the change of the migration velocity in this paper i derive kinematic and dynamic laws for the case of prestack residual migration from simple geometric principles the main theoretical result is a decomposition of prestack velocity continuation into three different components corresponding to residual normal moveout residual dip moveout and residual zero offset migration i analyze the contribution and properties of each of the three components separately this theory forms the basis for constructing efficient finite difference and spectral algorithms for time migration velocity ana

neural nets with superlinear vc dimension it has been known for quite a while that the vapnik chervonenkis dimension vc dimension of a feedforward neural net with linear threshold gates is at most o w log w where w is the total number of weights in the neural net we show in this paper that this bound is in fact asymptotically optimal more precisely we exhibit for any depth d 3 a large class of feedforward neural nets of depth d with w weights that have vc dimension w log w this lower bound holds even if the inputs are restricted to boolean values the proof of this result relies on a new method that allows us to encode more program bits in the weights of a neural net than previously thought possible

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

learning by distances a model of learning by distances is presented in this model a concept is a point in a metric space at each step of the learning process the student guesses a hypothesis and receives from the teacher an approximation of its distance to the target a notion of a distance measuring the proximity of a hypothesis to the correct answer is common to many models of learnability by focusing on this fundamental aspect we discover some general and simple tools for the analysis of learnability tasks as a corollary we present new learning algorithms for valiant s pac scenario with any given distribution these algorithms can learn any pac learnable class and in some cases settle for significantly less information than the usual labeled examples insight gained by the new model is applied to show that every class of subsets c that has a finite vc dimension is pac learnable with respect to any fixed distribution previously known results of this nature were subject to complicated measurability constraints

lower bounds in pattern recognition and learning lower bounds are derived for the performance of any pattern recognition algorithm which using training data selects a discrimination rule from a certain class of rules the bounds involve the vapnik chervonenkis dimension of the class and l the minimal error probability within the class we provide lower bounds when l 0 the usual assumption in valiant s theory of learning and l 0

time migration velocity analysis by velocity continuation time migration velocity analysis can be performed by velocity continuation an incremental process that transforms migrated seismic sections according to changes in the migration velocity velocity continuation enhances residual normal moveout correction by properly taking into account both vertical and lateral movements of events on seismic images finite difference and spectral algorithms provide efficient practical implementations for velocity continuation synthetic and field data examples demonstrate the performance of the method and confirm theoretical expectations

on the necessity of irrelevant variables this work explores the effects of relevant and irrelevant boolean variables on the accuracy of classifiers the analysis uses the assumption that the variables are conditionally independent given the class and focuses on a natural family of learning algorithms for such sources when the relevant variables have a small advantage over random guessing the main result is that algorithms relying predominately on irrelevant variables have error probabilities that quickly go to 0 in situations where algorithms that limit the use of irrelevant variables have errors bounded below by a positive constant we also show that accurate learning is possible even when there are so few examples that one cannot determine with high confidence whether or not any individual variable is relevant

quantifying inductive bias ai learning algorithms and valiant s learning framework abstract we show that the notion of inductive bias in concept learning can be quantified in a way that directly relates to learning performance in the framework recently introduced by valiant our measure of bias is based on the growth function introduced by vapnik and chervonenkis and on the vapnik chervonenkis dimension we measure some common language biases including restriction to conjunctive concepts conjunctive concepts with internal disjunction k dnf and k cnf concepts we also measure certain types of bias that result from a preference for simpler hypotheses using these bias measurements we analyze the performance of the classical learning algorithm for conjunctive concepts from the perspective of valiant s learning framework we then augment this algorithm with a hypothesis simplification routine that uses a greedy heuristic and show how this improves learning performance on simpler target concepts improved learning algorithms are also developed for conjunctive concepts with internal disjunction k dnf and k cnf concepts we show that all our algorithms are within a logarithmic factor of optimal in terms of the number of examples they require to achieve a given level of learning performance in the valiant framework our results hold for arbitrary attribute based instance spaces defined by either tree structured or linear attributes

computational limitations on learning from examples the computational complexity of learning boolean concepts from examples is investigated it is shown for various classes of concept representations that these cannot be learned feasibly in a distribution free sense unless r np these classes include a disjunctions of two monomials b boolean threshold functions and c boolean formulas in which each variable occurs at most once relationships between learning of heuristics and finding approximate solutions to np hard optimization problems are given

fast time to depth conversion and interval velocity estimation in the case of weak lateral variations time domain processing has a long history in seismic imaging and has always been a powerful workhorse that is routinely used it generally leads to an expeditious construction of the subsurface velocity model in time which can later be expressed in the cartesian depth coordinates via a subsequent time to depth conversion the conventional practice of such a conversion is done using dix inversion which is exact in the case of laterally homogeneous media for other media with lateral heterogeneity the time to depth conversion involves solving a more complex system of partial differential equations pdes we have developed an efficient alternative for time to depth conversion and interval velocity estimation based on the assumption of weak lateral velocity variations by considering only first order perturbative effects from lateral variations the exact system of pdes required to accomplish the exact conversion reduces to a simpler system that can be solved efficiently in a layer stripping downward stepping fashion numerical synthetic and field data examples show that our method can achieve reasonable accuracy and is significantly more efficient than previously proposed methods with a speedup by an order of magnitude