p sufficient statistics for pac learning k term dnf formulas through enumeration working in the framework of pac learning theory we present special statistics for accomplishing in polynomial time proper learning of dnf boolean formulas having a fixed number of monomials our statistics turn out to be near sufficient for a large family of distribution laws that we call butterfly distributions we develop a theory of most powerful learning for analyzing the performance of learning algorithms with particular reference to trade offs between power and computational costs focusing attention on sample and time complexity we prove that our algorithm works as efficiently as the best algorithms existing in the literature while the latter only take care of subclasses of our family of distributions

learnability with respect to fixed distributions consider a first order typed language with semantics s for expressions and types adding subtyping means that a partial order on types is defined and that the typing rules are extended to the effect that expression e has type t whenever e has type s and s t we show how to adapt the semantics s in a simple set theoretic way obtaining a semantics s that satisfies in addition to some obvious requirements also the property that s s is included in s t whenever s t

learning by canonical smooth estimation i simultaneous estimation this paper examines the problem of learning from examples in a framework that is based on but more general than valiant s probably approximately correct pac model for learning in our framework the learner observes examples that consist of sample points drawn and labeled according to a fixed unknown probability distribution based on this empirical data the learner must select from a set of candidate functions a particular function or hypothesis that will accurately predict the labels of future sample points the expected mismatch between a hypothesis prediction and the label of a new sample point is called the hypothesis generalization error following the pioneering work of vapnik and chervonenkis others have attacked this sort of learning problem by finding hypotheses that minimize the relative frequency based empirical error estimate we generalize this approach by examining the simultaneous estimation problem we demonstrate how one can learn from such a simultaneous error estimate and propose a new class of estimators called smooth estimators we characterize the class of simultaneous estimation problems solvable by a smooth estimator

effects of domain characteristics on instance based learning algorithms this paper presents average case analyses of instance based learning algorithms the algorithms analyzed employ a variant of k nearest neighbor classifier k nn our analysis deals with a monotone m of n target concept with irrelevant attributes and handles three types of noise relevant attribute noise irrelevant attribute noise and class noise we formally represent the expected classification accuracy of k nn as a function of domain characteristics including the number of training instances the number of relevant and irrelevant attributes the threshold number in the target concept the probability of each attribute the noise rate for each type of noise and k we also explore the behavioral implications of the analyses by presenting the effects of domain characteristics on the expected accuracy of k nn and on the optimal value of k for artificial domains

characterizations of monotonic and dual monotonic language learning the present paper deals with monotonic and dual monotonic language learning from positive as well as from positive and negative examples the three notions of monotonicity reflect different formalizations of the requirement that the learner has to produce better and better generalizations when fed more and more data on the concept to be learned the three versions of dual monotonicity describe the concept that the inference device has to produce specializations that fit better and better to the target language we characterize strong monotonic monotonic weak monotonic dual strong monotonic dual monotonic and monotonic dual monotonic as well as finite language learning from positive data in terms of recursively generable finite sets these characterizations provide a unifying framework for learning from positive data under the various monotonicity constraints moreover they yield additional insight into the problem of what a natural learning algorithm should look like

on learning read k satisfy j dnf we study the learnability of read k satisfy j rksj dnf formulas these are boolean formulas in disjunctive normal form dnf in which the maximum number of occurrences of a variable is bounded by k and the number of terms satisfied by any assignment is at most j after motivating the investigation of this class of dnf formulas we present an algorithm that for any unknown rksj dnf formula to be learned with high probability finds a logically equivalent dnf formula using the well studied protocol of equivalence and membership queries the algorithm runs in polynomial time for k cdot j o log n over log log n where n is the number of input variables

property testing and its connection to learning and approximation in this paper we consider the question of determining whether a function f has property p or is e far from any function with property p a property testing algorithm is given a sample of the value of f on instances drawn according to some distribution in some cases it is also allowed to query f on instances of its choice we study this question for different properties and establish some connections to problems in learning theory and approximation in particular we focus our attention on testing graph properties given access to a graph g in the form of being able to query whether an edge exists or not between a pair of vertices we devise algorithms to test whether the underlying graph has properties such as being bipartite k colorable or having a p clique clique of density p with respect to the vertex set our graph property testing algorithms are probabilistic and make assertions that are correct with high probability while making a number of queries that is independent of the size of the graph moreover the property testing algorithms can be used to efficiently i e in time linear in the number of vertices construct partitions of the graph that correspond to the property being tested if it holds for the input graph

logical analysis of binary data with missing bits we model a given pair of sets of positive and negative examples each of which may contain missing components as a partially defined boolean function with missing bits pbmb t f where email protected 0 1 n and email protected 0 1 n and stands for a missing bit then we consider the problem of establishing a boolean function an extension f 0 1 n 0 1 belonging to a given function class c such that f is true respectively false for every vector in t respectively in f this is a fundamental problem encountered in many areas such as learning theory pattern recognition example based knowledge bases logical analysis of data knowledge discovery and data mining in this paper depending upon how to deal with missing bits we formulate three types of extensions called robust consistent and most robust extensions for various classes of boolean functions such as general positive horn threshold decomposable and k dnf the complexity of the associated problems are then clarified some of them are solvable in polynomial time while the others are np hard

scenario reduction for stochastic programs with conditional value at risk in this paper we discuss scenario reduction methods for risk averse stochastic optimization problems scenario reduction techniques have received some attention in the literature and are used by practitioners as such methods allow for an approximation of the random variables in the problem with a moderate number of scenarios which in turn make the optimization problem easier to solve the majority of works for scenario reduction are designed for classical risk neutral stochastic optimization problems however it is intuitive that in the risk averse case one is more concerned with scenarios that correspond to high cost by building upon the notion of effective scenarios recently introduced in the literature we formalize that intuitive idea and propose a scenario reduction technique for stochastic optimization problems where the objective function is a conditional value at risk numerical results presented with problems from the literature illustrate the performance of the method and indicate the cases where we expect it to perform well

identification criteria and lower bounds for perceptron like learning rules perceptron like learning rules are known to require exponentially many correction steps in order to identify boolean threshold functions exactly we introduce criteria that are weaker than exact identification and investigate whether learning becomes significantly faster if exact identification is replaced by one of these criteria probably approximately correct pac identification order identification and sign identification pac identification is based on the learning paradigm introduced by valiant 1984 and is known to be easier than exact identification order identification uses the fact that each threshold function induces an ordering relation on the input variables that can be represented by weights of linear size sign identification is based on a property of threshold functions known as unateness and requires only weights of constant size we show that perceptron like learning rules cannot satisfy these criteria when the number of correction steps is to be bounded by a polynomial we also pre

how to construct random functions a constructive theory of randomness for functions based on computational complexity is developed and a pseudorandom function generator is presented this generator is a deterministic polynomial time algorithm that transforms pairs g r where g is any one way function and r is a random k bit string to polynomial time computable functions r 1 2 k 1 2 k these r s cannot be distinguished from random functions by any probabilistic polynomial time algorithm that asks and receives the value of a function at arguments of its choice the result has applications in cryptography random constructions and complexity theory

convexity and decomposition of mean risk stochastic programs traditional stochastic programming is risk neutral in the sense that it is concerned with the optimization of an expectation criterion a common approach to addressing risk in decision making problems is to consider a weighted mean risk objective where some dispersion statistic is used as a measure of risk we investigate the computational suitability of various mean risk objective functions in addressing risk in stochastic programming models we prove that the classical mean variance criterion leads to computational intractability even in the simplest stochastic programs on the other hand a number of alternative mean risk functions are shown to be computationally tractable using slight variants of existing stochastic programming decomposition algorithms we propose decomposition based parametric cutting plane algorithms to generate mean risk efficient frontiers for two particular classes of mean risk objectives

parameterized learnability of juntas we study the parameterized complexity of learning k juntas and some variations of juntas we show the hardness of learning k juntas and subclasses of k juntas in the pac model by reductions from a w 2 complete problem on the other hand as a consequence of a more general result we show that k juntas are exactly learnable with improper equivalence queries and access to a w p oracle

on the nonlearnability of a single spiking neuron we study the computational complexity of training a single spiking neuron n with binary coded inputs and output that in addition to adaptive weights and a threshold has adjustable synaptic delays a synchronization technique is introduced so that the results concerning the nonlearnability of spiking neurons with binary delays are generalized to arbitrary real valued delays in particular the consistency problem for n with programmable weights a threshold and delays and its approximation version are proven to be np complete it follows that the spiking neurons with arbitrary synaptic delays are not properly pac learnable and do not allow robust learning unless rp np in addition the representation problem for n a question whether an n variable boolean function given in dnf or as a disjunction of o n threshold gates can be computed by a spiking neuron is shown to be conp hard

interpolation and approximation of sparse multivariate polynomials over gf 2 a function f 0 1 n to 0 1 is called t sparse if the n variable polynomial representation of f over gf 2 contains at most t monomials such functions are uniquely determined by their values at the so called critical set of all binary n tuples of hamming weight geqq n lfloor log 2 t rfloor 1 an algorithm is presented for interpolating any t sparse function f given the values of f at the critical set the time complexity of the proposed algorithm is proportional to n t and the size of the critical set then the more general problem of approximating 1 sparse functions is considered in which case the approximating function may differ from f at a fraction varepsilon of the space 0 1 n it is shown that o t varepsilon cdot n evaluation points are sufficient for the deterministic varepsilon approximation of any t sparse function and that an order t varepsilon alpha t varepsilon cdot log n points are necessary for this purpose wh

scenario tree generation for multiperiod financial optimization by optimal discretization multiperiod financial optimization is usually based on a stochastic model for the possible market situations there is a rich literature about modeling and estimation of continuous state financial processes but little attention has been paid how to approximate such a process by a discrete state scenario process and how to measure the pertaining approximation error in this paper we show how a scenario tree may be constructed in an optimal manner on the basis of a simulation model of the underlying financial process by using a stochastic approximation technique consistency relations for the tree may also be taken into account

inductive inference of approximations in this paper we investigate inductive inference identification criteria which permit infinitely many errors in explanations but which require that the density of these errors be no more than a certain prespectified amount we introduce three hierarchies of such criteria each of which has the same order type as the real unit interval these three hierarchies are progressively more strict in the way they measure density of errors of explanations the strictest of the three turns out to have all of its members save one incomparable to the identification criterion which permits finitely many errors in explanations

probably approximately correct learning in fuzzy classification systems an efficient method for learning trapezoidal membership functions for fuzzy predicates is presented positive and negative examples of one class are given together with a system of classification rules the learned membership functions can be used for the fuzzy predicates occurring in the given rules to classify further examples we show that the obtained classification is approximately correct with high probability this justifies the obtained fuzzy sets within one particular classification problem instead of relying on a subjective meaning of fuzzy predicates as normally done by a domain expert

the minimum consistent dfa problem cannot be approximated within any polynomial the minimum consistent dfa problem is that of finding a dfa with as few states as possible that is consistent with a given sample a finite collection of words each labeled as to whether the dfa found should accept or reject assuming that p np it is shown that for any constant k no polynomial time algorithm can be guaranteed to find a consistent dfa with fewer than opt k states where opt is the number of states in the minimum state dfa consistent with the sample this result holds even if the alphabet is of constant size two and if the algorithm is allowed to produce an nfa a regular expression or a regular grammar that is consistent with the sample a similar nonapproximability result is presented for the problem of finding small consistent linear grammars for the case of finding minimum consistent dfas when the alphabet is not of constant size but instead is allowed to vary with the problem specification the slightly stronger lower bound on approximability of opt 1 e log log opt is shown for any e 0

error free and best fit extensions of partially defined boolean functions abstract in this paper we address a fundamental problem related to the induction of boolean logic given a set of data represented as a set of binary truen vectors or positive examples and a set of falsen vectors or negative examples we establish a boolean function or an extension f so thatfis true resp false in every given true resp false vector we shall further require that such an extension belongs to a certain specified class of functions e g class of positive functions class of horn functions and so on the class of functions represents our a priori knowledge or hypothesis about the extensionf which may be obtained from experience or from the analysis of mechanisms that may or may not cause the phenomena under consideration the real world data may contain errors e g measurement and classification errors might come in when obtaining data or there may be some other influential factors not represented as variables in the vectors in such situations we have to give up the goal of establishing an extension that is perfectly consistent with the given data and we are satisfied with an extensionfhaving the minimum number of misclassifications both problems i e the problem of finding an extension within a specified class of boolean functions and the problem of finding a minimum error extension in that class will be extensively studied in this paper for certain classes we shall provide polynomial algorithms and for other cases we prove their np hardness

computation of invariant tori by the method of characteristics in this paper we present a technique for the numerical approximation of a branch of invariant tori of finite dimensional ordinary differential equations systems our approach is a discrete version of the graph transform technique used in analytical work by fenichel indiana univ math j 21 1971 pp 193 226 in contrast to our previous work l dieci j lorenz and r d russell siam j sci statist comput 12 1991 pp 607 647 the method presented here does not require a priori knowledge of a suitable coordinate system for the branch of invariant tori but determines and updates such a coordinate system during a continuation process we give general convergence results for the method and present its algorithmic description we also show how the method performs on two physically important nonlinear problems a system of two coupled oscillators and the forced van der pol oscillator in the latter case we discuss some modifications needed to approximate an invariant curve for the poincare map

scenario reduction in stochastic programming an approach using probability metrics given a convex stochastic programming problem with a discrete initial probability distribution the problem of optimal scenario reduction is stated as follows determine a scenario subset of prescribed cardinality and a probability measure based on this set that is the closest to the initial distribution in terms of a natural or canonical probability metric arguments from stability analysis indicate that fortet mourier type probability metrics may serve as such canonical metrics efficient algorithms are developed that determine optimal reduced measures approximately numerical experience is reported for reductions of electrical load scenario trees for power management under uncertainty for instance it turns out that after 50 reduction of the scenario tree the optimal reduced tree still has about 90 relative accuracy

on the power of probabilistic strategies in inductive inference abstract inductive inference of programs of recursive functions from input output examples by probabilistic strategies with an a priori bound n n of changes of hypotheses is investigated advantages of probabilistic strategies over deterministic ones are shown concerning their power in principle for every n 2 with probability arbitrarily close to 1 inference of function classes which cannot be inferred by any deterministic strategy with n changes of hypotheses as well as their computational complexity linear speed up of the number of changes of hypotheses necessary for inference by deterministic strategies

learning curves of the clipped hebb rule for networks with binary weights networks with binary weights are very important from both the theoretical and practical points of view in this paper we investigate the clipped hebb rule for learning different networks of non overlapping binary perceptrons under uniform distribution we calculate exactly its learning curves in the limit of a large number of inputs where the average behaviour becomes the typical behaviour this calculation is performed using only very simple counting arguments and the central limit theorem the results indicate that the clipped hebb rule does indeed learn this class of networks in particular the generalization rates converge extremely rapidly often exponentially to perfect generalization these results are very encouraging given the simplicity of the learning rule the analytic expression of the learning curves are in excellent agreement with the numerical simulations

a comparison of three methods for selecting values of input variables in the analysis of output from a computer code two types of sampling plans are examined as alternatives to simple random sampling in monte carlo studies these plans are shown to be improvements over simple random sampling with respect to variance for a class of estimators which includes the sample mean and the empirical distribution function

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

identifying effective scenarios in distributionally robust stochastic programs with total variation distance traditional stochastic programs assume that the probability distribution of uncertainty is known however in practice the probability distribution oftentimes is not known or cannot be accurately approximated one way to address such distributional ambiguity is to work with distributionally robust convex stochastic programs drsps which minimize the worst case expected cost with respect to a set of probability distributions in this paper we analyze the case where there is a finite number of possible scenarios and study the question of how to identify the critical scenarios resulting from solving a drsp we illustrate that not all but only some scenarios might have effect on the optimal value and we formally define this notion for our general class of problems in particular we examine problems where the distributional ambiguity is modeled by the so called total variation distance we propose easy to check conditions to identify effective and ineffective scenarios for that class of problems computational results show that identifying effective scenarios provides useful insight on the underlying uncertainties of the problem

stability of varepsilon approximate solutions to convex stochastic programs an analysis of convex stochastic programs is provided when the underlying probability distribution is subjected to small perturbations it is shown in particular that varepsilon approximate solution sets of convex stochastic programs behave lipschitz continuously with respect to certain distances of probability distributions that are generated by the relevant integrands it is shown that these results apply to linear two stage stochastic programs with random recourse we discuss the consequences on associating fortet mourier metrics to two stage models and on the asymptotic behavior of empirical estimates of such models respectively

scenario tree modeling for multistage stochastic programs an important issue for solving multistage stochastic programs consists in the approximate representation of the multivariate stochastic input process in the form of a scenario tree in this paper we develop stability theory based heuristics for generating scenario trees out of an initial set of scenarios they are based on forward or backward algorithms for tree generation consisting of recursive scenario reduction and bundling steps conditions are established implying closeness of optimal values of the original process and its tree approximation respectively by relying on a recent stability result in heitsch romisch and strugarek siam j optim 17 511 525 2006 for multistage stochastic programs numerical experience is reported for constructing multivariate scenario trees in electricity portfolio management

comparison of identification criteria for machine inductive inference abstract a natural pl 1 hierarchy of successively more general criteria of success for inductive inference machines is described based on the size of sets of anomalies in programs synthesized by such machines these criteria are compared to others in the literature some of our results are interpreted as tradeoff results or as showing the inherent relative computational complexity of certain processes and others are interpreted from a positivistic mechanistic philosophical stance as theorems in philosophy of science the techniques of recursive function theory are employed including ordinary and infinitary recursion theorems

on learning perceptrons with binary weights we present an algorithm that pac learns any perceptron with binary weights and arbitrary threshold under the family of product distributions the sample complexity of this algorithm is of o n e 4 ln n and its running time increases only linearly with the number of training examples the algorithm does not try to find an hypothesis that agrees with all of the training examples rather it constructs a binary perceptron based on various probabilistic estimates obtained from the training examples we show that under the restricted case of the uniform distribution and zero threshold the algorithm reduces to the well known clipped hebb rule we calculate exactly the average generalization rate i e the learning curve of the algorithm under the uniform distribution in the limit of an infinite number of dimensions we find that the error rate decreases exponentially as a function of the number of training examples hence the average case analysis gives a sample complexity of o n ln 1 e a large improvement over the pac learning analysis the analytical expression of the learning curve is in excellent agreement with the extensive numerical simulations in addition the algorithm is very robust with respect to classification noise

numerical approximation of rough invariant curves of planar maps in this paper we describe a simple algorithm for the approximation of an invariant curve gamma of a planar map f we are particularly interested in the case where gamma is attracting but not smooth results for the delayed logistic map illustrate the performance of the algorithm

fast identification of geometric objects with membership queries we investigate the number of membership queries that are needed to identify polygons i e intersections of halfplanes over a two dimensional grid 0 n 1 2 we exhibit a learning algorithm that learns 100 correctly while requiring no random examples and not more membership queries than previous algorithms needed in addition to their random examples for probably almost correctly learning even for moderate values of furthermore the learning algorithm in this paper uses only grid points for its membership queries this appears to be appropriate in situations where the probing device has only a limited resolution and for applications to the set of pixels on a two dimensional screen the learning algorithm that is described in this paper overcomes a fundamental obstacle related to the zigzag borderline that is generated by a halfplane over a discrete grid this obstacle had thwarted previous attempts to show that in angluin s model for on line learning with equivalence and membership queries one can learn in an efficient manner even those classes of geometric objects which cannot be learnt fast by equivalence queries alone such as rectangles in general position

risk averse two stage stochastic linear programming modeling and decomposition we formulate a risk averse two stage stochastic linear programming problem in which unresolved uncertainty remains after the second stage the objective function is formulated as a composition of conditional risk measures we analyze properties of the problem and derive necessary and sufficient optimality conditions next we construct a new decomposition method for solving the problem that exploits the composite structure of the objective function we illustrate its performance on a portfolio optimization problem

computation of lyapunov type numbers for invariant curves of planar maps invariant manifolds play an important role in the study of dynamical systems and it is often crucial to know when they persist under small perturbations of the system sufficient conditions for persistence have been formulated in the literature in terms of so called lyapunov type numbers which measure and compare attractivity rates the numerical evaluations of these numbers in a simple setting is the subject of this paper we consider planar diffeomorphisms depending on a parameter the diffeomorphisms have invariant curves which deform as the parameter changes the lyapunov type numbers are then monitored as functions of parameter and are related to the dynamics on the invariant curves as an example we consider the delayed logistic map we also describe how the invariant curves have been computed

scenario reduction in stochastic programming given a convex stochastic programming problem with a discrete initial probability distribution the problem of optimal scenario reduction is stated as follows determine a scenario subset of prescribed cardinality and a probability measure based on this set that is the closest to the initial distribution in terms of a natural or canonical probability metric arguments from stability analysis indicate that fortet mourier type probability metrics may serve as such canonical metrics efficient algorithms are developed that determine optimal reduced measures approximately numerical experience is reported for reductions of electrical load scenario trees for power management under uncertainty for instance it turns out that after 50 reduction of the scenario tree the optimal reduced tree still has about 90 relative accuracy

equivalence of models for polynomial learnability abstract in this paper we consider several variants of valiant s learnability model that have appeared in the literature we give conditions under which these models are equivalent in terms of the polynomially learnable concept classes they define these equivalences allow comparisons of most of the existing theorems in valiant style learnability and show that several simplifying assumptions on polynomial learning algorithms can be made without loss of generality we also give a useful reduction of learning problems to the problem of finding consistent hypotheses and give comparisons and equivalences between valiant s model and the prediction learning models of haussler littlestone and warmuth in 29th annual ieee symposium on foundations of computer science 1988

measuring teachability using variants of the teaching dimension in a typical algorithmic learning model a learner has to identify a target object from partial information conversely in a teaching model a teacher has to give information that allows the learners to identify a target object we devise two variants of the classical teaching model for boolean concept classes based on the teaching dimension and describe them by teaching dimension like combinatorial parameters in the first model the learners choose consistent hypotheses with least complexity we show that 1 decision lists are the harder to teach the longer they are and that 2 term dnfs are the harder to teach the more terms they have this contrasts with the teachability results for these classes in the teaching dimension model in our second model the learners choose consistent hypotheses based on the assumption that the teacher is optimal we show that monomials can be taught with a linear number of examples whereas some 1 decision lists need exponentially many

bifurcations from an invariant circle for two parameter families of maps of the plane a computer assisted study we consider a two parameter family of maps of the plane to itself each map has a fixed point in the first quadrant and is a diffeomorphism in a neighborhood of this point for certain parameter values there is a hopf bifurcation to an invariant circle which is smooth for parameter values in a neighborhood of the bifurcation point however computer simulations show that the corresponding invariant set fails to be even topologically a circle for parameter values far from the bifurcation point this paper is an attempt to elucidate some of the mechanisms involved in this loss of smoothness and alteration of topological type

loading deep networks is hard the loading problem formulated by j s judd seems to be a relevant model for supervised connectionist learning of the feedforward networks from the complexity point of view it is known that loading general network architectures is np complete intractable when the training tasks are also general many strong restrictions on architectural design and or on the tasks do not help to avoid the intractability of loading judd concentrated on the width expanding architectures with constant depth and found a polynomial time algorithm for loading restricted shallow architectures he suppressed the effect of depth on loading complexity and left as an open prototypical computational problem the loading of easy regular triangular architectures that might capture the crux of depth difficulties we have proven this problem to be np complete this result does not give much hope for the existence of an efficient algorithm for loading deep networks

numerical computation of invariant circles of maps abstract a method for the numerical computation of invariant circles of maps is presented along with appropriate techniques for its implementation the method involves solution of a functional equation by discretization and newton iteration the resulting algorithm is applied to a map of the cylinder and some examples of bifurcations of invariant circles are illustrated generalization of the method to the computation of invariant circles in more than two dimensions as well as sections of invariant tori is discussed

some dichotomy theorems for neural learning problems the computational complexity of learning from binary examples is investigated for linear threshold neurons we introduce combinatorial measures that create classes of infinitely many learning problems with sample restrictions we analyze how the complexity of these problems depends on the values for the measures the results are established as dichotomy theorems showing that each problem is either np complete or solvable in polynomial time in particular we consider consistency and maximum consistency problems for neurons with binary weights and maximum consistency problems for neurons with arbitrary weights we determine for each problem class the dividing line between the np complete and polynomial time solvable problems moreover all efficiently solvable problems are shown to have constructive algorithms that require no more than linear time on a random access machine model similar dichotomies are exhibited for neurons with bounded threshold the results demonstrate on the one hand that the consideration of sample constraints can lead to the discovery of new efficient algorithms for non trivial learning problems on the other hand hard learning problems may remain intractable even for severely restricted samples

training a single sigmoidal neuron is hard we first present a brief survey of hardness results for training feedforward neural networks these results are then completed by the proof that the simplest architecture containing only a single neuron that applies a sigmoidal activation function r satisfying certain natural axioms e g the standard logistic sigmoid or saturated linear function to the weighted sum of n inputs is hard to train in particular the problem of finding the weights of such a unit that minimize the quadratic training error within 2 or its average over a training set within 5 2 12n of its infimum proves to be np hard hence the well known backpropagation learning algorithm appears not to be efficient even for one neuron which has negative consequences in constructive learning

generating scenario trees for multistage decision problems in models of decision making under uncertainty we often are faced with the problem of representing the uncertainties in a form suitable for quantitative models if the uncertainties are expressed in terms of multivariate continuous distributions or a discrete distribution with far too many outcomes we normally face two possibilities either creating a decision model with internal sampling or trying to find a simple discrete approximation of the given distribution that serves as input to the model this paper presents a method based on nonlinear programming that can be used to generate a limited number of discrete outcomes that satisfy specified statistical properties users are free to specify any statistical properties they find relevant and the method can handle inconsistencies in the specifications the basic idea is to minimize some measure of distance between the statistical properties of the generated outcomes and the specified properties we illustrate the method by single and multiple period problems the results are encouraging in that a limited number of generated outcomes indeed have statistical properties that are close to or equal to the specifications we discuss how to verify that the relevant statistical properties are captured in these specifications and argue that what are the relevant properties will be problem dependent

on the complexity of learning strings and sequences abstract it is shown that strings sequences cannot be learned by strings sequences in valiant s distribution free pac learning model assuming rp np

a cutting surface algorithm for semi infinite convex programming with an application to moment robust optimization we present and analyze a central cutting surface algorithm for general semi infinite convex optimization problems and use it to develop a novel algorithm for distributionally robust optimization problems in which the uncertainty set consists of probability distributions with given bounds on their moments moments of arbitrary order as well as nonpolynomial moments can be included in the formulation we show that this gives rise to a hierarchy of optimization problems with decreasing levels of risk aversion with classic robust optimization at one end of the spectrum and stochastic programming at the other although our primary motivation is to solve distributionally robust optimization problems with moment uncertainty the cutting surface method for general semi infinite convex programs is also of independent interest the proposed method is applicable to problems with nondifferentiable semi infinite constraints indexed by an infinite dimensional index set examples comparing the cutting surface algorit

version spaces and the consistency problem a version space is a collection of concepts consistent with a given set of positive and negative examples mitchell artificial intelligence 18 1982 203 226 proposed representing a version space by its boundary sets the maximally general g and maximally specific consistent concepts s for many simple concept classes the size of g and s is known to grow exponentially in the number of positive and negative examples this paper argues that previous work on alternative representations of version spaces has disguised the real question underlying version space reasoning we instead show that tractable reasoning with version spaces turns out to depend on the consistency problem i e determining if there is any concept consistent with a set of positive and negative examples indeed we show that tractable version space reasoning is possible if and only if there is an efficient algorithm for the consistency problem our observations give rise to new concept classes for which tractable version space reasoning is now possible e g 1 decision lists monotone depth two formulas and halfspaces

learning decision trees from random examples needed for learning abstract we define the rank of a decision tree and show that for any fixed r the class of all decision trees of rank at most r on n boolean variables is learnable from random examples in time polynomial in n and linear in 1 and log 1 where is the accuracy parameter and is the confidence parameter using a suitable encoding of variables rivest s polynomial learnability result for decision lists can be interpreted as a special case of this result for rank 1 as another corollary we show that decision trees on n boolean variables of size polynomial in n are learnable from random examples in time linear in n o log n 1 and log 1 as a third corollary we show that boolean functions that have polynomial size dnf expressions for both their positive and their negative instances are learnable from random examples in time linear in n o log n 2 1 and log 1

computational limitations on learning from examples the computational complexity of learning boolean concepts from examples is investigated it is shown for various classes of concept representations that these cannot be learned feasibly in a distribution free sense unless r np these classes include a disjunctions of two monomials b boolean threshold functions and c boolean formulas in which each variable occurs at most once relationships between learning of heuristics and finding approximate solutions to np hard optimization problems are given