p sufficient statistics for pac learning k term dnf formulas through enumeration working in the framework of pac learning theory we present special statistics for accomplishing in polynomial time proper learning of dnf boolean formulas having a fixed number of monomials our statistics turn out to be near sufficient for a large family of distribution laws that we call butterfly distributions we develop a theory of most powerful learning for analyzing the performance of learning algorithms with particular reference to trade offs between power and computational costs focusing attention on sample and time complexity we prove that our algorithm works as efficiently as the best algorithms existing in the literature while the latter only take care of subclasses of our family of distributions

adaptive bem with inexact pcg solver yields almost optimal computational costs we consider the preconditioned conjugate gradient method pcg with optimal preconditioner in the frame of the boundary element method for elliptic first kind integral equations our adaptive algorithm steers the termination of pcg as well as the local mesh refinement besides convergence with optimal algebraic rates we also prove almost optimal computational complexity in particular we provide an additive schwarz preconditioner which can be computed in linear complexity and which is optimal in the sense that the condition numbers of the preconditioned systems are uniformly bounded as model problem serves the 2d or 3d laplace operator and the associated weakly singular integral equation with energy space widetilde h 1 2 gamma the main results also hold for the hyper singular integral equation with energy space h 1 2 gamma

learnability with respect to fixed distributions consider a first order typed language with semantics s for expressions and types adding subtyping means that a partial order on types is defined and that the typing rules are extended to the effect that expression e has type t whenever e has type s and s t we show how to adapt the semantics s in a simple set theoretic way obtaining a semantics s that satisfies in addition to some obvious requirements also the property that s s is included in s t whenever s t

source separation using a criterion based on second order statistics it is often assumed that blind separation of dynamically mixed sources cannot be done with second order statistics it is shown that separation of dynamically mixed sources indeed can be performed using second order statistics only a criterion based on second order statistics for the purpose of separating crosswise mixtures is stated the criterion is used in order to derive a gradient based separation algorithm as well as a newton type separation algorithm the uniqueness of the solution representing the separation is also investigated this reveals that 1 the channel system is parameter identifiable under weak conditions and 2 if the sources have the same color there exists at most two solutions the local convergence behavior of the proposed algorithm is studied and reveals a sufficient condition for local convergence furthermore the estimates of the channel system are shown to be consistent or to locally minimize the criterion

on computation with pulses we explore the computational power of formal models for computation with pulses such models are motivated by realistic models for biological neurons and by related new types of vlsi pulse stream vlsi in preceding work it was shown that the computational power of formal models for computation with pulses is quite high if the pulses arriving at a computational unit have an approximately linearly rising or linearly decreasing initial segment this property is satisfied by common models for biological neurons on the other hand several implementations of pulse stream vlsi employ pulses that are approximately piecewise constant i e step functions in this article we investigate the relevance of the shape of pulses in formal models for computation with pulses the results show that the computational power drops significantly if one replaces pulses with linearly rising or decreasing initial segments by piecewise constant pulses we provide an exact characterization of the latter model in terms of a weak version of a random access machine ram we also compare the language recognition capability of a recurrent version of this model with that of deterministic finite automata and turing machines

characterizations of monotonic and dual monotonic language learning the present paper deals with monotonic and dual monotonic language learning from positive as well as from positive and negative examples the three notions of monotonicity reflect different formalizations of the requirement that the learner has to produce better and better generalizations when fed more and more data on the concept to be learned the three versions of dual monotonicity describe the concept that the inference device has to produce specializations that fit better and better to the target language we characterize strong monotonic monotonic weak monotonic dual strong monotonic dual monotonic and monotonic dual monotonic as well as finite language learning from positive data in terms of recursively generable finite sets these characterizations provide a unifying framework for learning from positive data under the various monotonicity constraints moreover they yield additional insight into the problem of what a natural learning algorithm should look like

effects of domain characteristics on instance based learning algorithms this paper presents average case analyses of instance based learning algorithms the algorithms analyzed employ a variant of k nearest neighbor classifier k nn our analysis deals with a monotone m of n target concept with irrelevant attributes and handles three types of noise relevant attribute noise irrelevant attribute noise and class noise we formally represent the expected classification accuracy of k nn as a function of domain characteristics including the number of training instances the number of relevant and irrelevant attributes the threshold number in the target concept the probability of each attribute the noise rate for each type of noise and k we also explore the behavioral implications of the analyses by presenting the effects of domain characteristics on the expected accuracy of k nn and on the optimal value of k for artificial domains

adaptive bem with optimal convergence rates for the helmholtz equation abstract we analyze an adaptive boundary element method for the weakly singular and hypersingular integral equations for the 2d and 3d helmholtz problem the proposed adaptive algorithm is steered by a residual error estimator and does not rely on any a priori information that the underlying meshes are sufficiently fine we prove convergence of the error estimator with optimal algebraic rates independently of the coarse initial mesh as a technical contribution we prove certain local inverse type estimates for the boundary integral operators associated with the helmholtz equation

approximation classes for adaptive higher order finite element approximation we provide an almost characterization of the approximation classes appearing when using adaptive finite elements of lagrange type of any fixed polynomial degree the characterization is stated in terms of besov regularity and requires the approximation within spaces with integrability indices below one this article generalizes to higher order finite elements the results presented for linear finite elements by binev et al bddp 2002

quasi optimal convergence rate of an adaptive discontinuous galerkin method we analyze an adaptive discontinuous finite element method adfem for symmetric second order linear elliptic operators the method is formulated on nonconforming meshes made of simplices or quadrilaterals with any polynomial degree and in any dimension geq2 we prove that the adfem is a contraction for the sum of the energy error and the scaled error estimator between two consecutive adaptive loops we design a refinement procedure that maintains the level of nonconformity uniformly bounded and prove that the approximation classes using continuous and discontinuous finite elements are equivalent the geometric decay and the equivalence of classes are instrumental in deriving the optimal cardinality of the adfem we show that the adfem and the afem on nonconforming meshes yields a decay rate of energy error plus oscillation in terms of the number of degrees of freedom as dictated by the best approximation for this combined nonlinear quantity

nonlinear blind source separation using a radial basis function network this paper proposes a novel neural network approach to blind source separation in nonlinear mixture the approach utilizes a radial basis function rbf neural network to approximate the inverse of the nonlinear mixing mapping which is assumed to exist and able to be approximated using an rbf network a contrast function which consists of the mutual information and partial moments of the outputs of the separation system is defined to separate the nonlinear mixture the minimization of the contrast function results in the independence of the outputs with desirable moments such that the original sources are separated properly two learning algorithms for the parametric rbf network are developed by using the stochastic gradient descent method and an unsupervised clustering method by virtue of the rbf neural network this proposed approach takes advantage of high learning convergence rate of weights in the hidden layer and output layer natural unsupervised learning characteristics modular structure and universal approximation capability simulation results are presented to demonstrate the feasibility robustness and computability of the proposed method

optimal adaptivity for the supg finite element method abstract for convection dominated problems the streamline upwind petrov galerkin method supg also named streamline diffusion finite element method sdfem ensures a stable finite element solution even on coarse meshes i e in the preasymptotic range based on some a posteriori error estimators from the literature we formulate an adaptive mesh refining algorithm for supg we prove that the adaptively generated supg solutions converge at asymptotically optimal rates towards the exact solution the main focus thus is on the mathematical proof that the supg stabilization does not spoil the asymptotic convergence behavior of the adaptive scheme

on learning read k satisfy j dnf we study the learnability of read k satisfy j rksj dnf formulas these are boolean formulas in disjunctive normal form dnf in which the maximum number of occurrences of a variable is bounded by k and the number of terms satisfied by any assignment is at most j after motivating the investigation of this class of dnf formulas we present an algorithm that for any unknown rksj dnf formula to be learned with high probability finds a logically equivalent dnf formula using the well studied protocol of equivalence and membership queries the algorithm runs in polynomial time for k cdot j o log n over log log n where n is the number of input variables

convergence of adaptive finite element methods for general second order linear elliptic pdes we prove convergence of adaptive finite element methods afems for general nonsymmetric second order linear elliptic pdes thereby extending the result of morin nochetto and siebert it siam j numer anal 38 2000 pp 466 488 it siam rev 44 2002 pp 631 658 the proof relies on quasi orthogonality which accounts for the bilinear form not being a scalar product together with novel error and oscillation reduction estimates which now do not decouple we show that afems are a contraction for the sum of energy error plus oscillation numerical experiments including oscillatory coefficients and both coercive and noncoercive convection diffusion pde illustrate the theory and yield optimal meshes

logical analysis of binary data with missing bits we model a given pair of sets of positive and negative examples each of which may contain missing components as a partially defined boolean function with missing bits pbmb t f where email protected 0 1 n and email protected 0 1 n and stands for a missing bit then we consider the problem of establishing a boolean function an extension f 0 1 n 0 1 belonging to a given function class c such that f is true respectively false for every vector in t respectively in f this is a fundamental problem encountered in many areas such as learning theory pattern recognition example based knowledge bases logical analysis of data knowledge discovery and data mining in this paper depending upon how to deal with missing bits we formulate three types of extensions called robust consistent and most robust extensions for various classes of boolean functions such as general positive horn threshold decomposable and k dnf the complexity of the associated problems are then clarified some of them are solvable in polynomial time while the others are np hard

blind separation of parametric nonlinear mixtures of possibly autocorrelated and non stationary sources in this paper we present a new method formulated in a maximum likelihood framework for blindly separating nonlinear mixtures of statistically independent signals our method exploits on the one hand the knowledge of the parametric model of the mixing transformation with unknown parameter values and on the other hand the possible structure of source signals i e their autocorrelation and or nonstationarity one of the main advantages of the proposed method is that it can be implemented even if the analytical expression of the inverse model is unknown the method is first addressed in a general configuration then detailed for two special cases i e a simple bijective toy model and a linear quadratic model the study of the toy model is interesting because of its simplicity and its global bijectivity which allows us to focus our efforts on parameter estimation the linear quadratic model is chosen due to its capacity to describe real world mixing phenomena simulation results using the toy model and using a subclass of the linear quadratic model i e the bilinear model show that taking into account the nonlinearity of the mixing transformations and the structure of signals considerably improves separation performance

how to construct random functions a constructive theory of randomness for functions based on computational complexity is developed and a pseudorandom function generator is presented this generator is a deterministic polynomial time algorithm that transforms pairs g r where g is any one way function and r is a random k bit string to polynomial time computable functions r 1 2 k 1 2 k these r s cannot be distinguished from random functions by any probabilistic polynomial time algorithm that asks and receives the value of a function at arguments of its choice the result has applications in cryptography random constructions and complexity theory

identification criteria and lower bounds for perceptron like learning rules perceptron like learning rules are known to require exponentially many correction steps in order to identify boolean threshold functions exactly we introduce criteria that are weaker than exact identification and investigate whether learning becomes significantly faster if exact identification is replaced by one of these criteria probably approximately correct pac identification order identification and sign identification pac identification is based on the learning paradigm introduced by valiant 1984 and is known to be easier than exact identification order identification uses the fact that each threshold function induces an ordering relation on the input variables that can be represented by weights of linear size sign identification is based on a property of threshold functions known as unateness and requires only weights of constant size we show that perceptron like learning rules cannot satisfy these criteria when the number of correction steps is to be bounded by a polynomial we also pre

parameterized learnability of juntas we study the parameterized complexity of learning k juntas and some variations of juntas we show the hardness of learning k juntas and subclasses of k juntas in the pac model by reductions from a w 2 complete problem on the other hand as a consequence of a more general result we show that k juntas are exactly learnable with improper equivalence queries and access to a w p oracle

on the nonlearnability of a single spiking neuron we study the computational complexity of training a single spiking neuron n with binary coded inputs and output that in addition to adaptive weights and a threshold has adjustable synaptic delays a synchronization technique is introduced so that the results concerning the nonlearnability of spiking neurons with binary delays are generalized to arbitrary real valued delays in particular the consistency problem for n with programmable weights a threshold and delays and its approximation version are proven to be np complete it follows that the spiking neurons with arbitrary synaptic delays are not properly pac learnable and do not allow robust learning unless rp np in addition the representation problem for n a question whether an n variable boolean function given in dnf or as a disjunction of o n threshold gates can be computed by a spiking neuron is shown to be conp hard

a goal oriented adaptive finite element method with convergence rates an adaptive finite element method is analyzed for approximating functionals of the solution of symmetric elliptic second order boundary value problems we show that the method converges and derive a favorable upper bound for its convergence rate and computational complexity we illustrate our theoretical findings with numerical results

interpolation and approximation of sparse multivariate polynomials over gf 2 a function f 0 1 n to 0 1 is called t sparse if the n variable polynomial representation of f over gf 2 contains at most t monomials such functions are uniquely determined by their values at the so called critical set of all binary n tuples of hamming weight geqq n lfloor log 2 t rfloor 1 an algorithm is presented for interpolating any t sparse function f given the values of f at the critical set the time complexity of the proposed algorithm is proportional to n t and the size of the critical set then the more general problem of approximating 1 sparse functions is considered in which case the approximating function may differ from f at a fraction varepsilon of the space 0 1 n it is shown that o t varepsilon cdot n evaluation points are sufficient for the deterministic varepsilon approximation of any t sparse function and that an order t varepsilon alpha t varepsilon cdot log n points are necessary for this purpose wh

adaptive fem with optimal convergence rates for a certain class of nonsymmetric and possibly nonlinear problems we analyze adaptive mesh refining algorithms for conforming finite element discretizations of certain nonlinear second order partial differential equations we allow continuous polynomials of arbitrary but fixed polynomial order the adaptivity is driven by the residual error estimator we prove convergence even with optimal algebraic convergence rates in particular our analysis covers general linear second order elliptic operators unlike prior works for linear nonsymmetric operators our analysis avoids the interior node property for the refinement and the differential operator has to satisfy a g rarding inequality only if the differential operator is uniformly elliptic no additional assumption on the initial mesh is posed

general purpose computation with neural networks a survey of complexity theoretic results we survey and summarize the literature on the computational aspects of neural network models by presenting a detailed taxonomy of the various models according to their complexity theoretic characteristics the criteria of classification include the architecture of the network feedforward versus recurrent time model discrete versus continuous state type binary versus analog weight constraints symmetric versus asymmetric network size finite nets versus infinite families and computation type deterministic versus probabilistic among others the underlying results concerning the computational power and complexity issues of perceptron radial basis function winner take all and spiking neural networks are briefly surveyed with pointers to the relevant literature in our survey we focus mainly on the digital computation whose inputs and outputs are binary in nature although their values are quite often encoded as analog neuron states we omit the important learning issues

inductive inference of approximations in this paper we investigate inductive inference identification criteria which permit infinitely many errors in explanations but which require that the density of these errors be no more than a certain prespectified amount we introduce three hierarchies of such criteria each of which has the same order type as the real unit interval these three hierarchies are progressively more strict in the way they measure density of errors of explanations the strictest of the three turns out to have all of its members save one incomparable to the identification criterion which permits finitely many errors in explanations

adaptive finite element methods with convergence rates adaptive finite element methods for numerically solving elliptic equations are used often in practice only recently 12 17 have these methods been shown to converge however this convergence analysis says nothing about the rates of convergence of these methods and therefore does in principle not guarantee yet any numerical advantages of adaptive strategies versus non adaptive strategies the present paper modifies the adaptive method of morin nochetto and siebert 17 for solving the laplace equation with piecewise linear elements on domains in 2 by adding a coarsening step and proves that this new method has certain optimal convergence rates in the energy norm which is equivalent to the h1 norm namely it is shown that whenever s 0 and the solution u is such that for each n 1 it can be approximated to accuracy o n s in the energy norm by a continuous piecewise linear function on a triangulation with n cells using complete knowledge of u then the adaptive algorithm constructs an approximation of the same type with the same asymptotic accuracy while using only information gained during the computational process moreover the number of arithmetic computations in the proposed method is also of order o n for each n 1 the construction and analysis of this adaptive method relies on the theory of nonlinear approximation

a new accurate residual based a posteriori error indicator for the bem in 2d acoustics in this work we construct a new reliable efficient and local a posteriori error estimate for the single layer and hyper singular boundary integral equations associated to the helmholtz equation in two dimensions it uses a localization technique based on a generic operator which is used to transport the residual into l2 under appropriate conditions on the construction of we show that it is asymptotically exact with respect to the energy norm of the error the single layer equation and the hyper singular equation are treated separately while the current analysis requires the boundary to be smooth numerical experiments show that the new error estimators also perform well for non smooth boundaries

equilibrated residual error estimator for edge elements reliable a posteriori error estimates without generic constants can be obtained by a comparison of the finite element solution with a feasible function for the dual problem a cheap computation of such functions via equilibration is well known for scalar equations of second order we simplify and modify the equilibration such that it can be applied to the curl curl equation and edge elements the construction is more involved for edge elements since the equilibration has to be performed on subsets with different dimensions for this reason raviart thomas elements are extended in the spirit of distributions

error free and best fit extensions of partially defined boolean functions abstract in this paper we address a fundamental problem related to the induction of boolean logic given a set of data represented as a set of binary truen vectors or positive examples and a set of falsen vectors or negative examples we establish a boolean function or an extension f so thatfis true resp false in every given true resp false vector we shall further require that such an extension belongs to a certain specified class of functions e g class of positive functions class of horn functions and so on the class of functions represents our a priori knowledge or hypothesis about the extensionf which may be obtained from experience or from the analysis of mechanisms that may or may not cause the phenomena under consideration the real world data may contain errors e g measurement and classification errors might come in when obtaining data or there may be some other influential factors not represented as variables in the vectors in such situations we have to give up the goal of establishing an extension that is perfectly consistent with the given data and we are satisfied with an extensionfhaving the minimum number of misclassifications both problems i e the problem of finding an extension within a specified class of boolean functions and the problem of finding a minimum error extension in that class will be extensively studied in this paper for certain classes we shall provide polynomial algorithms and for other cases we prove their np hardness

on the power of probabilistic strategies in inductive inference abstract inductive inference of programs of recursive functions from input output examples by probabilistic strategies with an a priori bound n n of changes of hypotheses is investigated advantages of probabilistic strategies over deterministic ones are shown concerning their power in principle for every n 2 with probability arbitrarily close to 1 inference of function classes which cannot be inferred by any deterministic strategy with n changes of hypotheses as well as their computational complexity linear speed up of the number of changes of hypotheses necessary for inference by deterministic strategies

optimal additive schwarz methods for the h p bem we propose and analyze an overlapping schwarz preconditioner for the p and h p boundary element method for the hypersingular integral equation in 3d we consider surface triangulations consisting of triangles the condition number is bounded uniformly in the mesh size h and the polynomial order p the preconditioner handles adaptively refined meshes and is based on a local multilevel preconditioner for the lowest order space numerical experiments on different geometries illustrate its robustness

convergence and quasi optimality of adaptive fem with inhomogeneous dirichlet data we consider the solution of a second order elliptic pde with inhomogeneous dirichlet data by means of adaptive lowest order fem as is usually done in practice the given dirichlet data are discretized by nodal interpolation as model example serves the poisson equation with mixed dirichlet neumann boundary conditions for error estimation we use an edge based residual error estimator which replaces the volume residual contributions by edge oscillations for 2d we prove convergence of the adaptive algorithm even with optimal convergence rate for 2d and 3d we show convergence if the nodal interpolation operator is replaced by the l 2 projection or the scott zhang quasi interpolation operator as a byproduct of the proof we show that the scott zhang operator converges pointwise to a limiting operator as the mesh is locally refined this property might be of independent interest besides the current application finally numerical experiments conclude the work

neural networks with local receptive fields and superlinear vc dimension local receptive field neurons comprise such well known and widely used unit types as radial basis function rbf neurons and neurons with center surround receptive field we study the vapnik chervonenkis vc dimension of feedforward neural networks with one hidden layer of these units for several variants of local receptive field neurons we show that the vc dimension of these networks is superlinear in particular we establish the bound w log k for any reasonably sized network with w parameters and k hidden nodes this bound is shown to hold for discrete center surround receptive field neurons which are physiologically relevant models of cells in the mammalian visual system for neurons computing a difference of gaussians which are popular in computational vision and for standard rbf neurons a major alternative to sigmoidal neurons in artificial neural networks the resutlt for rbf neural networks is of particular interest since it answers a question that has been open for several years the results also give rise to lower bounds for networks with fixed input dimension regarding constants all bounds are larger than those known thus far for similar architectures with sigmoidal neurons the superlinear lower bounds contrast with linear upper bounds for single local receptive field neurons also derived here

learning curves of the clipped hebb rule for networks with binary weights networks with binary weights are very important from both the theoretical and practical points of view in this paper we investigate the clipped hebb rule for learning different networks of non overlapping binary perceptrons under uniform distribution we calculate exactly its learning curves in the limit of a large number of inputs where the average behaviour becomes the typical behaviour this calculation is performed using only very simple counting arguments and the central limit theorem the results indicate that the clipped hebb rule does indeed learn this class of networks in particular the generalization rates converge extremely rapidly often exponentially to perfect generalization these results are very encouraging given the simplicity of the learning rule the analytic expression of the learning curves are in excellent agreement with the numerical simulations

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

minimal nonlinear distortion principle for nonlinear independent component analysis it is well known that solutions to the nonlinear independent component analysis ica problem are highly non unique in this paper we propose the minimal nonlinear distortion mnd principle for tackling the ill posedness of nonlinear ica problems mnd prefers the nonlinear ica solution with the estimated mixing procedure as close as possible to linear among all possible solutions it also helps to avoid local optima in the solutions to achieve mnd we exploit a regularization term to minimize the mean square error between the nonlinear mixing mapping and the best fitting linear one the effect of mnd on the inherent trivial and non trivial indeterminacies in nonlinear ica solutions is investigated moreover we show that local mnd is closely related to the smoothness regularizer penalizing large curvature which provides another useful regularization condition for nonlinear ica experiments on synthetic data show the usefulness of the mnd principle for separating various nonlinear mixtures finally as an application we use nonlinear ica with mnd to separate daily returns of a set of stocks in hong kong and the linear causal relations among them are successfully discovered the resulting causal relations give some interesting insights into the stock market such a result can not be achieved by linear ica simulation studies also verify that when doing causality discovery sometimes one should not ignore the nonlinear distortion in the data generation procedure even if it is weak

comparison of identification criteria for machine inductive inference abstract a natural pl 1 hierarchy of successively more general criteria of success for inductive inference machines is described based on the size of sets of anomalies in programs synthesized by such machines these criteria are compared to others in the literature some of our results are interpreted as tradeoff results or as showing the inherent relative computational complexity of certain processes and others are interpreted from a positivistic mechanistic philosophical stance as theorems in philosophy of science the techniques of recursive function theory are employed including ordinary and infinitary recursion theorems

nonlinear blind source separation using kernels we derive a new method for solving nonlinear blind source separation bss problems by exploiting second order statistics in a kernel induced feature space this paper extends a new and efficient closed form linear algorithm to the nonlinear domain using the kernel trick originally applied in support vector machines svms this technique could likewise be applied to other linear covariance based source separation algorithms experiments on realistic nonlinear mixtures of speech signals gas multisensor data and visual disparity data illustrate the applicability of our approach

on learning perceptrons with binary weights we present an algorithm that pac learns any perceptron with binary weights and arbitrary threshold under the family of product distributions the sample complexity of this algorithm is of o n e 4 ln n and its running time increases only linearly with the number of training examples the algorithm does not try to find an hypothesis that agrees with all of the training examples rather it constructs a binary perceptron based on various probabilistic estimates obtained from the training examples we show that under the restricted case of the uniform distribution and zero threshold the algorithm reduces to the well known clipped hebb rule we calculate exactly the average generalization rate i e the learning curve of the algorithm under the uniform distribution in the limit of an infinite number of dimensions we find that the error rate decreases exponentially as a function of the number of training examples hence the average case analysis gives a sample complexity of o n ln 1 e a large improvement over the pac learning analysis the analytical expression of the learning curve is in excellent agreement with the extensive numerical simulations in addition the algorithm is very robust with respect to classification noise

quasi optimal convergence rate for an adaptive finite element method we analyze the simplest and most standard adaptive finite element method afem with any polynomial degree for general second order linear symmetric elliptic operators as is customary in practice the afem marks exclusively according to the error estimator and performs a minimal element refinement without the interior node property we prove that the afem is a contraction for the sum of the energy error and the scaled error estimator between two consecutive adaptive loops this geometric decay is instrumental to derive the optimal cardinality of the afem we show that the afem yields a decay rate of the energy error plus oscillation in terms of the number of degrees of freedom as dictated by the best approximation for this combined nonlinear quantity

on the complexity of learning for spiking neurons with temporal coding abstract spiking neurons are models for the computational units in biological neural systems where information is considered to be encoded mainly in the temporal patterns of their activity in a network of spiking neurons a new set of parameters becomes relevant which has no counterpart in traditional neural network models the time that a pulse needs to travel through a connection between two neurons also known as delay of a connection it is known that these delays are tuned in biological neural systems through a variety of mechanisms in this article we consider the arguably most simple model for a spiking neuron which can also easily be implemented in pulsed vlsi we investigate the vapnik chervonenkis vc dimension of networks of spiking neurons where the delays are viewed as programmable parameters and we prove tight bounds for this vc dimension thus we get quantitative estimates for the diversity of functions that a network with fixed architecture can compute with different settings of its delays in particular it turns out that a network of spiking neurons with k adjustable delays is able to compute a much richer class of functions than a threshold circuit with k adjustable weights the results also yield bounds for the number of training examples that an algorithm needs for tuning the delays of a network of spiking neurons results about the computational complexity of such algorithms are also given

fast identification of geometric objects with membership queries we investigate the number of membership queries that are needed to identify polygons i e intersections of halfplanes over a two dimensional grid 0 n 1 2 we exhibit a learning algorithm that learns 100 correctly while requiring no random examples and not more membership queries than previous algorithms needed in addition to their random examples for probably almost correctly learning even for moderate values of furthermore the learning algorithm in this paper uses only grid points for its membership queries this appears to be appropriate in situations where the probing device has only a limited resolution and for applications to the set of pixels on a two dimensional screen the learning algorithm that is described in this paper overcomes a fundamental obstacle related to the zigzag borderline that is generated by a halfplane over a discrete grid this obstacle had thwarted previous attempts to show that in angluin s model for on line learning with equivalence and membership queries one can learn in an efficient manner even those classes of geometric objects which cannot be learnt fast by equivalence queries alone such as rectangles in general position

optimal convergence behavior of adaptive fem driven by simple h h 2 type error estimators abstract for some poisson type model problem we prove that adaptive fem driven by the h h 2 type error estimators from ferraz leite et al 2010 leads to convergence with optimal algebraic convergence rates besides the implementational simplicity another striking feature of these estimators is that they can provide guaranteed lower bounds for the energy error with known efficiency constant 1

equivalence of models for polynomial learnability abstract in this paper we consider several variants of valiant s learnability model that have appeared in the literature we give conditions under which these models are equivalent in terms of the polynomially learnable concept classes they define these equivalences allow comparisons of most of the existing theorems in valiant style learnability and show that several simplifying assumptions on polynomial learning algorithms can be made without loss of generality we also give a useful reduction of learning problems to the problem of finding consistent hypotheses and give comparisons and equivalences between valiant s model and the prediction learning models of haussler littlestone and warmuth in 29th annual ieee symposium on foundations of computer science 1988

time structure of the activity in neural network models several neural network models in continuous time are reconsidered in the framework of a general mean field theory which is exact in the limit of a large and fully connected network the theory assumes pointlike spikes which are generated by a renewal process the effect of spikes on a receiving neuron is described by a linear response kernel which is the dominant term in a weak coupling expansion it is shown that the resulting spike response model is the most general renewal model with linear inputs the standard integrate and fire model forms a special case in a network structure with several pools of identical spiking neurons the global states and the dynamic evolution are determined by a nonlinear integral equation which describes the effective interaction within and between different pools we derive explicit stability criteria for stationary incoherent and oscillatory coherent solutions it is shown that the stationary state of noiseless systems is almost always unstable noise suppresses fast oscillations and stabilizes the system furthermore collective oscillations are stable only if the firing occurs while the synaptic potential is increasing in particular collective oscillations in a network with delayless excitatory interaction are at most semistable inhibitory interactions with short delays or excitatory interactions with long delays lead to stable oscillations our general results allow a straightforward application to different network models with spiking neurons furthermore the theory allows an estimation of the errors introduced in firing rate or graded response models

measuring teachability using variants of the teaching dimension in a typical algorithmic learning model a learner has to identify a target object from partial information conversely in a teaching model a teacher has to give information that allows the learners to identify a target object we devise two variants of the classical teaching model for boolean concept classes based on the teaching dimension and describe them by teaching dimension like combinatorial parameters in the first model the learners choose consistent hypotheses with least complexity we show that 1 decision lists are the harder to teach the longer they are and that 2 term dnfs are the harder to teach the more terms they have this contrasts with the teachability results for these classes in the teaching dimension model in our second model the learners choose consistent hypotheses based on the assumption that the teacher is optimal we show that monomials can be taught with a linear number of examples whereas some 1 decision lists need exponentially many

equivariant adaptive source separation source separation consists of recovering a set of independent signals when only mixtures with unknown coefficients are observed this paper introduces a class of adaptive algorithms for source separation that implements an adaptive version of equivariant estimation and is henceforth called equivariant adaptive separation via independence easi the easi algorithms are based on the idea of serial updating this specific form of matrix updates systematically yields algorithms with a simple structure for both real and complex mixtures most importantly the performance of an easi algorithm does not depend on the mixing matrix in particular convergence rates stability conditions and interference rejection levels depend only on the normalized distributions of the source signals closed form expressions of these quantities are given via an asymptotic performance analysis the theme of equivariance is stressed throughout the paper the source separation problem has an underlying multiplicative structure the parameter space forms a matrix multiplicative group we explore the favorable consequences of this fact on implementation performance and optimization of easi algorithms

vc dimension of an integrate and fire neuron model we compute the vc dimension of a leaky integrate and fire neuron model the vc dimension quantifies the ability of a function class to partition an input pattern space and can be considered a measure of computational capacity in this case the function class is the class of integrate and fire models generated by varying the integration time constant t and the threshold the input space they partition is the space of continuous time signals and the binary partition is specified by whether or not the model reaches threshold at some specified time we show that the vc dimension diverges only logarithmically with the input signal bandwidth n we also extend this approach to arbitrary passive dendritic trees the main contributions of this work are 1 it offers a novel treatment of computational capacity of this class of dynamic system and 2 it provides a framework for analyzing the computational capabilities of the dynamic systems defined by networks of spiking neurons

the completion of locally refined simplicial partitions created by bisection recently in found comput math 7 2 2007 245 269 we proved that an adaptive finite element method based on newest vertex bisection in two space dimensions for solving elliptic equations which is essentially the method from sinum 38 2000 466 488 by morin nochetto and siebert converges with the optimal rate the number of triangles n in the output partition of such a method is generally larger than the number m of triangles that in all intermediate partitions have been marked for bisection because additional bisections are needed to retain conforming meshes a key ingredient to our proof was a result from numer math 97 2004 219 268 by binev dahmen and devore saying that n n 0 cm for some absolute constant c where n 0 is the number of triangles from the initial partition that have never been bisected in this paper we extend this result to bisection algorithms of n simplices with that generalizing the result concerning optimality of the adaptive finite element method to general space dimensions

adaptive fem with coarse initial mesh guarantees optimal convergence rates for compactly perturbed elliptic problems we prove that for compactly perturbed elliptic problems where the corresponding bilinear form satisfies a garding inequality adaptive mesh refinement is capable of overcoming the preasymptotic behavior and eventually leads to convergence with optimal algebraic rates as an important consequence of our analysis one does not have to deal with the a priori assumption that the underlying meshes are sufficiently fine hence the overall conclusion of our results is that adaptivity has stabilizing effects and can overcome possibly pessimistic restrictions on the meshes in particular our analysis covers adaptive mesh refinement for the finite element discretization of the helmholtz equation from where our interest originated

learning boolean formulas efficient distribution free learning of boolean formulas from positive and negative examples is considered it is shown that classes of formulas that are efficiently learnable from only positive examples or only negative examples have certain closure properties a new substitution technique is used to show that in the distribution free case learning dnf disjunctive normal form formulas is no harder than learning monotone dnf we prove that monomials cannot be efficiently learned from negative examples alone even if the negative examples are uniformly distributed it is also shown that if the examples are drawn from uniform distributions then the class of dnf in which each variable occurs at most once is efficiently weakly learnable i e individual examples are correctly classified with a probability larger than 1 2 1 p where p is a polynomial in the relevant parameters of the learning problem we then show an equivalence between the notion of weak learning and the notion of group learning where a group of examples of polynomial size either all positive or all negative must be correctly classified with high probability

data oscillation and convergence of adaptive fem data oscillation is intrinsic information missed by the averaging process associated with finite element methods fem regardless of quadrature ensuring a reduction rate of data oscillation together with an error reduction based on a posteriori error estimators we construct a simple and efficient adaptive fem for elliptic partial differential equations pdes with linear rate of convergence without any preliminary mesh adaptation nor explicit knowledge of constants any prescribed error tolerance is thus achieved in a finite number of steps a number of numerical experiments in two and three dimensions yield quasi optimal meshes along with a competitive performance

classification of linearly nonseparable patterns by linear threshold elements learning and convergence properties of linear threshold elements or perceptrons are well understood for the case where the input vectors or the training sets to the perceptron are linearly separable little is known however about the behavior of the perceptron learning algorithm when the training sets are linearly nonseparable we present the first known results on the structure of linearly nonseparable training sets and on the behavior of perceptrons when the set of input vectors is linearly nonseparable more precisely we show that using the well known perceptron learning algorithm a linear threshold element can learn the input vectors that are provably learnable and identify those vectors that cannot be learned without committing errors we also show how a linear threshold element can be used to learn large linearly separable subsets of any given nonseparable training set in order to develop our results we first establish formal characterizations of linearly nonseparable training sets and define learnable structures for such patterns we also prove computational complexity results for the related learning problems next based on such characterizations we show that a perceptron does the best one can expect for linearly nonseparable sets of input vectors and learns as much as is theoretically possible

some dichotomy theorems for neural learning problems the computational complexity of learning from binary examples is investigated for linear threshold neurons we introduce combinatorial measures that create classes of infinitely many learning problems with sample restrictions we analyze how the complexity of these problems depends on the values for the measures the results are established as dichotomy theorems showing that each problem is either np complete or solvable in polynomial time in particular we consider consistency and maximum consistency problems for neurons with binary weights and maximum consistency problems for neurons with arbitrary weights we determine for each problem class the dividing line between the np complete and polynomial time solvable problems moreover all efficiently solvable problems are shown to have constructive algorithms that require no more than linear time on a random access machine model similar dichotomies are exhibited for neurons with bounded threshold the results demonstrate on the one hand that the consideration of sample constraints can lead to the discovery of new efficient algorithms for non trivial learning problems on the other hand hard learning problems may remain intractable even for severely restricted samples

decay rates of adaptive finite elements with dorfler marking we investigate the decay rate for an adaptive finite element discretization of a second order linear symmetric elliptic pde we allow for any kind of estimator that is locally equivalent to the standard residual estimator this includes in particular hierarchical estimators estimators based on the solution of local problems estimators based on local averaging equilibrated residual estimators the zz estimator etc the adaptive method selects elements for refinement with dorfler marking and performs a minimal refinement in that no interior node property is needed based on the local equivalence to the residual estimator we prove an error reduction property in combination with minimal dorfler marking this yields an optimal decay rate in terms of degrees of freedom

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

training a single sigmoidal neuron is hard we first present a brief survey of hardness results for training feedforward neural networks these results are then completed by the proof that the simplest architecture containing only a single neuron that applies a sigmoidal activation function r satisfying certain natural axioms e g the standard logistic sigmoid or saturated linear function to the weighted sum of n inputs is hard to train in particular the problem of finding the weights of such a unit that minimize the quadratic training error within 2 or its average over a training set within 5 2 12n of its infimum proves to be np hard hence the well known backpropagation learning algorithm appears not to be efficient even for one neuron which has negative consequences in constructive learning

a convergent adaptive algorithm for poisson s equation we construct a converging adaptive algorithm for linear elements applied to poisson s equation in two space dimensions starting from a macro triangulation we describe how to construct an initial triangulation from a priori information then we use a posteriors error estimators to get a sequence of refined triangulation and approximate solutions it is proved that the error measured in the energy norm decreases at a constant rate in each step until a prescribed error bound is reached extension to higher order elements in two space dimension and numerical results are included

on the complexity of learning strings and sequences abstract it is shown that strings sequences cannot be learned by strings sequences in valiant s distribution free pac learning model assuming rp np

version spaces and the consistency problem a version space is a collection of concepts consistent with a given set of positive and negative examples mitchell artificial intelligence 18 1982 203 226 proposed representing a version space by its boundary sets the maximally general g and maximally specific consistent concepts s for many simple concept classes the size of g and s is known to grow exponentially in the number of positive and negative examples this paper argues that previous work on alternative representations of version spaces has disguised the real question underlying version space reasoning we instead show that tractable reasoning with version spaces turns out to depend on the consistency problem i e determining if there is any concept consistent with a set of positive and negative examples indeed we show that tractable version space reasoning is possible if and only if there is an efficient algorithm for the consistency problem our observations give rise to new concept classes for which tractable version space reasoning is now possible e g 1 decision lists monotone depth two formulas and halfspaces

axioms of adaptivity this paper aims first at a simultaneous axiomatic presentation of the proof of optimal convergence rates for adaptive finite element methods and second at some refinements of particular questions like the avoidance of discrete lower bounds inexact solvers inhomogeneous boundary data or the use of equivalent error estimators solely four axioms guarantee the optimality in terms of the error estimators compared to the state of the art in the temporary literature the improvements of this article can be summarized as follows first a general framework is presented which covers the existing literature on optimality of adaptive schemes the abstract analysis covers linear as well as nonlinear problems and is independent of the underlying finite element or boundary element method second efficiency of the error estimator is neither needed to prove convergence nor quasi optimal convergence behavior of the error estimator in this paper efficiency exclusively characterizes the approximation classes involved in terms of the best approximation error and data resolution and so the upper bound on the optimal marking parameters does not depend on the efficiency constant third some general quasi galerkin orthogonality is not only sufficient but also necessary for the r linear convergence of the error estimator which is a fundamental ingredient in the current quasi optimality analysis due to stevenson 2007 finally the general analysis allows for equivalent error estimators and inexact solvers as well as different non homogeneous and mixed boundary conditions

learning decision trees from random examples needed for learning abstract we define the rank of a decision tree and show that for any fixed r the class of all decision trees of rank at most r on n boolean variables is learnable from random examples in time polynomial in n and linear in 1 and log 1 where is the accuracy parameter and is the confidence parameter using a suitable encoding of variables rivest s polynomial learnability result for decision lists can be interpreted as a special case of this result for rank 1 as another corollary we show that decision trees on n boolean variables of size polynomial in n are learnable from random examples in time linear in n o log n 1 and log 1 as a third corollary we show that boolean functions that have polynomial size dnf expressions for both their positive and their negative instances are learnable from random examples in time linear in n o log n 2 1 and log 1

an information maximization approach to blind separation and blind deconvolution we derive a new self organizing learning algorithm that maximizes the information transferred in a network of nonlinear units the algorithm does not assume any knowledge of the input distributions and is defined here for the zero noise limit under these conditions information maximization has extra properties not found in the linear case linsker 1989 the nonlinearities in the transfer function are able to pick up higher order moments of the input distributions and perform something akin to true redundancy reduction between units in the output representation this enables the network to separate statistically independent components in the inputs a higher order generalization of principal components analysis we apply the network to the source separation or cocktail party problem successfully separating unknown mixtures of up to 10 speakers we also show that a variant on the network architecture is able to perform blind deconvolution cancellation of unknown echoes and reverberation in a speech signal finally we derive dependencies of information transfer on time delays we suggest that information maximization provides a unifying framework for problems in blind signal processing

computational limitations on learning from examples the computational complexity of learning boolean concepts from examples is investigated it is shown for various classes of concept representations that these cannot be learned feasibly in a distribution free sense unless r np these classes include a disjunctions of two monomials b boolean threshold functions and c boolean formulas in which each variable occurs at most once relationships between learning of heuristics and finding approximate solutions to np hard optimization problems are given

inhomogeneous dirichlet conditions in a priori and a posteriori finite element error analysis the numerical solution of elliptic boundary value problems with finite element methods requires the approximation of given dirichlet data ud by functions ud h in the trace space of a finite element space on d in this paper quantitative a priori and a posteriori estimates are presented for two choices of ud h namely the nodal interpolation and the orthogonal projection in l2 d onto the trace space two corresponding extension operators allow for an estimate of the boundary data approximation in global h1 and l2 a priori and a posteriori error estimates the results imply that the orthogonal projection leads to better estimates in the sense that the influence of the approximation error on the estimates is of higher order than for the nodal interpolation