a steerable complex wavelet construction and its application to image denoising this work addresses the design of a novel complex steerable wavelet construction the generation of transform space feature measurements associated with corner and edge presence and orientation properties and the application of these measurements directly to image denoising the decomposition uses pairs of bandpass filters that display symmetry and antisymmetry about a steerable axis of orientation while the angular characterization of the bandpass filters is similar to those previously described the radial characteristic is new as is the manner of constructing the interpolation functions for steering the complex filters have been engineered into a multirate system providing a synthesis and analysis subband filtering system with good reconstruction properties although the performance of our proposed denoising strategy is currently below that of recently reported state of the art techniques in denoising it does compare favorably with wavelet coring approaches employing global thresholds and with an oracle shrinkage technique and presents a very promising avenue for exploring structure based denoising in the wavelet domain

complete discrete 2 d gabor transforms by neural networks for image analysis and compression a three layered neural network is described for transforming two dimensional discrete signals into generalized nonorthogonal 2 d gabor representations for image analysis segmentation and compression these transforms are conjoint spatial spectral representations which provide a complete image description in terms of locally windowed 2 d spectral coordinates embedded within global 2 d spatial coordinates in the present neural network approach based on interlaminar interactions involving two layers with fixed weights and one layer with adjustable weights the network finds coefficients for complete conjoint 2 d gabor transforms without restrictive conditions in wavelet expansions based on a biologically inspired log polar ensemble of dilations rotations and translations of a single underlying 2 d gabor wavelet template image compression is illustrated with ratios up to 20 1 also demonstrated is image segmentation based on the clustering of coefficients in the complete 2 d gabor transform

generalization properties of multilayered neural networks generalization properties of multilayered neural networks with binary couplings are studied in the high temperature limit the transition to the perfect generalization phase is evaluated for systems with an arbitrary number of layers it is found that the thermodynamic transition occurs for a number of examples lower than for the perceptron but the opposite occurs for the transition in which the poor generalization solution disappears the generalization error is also decomposed according to the contributions coming from different numbers of hidden neurons that have a wrong sign in the internal field this allows the authors to describe the generalization behaviour of the hidden neurons

time series based bifurcation diagram reconstruction abstract we consider the problem of reconstructing bifurcation diagrams bds of maps using time series this study goes along the same line of ideas presented by tokunaga et al physica d 79 1994 348 and tokuda et al physica d 95 1996 380 the aim is to reconstruct the bd of a dynamical system without the knowledge of its functional form and its dependence on the parameters instead time series at different parameter values assumed to be available are used a three layer fully connected neural network is employed in the approximation of the map the task of the network is to learn the dynamics of the system as function of the parameters from the available time series we determine a class of maps for which one can always find a linear subspace in the weight space of the network where the network s bifurcation structure is qualitatively the same as the bifurcation structure of the map we discuss a scheme in locating this subspace using the time series we further discuss how to recognize time series generated by this class of maps finally we propose an algorithm in reconstructing the bds of this class of maps using predictor functions obtained by neural network this algorithm is flexible so that other classes of predictors apart from neural networks can be used in the reconstruction

inferring hidden states in langevin dynamics on large networks average case performance we present average performance results for dynamical inference problems in large networks where a set of nodes is hidden while the time trajectories of the others are observed examples of this scenario can occur in signal transduction and gene regulation networks we focus on the linear stochastic dynamics of continuous variables interacting via random gaussian couplings of generic symmetry we analyze the inference error given by the variance of the posterior distribution over hidden paths in the thermodynamic limit and as a function of the system parameters and the ratio ensuremath alpha between the number of hidden and observed nodes by applying kalman filter recursions we find that the posterior dynamics is governed by an effective drift that incorporates the effect of the observations we present two approaches for characterizing the posterior variance that allow us to tackle respectively equilibrium and nonequilibrium dynamics the first appeals to random matrix theory and reveals average spectral properties of the inference error and typical posterior relaxation times the second is based on dynamical functionals and yields the inference error as the solution of an algebraic equation

perturbative treatments and learning techniques the replica trick rt of statistical mechanics has become a very useful tool for the investigation of complex systems in general and in particular for studying learning and generalization processes in neural networks nn 1 2 the trick overcomes the difficulty of performing an ensemble average of the logarithm of the partition function z by that of averaging z over n replicas of the original network with n a very large integer for feedforward single layered nn 3 4 the rt ansatz has been proved to be a valuable tool for the learning of a rule on the basis of a suitable set of examples in particular the full quenched theory has been carefully studied within the framework of a replica symmetry approximation 5 however as the temperature drops symmetry breakings may invalidate this approximation so that an approach to the full quenched theory that incorporates disorder effects due to improper examples may be of some interest in the present effort we wish to study in the spirit of a second order approximation noise effects in the training set unavoidable in any realistic setting the noise will here be the result of letting just part of the examples to be produced by the perceptron teacher pt the rest are to be randomly selected bad examples two types of situations are to be confronted learnable and unlearnable rules 6 for the former there is at least a vector in the concomitant weight space that can learn the rule in an exact fashion the latter arises mostly in cases of architectural mismatch in such a situation the training error can never vanish the question to be answered is can a perceptron trained under these circumstances correctly respond to queries posed by a pt in other words is the rule underlying the good examples a learnable one we will show here that these questions can be adequately dealt with the paper is organized as follows section ii is devoted to a brief recapitulation of basic concepts concerning the rt while sec iii deals with the thermodynamics of the situation that interests us here a second order high temperature approximation is derived boolean perceptrons with ising weights on the learning curves are the subject of sec iv finally some conclusions are drawn in sec v ii the replica method

vapnik chervonenkis bounds for generalization the authors review the vapnik and chervonenkis theorem as applied to the problem of generalization by combining some of the technical modifications proposed in the literature they derive tighter bounds and a new version of the theorem bounding the accuracy in the estimation of generalization probabilities from finite samples a critical discussion and comparison with the results from statistical mechanics is given

approximations of solutions to neutral functional differential equations with nonlocal history conditions abstract this work is concerned with a class of neutral functional differential equations with nonlocal history conditions in a hilbert space the approximation of solution to a class of such problems is studied moreover the convergence of faedo galerkin approximation of solution is shown for illustration an example is worked out

four types of learning curves if machines are learning to make decisions given a number of examples the generalization error e t is defined as the average probability that an incorrect decision is made for a new example by a machine when trained with t examples the generalization error decreases as t increases and the curve e t is called a learning curve the present paper uses the bayesian approach to show that given the annealed approximation learning curves can be classified into four asymptotic types if the machine is deterministic with noiseless teacher signals then 1 e at 1 when the correct machine parameter is unique and 2 e at 2 when the set of the correct parameters has a finite measure if the teacher signals are noisy then 3 e at 1 2 for a deterministic machine and 4 e c at 1 for a stochastic machine

a class of linear partial neutral functional differential equations with nondense domain abstract we consider a system of linear partial neutral functional differential equations with nondense domain a natural generalized notion of solutions is provided by the integral solutions we derive a variation of constants formula which allows us to transform the integral solutions of the neutral equation to solutions of an abstract volterra integral equation the basic existence and uniqueness results are given and the solutions are shown to generate an integrated semigroup

a numerical study on learning curves in stochastic multilayer feedforward networks the universal asymptotic scaling laws proposed by amari et al are studied in large scale simulations using a cm5 small stochastic multilayer feedforward networks trained with backpropagation are investigated in the range of a large number of training patterns t the asymptotic generalization error scales as 1 t as predicted for a medium range t a faster 1 t2 scaling is observed this effect is explained by using higher order corrections of the likelihood expansion it is shown for small t that the scaling law changes drastically when the network undergoes a transition from strong overfitting to effective learning

linear readout of object manifolds objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location orientation and intensity what makes certain sensory representations better suited for invariant decoding of objects by downstream networks we present a theory that characterizes the ability of a linear readout network the perceptron to classify objects from variable neural responses we show how the readout perceptron capacity depends on the dimensionality size and shape of the object manifolds in its input neural representation

learning of higher order perceptrons with tunable complexities we study learning from examples by higher order perceptrons which realize polynomially separable rules the model complexities of the networks are made tunable by varying the relative orders of different monomial terms we analyse the learning curves of higher order perceptrons when the gibbs algorithm is used for training it is found that learning occurs in a stepwise manner this is because the number of examples needed to constrain the corresponding phase space component scales differently

the design and use of steerable filters the authors present an efficient architecture to synthesize filters of arbitrary orientations from linear combinations of basis filters allowing one to adaptively steer a filter to any orientation and to determine analytically the filter output as a function of orientation steerable filters may be designed in quadrature pairs to allow adaptive control over phase as well as orientation the authors show how to design and steer the filters and present examples of their use in the analysis of orientation and phase angularly adaptive filtering edge detection and shape from shading one can also build a self similar steerable pyramid representation the same concepts can be generalized to the design of 3 d steerable filters

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

on the generalization ability of diluted perceptrons a linearly separable boolean function is teamed by a diluted perceptron with optimal stability a different level of dilution is allowed for teacher and student perceptron the learning algorithms used were the optimal annealed dilution and hebbian dilution the generalization ability i e the probability to recognize a pattern which has not been learned before is calculated in replica symmetry

global existence for semilinear evolution equations with nonlocal conditions abstract in this paper we study the global existence of solutions for semilinear evolution equations with nonlocal conditions via a fixed point analysis approach using the leray schauder alternative we derive conditions under which a solution exists globally

primitive features by steering quadrature and scale the impulse response of neurons in the visual cortex of the mammalian brain has been known for some time how to make use of these as filters has led to many hypotheses the response of a single filter is ambiguous because the result depends on stimulus type contrast position orientation and scale we show that a set of quadrature filters at sparse positions can be constructed so that it is possible to disambiguate the 2d responses of the individual filters detecting edges is not the goal of the present work rather we seek to detect relevant edges thus we make the assumption that at the scale of interest a local image patch consists predominantly of an edge or a bar when this patch is processed by five or seven oriented filters one can compute the exact orientation and centroid position of the feature when the set of filters is applied at two different scales it is possible to distinguish edges from ridges and to identify the polarity intensity and width it is also possible to find corners and blobs these computations are stable under image shifts in position and orientation and can be made to subpixel resolution

existence of solutions for nonlocal impulsive partial functional integrodifferential equations via fractional operators in this paper by using the leray schauder alternative we have investigated the existence of mild solutions to first order impulsive partial functional integrodifferential equations with nonlocal conditions in an a norm we assume that the linear part generates an analytic compact bounded semigroup and that the nonlinear part is a lipschitz continuous function with respect to the fractional power norm of the linear part an example is also given to illustrate our main results

anti pairing in learning of a neural network with redundant hidden units we study the statistical mechanics of learning from examples between the two layered committee machines with different numbers of hidden units using the replica theory the number m of hidden units of the student network is larger than the number mt of those of the target network called the teacher we choose the networks to have binary synaptic weights 1 which makes it possible to compare the calculation with the monte carlo simulation we propose an effective teacher as a virtual target network which has the same m hidden units as the student and gives identical outputs with those of the original teacher this is a way of making a conjecture for a ground state of a thermodynamic system given by the weights of the effective teacher in our study we suppose that the weights on mt hidden units of the effective teacher are the same as those of the original teacher while those on m mt redundant hidden units are composed of anti pairs 1 1 with probability 1 p in the limit p 0 for p 0 exact there are no terms related to the effective teacher in the calculation for the contributions of anti pairs to outputs are exactly cancelled in the limit p 0 however we find that the learnt weights of the student are actually equivalent to those of the suggested effective teacher which is not possible from the calculation for p 0 p plays the role of a symmetry breaking parameter for anti pairing ordering which is analogous to the magnetic field for the ising model a first order phase transition is found to be signalled by breaking of symmetry in permuting hidden units above a critical number of examples the student is shown to learn perfectly the effective teacher anti pairing can be measured by a set of order parameters zero in the permutation symmetric phase and nonzero in the permutation symmetry breaking phase results from the monte carlo simulation are shown to be in good agreement with those from the replica calculation

existence and regularity of solutions for some neutral partial differential equations with nonlocal conditions abstract in this work we establish several results about the existence and regularity of solutions for some neutral partial differential equations with nonlocal conditions we assume that the linear part generates a strongly continuous semigroup on a general banach space

neural network training without spurious minima physics department national university la plata casilla de correro 727 1900 la plata argentina received 2 january 1996 in training student perceptions recourse to information theory concepts allows one to select the best workinghypothesis and obtain an exact solution for the associated probability distribution we apply this trainingscheme to perceptions with binary weights and show that no phase transition ensues by recourse to ourapproach fast learning is guaranteed and trapping by spurious local minima is avoided s1063 651x 96 07505 8 pacs number s 87 10 1e 05 20 2y 02 70 2ci introduction

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state

tight bounds on transition to perfect generalization in perceptrons sudden transition to perfect generalization in binary perceptrons is investigated building on recent theoretical works of gardner and derrida 1989 and baum and lyuu 1991 we show the followi

the dynamics of a genetic algorithm for a simple learning problem a formalism for describing the dynamics of genetic algorithms gas using method s from statistical mechanics is applied to the problem of generalization in a perceptron with binary weights the dynamics are solved for the case where a new batch of training patterns is presented to each population member each generation which considerably simplifies the calculation the theory is shown to agree closely to simulations of a real ga averaged over many runs accurately predicting the mean best solution found for weak selection and large problem size the difference equations describing the dynamics can be expressed analytically and we find that the effects of noise due to the finite size of each training batch can be removed by increasing the population size appropriately if this population resizing is used one can deduce the most computationally efficient size of training batch each generation for independent patterns this choice also gives the minimum total number of training patterns used although using independent patterns is a very inefficient use of training patterns in general this work may also prove useful for determining the optimum batch size in the case where patterns are recycled

how tight are the vapnik chervonenkis bounds we describe a series of numerical experiments that measure the average generalization capability of neural networks trained on a variety of simple functions these experiments are designed to test the relationship between average generalization performance and the worst case bounds obtained from formal learning theory using the vapnik chervonenkis vc dimension blumer et al 1989 haussler et al 1990 recent statistical learning theories tishby et al 1989 schwartz et al 1990 suggest that surpassing these bounds might be possible if the spectrum of possible generalizations has a gap near perfect performance we indeed find that in some cases the average generalization is significantly better than the vc bound the approach to perfect performance is exponential in the number of examples m rather than the 1 m result of the bound however in these cases we have not found evidence of the gap predicted by the above statistical theories in other cases we do find the 1 m behavior of the vc bound

dynamic proximity of spatio temporal sequences recurrent networks can generate spatio temporal neural sequences of very large cycles having an apparent random behavior nonetheless a proximity measure between these sequences may be defined through comparison of the synaptic weight matrices that generate them following the dynamic neural filter dnf formalism we demonstrate this concept by comparing teacher and student recurrent networks of binary neurons we show that large sequences providing a training set well exceeding the cover limit allow for good determination of the synaptic matrices alternatively assuming the matrices to be known very fast determination of the biases can be achieved thus a spatio temporal sequence may be regarded as spatio temporal encoding of the bias vector we introduce a linear support vector machine svm variant of the dnf in order to specify an optimal weight matrix this approach allows us to deal with noise spatio temporal sequences generated by different dnfs with the same number of neurons may be compared by calculating correlations of the synaptic matrices of the reconstructed dnfs other types of spatio temporal sequences need the introduction of hidden neurons and or the use of a kernel variant of the svm approach the latter is being defined as a recurrent support vector network rsvn

the transition to perfect generalization in perceptrons several recent papers gardner and derrida 1989 gyrgyi 1990 sompolinsky et al 1990 have found using methods of statistical physics that a transition to perfect generalization occurs in training a simple perceptron whose weights can only take values 1 we give a rigorous proof of such a phenomena that is we show for 2 0821 that if at least n examples are drawn from the uniform distribution on 1 1 n and classified according to a target perceptron wt 1 1 n as positive or negative according to whether wtx is nonnegative or negative then the probability is 2 n that there is any other such perceptron consistent with the examples numerical results indicate further that perfect generalization holds for as low as 1 5

existence of some neutral partial differential equations with infinite delay this study intends to investigate a class of quasi linear partial neutral functional differential equations with infinite delay we assume that the linear part generates an analytic compact semigroup and the nonlinear part satisfies certain conditions a sufficient condition is given to ensure the existence of mild and classical solutions finally an example is given to illustrate our abstract results

generalization from educated teachers the authors study the generalization ability of a perceptron as a function of the structural similarity between teacher and student to this end they consider a teacher perceptron b designed to implement a certain number of random input output mappings and a student perceptron bold j trained with input output examples generated by the teacher if the pattern sets of teacher and student overlap different combinations of teacher and student learning rules result in different generalization probabilities generalization can be facilitated or can be hindered depending on the combination of learning rules and the storage levels of teacher and student analytical calculations of the generalization ability for several combinations of teacher and student learning rules are in very good agreement with numerical simulations

perfect learning and power law in learning from stochastic examples by ising perceptrons analysis under one step replica symmetry breaking ansatz we study learning from stochastic examples by perceptrons with ising weights in the framework of statistical mechanics under the one step replica symmetry breaking ansatz the behaviour of learning curves are classified according to some local property of the rules by which examples are drawn the conditions for the existence of perfect learning together with other behaviours of the learning curves are given precisely the results agree with those obtained by seung 1995 using a refined annealled approximation

on line learning in the ising perceptron on line learning of both binary and continuous rules in an ising space is studied learning is achieved by using an artificial parameter a weight vector which is constrained to the surface of a hypersphere spherical constraint in the case of a binary rule the generalization error decays to zero super exponentially as exp c 2 where is the number of examples divided by n the size of the input vector and c 0 much faster learning is obtained in the case of continuous activation functions where the generalization error decays as exp e the number of steps required for perfect learning is estimated for both scenarios and compared with simulations

existence results for impulsive neutral stochastic functional integro differential inclusions with nonlocal initial conditions in this paper we prove the existence of mild solutions for a class of impulsive neutral stochastic functional integro differential inclusions with nonlocal initial conditions and resolvent operators sufficient conditions for the existence are derived with the help of the fixed point theorem for multi valued operators due to dhage and the fractional power of operators an example is provided to illustrate the theory

on learning perceptrons with binary weights we present an algorithm that pac learns any perceptron with binary weights and arbitrary threshold under the family of product distributions the sample complexity of this algorithm is of o n e 4 ln n and its running time increases only linearly with the number of training examples the algorithm does not try to find an hypothesis that agrees with all of the training examples rather it constructs a binary perceptron based on various probabilistic estimates obtained from the training examples we show that under the restricted case of the uniform distribution and zero threshold the algorithm reduces to the well known clipped hebb rule we calculate exactly the average generalization rate i e the learning curve of the algorithm under the uniform distribution in the limit of an infinite number of dimensions we find that the error rate decreases exponentially as a function of the number of training examples hence the average case analysis gives a sample complexity of o n ln 1 e a large improvement over the pac learning analysis the analytical expression of the learning curve is in excellent agreement with the extensive numerical simulations in addition the algorithm is very robust with respect to classification noise

learning from examples in fully connected committee machines the problem of learning from examples in two layer neural networks is studied within the framework of statistical mechanics a fully connected committee machine is trained to implement a task which is not linearly separable the generalization error as a function of the number of training examples per adjustable weight is calculated in the annealed approximation for both binary and continuous weights we find a first order transition with a discontinuous drop in the generalization error the transitions occur due to a specialization of the hidden units from a symmetric state to one with each hidden unit in the student network specialized on a corresponding unit in the teacher network the symmetric states of poor generalization remain metastable even for large training sets

existence for impulsive neutral integrodifferential inclusions with nonlocal initial conditions via fractional operators abstract this paper is mainly concerned with existence of mild solutions for a first order impulsive neutral integrodifferential inclusions with nonlocal initial conditions in norm we assume that the linear part generates an analytic semigroup and transform it into an integral inclusion by using a fixed point theorem for multi valued maps due to dhage a main existence theorem is established finally we present an example to illustrate this main theorem

existence for a class of partial differential equations with nonlocal conditions abstract this paper is concerned with the existence of mild and classical solutions of nonlocal cauchy problems we assume that the linear part generates an analytic compact bounded semigroup and that the nonlinear part is a holder continuous function with respect to the fractional power norm of the linear part considering the nonlocal conditions in the norm we generalize the recent conclusions on this topic the results obtained will be applied to a class of functional partial differential equations with nonlocal conditions in 1 2 norm

selecting examples for perceptrons a technique is introduced for analysing the learning process of perceptrons which continually select their own examples the most efficient training algorithm yet devised the authors predict a 42 reduction in the number of examples wasted in training an ising perceptron compared to the case in which examples are random optimally stable spherical perceptrons may also be taught significantly more efficiently and their results compare well with an existing numerical simulation

towards more practical average bounds on supervised learning in this paper we describe a method which enables us to study the average generalization performance of learning directly via hypothesis testing inequalities the resulting theory provides a unified viewpoint of average case learning curves of concept learning and regression in realistic learning problems not necessarily within the bayesian framework the advantages of the theory are that it alleviates the practical pessimism frequently claimed for the results of the vapnik chervonenkis vc theory and its alike and provides general insights into generalization besides the bounds on learning curves are directly related to the number of adjustable system weights although the theory is based on an approximation assumption and cannot apply to the worst case learning setting the precondition of the assumption is mild and the approximation itself is only a sufficient condition for the validity of the theory we illustrate the results with numerical simulations and apply the theory to examining the generalization ability of combination of neural networks

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

temporal evolution of generalization during learning in linear networks we study generalization in a simple framework of feedforward linear networks with n inputs and n outputs trained from examples by gradient descent on the usual quadratic error function we derive analytical results on the behavior of the validation function corresponding to the lms error function calculated on a set of validation patterns we show that the behavior of the validation function depends critically on the initial conditions and on the characteristics of the noise under certain simple assumptions if the initial weights are sufficiently small the validation function has a unique minimum corresponding to an optimal stopping time for training for which simple bounds can be calculated there exists also situations where the validation function can have more complicated and somewhat unexpected behavior such as multiple local minima at most n of variable depth and long but finite plateau effects additional results and possible extensions are briefly discussed

one step replica symmetry breaking solution for a perceptron learning with weight mismatch we investigate the properties of the one step replica symmetry breaking 1rsb solution for a perceptron learning from examples with weight mismatch where the entropy zero line crosses the almeida thouless at line of the rs solution for a small number of examples we find the optimal 1rsb solution which has the maximum free energy non negative entropy and satisfies the stability condition the at criterion for the 1rsb solution the transition from rs to 1rsb is continuous or discontinuous depending on whether the rs at line is above or below the zero entropy line however for a relatively large number of examples the 1rsb solution which maximizes the free energy becomes unstable and should be replaced by higher step rsb solutions we also obtain the at line for the 1rsb solution

on line learning in a discrete state space on line learning of a rule given by an n dimensional ising perceptron is considered for the case when the student is constrained to take values in a discrete state space of size for l 2 no on line algorithm can achieve a finite overlap with the teacher in the thermodynamic limit however if l is on the order of hebbian learning does achieve a finite overlap

a computational approach to edge detection this paper describes a computational approach to edge detection the success of the approach depends on the definition of a comprehensive set of goals for the computation of edge points these goals must be precise enough to delimit the desired behavior of the detector while making minimal assumptions about the form of the solution we define detection and localization criteria for a class of edges and present mathematical forms for these criteria as functionals on the operator impulse response a third criterion is then added to ensure that the detector has only one response to a single edge we use the criteria in numerical optimization to derive detectors for several common image features including step edges on specializing the analysis to step edges we find that there is a natural uncertainty principle between detection and localization performance which are the two main goals with this principle we derive a single operator shape which is optimal at any scale the optimal detector has a simple approximate implementation in which edges are marked at maxima in gradient magnitude of a gaussian smoothed image we extend this simple detector using operators of several widths to cope with different signal to noise ratios in the image we present a general method called feature synthesis for the fine to coarse integration of information from operators at different scales finally we show that step edge detector performance improves considerably as the operator point spread function is extended along the edge

shiftable multiscale transforms one of the major drawbacks of orthogonal wavelet transforms is their lack of translation invariance the content of wavelet subbands is unstable under translations of the input signal wavelet transforms are also unstable with respect to dilations of the input signal and in two dimensions rotations of the input signal the authors formalize these problems by defining a type of translation invariance called shiftability in the spatial domain shiftability corresponds to a lack of aliasing thus the conditions under which the property holds are specified by the sampling theorem shiftability may also be applied in the context of other domains particularly orientation and scale jointly shiftable transforms that are simultaneously shiftable in more than one domain are explored two examples of jointly shiftable transforms are designed and implemented a 1 d transform that is jointly shiftable in position and scale and a 2 d transform that is jointly shiftable in position and orientation the usefulness of these image representations for scale space analysis stereo disparity measurement and image enhancement is demonstrated

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

existence of solutions for neutral functional differential evolution equations with nonlocal conditions abstract in this paper by using fractional power of operators and sadovskii s fixed point theorem we study the existence of mild and strong solutions of semilinear neutral functional differential evolution equations with nonlocal conditions the results we obtained are a generalization and continuation of the recent results on this issue in the end an example is given to show the application of our results

adaptive pseudo dilation for gestalt edge grouping and contour detection we consider the problem of detecting object contours in natural images in many cases local luminance changes turn out to be stronger in textured areas than on object contours therefore local edge features which only look at a small neighborhood of each pixel cannot be reliable indicators of the presence of a contour and some global analysis is needed we introduce a new morphological operator called adaptive pseudo dilation apd which uses context dependent structuring elements in order to identify long curvilinear structure in the edge map we show that grouping edge pixels as the connected components of the output of apd results in a good agreement with the gestalt law of good continuation the novelty of this operator is that dilation is limited to the voronoi cell of each edge pixel an efficient implementation of apd is presented the grouping algorithm is then embedded in a multithreshold contour detector at each threshold level small groups of edges are removed and contours are completed by means of a generalized reconstruction from markers the use of different thresholds makes the algorithm much less sensitive to the values of the input parameters both qualitative and quantitative comparison with existing approaches prove the superiority of the proposed contour detector in terms of larger amount of suppressed texture and more effective detection of low contrast contours

decision theoretic generalizations of the pac model for neural net and other learning applications we describe a generalization of the pac learning model that is based on statistical decision theory in this model the learner receives randomly drawn examples each example consisting of an instance x in x and an outcome y in y and tries to find a hypothesis h x a where h in h that specifies the appropriate action a in a to take for each instance x in order to minimize the expectation of a loss l y a here x y and a are arbitrary sets l is a real valued function and examples are generated according to an arbitrary joint distribution on x times y special cases include the problem of learning a function from x into y the problem of learning the conditional probability distribution on y given x regression and the problem of learning a distribution on x density estimation we give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces h and show how this implies learnability with bounded sample size disregarding computational complexity as an application we give distribution independent upper bounds on the sample size needed for learning with feedforward neural networks our theorems use a generalized notion of vc dimension that applies to classes of real valued functions adapted from pollard s work and a notion of capacity and metric dimension for classes of functions that map into a bounded metric space supersedes 89 30 and 90 52 also in information and computation vol 100 no 1 september 1992

how to use expert advice we analyze algorithms that predict a binary value by combining the predictions of several prediction strategies called experts our analysis is for worst case situations i e we make no assumptions about the way the sequence of bits to be predicted is generated we measure the performance of the algorithm by the difference between the expected number of mistakes it makes on the bit sequence and the expected number of mistakes made by the best expert on this sequence where the expectation is taken with respect to the randomization in the predictins we show that the minimum achievable difference is on the order of the square root of the number of mistakes of the best expert and we give efficient algorithms that achieve this our upper and lower bounds have matching leading constants in most cases we then show how this leads to certain kinds of pattern recognition learning algorithms with performance bounds that improve on the best results currently know in this context we also compare our analysis to the case in which log loss is used instead of the expected number of mistakes

stochastic dynamics of supervised learning the stochastic evolution of adiabatic slow backpropagation training of a neural network is discussed and a fokker planck equation for the post training distribution function in the network space is derived the distribution obtained differs from the one given by radons et al 1990 studying the character of the post training distribution the authors find that except under very special circumstances the distribution will be non gibbsian the validity of the present approach is tested on a simple backpropagation learning system in one dimension which can be solved analytically as well implications of the fokker planck approach for general situations are examined in the local linear approximation surprisingly they find that the post training distribution is isotropic close to its peak hence simpler than the corresponding gibbs distribution

generalization in a two layer neural network generalization in a fully connected two layer neural network with n input nodes m hidden nodes a single output node and binary weights is studied in the annealed approximation when the number of examples is the order of n the generalization error approaches a plateau and the system is in a permutation symmetric phase when the number of examples is of the order of mn the system undergoes a first order phase transition to perfect generalization and the permutation symmetry breaks results of the computer simulation show good agreement with analytic calculation

statistical theory of learning curves under entropic loss criterion the present paper elucidates a universal property of learning curves which shows how the generalization error training error and the complexity of the underlying stochastic machine are related and how the behavior of a stochastic machine is improved as the number of training examples increases the error is measured by the entropic loss it is proved that the generalization error converges to h0 the entropy of the conditional distribution of the true machine as h0 m 2t while the training error converges as h0 m 2t where t is the number of examples and m shows the complexity of the network when the model is faithful implying that the true machine is in the model m is reduced to m the number of modifiable parameters this is a universal law because it holds for any regular machine irrespective of its structure under the maximum likelihood estimator similar relations are obtained for the bayes and gibbs learning algorithms these learning curves show the relation among the accuracy of learning the complexity of a model and the number of training examples

average case learning curves for radial basis function networks the application of statistical physics to the study of the learning curves of feedforward connectionist networks has to date been concerned mostly with perceptron like networks recent work has extended the theory to networks such as committee machines and parity machines and an important direction for current and future research is the extension of this body of theory to further connectionist networks in this article we use this formalism to investigate the learning curves of gaussian radial basis function networks rbfns having fixed basis functions these networks have also been called generalized linear regression models we address the problem of learning linear and nonlinear realizable and unrealizable target rules from noise free training examples using a stochastic training algorithm expressions for the generalization error defined as the expected error for a network with a given set of parameters are derived for general gaussian rbfns for which all parameters including centers and sp