properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations rectified linear units relus have become the main model for the neural units in current deep learning systems this choice was originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent learning in networks composed of multiple layers here we provide analytical results on the effects of relus on the capacity and on the geometrical landscape of the solution space in two layer neural networks with either binary or real valued weights we study the problem of storing an extensive number of random patterns and find that quite unexpectedly the capacity of the network remains finite as the number of neurons in the hidden layer increases at odds with the case of threshold units in which the capacity diverges possibly more important a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure while the majority of solutions are close in distance but still isolated there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units these solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs the analytical results are corroborated by numerical findings

generalization and capacity of extensively large two layered perceptrons the generalization ability and storage capacity of a treelike two layered neural network with a number of hidden units scaling as the input dimension is examined the mapping from the input to the hidden layer is via boolean functions the mapping from the hidden layer to the output is done by a perceptron the analysis is within the replica framework where an order parameter characterizing the overlap between two networks in the combined space of boolean functions and hidden to output couplings is introduced the maximal capacity of such networks is found to scale linearly with the logarithm of the number of boolean functions per hidden unit the generalization process exhibits a first order phase transition from poor to perfect learning for the case of discrete hidden to output couplings the critical number of examples per input dimension ensuremath alpha c at which the transition occurs again scales linearly with the logarithm of the number of boolean functions in the case of continuous hidden to output couplings the generalization error decreases according to the same power law as for the perceptron with the prefactor being different

a harmonic retrieval framework for discontinuous motion estimation motion discontinuities arise when there are occlusions or multiple moving objects in the scene that is imaged conventional regularization techniques use smoothness constraints but are not applicable to motion discontinuities in this paper we show that discontinuous or multiple motion estimation can be viewed as a multicomponent harmonic retrieval problem from this viewpoint a number of established techniques for harmonic retrieval ran be applied to solve the challenging problem of discontinuous or multiple motion compared with existing techniques the resulting algorithm is not iterative which not only implies computational efficiency but also obviates concerns regarding convergence or local minima it also adds flexibility to spatio temporal techniques which have suffered from lack of explicit modeling of discontinuous motion experimental verification of our framework on both synthetic data as well as real image data is provided

the statistical mechanics of constructive algorithms the storage capacity of multilayer networks with overlapping receptive fields is studied for constructive algorithms using boolean perceptrons as their basic building block which have been investigated within a replica framework the assumption of weak coupling between subsequently constructed perceptrons is verified within a replica symmetric rs ansatz and shown to be negligible in most cases in comparison with correction due to replica symmetry breaking rsb in individual perceptrons the capacities of a tiling like and variants of the upstart algorithm are then calculated within rs and one step rsb with the quenched average taken over the individual units separately for networks with up to k 4000 and k 600 units respectively within this treatment the storage capacity seems to exhibit a power law behaviour in with an exponent n that may depend on the algorithm and the stability however due to finite size effects in k reliable estimates of n could not be extracted nevertheless the results strongly indicate that n should be strictly smaller than 1 within one step rsb whereas within rs the mitchison durbin bound is violated for finite k and n 1 may hold asymptotically

generalization properties of multilayered neural networks generalization properties of multilayered neural networks with binary couplings are studied in the high temperature limit the transition to the perfect generalization phase is evaluated for systems with an arbitrary number of layers it is found that the thermodynamic transition occurs for a number of examples lower than for the perceptron but the opposite occurs for the transition in which the poor generalization solution disappears the generalization error is also decomposed according to the contributions coming from different numbers of hidden neurons that have a wrong sign in the internal field this allows the authors to describe the generalization behaviour of the hidden neurons

a fast scalable algorithm for discontinuous optical flow estimation multiple moving objects partially occluded objects or even a single object moving against the background gives rise to discontinuities in the optical flow field in corresponding image sequences while uniform global regularization based moderately fast techniques cannot provide accurate estimates of the discontinuous flow field statistical optimization based accurate techniques suffer from excessive solution time a weighted anisotropic smoothness based numerically robust algorithm is proposed that can generate discontinuous optical flow field with high speed and linear computational complexity weighted sum of the first order spatial derivatives of the flow field is used for regularization less regularization is performed where strong gradient information is available the flow field at any point is interpolated more from those at neighboring points along the weaker intensity gradient component such intensity gradient weighted regularization leads to euler lagrange equations with strong anisotropies coupled with discontinuities in their coefficients a robust multilevel iterative technique that recursively generates coarse level problems based on intensity gradient weighted smoothing weights is employed to estimate discontinuous optical flow field experimental results are presented to demonstrate the efficacy of the proposed technique

categorization and effective perceptron learning in feed forward neural networks in the initial stages of back propagation learning weights in the first synaptic layer are small and hidden neurons operate in the linear regime so the input output map is then the same as that of a perceptron as the weights increase the hidden neurons begin to saturate each saturation pattern represents a particular category of input vectors for which the neural network nn locally behaves as a perceptron the effective perceptron weights for each category are functions of the synaptic weights of the nn we define an internal temperature as the inverse of the norm of the first synaptic matrix and an entropy function that measures the disorder of the hidden neuron layer the learning curves are recast in the language of a condensation process the entropy drops abruptly when the hidden neurons begin to saturate marking a clear transition from the global perceptron state to an ordered state characterized by different local effective perceptrons in different categories the problem of optimizing the effective perceptrons in different categories is frustrated

a fully connected committee machine learning unrealizable rules we study generalization in a large fully connected committee machine with continuous weights trained on patterns with outputs generated by a teacher of the same structure but corrupted by noise the corruption is due to additive gaussian noise applied in the input layer or the hidden layer of the teacher contrary to related cases in the presence of input noise the generalization error epsilon g is not minimized by the teacher s weights for small values of the load parameter alpha the student is in a permutation symmetric phase as alpha increases three additional phases emerge the large alpha theory of the stable phase is similar to the tree committee machine in particular at zero temperature in the presence of noise epsilon g does not approach its minimal value epsilon min and the student s weights do not converge to those of the teacher for a positive temperature epsilon g epsilon min decays as a power of alpha the exponent being the same as in the corresponding case of the tree however for all values of alpha an at least metastable phase exists which is permutation symmetric with respect to the teacher

learning and generalization in radial basis function networks the two layer radial basis function network with fixed centers of the basis functions is analyzed within a stochastic training paradigm various definitions of generalization error are considered and two such definitions are employed in deriving generic learning curves and generalization properties both with and without a weight decay term the generalization error is shown analytically to be related to the evidence and via the evidence to the prediction error and free energy the generalization behavior is explored the generic learning curve is found to be inversely proportional to the number of training pairs presented optimization of training is considered by minimizing the generalization error with respect to the free parameters of the training algorithms finally the effect of the joint activations between hidden layer units is examined and shown to speed training

robust filtering for 2 d discrete time linear systems with convex bounded parameter uncertainty this paper is concerned with the problems of robust h and h 2 filtering for 2 dimensional 2 d discrete time linear systems described by a fornasini marchesini second model with matrices that depend affinely on convex bounded uncertain parameters by a suitable transformation the system is represented by an equivalent difference algebraic representation a parameter dependent lyapunov function approach is then proposed for the design of 2 d stationary discrete time linear filters that ensure either a prescribed h performance or h 2 performance for all admissible uncertain parameters the filter designs are given in terms of linear matrix inequalities numerical examples illustrate the effectiveness of the proposed filter design methods

computational capabilities of multilayer committee machines we obtained an analytical expression for the computational complexity of many layered committee machines with a finite number of hidden layers l 8 using the generalization complexity measure introduced by franco et al 2006 ieee trans neural netw 17 578 although our result is valid in the large size limit and for an overlap synaptic matrix that is ultrametric it provides a useful tool for inferring the appropriate architecture a network must have to reproduce an arbitrary realizable boolean function

vapnik chervonenkis dimension bounds for two and three layer networks we show that the vapnik chervonenkis dimension of the class of functions that can be computed by arbitrary two layer or some completely connected three layer threshold networks with real inputs is at least linear in the number of weights in the network in valiant s probably approximately correct learning framework this implies that the number of random training examples necessary for learning in these networks is at least linear in the number of weights

a simple scheme for motion boundary detection abstract a new approach is developed for detecting motion boundaries and independent motion without explicit estimation of the motion field the formulation is based on the differential optical flow equation and uses a piecewise linear model for the image intensity data compared with previous techniques our approach has greater simplicity and makes less specific assumptions on the motion field experimental results on real image sequences and real time implementation are included

learning in ultrametric committee machines the problem of learning by examples in ultrametric committee machines ucms is studied within the framework of statistical mechanics using the replica formalism we calculate the average generalization error in ucms with l hidden layers and for a large enough number of units in most of the regimes studied we find that the generalization error as a function of the number of examples presented develops a discontinuous drop at a critical value of the load parameter we also find that when l 1 a number of teacher networks with the same number of hidden layers and different overlaps induce learning processes with the same critical points

robust learning and generalization with support vector machines in this paper we study the typical learning properties of the recently proposed support vector machines svms the generalization error on linearly separable tasks the capacity the typical number of support vectors the margin and the robustness or noise tolerance of a class of svms are determined in the framework of statistical mechanics the robustness is shown to be closely related to the generalization properties of these machines

bayesian estimation of motion vector fields a stochastic approach to the estimation of 2d motion vector fields from time varying images is presented the formulation involves the specification of a deterministic structural model along with stochastic observation and motion field models two motion models are proposed a globally smooth model based on vector markov random fields and a piecewise smooth model derived from coupled vector binary markov random fields two estimation criteria are studied in the maximum a posteriori probability map estimation the a posteriori probability of motion given data is maximized whereas in the minimum expected cost mec estimation the expectation of a certain cost function is minimized both algorithms generate sample fields by means of stochastic relaxation implemented via the gibbs sampler two versions are developed one for a discrete state space and the other for a continuous state space the map estimation is incorporated into a hierarchical environment to deal efficiently with large displacements

anti pairing in learning of a neural network with redundant hidden units we study the statistical mechanics of learning from examples between the two layered committee machines with different numbers of hidden units using the replica theory the number m of hidden units of the student network is larger than the number mt of those of the target network called the teacher we choose the networks to have binary synaptic weights 1 which makes it possible to compare the calculation with the monte carlo simulation we propose an effective teacher as a virtual target network which has the same m hidden units as the student and gives identical outputs with those of the original teacher this is a way of making a conjecture for a ground state of a thermodynamic system given by the weights of the effective teacher in our study we suppose that the weights on mt hidden units of the effective teacher are the same as those of the original teacher while those on m mt redundant hidden units are composed of anti pairs 1 1 with probability 1 p in the limit p 0 for p 0 exact there are no terms related to the effective teacher in the calculation for the contributions of anti pairs to outputs are exactly cancelled in the limit p 0 however we find that the learnt weights of the student are actually equivalent to those of the suggested effective teacher which is not possible from the calculation for p 0 p plays the role of a symmetry breaking parameter for anti pairing ordering which is analogous to the magnetic field for the ising model a first order phase transition is found to be signalled by breaking of symmetry in permuting hidden units above a critical number of examples the student is shown to learn perfectly the effective teacher anti pairing can be measured by a set of order parameters zero in the permutation symmetric phase and nonzero in the permutation symmetry breaking phase results from the monte carlo simulation are shown to be in good agreement with those from the replica calculation

replica symmetry breaking in perceptrons in the problem of optimization of pattern stabilization in perceptrons the replica symmetric ansatz is known to be mathematically unstable for storage capacities greater than some in this paper we demonstrate that for greater than the one step replica symmetry broken rsb solution is also unstable we further show that in this region full rsb is necessary for an exact solution direct evaluation of the two step rsb solution yields a minimum storage error which is only slightly greater than the one step rsb which itself is greater than that given by the unstable replica symmetric ansatz by a much larger amount

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

beyond storage capacity in a single model neuron continuous replica symmetry breaking a single mcculloch pitts neuron that is the simple perceptron is studied with focus on the region beyond storage capacity it is shown that parisi s hierarchical ansatz for the overlap matrix of the synaptic couplings with so called continuous replica symmetry breaking is a solution and as we propose it is the exact one to the equilibrium problem we describe some of the most salient features of the theory and give results about the low temperature region in particular the basics of the parisi technique and the way to calculate thermodynamical expectation values is explained we have numerically extremized the replica free energy functional for some parameter settings and thus obtained the order parameter function i e the probability distribution of overlaps that enabled us to evaluate the probability density of the local stability parameter we also performed a simulation and found a local stability density closer to the theoretical curve than previous numerical results were

learning and generalization in a two layer neural network the role of the vapnik chervonvenkis dimension bounds for the generalization ability of neural networks based on vapnik chervonenkis vc theory are compared with statistical mechanics results for the case of the parity machine for fixed phase space dimension the vc dimension can grows arbitrarily by increasing the number k of hidden units generalization is impossible up to a critical number of training examples that grows with the vc dimension the asymptotic decrease of the generalization error mathrm ensuremath varepsilon mathit g comes out independent of k and the vc bounds strongly overestimate mathrm ensuremath varepsilon mathit g this shows that phase space dimension and vc dimension can play independent and different roles for the generalization process

storage capacity of a constructive learning algorithm upper and lower bounds for the typical storage capacity of a constructive algorithm the tilinglike learning algorithm for the parity machine biehl m and opper m 1991 phys rev a 44 6888 are determined in the asymptotic limit of large training set sizes the properties of a perceptron with threshold learning a training set of patterns having a biased distribution of targets needed as an intermediate step in the capacity calculation are determined analytically the lower bound for the capacity determined with a cavity method is proportional to the number of hidden units the upper bound obtained with the hypothesis of replica symmetry is close to the one predicted by mitchinson and durbin 1989 biol cybern 60 345

learning a rule in a multilayer neural network the problem of learning from examples in multilayer networks is studied within the framework of statistical mechanics using the replica formalism we calculate the average generalization error of a fully connected committee machine in the limit of a large number of hidden units if the number of training examples is proportional to the number of inputs in the network the generalization error as a function of the training set size approaches a finite value if the number of training examples is proportional to the number of weights in the network we find first order phase transitions with a discontinuous drop in the generalization error for both binary and continuous weights

weight space structure and generalization in the reversed wedge perceptron the generalization ability of the reversed wedge perceptron serving as a toy model for multilayer neural networks is investigated we analyse the decomposition of the version space into disjoint cells belonging to different internal representations defined by the signs of the aligning fields the version space is characterized by the number and size of these cells and their typical overlap with the teacher network for a small training set the system is unable to detect the structure of the patterns induced by the teacher accordingly it performs as if storing random input output patterns with very low generalization ability and a large misfit in the internal representation with increasing training set size cells with large misfit are eliminated at a much higher rate than those with internal representation similar to that of the teacher this results eventually in the discontinuous phase transition to good generalization typical for multilayer neural networks

phase transitions in the generalization behaviour of multilayer perceptrons ii the influence of noise we extend our study of phase transitions in the generalization behaviour of multilayer perceptrons with non overlapping receptive fields to the problem of the influence of noise concerning for example the input units and or the couplings between the input units and the hidden units of the second layer input noise or the final output unit output noise without output noise the output itself is given by a general permutation invariant boolean function of the outputs of the hidden units as a result we find that the phase transitions which we found in the deterministic case mostly persist in the presence of noise the influence of the noise on the position of the phase transition as well as on the behaviour in other regimes of the loading parameter can often be described by a simple rescaling of depending on the strength and type of the noise we then consider the problem of the optimal noise level for gibbsian and bayesian learning looking at replica symmetry breaking as well finally we consider the question of why learning with errors is useful at all

learnability and the vapnik chervonenkis dimension valiant s learnability model is extended to learning classes of concepts defined by regions in euclidean space e n the methods in this paper lead to a unified treatment of some of valiant s results along with previous results on distribution free convergence of certain pattern recognition algorithms it is shown that the essential condition for distribution free learnability is finiteness of the vapnik chervonenkis dimension a simple combinatorial parameter of the class of concepts to be learned using this parameter the complexity and closure properties of learnable classes are analyzed and the necessary and sufficient conditions are provided for feasible learnability

noisy regression and classification with continuous multilayer networks we investigate zero temperature gibbs learning for two classes of unrealizable rules which play an important role in practical applications of multilayer neural networks with differentiable activation functions classification problems and noisy regression problems considering one step of replica symmetry breaking we surprisingly find that for sufficiently large training sets the stable state is replica symmetric even though the target rule is unrealizable furthermore the classification problem is shown to be formally equivalent to the noisy regression problem

an error weighted regularization algorithm for image motion field estimation local motion measurement errors are used to guide the global smoothing process in order to preserve motion field discontinuities a field smoothing algorithm based on matching error weighting is proposed the added computation is minimal since it uses byproducts of the local measurement process the error weighting functional provides significantly improved motion field estimates as measured by motion compensated interpolation performance however the mean square reconstruction error is somewhat higher than that obtained by performing the much more computationally expensive stochastic optimization

analytical and numerical study of internal representations in multilayer neural networks with binary weights we study the weight space structure of the parity machine with binary weights by deriving the distribution of volumes associated to the internal representations of the learning examples the learning behavior and the symmetry breaking transition are analyzed and the results are found to be in very good agreement with the extended numerical simulations s1063 651x 96 01207 x

h sub spl infin filtering of 2 d discrete systems this paper deals with h sub spl infin filtering of two dimensional 2 d linear discrete systems described by a 2 d local state space lss fornasini marchesini 1978 second model several versions of the bounded real lemma of the 2 d discrete systems are established the 2 d bounded real lemma allows us to solve the finite horizon and infinite horizon h sub spl infin filtering problems using a riccati difference equation or a riccati inequality approach further a solution to the infinite horizon h sub spl infin filtering problem based on a linear matrix inequality lmi approach is developed our results extend existing work for one dimensional 1 d systems to the 2 d case and give a state space solution to the bounded realness of 2 d discrete systems as well as 2 d h sub spl infin filtering for the first time numerical examples are given to demonstrate the riccati difference equation approach to the 2 d finite horizon h sub spl infin filtering problem and the lmi approach to the 2 d infinite horizon h sub spl infin filtering problem

storage capacity of the fully connected committee machine the storage capacity that is the number of patterns which can be stored per weight is calculated for the fully connected committee machine with real couplings and k hidden units from the vanishing of the entropy of the internal representations and it is found to diverge as

an investigation of smoothness constraints for the estimation of displacement vector fields from image sequences a mapping between one frame from an image sequence and the preceding or following frame can be represented as a displacement vector field in most situations the mere gray value variations do not provide sufficient information in order to estimate such a displacement vector field supplementary constraints are necessary for example the postulate that a displacement vector field varies smoothly as a function of the image position taken as a general requirement this creates difficulties at gray value transitions which correspond to occluding contours nagel therefore introduced the oriented smoothness requirement which restricts variations of the displacement vector field only in directions with small or no variation of gray values this contribution reports results of an investigation about how such an oriented smoothness constraint may be formulated and evaluated

storage capacity of ultrametric committee machines the problem of computing the storage capacity of a feed forward network with l hidden layers n inputs and k units in the first hidden layer is analyzed using techniques from statistical mechanics we found that the storage capacity strongly depends on the network architecture c log k 1 1 2l and that the number of units k limits the number of possible hidden layers l through the relationship 2l 1 2log k

learning in feedforward layered networks the tiling algorithm the authors propose a new algorithm which builds a feedforward layered network in order to learn any boolean function of n boolean units the number of layers and the number of hidden units in each layer are not prescribed in advance they are outputs of the algorithm it is an algorithm for growth of the network which adds layers and units inside a layer at will until convergence the convergence is guaranteed and numerical tests of this strategy look promising

phase transitions in the generalization behaviour of multilayer neural networks we study the generalization ability of multilayer neural networks with tree architecture for the case of random training sets the first layer consists of k spherical perceptrons with binary output a boolean function b computes the final output from the k values produced by the first layer we first calculate the learning behaviour of gibbs learning in the case of learnable rules where teacher and student have the same architecture and the boolean function b is permutation symmetric with respect to the hidden units in the asymptotic case of high loading alpha to infinity as usual alpha is the loading parameter we find that the generalization error vanishes in the same way for all b the reason being is that there are two effects cancelling each other in the opposite limit for small alpha we find qualitatively different behaviour i e some networks undergo a phase transition we show how these differences in the behaviour depend on certain characteristics of the boolean function b we then study the bayes algorithm which generalizes according to the majority decision of a certain ensemble of machines and find that the parity function plays a special role if the teacher is a parity machine there always exists a single student parity machine that generalizes as well as the bayes algorithm

polynomial solution of 2d kalman bucy filtering problem the 2d version of the kalman bucy filtering problem is formulated and then solved via 2d polynomial methods the optimal filter is restricted to be a linear causal system the design procedure is shown to consist of one 2d spectral factorization equation only in fact it works for n d signals n 2 as well

on line learning in parity machines the optimal algorithm for on line learning in the tree k parity machine is studied we introduce a set of recursion relations for the relevant probability distributions which permit study of the general k case the generalization error curve is determined and shown to decay to zero for large as even in the presence of noise there is no critical noise level the dynamics of on line learning is studied analytically near the origin in the absence of previous knowledge the learning dynamics has a fixed point at previous knowledge is needed in at least k 1 branches for the learning to take place

explicit symmetries and the capacity of multilayer neural networks calculating the capacity and generalization capabilities of feed forward multilayer neural networks requires the use of replica symmetry breaking methods making the calculation practically unfeasible replica symmetry is broken because the configuration space is disconnected which is clearly the case in the capacity limit where the configuration space shrinks to isolated points moreover there is no knowledge about the number of replica symmetry breaking steps required to obtain reliable results novel approaches to tackle the capacity calculation of feed forward neural networks avoiding the use of replica symmetry breaking methods are presented in this paper the basic idea behind these approaches is that breaking explicit symmetries of the network prior to the capacity calculation itself restores order parameter symmetry at least to a good approximation and therefore enables the use of the replica symmetry ansatz two methods are presented for breaking the explicit symmetries and restoring replica symmetry one restricts relations between the various weight elements while the other restricts the values of the order parameters these methods which are demonstrated in this work via the capacity calculation of feed forward neural networks are applicable to a variety of capacity learning and generalization capability calculations of such nets we examine an approximation for carrying out the multi dimensional gaussian integrals appearing during the calculation as well as exact results for some simple cases numerical results obtained for nets with one to six hidden neurons using the downhill simplex and adaptive simulated annealing optimization algorithms are in good agreement with simulation results

learning by on line gradient descent we study on line gradient descent learning in multilayer networks analytically and numerically the training is based on randomly drawn inputs and their corresponding outputs as defined by a target rule in the thermodynamic limit we derive deterministic differential equations for the order parameters of the problem which allow an exact calculation of the evolution of the generalization error first we consider a single layer perceptron with sigmoidal activation function learning a target rule defined by a network of the same architecture for this model the generalization error decays exponentially with the number of training examples if the learning rate is sufficiently small however if the learning rate is increased above a critical value perfect learning is no longer possible for architectures with hidden layers and fixed hidden to output weights such as the parity and the committee machine we find additional effects related to the existence of symmetries in these problems

a computational framework and an algorithm for the measurement of visual the robust measurement of visual motion from digitized image sequences has been an important but difficult problem in computer vision this paper describes a hierarchical computational framework for the determination of dense displacement fields from a pair of images and an algorithm consist ent with that framework our framework is based on the separation of the image intensity information as well as the process of measuring motion according to scale the large scale intensity information is first used to obtain rough estimates of image motion which are then refined by using intensity information at smaller scales the estimates are in the form of displacement or velocity vectors for pixels and are accompanied by a direction dependent confidence measure a smoothness constraint is employed to propagate the measurements with high confidence to their neigboring areas where the confidences are low at all levels the computations are pixel parallel uniform across the image and based on information from a small neighborhood of a pixel for our algorithm the local displacement vectors are determind by mini mizing the sum of squared differences ssd of intensities the confidence measures are derived from the shape of the ssd surface and the smoothness constraint is cast in the form of energy minimization results of applying our algorithm to pairs of real images are included in addition to our own

a thermal perceptron learning rule the thermal perceptron is a simple extension to rosenblatt s perceptron learning rule for training individual linear threshold units it finds stable weights for nonseparable problems as well as separable ones experiments indicate that if a good initial setting for a temperature parameter to has been found then the thermal perceptron outperforms the pocket algorithm and methods based on gradient descent the learning rule stabilizes the weights learns over a fixed training period for separable problems it finds separating weights much more quickly than the usual rules

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

a large committee machine learning noisy rules statistical mechanics is used to study generalization in a tree committee machine with k hidden units and continuous weights trained on examples generated by a teacher of the same structure but corrupted by noise the corruption is due to additive gaussian noise applied in the input layer or the hidden layer of the teacher in the large k limit the generalization error eg as function of the number of patterns per adjustable parameter shows a qualitatively similar behavior for the two cases it does not approach its optimal value and is nonmonotonic if training is done at zero temperature this remains true even when replica symmetry breaking is taken into account training at a fixed positive temperature leads within the replica symmetric theory to an k decay of eg toward its optimal value the value of k is calculated and found to depend on the model of noise by scaling the temperature with the value of k can be increased to an optimal value kopt however at one step of replica symmetry breaking at a fixed positive temperature eg decays as kopt so although eg will approach its optimal value with increasing sample size for any fixed k the convergence is only uniform in k when training at a positive temperature

low temperature properties of directed walks with random self interactions we consider a simple model for a chain with random self interactions each monomer on the chain carries a charge which is a random variable and the configurations of the chain are directed walks we show that at zero temperature the end to end distance r and the energy e of chains with total charge q approximately nx scale as r approximately nnu x and e approximately ntheta x the exponents thus depend on the total charge through the parameter x these exponents are measured numerically and compared to the predictions of some simple arguments at a finite temperature r is extensive and appears to be independent of x

diophantine equations in control a survey abstract this survey is also a tutorial whose aim is to explain the role of diophantine equations in the synthesis of feedback control systems these are linear equations in a ring and result from a fractional representation of the systems involved the cornerstone of the exposition is a simple parametrization of all stabilizing controllers for a given plant one can then choose in principle the best controllers for various applications these ideas evolved from early attempts to use polynomial equations in the design of discrete time linear systems by now they have been extended to continuous time infinite dimensional time varying and non linear systems

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

simplifying neural networks by soft weight sharing one way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of nonzero weights we propose a more complicated penalty term in which the distribution of weight values is modeled as a mixture of multiple gaussians a set of weights is simple if the weights have high probability density under the mixture model this can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values since we do not know the appropriate means or variances of the clusters in advance we allow the parameters of the mixture model to adapt at the same time as the network learns simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms

unsupervised learning of binary vectors a gaussian scenario we study a model of unsupervised learning where the real valued data vectors are isotropically distributed except for a single symmetry breaking binary direction bin 1 1 n onto which the projections have a gaussian distribution we show that a candidate vector j undergoing gibbs learning in this discrete space approaches the perfect match j b exponentially in addition to the second order retarded learning phase transition for unbiased distributions we show that first order transitions can also occur extending the known result that the center of mass of the gibbs ensemble has bayes optimal performance we show that taking the sign of the components of this vector clipping leads to the vector with optimal performance in the binary space these upper bounds are shown generally not to be saturated with the technique of transforming the components of a special continuous vector except in asymptotic limits and in a special linear case simulations are presented which are in excellent agreement with the theoretical results

learning the unlearnable three neural network training algorithms are presented which are robust to nonlearnable problems the first algorithm converges to the gardner stability limit if the learning problem is linearly separable and otherwise finds a locally maximally stable solution the second algorithm is a robust version of rosenblatt s perceptron learning algorithm which will converge to a solution of the learning problem if one exists and otherwise will converge locally to a solution with a certain fraction of wrongly mapped patterns the third algorithm is suited most favourably to unlearnable problems it will always find a solution if the problem is learnable and otherwise it locally maximizes the number of patterns which are stored correctly the error rate of this algorithm and other known algorithms for unlearnable problems are compared for two benchmark problems proofs of the existence of solutions are given convergence is proven as well to be global in the case of learnable and local in unlearnable cases

learning by dilution in a neural network a perceptron with n random weights can store of the order of n patterns by removing a fraction of the weights without changing their strengths the critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated a simple hebb like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability

polynomial approach to wiener filtering a rigorous derivation is given of the polynomial method for finding wiener filters in both discrete time and continuous time in each case the method involves one spectral factorization and the solution of one diophantine equation the theory is applicable in multivariable situations but simple scalar examples are included here to compare the polynomial approach with the basic wiener method

threshold induced phase transitions in perceptrons error rates of a boolean perceptron with threshold and either spherical or ising constraint on the weight vector are calculated for storing patterns from biased input and output distributions derived within a one step replica symmetry breaking rsb treatment for unbiased output distribution and non zero stability of the patterns we find a critical load p above which two solutions to the saddlepoint equations appear one with higher free energy and zero threshold and a dominant solution with non zero threshold we examine this second order phase transition and the dependence of p on the required pattern stability for both one step rsb and replica symmetry rs in the spherical case and for one step rsb in the ising case

weight decay induced phase transitions in multilayer neural networks we investigate layered neural networks with differentiable activation function and student vectors without normalization constraint by means of equilibrium statistical physics we consider the learning of perfectly realizable rules and find that the length of student vectors becomes infinite unless a proper weight decay term is added to the energy then the system undergoes a first order phase transition between states with very long student vectors and states where the lengths are comparable to those of the teacher vectors additionally in both configurations there is a phase transition between a specialized and an unspecialized phase an anti specialized phase with long student vectors exists in networks with a small number of hidden units

polynomial systems approach to optimal linear filtering and prediction the solution of the optimal linear estimation problem is considered using a polynomial matrix description for the discrete system the filter or predictor is given by the solution of two diophantine equations and is equivalent to the state equation form of the steady state kalman filter or the transfer function matrix form of the wiener filter the pole zero properties of the optimal filter are more obvious in the polynomial representation and new insights into the disturbance rejection properties of the filter are obtained the plant or signal model can be stable or unstable and allowance is made for both control and disturbance input subsystems and white and coloured measurement noise or an output disturbance subsystem the model structure was determined by the needs of several industrial filtering problems the polynomial form of filter may easily be included in a self tuning algorithm and a simple adaptive estimator is described

neural nets with superlinear vc dimension it has been known for quite a while that the vapnik chervonenkis dimension vc dimension of a feedforward neural net with linear threshold gates is at most o w log w where w is the total number of weights in the neural net we show in this paper that this bound is in fact asymptotically optimal more precisely we exhibit for any depth d 3 a large class of feedforward neural nets of depth d with w weights that have vc dimension w log w this lower bound holds even if the inputs are restricted to boolean values the proof of this result relies on a new method that allows us to encode more program bits in the weights of a neural net than previously thought possible

generalization in a two layer neural network generalization in a fully connected two layer neural network with n input nodes m hidden nodes a single output node and binary weights is studied in the annealed approximation when the number of examples is the order of n the generalization error approaches a plateau and the system is in a permutation symmetric phase when the number of examples is of the order of mn the system undergoes a first order phase transition to perfect generalization and the permutation symmetry breaks results of the computer simulation show good agreement with analytic calculation

learning from examples in large neural networks a statistical mechanical theory of learning from examples in layered networks at finite temperature is studied when the training error is a smooth function of continuously varying weights the generalization error falls off asymptotically as the inverse number of examples by analytical and numerical studies of single layer perceptrons we show that when the weights are discrete the generalization error can exhibit a discontinuous transition to perfect generalization for intermediate sizes of the example set the state of perfect generalization coexists with a metastable spin glass state