denseness of the generalized eigenvectors of an h s discrete operator let t be a closed densely defined linear operator in a hilbert space h and assume there exists 0 p t such that r 0 t is a hilbert schmidt operator the operator t is a special type of discrete operator a so called h s discrete operator which is shown to be a fredholm operator in h with fredholm set equal to the whole complex plane let t i i 1 where the operator t i has ascent mi let pi i 1 2 be the projection of h onto the generalized eigenspace n t i mi along r t i mi and let s and m be the subspaces of h consisting of all x h such that x i 1 pix and such that pix 0 for i 1 2 respectively sufficient conditions are introduced which guarantee that s h and m 0 these conditions require that r t be bounded on certain rays in the complex plane and r t 0 as on at least one of the rays but specific decay rates for r t are not necessary

storage of sets of correlated data in neural network memories in most applications of neural network models one has to consider sets of correlated data the authors study the problem of the storage of associated patterns in neural network memories they consider two basic types of correlation semantic ones among the different patterns and spatial ones among the different sites of the network they apply gardner s program to evaluate optimal storage conditions for both kinds of correlation the spatial ones worsen generally speaking the storage properties of a simple perceptron while they improve them for the hopfield network in the case of semantic correlations they obtain bounds of the critical capacity for both kinds of networks the storage ratio of the perceptron may be significantly increased in this case

structure of metastable states in the hopfield model an upper bound for the number of metastable states in the hopfield model is calculated as a function of the hamming fraction from an input pattern for all finite values of alpha the ratio of number of patterns to nodes the hamming fraction from the input pattern to the nearest metastable state is infinite when alpha 0 113 the bound also implies that there is a gap between a set of states close to the input pattern and another set centred around the hamming fraction 0 5 from it

thermodynamic properties of fully connected q ising neural networks the thermodynamic and retrieval properties of fully connected q ising networks are studied in the replica symmetric mean field approximation in particular capacity gain parameter and capacity temperature phase diagrams are derived for q 3 4 and q infinity and different distributions of the stored patterns furthermore the optimal gain function is determined in order to obtain the best performance where appropriate the results are compared with the diluted and layered versions of these models

the statistical mechanics of constructive algorithms the storage capacity of multilayer networks with overlapping receptive fields is studied for constructive algorithms using boolean perceptrons as their basic building block which have been investigated within a replica framework the assumption of weak coupling between subsequently constructed perceptrons is verified within a replica symmetric rs ansatz and shown to be negligible in most cases in comparison with correction due to replica symmetry breaking rsb in individual perceptrons the capacities of a tiling like and variants of the upstart algorithm are then calculated within rs and one step rsb with the quenched average taken over the individual units separately for networks with up to k 4000 and k 600 units respectively within this treatment the storage capacity seems to exhibit a power law behaviour in with an exponent n that may depend on the algorithm and the stability however due to finite size effects in k reliable estimates of n could not be extracted nevertheless the results strongly indicate that n should be strictly smaller than 1 within one step rsb whereas within rs the mitchison durbin bound is violated for finite k and n 1 may hold asymptotically

storage capacity of a neural network with state dependent synapses the storage capacity of the hopfield network is limited to p n c 0 138 p number of patterns n number of neurons beyond which the contribution of weakly correlated patterns surpasses that of the desired pattern this contribution can be eliminated by introducing a threshold a pattern correlation below this threshold is simply set to zero in the synapses we solve the mean field equations and derive the critical value of the threshold required to stabilize p n patterns with c

parallel dynamics of the fully connected blume emery griffiths neural network abstract the parallel dynamics of the fully connected blume emery griffiths neural network model is studied at zero temperature using a probabilistic approach a recursive scheme is found determining the complete time evolution of the order parameters taking into account all feedback correlations it is based upon the evolution of the distribution of the local field the structure of which is determined in detail as an illustrative example explicit analytic formula are given for the first few time steps of the dynamics furthermore equilibrium fixed point equations are derived and compared with the thermodynamic approach the analytic results find excellent confirmation in extensive numerical simulations

optimal storage capacity of neural networks at finite temperatures gardner s analysis 1989 of the optimal storage capacity of neural networks is extended to study finite temperature effects the typical volume of the space of interactions is calculated for strongly diluted networks as a function of the storage ratio alpha temperature t and the tolerance parameter m from which the optimal storage capacity alpha c is obtained as a function of t and m at zero temperature it is found that alpha c 2 regardless of m while alpha c in general increases with the tolerance at finite temperatures the authors show how the best performance for given alpha and t is obtained which reveals a first order transition from high quality performance to a low quality one at low temperatures an approximate criterion for recalling which is valid near m 1 is also discussed

spectral theory of two point differential operators determined by d2 i spectral properties abstract the necessary tools for the complete resolution of the spectral theory for all linear two point differential operators l in l2 0 1 determined by d2 and by boundary values b1 b2 are developed all work is based on six numerical parameters defined in terms of the coefficients of b1 b2 these quantities are utilized in the development of the characteristic determinant of l and the green s function of i l and in the development of the projections of l2 0 1 onto the generalized eigenspaces of l they are used in the study of the decay rates of the resolvent operator r l as and they are used to identify the 13 cases treated in developing the spectral theory for d2 all results pertaining to the spectrum l of l to the algebraic multiplicities of the eigenvalues l to the ascents of the operators i l to the boundedness of the family of all finite sums of the projections associated with l to the denseness of the generalized eigenfunctions and to bases consisting of generalized eigenfunctions are summarized in a table the proofs of these results are the subject of the sequel part ii

retrieval phase diagrams for attractor neural networks with optimal interactions the authors consider the retrieval properties of attractor neural networks whose synaptic matrices have been constructed to maximise the number of patterns which can be stored in a perceptron satisfying certain constraints retrieval is studied in the absence as well as in the presence of fast noise temperature the discussion is restricted to dilute networks for which dynamical equations for the overlaps are available when the patterns are stored with a prescribed lower limit on the stability parameter on every site the full stability of the perceptron ensures the existence of an attractor with perfect retrieval it is found that the curve of critical storage capacity alpha against temperature t as a line of first order transitions for high values of alpha and becomes second order for low alpha at a point of a tricritical nature the phase diagram is compared with the dilute hopfield model it is found that at high synaptic noise levels the diluted hopfield net stores more effectively than the network trained for optimal perceptron storage when a given fraction of sites is allowed to violate the stability bound the solution of the perceptron learning problem does not ensure the existence of an attractor of finite overlap even in the absence of noise this case is studied separately for t 0 and for finite t

optimal capacity of graded response perceptrons optimal capacities of perceptrons with graded input output relations are computed within the gardner approach 1988 the influence of desired output precision stability with respect to input errors and output pattern statistics are analysed and discussed

maximum likelihood image estimation using photon correlated beams a theory is presented addressing the fundamental limits of image estimation in a setup that uses two photon correlated beams these beams have the property that their photon arrivals as a point process are ideally synchronized in time and space the true image represents the spatial distribution of the optical transmittance or reflectance of an object in this setup one beam is used to probe the image while the other is used as a reference providing additional information on the actual number of photons impinging on the object this additional information is exploited to reduce the effect of quantum noise associated with the uncertainty in the number of photons per pixel a stochastic model for the joint statistics of the two observation matrices is developed and used to obtain a local maximum likelihood estimator of the image the model captures the nonideal nature of the correlation between the photons of the beams by means of a simple random translation model the mean square error of the estimator is evaluated and compared to the corresponding conventional techniques conditions for the performance advantage of the proposed estimator are examined in terms of key system parameters the theoretical predictions are demonstrated by means of simulation

learning grey toned patterns in neural networks the problem of learning multi state patterns in neural networks is investigated an analysis of the space of couplings gardner approach yields the distribution of local fields the critical storage capacity alpha c and the minimum number of errors for an overloaded network for noisy local fields the classification error is minimized if the local fields of the patterns are allowed to lie in intervals of finite width a fast converging adaptive learning algorithm is presented which finds the coupling matrix of optimal stability

statistical mechanics for neural networks with continuous time dynamics the present paper is intended to summariz our rent knowledge about the long time behaviour of nehwrks of graded response neurons with continuous time dynamics we demonstrate the workings of our previously developed statistical mechanical approach to continuous time dynamics by applying it to networks with various forms of synaptic organization leaming rules and neural composition neuron types as encoded in gain functions as well as to networks varying with respect to the ensemble of stored data unbiased and low activity patterns we present phase diagrams and compute distributions of local fields for a variety of examples local field distributions are found to deviate from the gaussian form obtained for stochastic neumns in the wntexi of the replica approach a solution to the low firing rates problem within the framework of nets of analogue neurons is also briefly discussed finally the statisticalmechanical approach to the analysis of continuowtime dynamics is extended to include effects of fast stochastic noise detailed balance solutions are shown to be unique and of canonical form governed by hamiltonians which exhibit a reciprocity relation between potential dynamics and fuing rate dynamics for the potential dynamics the hamiltonian is given by the lyapounw function of the system expressed in terms of the fuing rates and it generates a gibbs distribution over fuing rate space for the noisy firinerate dynamics the same lyapounov function now expressed in terms of neural potentials generates a gibbs distribution over the space of these potentials as a consequence the firing rate dynamics will freeze in configurations saturating the neural input xtput relations whenever such saturation levels exist both types of stationary distribution are shown to exist only under unrealistic assumptions about the noise in the system

spectral decomposition of a hilbert space by a fredholm operator abstract let t be a fredholm operator in a hilbert space h with spectrum t i i 1 let t i t i m i where m i is the ascent of the operator t i i i t and let p i denote the projection of h onto the generalized eigenspace n t i along r t i in this paper it is shown that s s and h s m topological direct sum iff there exists m 0 such that i 1 n p i m n 1 2 where s x h i 1 x i x i n t i and m is the zero or infinite dimensional subspace i 1 r t i this result is then applied to a differential operator l in l 2 0 1 showing that s s l 2 0 1 for this l also an example with m 0 is presented

eigenfunction expansions for the nonspectral differential operators determined by d2 abstract for a nonspectral differential operator l determined by d2 sufficient conditions are established for the l2 expansion of a function u l2 0 1 in terms of the eigenfunctions of l in some cases both necessary and sufficient conditions are derived

maximum likelihood reconstruction for emission tomography previous models for emission tomography et do not distinguish the physics of et from that of transmission tomography we give a more accurate general mathematical model for et where an unknown emission density x y z generates and is to be reconstructed from the number of counts n d in each of d detector units d within the model we give an algorithm for determining an estimate of which maximizes the probability p n of observing the actual detector count data n over all possible densities let independent poisson variables n b with unknown means b b 1 b represent the number of unobserved emissions in each of b boxes pixels partitioning an object containing an emitter suppose each emission in box b is detected in detector unit d with probability p b d d 1 d with p b d a one step transition matrix assumed known we observe the total number n n d of emissions in each detector unit d and want to estimate the unknown b b 1 b for each the observed data n has probability or likelihood p n the em algorithm of mathematical statistics starts with an initial estimate 0 and gives the following simple iterative procedure for obtaining a new estimate new from an old estimate old to obtain k k 1 2 new b old b dd 1 n d p b d old b p b d b 1 b

from statistical mechanics to neural networks and back abstract the neural network models which have attracted so much attention in the physics community consist of a large assembly of two state elements highly interconnected by ferromagnetic and antiferromagnetic interactions methods of statistical mechanics have been applied to these models bringing new discoveries and surprises to neural network science at the same time neural network models have posed new problems and incentives to statistical mechanics one of the basic problems is that of learning which in the context of these models means a choice of the interactions for which the network dynamics leads to specific attractors the main emphasis in this paper is on the essential developments and results on this problem derived from methods of statistical mechanics

storage capacity and optimal learning of potts model perceptrons by a cavity method by means of a general formulation for the optimal learning capacity of perceptrons with multi state neurons and real valued couplings with spherical constraints which we derive by a cavity method we calculate the optimal learning capacity alpha c q k pmax n q 1 for perceptrons with a q resp q state potts model input resp output neurons as a function of q and the stability parameter k among other results the asymptote for q to infinity is found and it is shown that for k 0 the information gain per coupling delta i alpha c in q q 1 converges slowly to 1 2 in this limit moreover for q to infinity the same asymptotics also apply for the simple case of hebbian learning

spectral representation of the resolvent of a discrete operator let t be a discrete linear operator in a hilbert space h with spectrum t ii 1 let r t denote the resolvent of t and let pi denote the projection of h onto the generalized eigenspace n i i t mj along r i i t mj where mi is the ascent of the operator ii t in this paper it is shown that r i 1 mjj 1 ni j 1pi j j 0 o jr 0 t j 1 i p in b h for all t where ni is the restriction of ii t to is the restriction of t to n i i t mj t d t i 1r ii t mj p i 1pi strong convergence and 0 is a fixed but arbitrary point in c this spectral representation is valid provided there exists m 0 such that i 1n pi m n 1 2 and generalizes results that apply to self adjoint normal and spectral operators the results of this paper are applied to represent the resolvent of a differential operator l in l2 0 1 having infinitely many eigenvalues with ascent mi 2 and are also applied to represent the resolvent of an operator t with p i

large systems of random linear equations with nonnegative solutions characterizing the solvable and the unsolvable phase large systems of linear equations are ubiquitous in science quite often e g when considering population dynamics or chemical networks the solutions must be nonnegative recently it has been shown that large systems of random linear equations exhibit a sharp transition from a phase where a nonnegative solution exists with probability one to one where typically no such solution may be found the critical line separating the two phases was determined by combining farkas lemma with the replica method here we show that the same methods remain viable to characterize the two phases away from criticality to this end we analytically determine the residual norm of the system in the unsolvable phase and a suitable measure of robustness of solutions in the solvable one our results are in very good agreement with numerical simulations

optimal capacity of the blume emery griffiths perceptron a blume emery griffiths perceptron model is introduced and its optimal capacity is calculated within the replica symmetric gardner approach as a function of the pattern activity and the embedding stability parameter the stability of the replica symmetric approximation is studied via the analog of the de almeida thouless line a comparison is made with other three state perceptrons

storing an extensive number of grey toned patterns in a neural network using multistate neurons grey toned patterns are pictures composed of pixels of several shades of grey the ability of neural networks using multistate neurons to store such patterns is systematically investigated if conventional generalizations of hopfield networks using analogue or soft neurons are considered it is impossible to stabilize these grey tones nevertheless it is shown that it can be done with networks that use neurons which have only a discrete set of possible activities this is demonstrated for the pseudo inverse rule for the synaptic couplings where only the stability of the patterns shrinks with increasing number q of grey tones one wants to store if the patterns are uncorrelated one can use the hebb rule and in this case the mean field theory is presented applying this rule the storage capacity decreases as q 2 with the number of grey tones

on the parallel dynamics of the q state potts and q ising neural networks using a probabilistic approach the parallel dynamics of theq state potts andq ising neural networks are studied at zero and at nonzero temperatures evolution equations are derived for the first time step and arbitraryq these formulas constitute recursion relations for the exact parallel dynamics of the extremely diluted asymmetric versions of these networks an explicit analysis including dynamical capacity temperature diagrams and the temperature dependence of the overlap is carried out forq 3 both types of models are compared

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

categorization in the pseudo inverse neural network we investigate analytically the emergence of the categorization ability in the pseudo inverse attractor neural network more pointedly we consider the problem of learning an extensive number of concepts by storing a finite number of examples s of each concept we find that there is a critical value beyond which the categorization error as measured by the average fraction of unstable sites in the concepts decreases monotonically with s

higher order corrections to the effective potential close to the jamming transition in the perceptron model in view of the results achieved in a previously related work a altieri s franz and g parisi j stat mech 2016 093301 10 1088 1742 5468 2016 09 093301 regarding a plefka like expansion of the free energy up to the second order in the perceptron model we improve the computation here focusing on the role of third order corrections the perceptron model is a simple example of constraint satisfaction problem falling in the same universality class as hard spheres near jamming and hence allowing us to get exact results in high dimensions for more complex settings our method enables to define an effective potential or thouless anderson palmer free energy namely a coarse grained functional which depends on the generalized forces and the effective gaps between particles the analysis of the third order corrections to the effective potential reveals that albeit irrelevant in a mean field framework in the thermodynamic limit they might instead play a fundamental role in considering finite size effects we also study the typical behavior of generalized forces and we show that two kinds of corrections can occur the first contribution arises since the system is analyzed at a finite distance from jamming while the second one is due to finite size corrections we nevertheless show that third order corrections in the perturbative expansion vanish in the jamming limit both for the potential and the generalized forces in agreement with the isostaticity argument proposed by wyart and coworkers finally we analyze the relevant scaling solutions emerging close to the jamming line which define a crossover regime connecting the control parameters of the model to an effective temperature

critical capacity of constrained perceptrons the authors calculate the critical capacity of a single layer perceptron with interactions ji constrained to take values on a general surface the critical capacity is expressed in terms of a geometrical property of the constraining surface

perceptrons above saturation we study the storage of random patterns by a perceptron above its storage capacity alpha c i e in the region where perfect storage becomes impossible we determine the minimal fraction of learning errors and the distribution of stabilities for different learning rules in one step replica symmetry breaking thereby we not only extend the known replica symmetric results to values of the storage capacity beyond the at line but also show that depending on the learning rule replica symmetry may be globally unstable already well below the at line as an example for possible implications we compare the results for the typical basins of attraction of an extremely diluted attractor neural network as given by replica symmetry and one step replica symmetry breaking

statistical mechanics of lossy data compression using a nonmonotonic perceptron the performance of a lossy data compression scheme for uniformly biased boolean messages is investigated via methods of statistical mechanics inspired by a formal similarity to the storage capacity problem in neural network research we utilize a perceptron of which the transfer function is appropriately designed in order to compress and decode the messages employing the replica method we analytically show that our scheme can achieve the optimal performance known in the framework of lossy compression in most cases when the code length becomes infinite the validity of the obtained results is numerically confirmed

optimal capacity of graded response perceptrons a replica symmetry breaking solution optimal capacities of perceptrons with graded input output relations are studied within the first step replica symmetry breaking gardner approach input data errors and a limited output precision are allowed in particular the role of non monotonicity in the input output relations on the breaking and on the overall performance is determined

replica symmetric theory of nonlinear analogue neural networks on the basis of the theory of the naive mean field model of spin glasses analogue neural networks of the hopfield type are investigated in the saturation limit the saddle point equations for the order parameters describing retrieval and spin glass phases of the networks are obtained by means of a statistical mechanical analysis within the framework of the replica symmetric theory and are shown to undergo a modification to that of ags theory due to the absence of the onsager reaction term in the tap equation based on the equations the memory storage capacity of the networks is analysed as a function of the analogue gain beta a small increase in the critical storage capacity is found for finite values of beta compared with that of the ising model networks with corresponding inverse temperature although the qualitative nature of the phase diagram is unchanged

three state neural network from mutual information to the hamiltonian the mutual information i of the three state neural network can be obtained exactly for the mean field architecture as a function of three macroscopic parameters the overlap the neural activity and the activity overlap i e the overlap restricted to the active neurons we perform an expansion of i on the overlap and the activity overlap around their values for neurons almost independent of the patterns from this expansion we obtain an expression for a hamiltonian which optimizes the retrieval properties of this system this hamiltonian has the form of a disordered blume emery griffiths model the dynamics corresponding to this hamiltonian is found as a special characteristic of such a network we see that information can survive even if no overlap is present hence the basin of attraction of the patterns and the retrieval capacity is much larger than for the hopfield network the extreme diluted version is analyzed the curves of information are plotted and the phase diagrams are built

generalization ability of perceptrons with continuous outputs in this paper we examine the influence of different input output relations on the generalization ability of a single layer perceptron the input output relations can be linear binary or sigmoid with this choice we take into account most of the cases which are of present interest the generalization problem will be realizable or unrealizable if the input output relations for teacher and student are identical or not we show that sometimes it can have a positive effect on the generalization ability if one learns with errors

image detection under low level illumination image detection under low light level conditions is treated as a hypothesis testing problem in which the observations are modeled as a shot noise process since computing the likelihood ratio for shot noise processes is not feasible the use of a one dimensional test statistic obtained by filtering and sampling the observations is proposed the filter is chosen to maximize a generalized signal to noise ratio the likelihood ratio for the one dimensional test statistic is evaluated numerically by inverting the corresponding characteristic function under each hypothesis

spectral theory of two point differential operators determined by d2 ii analysis of cases abstract in part i the necessary tools are developed for the complete resolution of the spectral theory for all linear two point differential operators l in l 2 0 1 determined by d 2 and by boundary values b 1 b 2 the development emphasizes six numerical parameters defined in terms of the coefficients of b 1 b 2 in this paper we state and prove all results pertaining to the spectrum l of l to the algebraic multiplicities of the eigenvalues l and the ascents of the operators i l to the boundedness of the family of all finite sums of the projections associated with l to the denseness of the generalized eigenfunctions and to the existence of bases consisting of generalized eigenfunctions we utilize the tools developed in part i to establish these results in particular this paper provides either the explicit or the asymptotic form for the eigenvalues of l provides explicit numbers for the algebraic multiplicities and ascents provides the explicit or the asymptotic form for the projections and provides explicit bounds on the family of all finite sums of these projections or shows that the family is unbounded

a simple derivation of a bound on the perceptron margin using singular value decomposition the perceptron is a simple supervised algorithm to train a linear classifier that has been analyzed and used extensively the classifier separates the data into two groups using a decision hyperplane with the margin between the data and the hyperplane determining the classifier s ability to generalize and its robustness to input noise exact results for the maximal size of the separating margin are known for specific input distributions and bounds exist for arbitrary distributions but both rely on lengthy statistical mechanics calculations carried out in the limit of infinite input size here we present a short analysis of perceptron classification using singular value decomposition we provide a simple derivation of a lower bound on the margin and an explicit formula for the perceptron weights that converges to the optimal result for large separating margins

multiplicity of metastable retrieval phases in networks of multistate neurons it is demonstrated that networks of multistate neurons storing an ensemble of multistate patterns exhibit a multiplicity of metastable retrieval phases these phases are described by solutions of fixed point equations with characteristic retrieval errors they emerge if the gain of the neural input output relation is varied the number of these phases increases asq2 with the numberq of gray levels available to each neuron implications for the optimal gain function and basins of attraction are briefly discussed again networks endowed with pseudoinverse couplings are found to perform better than networks with hebbian couplings at moderate loading levels phases with retrieval errors are destabilized whereas the error free phase remains stable up to the theoretically possible maximum if the gain parameter is properly chosen

iterative reweighted total generalized variation based poisson noise removal model total generalized variation tgv regularization model is one of the most effective methods for denoising and eliminating staircase effect however the tgv regularization model tends to blur edges as the existence of high order derivative in order to avoid the staircase effect while alleviating the edge blurring an iterative reweighted tgv based poisson noise removal model is presented under the assumption that each pixel of noisy image follows a poisson distribution the weight function incorporated in the tgv regularization term is derived from the expectation maximization em algorithm we design a new iterative weighted primal dual algorithm which is an improvement of the classic iterative reweighted algorithm numerical experimental results show the better performance of our model in removing noise effectively while preserving edges and eliminating staircase effect

on the capacity per synapse the optimal storage capacity of a perceptron with a finite fraction of sign constrained weights which are prescribed a priori is examined the storage capacity is calculated by considering the fractional volume of weights which can store a set of alpha n random patterns where n is the size of the input it is found that in the case where 1 s n weights are sign constrained the capacity is 1 s 2 alpha gc k where alpha gc k is the maximal storage capacity in gardner s case and k is a stability parameter

storage capacity of a diluted neural network with ising couplings the capacity for storing random patterns in a diluted neural network is determined following the method of gardner 1988 the non zero coupling coefficients are restricted to take on the ising values or 1 different degrees of dilution are considered the maximum values alpha c 1 17 for the storage capacity is obtained when each neuron is connected to 63 of the other neurons

storage capacity of ultrametric committee machines the problem of computing the storage capacity of a feed forward network with l hidden layers n inputs and k units in the first hidden layer is analyzed using techniques from statistical mechanics we found that the storage capacity strongly depends on the network architecture c log k 1 1 2l and that the number of units k limits the number of possible hidden layers l through the relationship 2l 1 2log k

capacity of diluted multi state neural networks the optimal storage capacity is studied for diluted networks with multi state neurons and continuous respectively discrete couplings within the replica symmetric gardner approach the gardner derrida line the de almeida thouless line and the zero entropy line are compared and the validity of the replica symmetric approximation is discussed in detail the distribution of the synaptic couplings is determined the results are analysed in terms of the number of states of the neuron the distribution of the stored patterns the amount of dilution and the number of discrete values for the couplings

on the capacity of neural networks with binary weights the authors study the critical capacity alpha c of multilayered networks with binary couplings they show that for any network presenting a tree like architecture after the first hidden layer no fixed internal representation is required using gardner s calculations they apply statistical mechanics to the simplest network with two layers of adaptive weights following the same approach as for the binary perceptron they find from the zero entropy point a critical capacity alpha c 0 92 they discuss the validity of this result in view of exhaustive search simulations on small networks

on a new method for solving irregular problems abstract we suggest a new approach to the investigation of boundary value problems with a parameter in the case when the parameter appears in both the equation and the boundary conditions the method is applied to the regge problem in spite of the fact that the regge problem is birkhoff irregular see g d birkhoff trans amer math soc 9 1908 373 395 the problem is coercive but with a defect with respect to the spectral parameter we also prove 2 fold completeness of root functions of the regge problem

the spectral theory of second order two point differential operators i a priori estimates for the eigenvalues and completeness this paper is the first part in a four part series which develops the spectral theory for a two point differential operator l in l 2 0 1 determined by a second order formal differential operator l d 2 pd q and by independent boundary values b 1 b 2 the differential operator l is classified as belonging to one of five cases cases 1 5 according to conditions satisfied by the coefficients of b 1 b 2 for cases 1 4 it is shown that if 2 is any eigenvalue of l with sufficiently large then lies in the interior of a horizontal strip cases 1 3 or the interior of a logarithmic strip case 4 and in each of these cases the generalised eigenfunctions of l are complete in l 2 0 1

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

spectral theory for a differential operator characteristic determinant and green s function abstract for a two point differential operator l in l2 a b it is shown that the green s function has the representation g t s h t s d for belonging to the resolvent set l where d is the characteristic determinant and h t s is an entire function in the variable admitting a power series expansion about any point 0 c this representation is given several applications first to calculate the coefficient operators in the laurent series for the resolvent r l about each point 0 in the spectrum l and second to relate the algebraic multiplicity v 0 of an eigenvalue 0 to the ascent m0 of the operator 0i l

gardner optimal capacity of the diluted blume emery griffiths neural network abstract the optimal capacity of a diluted blume emery griffiths neural network is studied as a function of the pattern activity and the embedding stability using the gardner entropy approach annealed dilution is considered cutting some of the couplings referring to the ternary patterns themselves and some of the couplings related to the active patterns both simultaneously synchronous dilution or independently asynchronous dilution through the de almeida thouless criterion it is found that the replica symmetric solution is locally unstable as soon as there is dilution the distribution of the couplings shows the typical gap with a width depending on the amount of dilution but this gap persists even in cases where a particular type of coupling plays no role in the learning process

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

phase diagram of restricted boltzmann machines and generalized hopfield networks with arbitrary priors restricted boltzmann machines are described by the gibbs measure of a bipartite spin glass which in turn can be seen as a generalized hopfield network this equivalence allows us to characterize the state of these systems in terms of their retrieval capabilities both at low and high load of pure states we study the paramagnetic spin glass and the spin glass retrieval phase transitions as the pattern i e weight distribution and spin i e unit priors vary smoothly from gaussian real variables to boolean discrete variables our analysis shows that the presence of a retrieval phase is robust and not peculiar to the standard hopfield model with boolean patterns the retrieval region becomes larger when the pattern entries and retrieval units get more peaked and conversely when the hidden units acquire a broader prior and therefore have a stronger response to high fields moreover at low load retrieval always exists below some critical temperature for every pattern distribution ranging from the boolean to the gaussian case

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

on the ability of the optimal perceptron to generalise a linearly separable boolean function is derived from a set of examples by a perceptron with optimal stability the probability to reconstruct a pattern which is not learnt is calculated analytically using the replica method

bayesian filtering of poisson noise using local statistics images recorded at low light levels inherently suffer from poisson noise a filter based on the maximum a posteriori probability map criterion is developed to remove this noise the filter is adaptive it responds to local changes in image statistics and thus removes the noise along the edges without significantly affecting the edge sharpness it does not require any a priori information about the original image because all the parameters needed for the filter are estimated from the noisy image by assuming local stationarity additionally the simple structure of the filter can be easily implemented in hardware

optimal storage of invariant sets of patterns in neural network memories the authors investigate optimal conditions for the storage of invariant sets of patterns in neural network memories the sets of patterns that they consider are highly correlated non random and invariant with respect to some symmetry transformations they generalize gardner s method to study the fractional volume in the space of neural interactions that allow for storage of invariant patterns they demonstrate that optimal storage conditions correspond to non trivial relations between the number of stored patterns their stability and other symmetry specific characteristics of patterns

learning by dilution in a neural network a perceptron with n random weights can store of the order of n patterns by removing a fraction of the weights without changing their strengths the critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated a simple hebb like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability

capacity of neural networks with discrete synaptic couplings the authors study the optimal storage capacity of neural networks with discrete local constraints on the synaptic couplings jij models with such constraints include those with binary couplings jij or 1 or jij 0 1 quantised couplings with larger synaptic range e g jij or 1 l or 2 l or 1 and in the limit continuous couplings confined to the hypercube mod jij mod or 1 box confinement they find that the optimal storage capacity alpha kappa is best determined by the vanishing of a suitably defined entropy as calculated in the replica symmetric approximation they also extend their results to cases with biased memories and make contact with sparse coding models

a dilution algorithm for neural networks a dilution algorithm to enlarge the storage capacity per synapse alpha eff of neural networks is proposed the algorithm is a hybrid method where hebb s rule is used to select a fraction of couplings to be removed afterwards the perceptron of optimal stability for the remaining couplings is learned the authors present an analytical calculation and the results of the numerical simulations in comparison with the fully connected or the randomly diluted perceptron the effective storage capacity alpha eff is remarkably enlarged

a replica analysis of the minimum weight solution to the linear equations problem we investigate through statistical mechanics techniques the problem of finding n tuples j ji j2 jn with a fixed fraction k of non vanishing entries that solve a set of p alpha n linear equations in particular we determine the region in the plane alpha k where these solutions exist with a probability of 1 in the limit of large n we also obtain at insight into the microscopic structure of these solutions by calculating the distribution of the probability of an arbitrary non vanishing entry ji moreover we evaluate analytically the performances of several easy to implement heuristic algorithms and compare them with the optimal solution a task that is suited very well to the statistical mechanics approach

the nonspectral birkhoff regular differential operators determined by d2 the class of nonspectral birkhoff regular differential operators l in l2 0 1 determined by d2 and by boundary values b1 b2 is reexamined for the family of projections p0k k 1m pk k k0 p k k k0 which map l2 0 1 onto the generalized eigenspaces of l the norms of the pk and p k tend to while in sharp contrast the family of all finite sums of the projections p0k k 1m pk p k k k0 is uniformly bounded in norm in addition the expansion u k 1m p0k u k k0 pk p k u is valid for all u l2 0 1 even though there exist u in l2 0 1 for which both series k k0 pku and k k0 p ku diverge

intensity estimation from shot noise data the estimation of the intensity function of a poisson driven shot noise process is addressed using a regularization technique where the data is modeled as a signal term plus a signal dependent noise term a new data based method for selecting a pair of regularization parameters is presented and compared with the minimum unbiased risk method the detail in the intensity function can be recovered by both methods but the new method does a better job at suppressing spurious oscillations

from shrinking to percolation in an optimization model a model of noise reduction for signal processing and other optimization tasks is introduced each noise source puts a symmetric constraint on the space of the signal vector within a tolerance bound when the number of noise sources increases sequences of transitions take place causing the solution space to vanish we find that the transition from an extended solution space to a shrunk space is retarded because of the symmetry of the constraints in contrast with the analogous problem of pattern storage for low tolerance the solution space vanishes by volume reduction whereas for high tolerance the vanishing becomes more and more like percolation

statistical mechanics of neural networks what are the differences between wide and narrow basins abstract we consider training noise in neural networks as a means of tuning the structure of retrieval basins and study how learning and retrieving properties depend on it the stability of the replica symmetric solution and the correlation in the weight space indicate that neural networks can be roughly classified into hebbian like and msn like msn meaning the maximally stable network re entrant retrieval noise robustness selectivity damage spreading and activity distribution all illustrate the differences in retrieval behaviours arising from the different basin structures

dilution in boolean perceptrons that learn from noisy examples we investigate the effect of dilution after learning on the generalization ability of single layer boolean perceptrons that learn from noisy examples we present a thorough comparison between the relative performances of several well known learning rules in particular we show that the effect of dilution is always deleterious and that the bayes algorithm always gives the best generalization performance

blind intensity estimation from shot noise data the estimation of the intensity function of an inhomogeneous poisson process is considered when the observable data consists of sampled shot noise that results from passing the poisson process through an unknown linear time invariant system the proposed method consists of first estimating a histogram of the underlying point process the estimated histogram is used to construct a kernel estimate of the intensity function an estimate of the unknown impulse response of the linear time invariant system is constructed via a regularized backsubstitution of a discrete time convolution with the estimated histogram