storage of sets of correlated data in neural network memories in most applications of neural network models one has to consider sets of correlated data the authors study the problem of the storage of associated patterns in neural network memories they consider two basic types of correlation semantic ones among the different patterns and spatial ones among the different sites of the network they apply gardner s program to evaluate optimal storage conditions for both kinds of correlation the spatial ones worsen generally speaking the storage properties of a simple perceptron while they improve them for the hopfield network in the case of semantic correlations they obtain bounds of the critical capacity for both kinds of networks the storage ratio of the perceptron may be significantly increased in this case

critical storage capacity of the j 1 neural network for neural networks in which the coupling jij are allowed to take on the values jij 1 or jij 1 the authors determine numerically the critical storage capacity for random unbiased patterns as a function of the stability they use an exact enumeration scheme based on the gray code and a continuous distribution for the patterns to control finite size effects results are presented for n 25 they indicate an optimal storage capacity of c 0 82 n

stability of the one step replica symmetry broken phase in neural networks the stability of the phase with one step replica symmetry breaking is studied in fully connected neural networks with modified pseudo inverse interactions the interaction matrix has an intermediate form between the hebb learning rule and the pseudo inverse one at low temperature there is a region of parameters where the one step replica symmetry broken solution exists fluctuations around this solution are analysed by a replica group representations approach and it is found that the solution is stable for all ranges of the parameters where it exists

generalization and capacity of extensively large two layered perceptrons the generalization ability and storage capacity of a treelike two layered neural network with a number of hidden units scaling as the input dimension is examined the mapping from the input to the hidden layer is via boolean functions the mapping from the hidden layer to the output is done by a perceptron the analysis is within the replica framework where an order parameter characterizing the overlap between two networks in the combined space of boolean functions and hidden to output couplings is introduced the maximal capacity of such networks is found to scale linearly with the logarithm of the number of boolean functions per hidden unit the generalization process exhibits a first order phase transition from poor to perfect learning for the case of discrete hidden to output couplings the critical number of examples per input dimension ensuremath alpha c at which the transition occurs again scales linearly with the logarithm of the number of boolean functions in the case of continuous hidden to output couplings the generalization error decreases according to the same power law as for the perceptron with the prefactor being different

properties of neural networks storing spatially correlated patterns the author studies the behaviour of a feedforward neural network supplied with spatially organized data this inner structure is taken into account by a matrix cij whose coefficients equal the average correlation between two pixels i and j of the input patterns the storage capacity alpha is computed as a function of the required stability and of the eigenvalues of c the author proposes a geometrical transformation allowing an intuitive interpretation of these results numerical simulations using real and binary patterns show a very good agreement with the theory finally the author analyses the synaptic couplings correlations resulting from the training of the network with structured patterns focusing on exponentially decreasing correlations one and two dimensions the author finds that they exhibit a mexican hat profile the excitatory centre size of which depends on alpha

optimal storage capacity of neural networks at finite temperatures gardner s analysis 1989 of the optimal storage capacity of neural networks is extended to study finite temperature effects the typical volume of the space of interactions is calculated for strongly diluted networks as a function of the storage ratio alpha temperature t and the tolerance parameter m from which the optimal storage capacity alpha c is obtained as a function of t and m at zero temperature it is found that alpha c 2 regardless of m while alpha c in general increases with the tolerance at finite temperatures the authors show how the best performance for given alpha and t is obtained which reveals a first order transition from high quality performance to a low quality one at low temperatures an approximate criterion for recalling which is valid near m 1 is also discussed

generalization learning in a perceptron with binary synapses we consider the generalization problem for a perceptron with binary synapses implementing the stochastic belief propagation inspired sbpi learning algorithm which we proposed earlier and perform a mean field calculation to obtain a differential equation which describes the behaviour of the device in the limit of a large number of synapses n we show that the solving time of sbpi is of order n sqrt log n while the similar well known clipped perceptron cp algorithm does not converge to a solution at all in the time frame we considered the analysis gives some insight into the ongoing process and shows that in this context the sbpi algorithm is equivalent to a new simpler algorithm which only differs from the cp algorithm by the addition of a stochastic unsupervised meta plastic reinforcement process whose rate of application must be less than sqrt 2 pi n for the learning to be achieved effectively the analytical results are confirmed by simulations

the binary perceptron and general aspects of non self averaged quantities abstract the possibility of a finite width distribution for the maximal capacity of the binary perceptron in the thermodynamic limit is discussed analytically and supported by a careful analysis of numerical simulations the results also indicate that the description of quenched random systems could take into account the possibility that in addition to non self averaged quantities other quantities such as the transition temperature might also be sample dependent

storage capacity of neural networks effect of the fluctuations of the number of active neurons per memory the storage capacity in an attractor neural network with excitatory couplings is shown to depend not only on the fraction of active neurons per pattern or coding rate but also on the fluctuations around this value in the thermodynamical limit the capacity is calculated in the case of exactly the same number of active neurons in every pattern for every coding level the capacity is increased with respect to the case of random patterns results are supported by numerical simulations done with an exhaustive search algorithm and partly solve in the sparse coding limit the paradox of the discrepancy of the capacity of the willshaw model with optimal capacity

correlation between eigenvalue spectra and dynamics of neural networks this letter presents a study of the correlation between the eigenvalue spectra of synaptic matrices and the dynamical properties of asymmetric neural networks with associative memories for this type of neural network it was found that there are essentially two different dynamical phases the chaos phase with almost all trajectories converging to a single chaotic attractor and the memory phase with almost all trajectories being attracted toward fixed point attractors acting as memories we found that if a neural network is designed in the chaos phase the eigenvalue spectrum of its synaptic matrix behaves like that of a random matrix i e all eigenvalues lie uniformly distributed within a circle in the complex plan and if it is designed in the memory phase the eigenvalue spectrum will split into two parts one part corresponds to a random background the other part equal in number to the memory attractors the mechanism for these phenomena is discussed in this letter

learning times of a perceptron that learns from examples we calculate the distribution of learning times of the optimal stability perceptron algorithm of krauth and mezard 1987 for the learning from noisy examples problem in particular we find that in the case of noiseless examples the average total number of learning steps scales with alpha 2 where alpha is the training set size although the number of examples that must effectively be learned tends to zero as alpha 1

perceptron learning with sign constrained weights the authors study neural network models in which the synaptic efficacies are restricted to have a prescribed set of signs it is proved that such neural networks can learn a set of random patterns by a perceptron like algorithm which respects the synaptic restrictions at every step in particular it shows that learning can take place iteratively in a network which obeys dale s rule i e in which neurons are exclusively excitatory or inhibitory the learning algorithm as well as its convergence theorem are stated in perceptron language and it is proved that the algorithm converges under the same conditions as required for an unconstrained perceptron numerical experiments show that these necessary conditions can actually be met for relatively large sets of patterns to be learned they then argue that the results are invariant under the distribution of the signs due to gauge invariance for random patterns as a consequence the same sets of random patterns can be learned by networks which have any fixed distribution of synaptic signs ranging from fully inhibitory to fully excitatory

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

critical capacity of constrained perceptrons the authors calculate the critical capacity of a single layer perceptron with interactions ji constrained to take values on a general surface the critical capacity is expressed in terms of a geometrical property of the constraining surface

empirical bayes analysis of a microarray experiment microarrays are a novel technology that facilitates the simultaneous measurement of thousands of gene expression levels a typical microarray experiment can produce millions of data points raising serious problems of data reduction and simultaneous inference we consider one such experiment in which oligonucleotide arrays were employed to assess the genetic effects of ionizing radiation on seven thousand human genes a simple nonparametric empirical bayes model is introduced which is used to guide the efficient reduction of the data to a single summary statistic per gene and also to make simultaneous inferences concerning which genes were affected by the radiation although our focus is on one specific experiment the proposed methods can be applied quite generally the empirical bayes inferences are closely related to the frequentist false discovery rate fdr criterion

vapnik chervonenkis dimension of neural networks with binary weights we investigate the vc dimension of the perceptron and simple two layer networks like the committee and the parity machine with weights restricted to values pm1 for binary inputs the vc dimension is determined by atypical pattern sets i e it cannot be found by replica analysis or numerical monte carlo sampling for small systems exhaustive enumerations yield exact results for systems that are too large for enumerations number theoretic arguments give lower bounds for the vc dimension for the ising perceptron the vc dimension is probably larger than n 2

the weight space of the binary perceptron with a view to finding features of the weight space of the binary perceptron that might be instructive for training binary synapse neural networks the maximally stable perceptron having binary valued weights is compared with continuous weight perceptrons for universal choices of stored patterns the fraction of synaptic weights correctly predicted by clipping the synapses of the continuous network is calculated in the thermodynamic limit and compared with simulation results for smaller systems numerical experiments show good agreement with theory but in addition indicate that those binary synapses likely to be wrongly predicted by weight clipping are predominantly those which are weakest in the continuous synapse perceptron although not rescuing training time from growing exponentially in the system size our results suggest ways of significantly accelerating the search for successful albeit possibly imperfect neural networks with discrete valued couplings

information capacity of a perceptron the authors study the information storage capacity of a simple perceptron in the error regime for random unbiased patterns the geometrical analysis gives a logarithmic dependence for the information content in the asymptotic limit in this case the statistical physics approach when used at the simplest level of replica theory does not give satisfactory results however for perceptrons with finite stability the information content can be simply calculated with statistical physics methods in a region above the critical storage level for biased as well as for unbiased patterns

on the capacity per synapse the optimal storage capacity of a perceptron with a finite fraction of sign constrained weights which are prescribed a priori is examined the storage capacity is calculated by considering the fractional volume of weights which can store a set of alpha n random patterns where n is the size of the input it is found that in the case where 1 s n weights are sign constrained the capacity is 1 s 2 alpha gc k where alpha gc k is the maximal storage capacity in gardner s case and k is a stability parameter

on the capacity of neural networks with binary weights the authors study the critical capacity alpha c of multilayered networks with binary couplings they show that for any network presenting a tree like architecture after the first hidden layer no fixed internal representation is required using gardner s calculations they apply statistical mechanics to the simplest network with two layers of adaptive weights following the same approach as for the binary perceptron they find from the zero entropy point a critical capacity alpha c 0 92 they discuss the validity of this result in view of exhaustive search simulations on small networks

capacity of diluted multi state neural networks the optimal storage capacity is studied for diluted networks with multi state neurons and continuous respectively discrete couplings within the replica symmetric gardner approach the gardner derrida line the de almeida thouless line and the zero entropy line are compared and the validity of the replica symmetric approximation is discussed in detail the distribution of the synaptic couplings is determined the results are analysed in terms of the number of states of the neuron the distribution of the stored patterns the amount of dilution and the number of discrete values for the couplings

duality between learning machines a bridge between supervised and unsupervised learning we exhibit a duality between two perceptrons that allows us to compare the theoretical analysis of supervised and unsupervised learning tasks the first perceptron has one output and is asked to learn a classification of p patterns the second dual perceptron has p outputs and is asked to transmit as much information as possible on a distribution of inputs we show in particular that the maximum information that can be stored in the couplings for the supervised learning task is equal to the maximum information that can be transmitted by the dual perceptron

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

a perceptron with a skeletal weight space a perceptron whose space of interactions can interpolate between spherically constrained and binary valued synapses is introduced and investigated as an associative neural network memory for maximally stable storage and where the weight space remains connected the critical storage capacity alpha c is found to be reduced by a factor determined solely by the geometry of the weight space and is shown to interpolate within the replica symmetric approximation between alphac 2 in the gardner model limit and alphac 4 pi various comparisons of the synaptic weights with those of the binary perceptron show that such differences as remain between this weight space and that of the true binary perceptron are crucial to obtaining alphac or 4 pi although these differences limit the use of such models in realizing optimal binary networks they may yet provide worthwhile binary systems by simple weight clipping simulation results are presented in support of the theoretical analyses

three unfinished works on the optimal storage capacity of networks the optimal storage properties of three different neural network models are studied for two of these models the architecture of the network is a perceptron with or j interactions whereas for the third model the output can be an arbitrary function of the inputs analytic bounds and numerical estimates of the optimal capacities and of the minimal fraction of errors are obtained for the first two models the third model can be solved exactly and the exact solution is compared to the bounds and to the results of numerical simulations used for the two other models

capacity of neural networks with discrete synaptic couplings the authors study the optimal storage capacity of neural networks with discrete local constraints on the synaptic couplings jij models with such constraints include those with binary couplings jij or 1 or jij 0 1 quantised couplings with larger synaptic range e g jij or 1 l or 2 l or 1 and in the limit continuous couplings confined to the hypercube mod jij mod or 1 box confinement they find that the optimal storage capacity alpha kappa is best determined by the vanishing of a suitably defined entropy as calculated in the replica symmetric approximation they also extend their results to cases with biased memories and make contact with sparse coding models

learning by dilution in a neural network a perceptron with n random weights can store of the order of n patterns by removing a fraction of the weights without changing their strengths the critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated a simple hebb like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability

replica symmetry breaking in neural networks with modified pseudo inverse interactions replica symmetry breaking is studied in fully connected neural networks with modified pseudo inverse interactions the interaction matrix has an intermediate form between the hebb learning rule and the pseudo inverse one at low temperatures there is a region of parameters where the replica symmetric solution is stable while its entropy is negative it indicates the existence of an alternative solution in which the replica symmetry is broken the one step replica symmetry breaking solution is found and its properties are analysed

associative memory on the puzzling sparse coding limit recent studies of the information capacity in a sparsely coded memory net has led to some contradictory results in the willshaw model where the couplings are binary 0 or 1 the maximal quantity of information that can be stored is 1n 2 approximately 0 69 bits per synapse on the other hand a calculation a la gardner 1988 for 0 1 couplings gives an upper bound for the maximal capacity of about 0 29 bits per synapse in this study the author considers two possible sources for this discrepancy the first one is that the criteria for defining the maximal capacity are different with or without a constraint of perfect errorless storage the second one is a difference in the choice of the probability distribution of the random patterns used to compute this capacity this analysis shows in particular that for the willshaw model the maximal information capacity is much larger when the number of active neurons is exactly the same in every stored pattern than when it is given only in average in addition he gives an argument showing that this result may be generic e g valid for any activity level and independent of the learning rule

gain of regularity for the kp i equation abstract in this paper we study the smoothness properties of solutions to the kp i equation we show that the equation s dispersive nature leads to a gain in regularity for the solution in particular if the initial data possesses certain regularity and sufficient decay as x then the solution u t will be smoother than for 0 t t where t is the existence time of the solution