storage of sets of correlated data in neural network memories in most applications of neural network models one has to consider sets of correlated data the authors study the problem of the storage of associated patterns in neural network memories they consider two basic types of correlation semantic ones among the different patterns and spatial ones among the different sites of the network they apply gardner s program to evaluate optimal storage conditions for both kinds of correlation the spatial ones worsen generally speaking the storage properties of a simple perceptron while they improve them for the hopfield network in the case of semantic correlations they obtain bounds of the critical capacity for both kinds of networks the storage ratio of the perceptron may be significantly increased in this case

structure of metastable states in the hopfield model an upper bound for the number of metastable states in the hopfield model is calculated as a function of the hamming fraction from an input pattern for all finite values of alpha the ratio of number of patterns to nodes the hamming fraction from the input pattern to the nearest metastable state is infinite when alpha 0 113 the bound also implies that there is a gap between a set of states close to the input pattern and another set centred around the hamming fraction 0 5 from it

the statistical mechanics of constructive algorithms the storage capacity of multilayer networks with overlapping receptive fields is studied for constructive algorithms using boolean perceptrons as their basic building block which have been investigated within a replica framework the assumption of weak coupling between subsequently constructed perceptrons is verified within a replica symmetric rs ansatz and shown to be negligible in most cases in comparison with correction due to replica symmetry breaking rsb in individual perceptrons the capacities of a tiling like and variants of the upstart algorithm are then calculated within rs and one step rsb with the quenched average taken over the individual units separately for networks with up to k 4000 and k 600 units respectively within this treatment the storage capacity seems to exhibit a power law behaviour in with an exponent n that may depend on the algorithm and the stability however due to finite size effects in k reliable estimates of n could not be extracted nevertheless the results strongly indicate that n should be strictly smaller than 1 within one step rsb whereas within rs the mitchison durbin bound is violated for finite k and n 1 may hold asymptotically

storage capacity of a neural network with state dependent synapses the storage capacity of the hopfield network is limited to p n c 0 138 p number of patterns n number of neurons beyond which the contribution of weakly correlated patterns surpasses that of the desired pattern this contribution can be eliminated by introducing a threshold a pattern correlation below this threshold is simply set to zero in the synapses we solve the mean field equations and derive the critical value of the threshold required to stabilize p n patterns with c

transpose free multiple lanczos and its application in pade approximation a transpose free two sided nonsymmetric lanczos method is developed for multiple starting vectors on both the left and right the method is mathematically equivalent to the two sided methods without look ahead or deflation steps but avoids the use of the transpose of the system matrix the method is applied to the computation of the matrix pade approximation to a linear dynamical system the result is a method which can be labeled transpose free matrix pade via lanczos tfmpvl under certain circumstances tfmpvl will actually reduce the total number of matrix vector products needed it is illustrated with some numerical examples

optimal storage capacity of neural networks at finite temperatures gardner s analysis 1989 of the optimal storage capacity of neural networks is extended to study finite temperature effects the typical volume of the space of interactions is calculated for strongly diluted networks as a function of the storage ratio alpha temperature t and the tolerance parameter m from which the optimal storage capacity alpha c is obtained as a function of t and m at zero temperature it is found that alpha c 2 regardless of m while alpha c in general increases with the tolerance at finite temperatures the authors show how the best performance for given alpha and t is obtained which reveals a first order transition from high quality performance to a low quality one at low temperatures an approximate criterion for recalling which is valid near m 1 is also discussed

learning and retrieval in attractor neural networks with noise abstract a recent study on noiseless learning and retrieval in attractor neural networks above saturation by griniasty and gutfreund is extended to take account for imperfect learning by means of a temperature 1 t violations of the constraint imposed on the local stabilities are taken into account by various cost functions the distribution of local stabilities and the fraction of errors during the learning stage are analysed the retrieval dynamics for the sparsely connected network is studied showing high retrieval overlap in reduced retrieval regions for finite t

gpbi cg generalized product type methods based on bi cg for solving nonsymmetric linear systems recently bi cgstab as a variant of bi cg has been proposed for solving nonsymmetric linear systems and its attractive convergence behavior has been confirmed in many numerical experiments bi cgstab can be characterized by its residual polynomial which consists of the product of the residual polynomial of bi cg with other polynomials generated from two term recurrence relations in this paper we propose a unified way to generalize a class of product type methods whose residual polynomials can be factored by the residual polynomial of bi cg and other polynomials with standard three term recurrence relations such product type methods which are based on bi cg can be regarded as generalizations of bi cgstab from the unified way the well known variants of the product type methods like cgs bi cgstab bi cgstab2 are reacquired again

retrieval phase diagrams for attractor neural networks with optimal interactions the authors consider the retrieval properties of attractor neural networks whose synaptic matrices have been constructed to maximise the number of patterns which can be stored in a perceptron satisfying certain constraints retrieval is studied in the absence as well as in the presence of fast noise temperature the discussion is restricted to dilute networks for which dynamical equations for the overlaps are available when the patterns are stored with a prescribed lower limit on the stability parameter on every site the full stability of the perceptron ensures the existence of an attractor with perfect retrieval it is found that the curve of critical storage capacity alpha against temperature t as a line of first order transitions for high values of alpha and becomes second order for low alpha at a point of a tricritical nature the phase diagram is compared with the dilute hopfield model it is found that at high synaptic noise levels the diluted hopfield net stores more effectively than the network trained for optimal perceptron storage when a given fraction of sites is allowed to violate the stability bound the solution of the perceptron learning problem does not ensure the existence of an attractor of finite overlap even in the absence of noise this case is studied separately for t 0 and for finite t

learning to control dynamical behaviour abstract control of dynamical behaviour requires the transformation of one dynamical sequence into another the ability of a neural network to perform this task depends on the complexity of a time series involved and on the connectivity of the network used scaling estimates of the information extracted during learning and the resulting control error achieved are compared with simulations of period doubling maps

path positivity and infinite coxeter groups abstract a coxeter graph is a connected graph each of whose edges is labeled with an integer 3 or with the adjacency matrix of a coxeter graph g denoted by a g aij is defined to be a square matrix of order v where a ij 2 cos p if the edge i j is labeled with the integer p and 0 if there is no edge joining vertex i with vertex j for any positive integer k we denote by pk the characteristic polynomial of the adjacency matrix of the path on k vertices a coxeter graph g is said to be path positive if for all positive integers k the matrix pk a g is entrywise nonnegative it is shown that with the exception of a few cases which are a b d e f h and i any coxeter graph is path positive the result can be interpreted as a new criterion for the infiniteness of a coxeter group

from statistical mechanics to neural networks and back abstract the neural network models which have attracted so much attention in the physics community consist of a large assembly of two state elements highly interconnected by ferromagnetic and antiferromagnetic interactions methods of statistical mechanics have been applied to these models bringing new discoveries and surprises to neural network science at the same time neural network models have posed new problems and incentives to statistical mechanics one of the basic problems is that of learning which in the context of these models means a choice of the interactions for which the network dynamics leads to specific attractors the main emphasis in this paper is on the essential developments and results on this problem derived from methods of statistical mechanics

categorization in the pseudo inverse neural network we investigate analytically the emergence of the categorization ability in the pseudo inverse attractor neural network more pointedly we consider the problem of learning an extensive number of concepts by storing a finite number of examples s of each concept we find that there is a critical value beyond which the categorization error as measured by the average fraction of unstable sites in the concepts decreases monotonically with s

large systems of random linear equations with nonnegative solutions characterizing the solvable and the unsolvable phase large systems of linear equations are ubiquitous in science quite often e g when considering population dynamics or chemical networks the solutions must be nonnegative recently it has been shown that large systems of random linear equations exhibit a sharp transition from a phase where a nonnegative solution exists with probability one to one where typically no such solution may be found the critical line separating the two phases was determined by combining farkas lemma with the replica method here we show that the same methods remain viable to characterize the two phases away from criticality to this end we analytically determine the residual norm of the system in the unsolvable phase and a suitable measure of robustness of solutions in the solvable one our results are in very good agreement with numerical simulations

higher order corrections to the effective potential close to the jamming transition in the perceptron model in view of the results achieved in a previously related work a altieri s franz and g parisi j stat mech 2016 093301 10 1088 1742 5468 2016 09 093301 regarding a plefka like expansion of the free energy up to the second order in the perceptron model we improve the computation here focusing on the role of third order corrections the perceptron model is a simple example of constraint satisfaction problem falling in the same universality class as hard spheres near jamming and hence allowing us to get exact results in high dimensions for more complex settings our method enables to define an effective potential or thouless anderson palmer free energy namely a coarse grained functional which depends on the generalized forces and the effective gaps between particles the analysis of the third order corrections to the effective potential reveals that albeit irrelevant in a mean field framework in the thermodynamic limit they might instead play a fundamental role in considering finite size effects we also study the typical behavior of generalized forces and we show that two kinds of corrections can occur the first contribution arises since the system is analyzed at a finite distance from jamming while the second one is due to finite size corrections we nevertheless show that third order corrections in the perturbative expansion vanish in the jamming limit both for the potential and the generalized forces in agreement with the isostaticity argument proposed by wyart and coworkers finally we analyze the relevant scaling solutions emerging close to the jamming line which define a crossover regime connecting the control parameters of the model to an effective temperature

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

ml n bicgstabt a ml n bicgstab variant with a transpose the 1980 idr method wesseling and sonneveld 1980 12 plays an important role in the history of krylov subspace methods it started the research of transpose free krylov subspace methods the ml n bicgstab method yeung 2012 is one of such methods in this paper we present a new ml n bicgstab variant that involves a transpose in its implementation comparison of this new algorithm with the existing ml n bicgstab algorithms and some other krylov subspace algorithms will be presented

ml k bicgstab a bicgstab variant based on multiple lanczos starting vectors we present a variant of the popular bicgstab method for solving nonsymmetric linear systems the method which we denote by ml k bicgstab is derived from a variant of the bicg method based on a lanczos process using multiple k 1 starting left lanczos vectors compared with the original bicgstab method our new method produces a residual polynomial which is of lower degree after the same number of steps but which also requires fewer matrix vector products to generate on average requiring only 1 1 k matvecs per step empirically it also seems to be more stable and more quickly convergent the new method can be implemented as a k term recurrence and can be viewed as a bridge connecting the arnoldi based fom gmres methods and the lanczos based bicgstab methods

disproof of a conjecture on the existence of the path recursive period for a connected graph abstract a sufficient and necessary condition on the existence of the path recursive period for a graph has been established in this paper this disproves the conjecture proposed in r shi path polynomials of a graph linear algebra appl 236 1996 181 187 some results presented in r shi path polynomials of a graph linear algebra appl 236 1996 181 187 r b bapat a k lal path positive graphs linear algebra appl 149 1991 125 149 have also been included

perceptrons above saturation we study the storage of random patterns by a perceptron above its storage capacity alpha c i e in the region where perfect storage becomes impossible we determine the minimal fraction of learning errors and the distribution of stabilities for different learning rules in one step replica symmetry breaking thereby we not only extend the known replica symmetric results to values of the storage capacity beyond the at line but also show that depending on the learning rule replica symmetry may be globally unstable already well below the at line as an example for possible implications we compare the results for the typical basins of attraction of an extremely diluted attractor neural network as given by replica symmetry and one step replica symmetry breaking

critical capacity of constrained perceptrons the authors calculate the critical capacity of a single layer perceptron with interactions ji constrained to take values on a general surface the critical capacity is expressed in terms of a geometrical property of the constraining surface

variants of bicgstab for matrices with complex spectrum recently van der vorst siam j sci statist comput 13 1992 pp 631 644 proposed for solving nonsymmetric linear systems az b a biconjugate gradient bicg based krylov space method called bicgstab that like the biconjugate gradient squared bicgs method of sonneveld does not require matrix vector multiplications with the transposed matrix a t and that has typically a much smoother convergence behavior than bicg and bicgs its nth residual polynomial is the product of the one of bicg i e the nth lanczos polynomial with a polynomial of the same degree with real zeros therefore nonreal eigenvalues of a are not approximated well by the second polynomial factor here the author presents for real nonsymmetric matrices a method bicgstab2 in which the second factor may have complex conjugate zeros moreover versions suitable for complex matrices are given for both methods

statistical mechanics of lossy data compression using a nonmonotonic perceptron the performance of a lossy data compression scheme for uniformly biased boolean messages is investigated via methods of statistical mechanics inspired by a formal similarity to the storage capacity problem in neural network research we utilize a perceptron of which the transfer function is appropriately designed in order to compress and decode the messages employing the replica method we analytically show that our scheme can achieve the optimal performance known in the framework of lossy compression in most cases when the code length becomes infinite the validity of the obtained results is numerically confirmed

exploiting bicgstab ell strategies to induce dimension reduction idr s p sonneveld and m b van gijzen siam j sci comput 31 2008 pp 1035 1062 and bicgstab ell g l g sleijpen and d r fokkema electron trans numer anal 1 1993 pp 11 32 are two of the most efficient short recurrence iterative methods for solving large nonsymmetric linear systems of equations which of the two is best depends on the specific problem class in this paper we describe idrstab a new method that combines the strengths of idr s and bicgstab ell to derive idrstab we extend the results that we reported on in g l g sleijpen p sonneveld and m b van gijzen appl numer math 2009 doi 10 1016 j apnum 2009 07 001 where we considered bi cgstab as an induced dimension reduction idr method we will analyze the relation between hybrid bi cg methods and idr and introduce the new concept of the sonneveld subspace as a common framework through numerical experiments we will show that idrstab can outperform both idr s and bicgstab ell

an implementation of the look ahead lanczos algorithm for non hermitian matrices the nonsymmetric lanczos method can be used to compute eigenvalues of large sparse non hermitian matrices or to solve large sparse non hermitian linear systems however the original lanczos algorithm is susceptible to possible breakdowns and potential instabilities an implementation of a look ahead version of the lanczos algorithm is presented that except for the very special situation of an incurable breakdown overcomes these problems by skipping over those steps in which a breakdown or near breakdown would occur in the standard process the proposed algorithm can handle look ahead steps of any length and requires the same number of matrix vector products and inner products as the standard lanczos process without look ahead

path positive graphs for any positive integer k let p denote the characteristic polynomial of the adjacency matrix of the path on k vertices we obtain certain results regarding t evaluated at a c the adjacency matrix of any given graph g for some special graphs g we describe the matrix p a g completely we call a graph g path positive if i a g is a nonnegative matrix for all positive integers k it is shown that if g is a connected graph with a vertex of degree at least four then g is path positive we conjecture that all connected graphs are path positive apart from a few exceptional cases described in the paper

a simple derivation of a bound on the perceptron margin using singular value decomposition the perceptron is a simple supervised algorithm to train a linear classifier that has been analyzed and used extensively the classifier separates the data into two groups using a decision hyperplane with the margin between the data and the hyperplane determining the classifier s ability to generalize and its robustness to input noise exact results for the maximal size of the separating margin are known for specific input distributions and bounds exist for arbitrary distributions but both rely on lengthy statistical mechanics calculations carried out in the limit of infinite input size here we present a short analysis of perceptron classification using singular value decomposition we provide a simple derivation of a lower bound on the margin and an explicit formula for the perceptron weights that converges to the optimal result for large separating margins

optimization by simulated annealing an experimental evaluation part ii graph coloring and number partitioning this is the second in a series of three papers that empirically examine the competitiveness of simulated annealing in certain well studied domains of combinatorial optimization simulated annealing is a randomized technique proposed by s kirkpatrick c d gelatt and m p vecchi for improving local optimization algorithms here we report on experiments at adapting simulated annealing to graph coloring and number partitioning two problems for which local optimization had not previously been thought suitable for graph coloring we report on three simulated annealing schemes all of which can dominate traditional techniques for certain types of graphs at least when large amounts of computing time are available for number partitioning simulated annealing is not competitive with the differencing algorithm of n karmarkar and r m karp except on relatively small instances moreover if running time is taken into account natural annealing schemes cannot even outperform multiple random runs of the local optimization algorithms on which they are based in sharp contrast to the observed performance of annealing on other problems

a block idr s method for nonsymmetric linear systems with multiple right hand sides the idr s based on the induced dimension reduction idr theorem is a new class of efficient algorithms for large nonsymmetric linear systems idr 1 is mathematically equivalent to bicgstab at the even idr 1 residuals and idr s with s 1 is competitive with most bi cg based methods for these reasons we extend the idr s to solve large nonsymmetric linear systems with multiple right hand sides in this paper a variant of the idr theorem is given at first then the block idr s an extension of idr s based on the variant idr s theorem is proposed by analysis the upper bound on the number of matrix vector products of block idr s is the same as that of the idr s for a single right hand side in generic case i e the total number of matrix vector products of idr s may be m times that of of block idr s where m is the number of right hand sides numerical experiments are presented to show the effectiveness of our proposed method

idr s a family of simple and fast algorithms for solving large nonsymmetric systems of linear equations we present idr s a new family of efficient short recurrence methods for large nonsymmetric systems of linear equations the new methods are based on the induced dimension reduction idr method proposed by sonneveld in 1980 idr s generates residuals that are forced to be in a sequence of nested subspaces although idr s behaves like an iterative method in exact arithmetic it computes the true solution using at most n n s matrix vector products with n the problem size and s the codimension of a fixed subspace we describe the algorithm and the underlying theory and present numerical experiments to illustrate the theoretical properties of the method and its performance for systems arising from different applications our experiments show that idr s is competitive with or superior to most bi cg based methods and outperforms bi cgstab when s 1

storage capacity of ultrametric committee machines the problem of computing the storage capacity of a feed forward network with l hidden layers n inputs and k units in the first hidden layer is analyzed using techniques from statistical mechanics we found that the storage capacity strongly depends on the network architecture c log k 1 1 2l and that the number of units k limits the number of possible hidden layers l through the relationship 2l 1 2log k

on the polynomial of a path let a pn be the adjacency matrix of the path on n vertices suppose that r is a polynomial of degree less than n and consider the matrix m r a pn we determine all polynomials for which m is the adjacency matrix of a graph

on the capacity per synapse the optimal storage capacity of a perceptron with a finite fraction of sign constrained weights which are prescribed a priori is examined the storage capacity is calculated by considering the fractional volume of weights which can store a set of alpha n random patterns where n is the size of the input it is found that in the case where 1 s n weights are sign constrained the capacity is 1 s 2 alpha gc k where alpha gc k is the maximal storage capacity in gardner s case and k is a stability parameter

instance space of the number partitioning problem within the replica framework we study analytically the instance space of the number partitioning problem this classic integer programming problem consists of partitioning a sequence of n positive real numbers a1 a2 an the instance into two sets such that the absolute value of the difference of the sums of aj over the two sets is minimized we show that regardless of the distribution of the instance entries there is an upper bound cn to the number of perfect random partitions i e partitions for which that difference is zero in particular in the case where the two sets have the same cardinality balanced partitions we find c moreover in the case of unbalanced partitions we show that perfect random partitions exist only if the difference between the cardinalities of the two sets scales like mn1 2 where m is of the order of 1

gbi cgstab s l idr s with higher order stabilization polynomials idr s is now recognized as one of the most effective methods often superior to other krylov subspace methods for large nonsymmetric linear systems of equations in this paper we propose an improvement upon idr s by incorporating a higher order stabilization polynomial into idr s the proposed algorithm named gbi cgstab s l shares desirable features with both idr s and bi cgstab l

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

on the capacity of neural networks with binary weights the authors study the critical capacity alpha c of multilayered networks with binary couplings they show that for any network presenting a tree like architecture after the first hidden layer no fixed internal representation is required using gardner s calculations they apply statistical mechanics to the simplest network with two layers of adaptive weights following the same approach as for the binary perceptron they find from the zero entropy point a critical capacity alpha c 0 92 they discuss the validity of this result in view of exhaustive search simulations on small networks

a look ahead lanczos algorithm for unsymmetric matrices abstract the two sided lanczos algorithm is known to suffer from serious breakdowns these occur when the associated moment matrix does not permit triangular factorization we modify the algorithm slightly so that it corresponds to using a 2 x 2 pivot in triangular factorization whenever a 1 x 1 pivot would be dangerous the incidence of breakdown is greatly reduced the price paid is that the tridiagonal matrix produced by the algorithm now has bumps whenever a 2 x 2 pivot is used author

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

gmres with deflated restarting a modification is given of the gmres iterative method for nonsymmetric systems of linear equations the new method deflates eigenvalues using wu and simon s thick restarting approach siam j matrix anal appl 22 2000 pp 602 616 it has the efficiency of implicit restarting but is simpler and does not have the same numerical concerns the deflation of small eigenvalues can greatly improve the convergence of restarted gmres also it is demonstrated that using harmonic ritz vectors is important because then the whole subspace is a krylov subspace that contains certain important smaller subspaces

phase diagram of restricted boltzmann machines and generalized hopfield networks with arbitrary priors restricted boltzmann machines are described by the gibbs measure of a bipartite spin glass which in turn can be seen as a generalized hopfield network this equivalence allows us to characterize the state of these systems in terms of their retrieval capabilities both at low and high load of pure states we study the paramagnetic spin glass and the spin glass retrieval phase transitions as the pattern i e weight distribution and spin i e unit priors vary smoothly from gaussian real variables to boolean discrete variables our analysis shows that the presence of a retrieval phase is robust and not peculiar to the standard hopfield model with boolean patterns the retrieval region becomes larger when the pattern entries and retrieval units get more peaked and conversely when the hidden units acquire a broader prior and therefore have a stronger response to high fields moreover at low load retrieval always exists below some critical temperature for every pattern distribution ranging from the boolean to the gaussian case

the lanczos biorthogonalization algorithm and other oblique projection methods for solving large unsymmetric systems many powerful methods for solving systems of equations can be regarded as projection methods most of the projection methods known for solving linear systems are orthogonal projection methods but little attention has been given to the class of nonorthogonal or oblique projection methods which is particularly attractive for large nonsymmetric systems the purpose of the paper is to present some methods in the general setting of oblique projection methods and to give some theoretical results some experiments comparing the various algorithms are reported

on the ability of the optimal perceptron to generalise a linearly separable boolean function is derived from a set of examples by a perceptron with optimal stability the probability to reconstruct a pattern which is not learnt is calculated analytically using the replica method

finite state neural networks a step toward the simulation of very large systems neural networks composed of neurons withq n states and synapses withq states are studied analytically and numerically analytically it is shown that these finite state networks are much more efficient at information storage than networks with continuous synapses in order to take the utmost advantage of networks with finite state elements a multineuron and multisynapse coding scheme is introduced which allows the simulation of networks having 1 0 109 couplings at a speed of 7 1 109 coupling evaluations per second on asingle processor of the cray ymp a local learning algorithm is also introduced which allows for the efficient training of large networks with finite state elements

optimal storage of invariant sets of patterns in neural network memories the authors investigate optimal conditions for the storage of invariant sets of patterns in neural network memories the sets of patterns that they consider are highly correlated non random and invariant with respect to some symmetry transformations they generalize gardner s method to study the fractional volume in the space of neural interactions that allow for storage of invariant patterns they demonstrate that optimal storage conditions correspond to non trivial relations between the number of stored patterns their stability and other symmetry specific characteristics of patterns

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

phase transition in the number partitioning problem universit at magdeburg institut fu r theoretische physik universit atsplatz 2 d 39106 magdeburg germany february 1 2008 number partitioning is an np complete problem of com binatorial optimization a statistical mechanics analysis re veals the existence of a phase transition that separates theeasy from the hard to solve instances and that re ects thepseudo polynomiality of number partitioning the phase di agram and the value of the typical ground state energy arecalculated 64 60 cn 02 60 pn 02 70 lq 89 80 h

on the convergence behavior of idr s and related methods an explanation is given of the convergence behavior of idr s methods the convergence mechanism of these algorithms has two components the first consists of damping properties of certain factors in the residual polynomials which becomes less important for large values of s the second component depends on the behavior of lanczos polynomials that occur in the theoretical description this part of the residual polynomials is related to lanczos methods with s left starting vectors as described in a paper by yeung and chan on their ml k bicgstab method in siam j sci comput 21 1999 pp 1263 1290 in this paper the behavior of the second component is compared with the full gmres method siam j sci statist comput 7 1986 pp 856 869 and an expected rate of convergence is given based on a random choice of the s shadow vectors

capacity of neural networks with discrete synaptic couplings the authors study the optimal storage capacity of neural networks with discrete local constraints on the synaptic couplings jij models with such constraints include those with binary couplings jij or 1 or jij 0 1 quantised couplings with larger synaptic range e g jij or 1 l or 2 l or 1 and in the limit continuous couplings confined to the hypercube mod jij mod or 1 box confinement they find that the optimal storage capacity alpha kappa is best determined by the vanishing of a suitably defined entropy as calculated in the replica symmetric approximation they also extend their results to cases with biased memories and make contact with sparse coding models

a replica analysis of the minimum weight solution to the linear equations problem we investigate through statistical mechanics techniques the problem of finding n tuples j ji j2 jn with a fixed fraction k of non vanishing entries that solve a set of p alpha n linear equations in particular we determine the region in the plane alpha k where these solutions exist with a probability of 1 in the limit of large n we also obtain at insight into the microscopic structure of these solutions by calculating the distribution of the probability of an arbitrary non vanishing entry ji moreover we evaluate analytically the performances of several easy to implement heuristic algorithms and compare them with the optimal solution a task that is suited very well to the statistical mechanics approach

a dilution algorithm for neural networks a dilution algorithm to enlarge the storage capacity per synapse alpha eff of neural networks is proposed the algorithm is a hybrid method where hebb s rule is used to select a fraction of couplings to be removed afterwards the perceptron of optimal stability for the remaining couplings is learned the authors present an analytical calculation and the results of the numerical simulations in comparison with the fully connected or the randomly diluted perceptron the effective storage capacity alpha eff is remarkably enlarged

learning by dilution in a neural network a perceptron with n random weights can store of the order of n patterns by removing a fraction of the weights without changing their strengths the critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated a simple hebb like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability

statistical mechanics of neural networks what are the differences between wide and narrow basins abstract we consider training noise in neural networks as a means of tuning the structure of retrieval basins and study how learning and retrieving properties depend on it the stability of the replica symmetric solution and the correlation in the weight space indicate that neural networks can be roughly classified into hebbian like and msn like msn meaning the maximally stable network re entrant retrieval noise robustness selectivity damage spreading and activity distribution all illustrate the differences in retrieval behaviours arising from the different basin structures

dilution in boolean perceptrons that learn from noisy examples we investigate the effect of dilution after learning on the generalization ability of single layer boolean perceptrons that learn from noisy examples we present a thorough comparison between the relative performances of several well known learning rules in particular we show that the effect of dilution is always deleterious and that the bayes algorithm always gives the best generalization performance

probabilistic analysis of the number partitioning problem given a sequence of n positive real numbers the number partitioning problem consists of partitioning them into two sets such that the absolute value of the difference of the sums of over the two sets is minimized in the case in which the s are statistically independent random variables uniformly distributed in the unit interval this np complete problem is equivalent to the problem of finding the ground state of an infinite range random antiferromagnetic ising model we employ the annealed approximation to derive analytical lower bounds to the average value of the difference for the best constrained and unconstrained partitions in the large n limit furthermore we calculate analytically the fraction of metastable states i e states that are stable against all single spin flips and found that it vanishes like