mean field message passing equations in the hopfield model and its generalizations motivated by recent progress in using restricted boltzmann machines as preprocessing algorithms for deep neural network we revisit the mean field equations belief propagation and thouless anderson palmer tap equations in the best understood of such machines namely the hopfield model of neural networks and we explicit how they can be used as iterative message passing algorithms providing a fast method to compute the local polarizations of neurons in the retrieval phase where neurons polarize in the direction of one memorized pattern we point out a major difference between the belief propagation and tap equations the set of belief propagation equations depends on the pattern which is retrieved while one can use a unique set of tap equations this makes the latter method much better suited for applications in the learning process of restricted boltzmann machines in the case where the patterns memorized in the hopfield model are not independent but are correlated through a combinatorial structure we show that the tap equations have to be modified this modification can be seen either as an alteration of the reaction term in tap equations or more interestingly as the consequence of message passing on a graphical model with several hidden layers where the number of hidden layers depends on the depth of the correlations in the memorized patterns this layered structure is actually necessary when one deals with more general restricted boltzmann machines

storage of sets of correlated data in neural network memories in most applications of neural network models one has to consider sets of correlated data the authors study the problem of the storage of associated patterns in neural network memories they consider two basic types of correlation semantic ones among the different patterns and spatial ones among the different sites of the network they apply gardner s program to evaluate optimal storage conditions for both kinds of correlation the spatial ones worsen generally speaking the storage properties of a simple perceptron while they improve them for the hopfield network in the case of semantic correlations they obtain bounds of the critical capacity for both kinds of networks the storage ratio of the perceptron may be significantly increased in this case

additive spectral method for fuzzy cluster analysis of similarity data including community structure and affinity matrices an additive spectral method for fuzzy clustering is proposed the method operates on a clustering model which is an extension of the spectral decomposition of a square matrix the computation proceeds by extracting clusters one by one which makes the spectral approach quite natural the iterative extraction of clusters also allows us to draw several stopping rules to the procedure this applies to several relational data types differently normalized network structure data the first eigenvector subtracted affinity between multidimensional vectors the pseudo inverse laplacian transformation and conventional relational data including in house data of similarity between research topics according to working of a research center the method is experimentally compared with several classic and recent techniques and shown to be competitive

replica symmetry breaking in noise optimal neural networks recent studies of optimization in neural networks trained with noisy data have shown that replica symmetric solutions are unstable in the low noise region of parameter space we calculate the 1 step replica symmetry broken solution in this region which joins the replica symmetric solution continuously at the de almeida thouless line these solutions yield satisfactory agreement with simulations for the aligning field distribution better than those given by the replica symmetric ansatz

numerical comparison of the rfcm and ap algorithms for clustering relational data we describe and compare two fuzzy clustering algorithms based on optimizing penalty functions of relational data more specifically the ap algorithm of windham is contrasted with the relational fuzzy c means approach through a numerical example on a small artificial data set defined by windham our results indicate that while the two algorithms seem to cluster objects using very different mathematical criteria the numerical results can be quite similar however a second numerical example using a slightly distorted version of windham s data seems to exhibit real differences between clusterings obtained by the two approaches

gaussian beam decomposition of high frequency wave fields in this paper we present a method of decomposing a highly oscillatory wave field into a sparse superposition of gaussian beams the goal is to extract the necessary parameters for a gaussian beam superposition from this wave field so that further evolution of the high frequency waves can be computed by the method of gaussian beams the methodology is described for r d with numerical examples for d 2 in the first example a field generated by an interface reflection of gaussian beams is decomposed into a superposition of gaussian beams the beam parameters are reconstructed to a very high accuracy the data in the second example is not a superposition of a finite number of gaussian beams the wave field to be approximated is generated by a finite difference method for a geometry with two slits the accuracy in the decomposition increases monotonically with the number of beams

higher order corrections to the effective potential close to the jamming transition in the perceptron model in view of the results achieved in a previously related work a altieri s franz and g parisi j stat mech 2016 093301 10 1088 1742 5468 2016 09 093301 regarding a plefka like expansion of the free energy up to the second order in the perceptron model we improve the computation here focusing on the role of third order corrections the perceptron model is a simple example of constraint satisfaction problem falling in the same universality class as hard spheres near jamming and hence allowing us to get exact results in high dimensions for more complex settings our method enables to define an effective potential or thouless anderson palmer free energy namely a coarse grained functional which depends on the generalized forces and the effective gaps between particles the analysis of the third order corrections to the effective potential reveals that albeit irrelevant in a mean field framework in the thermodynamic limit they might instead play a fundamental role in considering finite size effects we also study the typical behavior of generalized forces and we show that two kinds of corrections can occur the first contribution arises since the system is analyzed at a finite distance from jamming while the second one is due to finite size corrections we nevertheless show that third order corrections in the perturbative expansion vanish in the jamming limit both for the potential and the generalized forces in agreement with the isostaticity argument proposed by wyart and coworkers finally we analyze the relevant scaling solutions emerging close to the jamming line which define a crossover regime connecting the control parameters of the model to an effective temperature

replica symmetry breaking in perceptrons in the problem of optimization of pattern stabilization in perceptrons the replica symmetric ansatz is known to be mathematically unstable for storage capacities greater than some in this paper we demonstrate that for greater than the one step replica symmetry broken rsb solution is also unstable we further show that in this region full rsb is necessary for an exact solution direct evaluation of the two step rsb solution yields a minimum storage error which is only slightly greater than the one step rsb which itself is greater than that given by the unstable replica symmetric ansatz by a much larger amount

weight space structure and generalization in the reversed wedge perceptron the generalization ability of the reversed wedge perceptron serving as a toy model for multilayer neural networks is investigated we analyse the decomposition of the version space into disjoint cells belonging to different internal representations defined by the signs of the aligning fields the version space is characterized by the number and size of these cells and their typical overlap with the teacher network for a small training set the system is unable to detect the structure of the patterns induced by the teacher accordingly it performs as if storing random input output patterns with very low generalization ability and a large misfit in the internal representation with increasing training set size cells with large misfit are eliminated at a much higher rate than those with internal representation similar to that of the teacher this results eventually in the discontinuous phase transition to good generalization typical for multilayer neural networks

statistical mechanics of lossy data compression using a nonmonotonic perceptron the performance of a lossy data compression scheme for uniformly biased boolean messages is investigated via methods of statistical mechanics inspired by a formal similarity to the storage capacity problem in neural network research we utilize a perceptron of which the transfer function is appropriately designed in order to compress and decode the messages employing the replica method we analytically show that our scheme can achieve the optimal performance known in the framework of lossy compression in most cases when the code length becomes infinite the validity of the obtained results is numerically confirmed

prestack gaussian beam depth migration kirchhoff migration is the most popular method of three dimensional prestack depth migration because of its flexibility and efficiency its effectiveness can become limited however when complex velocity structure causes multipathing of seismic energy an alternative is gaussian beam migration which is an extension of kirchhoff migration that overcomes many of the problems caused by multipathing unlike first arrival and most energetic arrival methods which retain only one traveltime this alternative method retains most arrivals by the superposition of gaussian beams this paper presents a prestack gaussian beam migration method that operates on common offset gathers the method is efficient because the computation of beam superposition isolates summations that do not depend on the seismic data and evaluates these integrals by considering their saddle points gaussian beam migration of the two dimensional marmousi test data set demonstrates the method s effectiveness for structural imaging in a case w

generalization from correlated sets of patterns in the perceptron generalization is a central aspect of learning theory here we proposea 13 a framework that explores an auxiliary task dependent notion of gener a 13 alization and attempts to quantitatively answer the following question a 13 given two sets of patterns with a given degree of dissimilarity how easilya 13 will a network be able to unify their interpretation this is quantifieda 13 by the volume of the configurations of synaptic weights that classify thea 13 two sets in a similar manner to show the applicability of our idea ina 13 a concrete setting we compute this quantity for the perceptron a sim a 13 ple binary classifier using the classical statistical physics approach in thea 13 replica symmetric ansatz in this case we show how an analytical ex a 13 pression measures the distance based capacity the maximum load ofa 13 patterns sustainable by the network at fixed dissimilarity between pat a 13 terns and fixed allowed number of errors this curve indicates that gen a 13 eralization is possible at any distance but with decreasing capacity wea 13 propose that a distance based definition of generalization may be usefula 13 in numerical experiments with real world neural networks and to explorea 13 computationally sub dominant sets of synaptic solutions

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

nerf c means non euclidean relational fuzzy clustering abstract the relational fuzzy c means rfcm algorithm can be used to cluster a set of n objects described by pair wise dissimilarity values if and only if there exist n points in r n 1 whose squared euclidean distances precisely match the given dissimilarity data this strong restriction on the dissimilarity data renders rfcm inapplicable to most relational clustering problems this paper substantially improves rfcm by generalizing it to the case of arbitrary symmetric dissimilarity data the generalization is obtained using a computationally efficient modification of the existing algorithm that is equivalent to applying a spreading transformation to the dissimilarity data while the method given applies specifically to dissimilarity data a simple transformation can be used to convert similarity relations into dissimilarity data so the method is applicable to any numerical relational data that are positive reflexive or anti reflexive and symmetric numerical examples illustrate and compare the present approach to problems that can be studied with alternatives such as the linkage algorithms

random geometric graphs in high dimension many machine learning algorithms used for dimensional reduction and manifold learning leverage on the computation of the nearest neighbors to each point of a data set to perform their tasks these proximity relations define a so called geometric graph where two nodes are linked if they are sufficiently close to each other random geometric graphs where the positions of nodes are randomly generated in a subset of r d offer a null model to study typical properties of data sets and of machine learning algorithms up to now most of the literature focused on the characterization of low dimensional random geometric graphs whereas typical data sets of interest in machine learning live in high dimensional spaces d 10 2 in this work we consider the infinite dimensions limit of hard and soft random geometric graphs and we show how to compute the average number of subgraphs of given finite size k e g the average number of k cliques this analysis highlights that local observables display different behaviors depending on the chosen ensemble soft random geometric graphs with continuous activation functions converge to the naive infinite dimensional limit provided by erdos renyi graphs whereas hard random geometric graphs can show systematic deviations from it we present numerical evidence that our analytical results exact in infinite dimensions provide a good approximation also for dimension d 10

storage capacities of committee machines with overlapping and non overlapping receptive fields we present theoretical investigations via the replica theory of the storage capacities of committee machines with a large number m of hidden units and spherical weights difficulties arise in the solution of this problem in the limit of large m in the case of overlapping receptive fields as the number of patterns increases both permutation symmetry and replica symmetry are broken which leads to the appearance of many order parameters and causes additional difficulty we observe that the relations among these order parameters yield a set of quantities which are small in the limit of large m making the asymptotic calculation tractable using the one step replica symmetry breaking scheme we compute the asymptotic value of the storage capacity per input unit in the limit of large m we find that the shift to the case of non overlapping receptive fields can be made easily we then find both values satisfy the bound of mitchison and durbin

learning by dilution in a neural network a perceptron with n random weights can store of the order of n patterns by removing a fraction of the weights without changing their strengths the critical storage capacity as a function of the concentration of the remaining bonds for random outputs and for outputs given by a teacher perceptron is calculated a simple hebb like dilution algorithm is presented which in the teacher case reaches the optimal generalization ability

threshold induced phase transitions in perceptrons error rates of a boolean perceptron with threshold and either spherical or ising constraint on the weight vector are calculated for storing patterns from biased input and output distributions derived within a one step replica symmetry breaking rsb treatment for unbiased output distribution and non zero stability of the patterns we find a critical load p above which two solutions to the saddlepoint equations appear one with higher free energy and zero threshold and a dominant solution with non zero threshold we examine this second order phase transition and the dependence of p on the required pattern stability for both one step rsb and replica symmetry rs in the spherical case and for one step rsb in the ising case

a dilution algorithm for neural networks a dilution algorithm to enlarge the storage capacity per synapse alpha eff of neural networks is proposed the algorithm is a hybrid method where hebb s rule is used to select a fraction of couplings to be removed afterwards the perceptron of optimal stability for the remaining couplings is learned the authors present an analytical calculation and the results of the numerical simulations in comparison with the fully connected or the randomly diluted perceptron the effective storage capacity alpha eff is remarkably enlarged

a replica analysis of the minimum weight solution to the linear equations problem we investigate through statistical mechanics techniques the problem of finding n tuples j ji j2 jn with a fixed fraction k of non vanishing entries that solve a set of p alpha n linear equations in particular we determine the region in the plane alpha k where these solutions exist with a probability of 1 in the limit of large n we also obtain at insight into the microscopic structure of these solutions by calculating the distribution of the probability of an arbitrary non vanishing entry ji moreover we evaluate analytically the performances of several easy to implement heuristic algorithms and compare them with the optimal solution a task that is suited very well to the statistical mechanics approach

from shrinking to percolation in an optimization model a model of noise reduction for signal processing and other optimization tasks is introduced each noise source puts a symmetric constraint on the space of the signal vector within a tolerance bound when the number of noise sources increases sequences of transitions take place causing the solution space to vanish we find that the transition from an extended solution space to a shrunk space is retarded because of the symmetry of the constraints in contrast with the analogous problem of pattern storage for low tolerance the solution space vanishes by volume reduction whereas for high tolerance the vanishing becomes more and more like percolation

storage capacity of a neural network with state dependent synapses the storage capacity of the hopfield network is limited to p n c 0 138 p number of patterns n number of neurons beyond which the contribution of weakly correlated patterns surpasses that of the desired pattern this contribution can be eliminated by introducing a threshold a pattern correlation below this threshold is simply set to zero in the synapses we solve the mean field equations and derive the critical value of the threshold required to stabilize p n patterns with c

a method of relational fuzzy clustering based on producing feature vectors using fastmap the first stage of organizing objects is to partition them into groups or clusters the clustering is generally done on individual object data representing the entities such as feature vectors or on object relational data incorporated in a proximity matrix this paper describes another method for finding a fuzzy membership matrix that provides cluster membership values for all the objects based strictly on the proximity matrix this is generally referred to as relational data clustering the fuzzy membership matrix is found by first finding a set of vectors that approximately have the same inter vector euclidian distances as the proximities that are provided these vectors can be of very low dimension such as 5 or less fuzzy c means fcm is then applied to these vectors to obtain a fuzzy membership matrix in addition two dimensional vectors are also created to provide a visual representation of the proximity matrix this allows comparison of the result of automatic clustering to visual clustering the method proposed here is compared to other relational clustering methods including nerfcm rouben s method and windhams a p method various clustering quality indices are also calculated for doing the comparison using various proximity matrices as input simulations show the method to be very effective and no more computationally expensive than other relational data clustering methods the membership matrices that are produced by the proposed method are less crisp than those produced by nerfcm and more representative of the proximity matrix that is used as input to the clustering process

performance of the bayesian online algorithm for the perceptron in this letter we derive continuum equations for the generalization error of the bayesian online algorithm bona for the one layer perceptron with a spherical covariance matrix using the rosenblatt potential and show by numerical calculations that the asymptotic performance of the algorithm is the same as the one for the optimal algorithm found by means of variational methods with the added advantage that the bona does not use any inaccessible information during learning

linear readout of object manifolds objects are represented in sensory systems by continuous manifolds due to sensitivity of neuronal responses to changes in physical features such as location orientation and intensity what makes certain sensory representations better suited for invariant decoding of objects by downstream networks we present a theory that characterizes the ability of a linear readout network the perceptron to classify objects from variable neural responses we show how the readout perceptron capacity depends on the dimensionality size and shape of the object manifolds in its input neural representation

gardner derrida neural networks with correlated patterns the storage properties of an optimal neural network with correlated patterns is studied allowing for a finite fraction of errors determined by the gardner derrida cost function a discontinuity in the probability distribution of the local stabilities is seen as a drastic decrease in the minimal fraction of errors there is also an enlargement of the domain in which the replica symmetric solution is stable allowing for higher storage capacities

critical capacity of constrained perceptrons the authors calculate the critical capacity of a single layer perceptron with interactions ji constrained to take values on a general surface the critical capacity is expressed in terms of a geometrical property of the constraining surface

storage capacity of ultrametric committee machines the problem of computing the storage capacity of a feed forward network with l hidden layers n inputs and k units in the first hidden layer is analyzed using techniques from statistical mechanics we found that the storage capacity strongly depends on the network architecture c log k 1 1 2l and that the number of units k limits the number of possible hidden layers l through the relationship 2l 1 2log k

error estimates for gaussian beam superpositions gaussian beams are asymptotically valid high frequency solutions to hyperbolic partial differential equations concentrated on a single curve through the physical domain they can also be extended to some dispersive wave equations such as the schrodinger equation superpositions of gaussian beams provide a powerful tool to generate more general high frequency solutions that are not necessarily concentrated on a single curve this work is concerned with the accuracy of gaussian beam superpositions in terms of the wavelength epsilon we present a systematic construction of gaussian beam superpositions for all strictly hyperbolic and schrodinger equations subject to highly oscillatory initial data of the form ae i phi epsilon through a careful estimate of an oscillatory integral operator we prove that the k th order gaussian beam superposition converges to the original wave field at a rate proportional to epsilon k 2 in the appropriate norm dictated by the well posedness estimate in particular we prove that the gaussian beam superposition converges at this rate for the acoustic wave equation in the standard epsilon scaled energy norm and for the schrodinger equation in the l 2 norm the obtained results are valid for any number of spatial dimensions and are unaffected by the presence of caustics we present a numerical study of convergence for the constant coefficient acoustic wave equation in r 2 to analyze the sharpness of the theoretical results

on the capacity of neural networks with binary weights the authors study the critical capacity alpha c of multilayered networks with binary couplings they show that for any network presenting a tree like architecture after the first hidden layer no fixed internal representation is required using gardner s calculations they apply statistical mechanics to the simplest network with two layers of adaptive weights following the same approach as for the binary perceptron they find from the zero entropy point a critical capacity alpha c 0 92 they discuss the validity of this result in view of exhaustive search simulations on small networks

statistical mechanics calculation of vapnik chervonenkis bounds for perceptrons using the replica technique we calculate the maximal possible difference between the learning and the generalization error of a perceptron learning a linearly separable boolean classification from examples we consider both spherical and ising constraints on the couplings of the perceptron investigate learnable as well as unlearnable problems and study the special situation where the class of perceptrons considered is restricted to the version space the results are compared with the vapnik chervonenkis bound and variants thereof we find that these bounds are asymptotically tight within logarithmic corrections

storage of correlated patterns in a perceptron we calculate the storage capacity of a perceptron for correlated gaussian patterns we find that the storage capacity alpha c can be less than 2 if similar patterns are mapped onto different outputs and vice versa as long as the patterns are in a general position we obtain in contrast to previous works that alpha c or 1 in agreement with cover s theorem numerical simulations confirm the results

learning strategy for the binary perceptron pursuing the work of penney and sherrington we determine the optimal continuous weight perceptron which on clipping correctly predicts the largest number of weights for the binary perceptron with maximum stability we calculate the fraction of correctly predicted binary weights when only the continuous weights stronger than a certain threshold are clipped we finally carry out simulations for a perceptron with 50 weights to test the practicability of different learning strategies

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

bayes optimal performance in a discrete space we study a simple model of unsupervised learning where the single symmetry breaking vector has binary components 1 we calculate exactly the bayes optimal performance of an estimator which is required to lie in the same discrete space we also show that except for very special cases such an estimator cannot be obtained by minimization of a class of variationally optimal potentials

optimal storage of invariant sets of patterns in neural network memories the authors investigate optimal conditions for the storage of invariant sets of patterns in neural network memories the sets of patterns that they consider are highly correlated non random and invariant with respect to some symmetry transformations they generalize gardner s method to study the fractional volume in the space of neural interactions that allow for storage of invariant patterns they demonstrate that optimal storage conditions correspond to non trivial relations between the number of stored patterns their stability and other symmetry specific characteristics of patterns

origin of the computational hardness for learning with binary synapses through supervised learning in a binary perceptron one is able to classify an extensive number of random patterns by a proper assignment of binary synaptic weights however to find such assignments in practice is quite a nontrivial task the relation between the weight space structure and the algorithmic hardness has not yet been fully understood to this end we analytically derive the franz parisi potential for the binary perceptron problem by starting from an equilibrium solution of weights and exploring the weight space structure around it our result reveals the geometrical organization of the weight space the weight space is composed of isolated solutions rather than clusters of exponentially many close by solutions the pointlike clusters far apart from each other in the weight space explain the previously observed glassy behavior of stochastic local search heuristics

statistical mechanics of neural networks what are the differences between wide and narrow basins abstract we consider training noise in neural networks as a means of tuning the structure of retrieval basins and study how learning and retrieving properties depend on it the stability of the replica symmetric solution and the correlation in the weight space indicate that neural networks can be roughly classified into hebbian like and msn like msn meaning the maximally stable network re entrant retrieval noise robustness selectivity damage spreading and activity distribution all illustrate the differences in retrieval behaviours arising from the different basin structures

dilution in boolean perceptrons that learn from noisy examples we investigate the effect of dilution after learning on the generalization ability of single layer boolean perceptrons that learn from noisy examples we present a thorough comparison between the relative performances of several well known learning rules in particular we show that the effect of dilution is always deleterious and that the bayes algorithm always gives the best generalization performance

generalization and capacity of extensively large two layered perceptrons the generalization ability and storage capacity of a treelike two layered neural network with a number of hidden units scaling as the input dimension is examined the mapping from the input to the hidden layer is via boolean functions the mapping from the hidden layer to the output is done by a perceptron the analysis is within the replica framework where an order parameter characterizing the overlap between two networks in the combined space of boolean functions and hidden to output couplings is introduced the maximal capacity of such networks is found to scale linearly with the logarithm of the number of boolean functions per hidden unit the generalization process exhibits a first order phase transition from poor to perfect learning for the case of discrete hidden to output couplings the critical number of examples per input dimension ensuremath alpha c at which the transition occurs again scales linearly with the logarithm of the number of boolean functions in the case of continuous hidden to output couplings the generalization error decreases according to the same power law as for the perceptron with the prefactor being different

robust fuzzy clustering of relational data popular relational data clustering algorithms relational dual of fuzzy c means rfcm non euclidean rfcm nerfcm both by hathaway et al and fanny by kaufman and rousseeuw are examined a new algorithm which is a generalization of fanny called the fuzzy relational data clustering frc algorithm is introduced having an identical objective functional as rfcm however the frc does not have the restriction of rfcm which is that the relational data is derived from euclidean distance as the measure of dissimilarity between the objects and it also does not have limitations of fanny including the use of a fixed membership exponent or a fuzzifier exponent m the frc algorithm is further improved by incorporating the concept of dave s object data noise clustering nc algorithm done by proposing a concept of noise dissimilarity next based on the constrained minimization which includes an inequality constraint for the memberships and corresponding kuhn tucker conditions a noise resistant frc algorithm is derived which works well for all types of non euclidean dissimilarity data thus it is shown that the extra computations for data expansion spl beta spread transformation required by the nerfcm algorithm are not necessary this new algorithm is called robust non euclidean fuzzy relational data clustering robust ne frc and its robustness is demonstrated through several numerical examples advantages of this new algorithm are faster convergence robustness against outliers and ability to handle all kinds of relational data including non euclidean the paper also presents a new and better interpretation of the noise class

a wavefront based gaussian beam method for computing high frequency wave propagation problems we present a novel wavefront method based on gaussian beams for computing high frequency wave propagation problems unlike standard geometrical optics gaussian beams compute the correct solution of the wave field also at caustics the method tracks a front of two canonical beams with two particular initial values for width and curvature in a fast post processing step from the canonical solutions we recreate any other gaussian beam with arbitrary initial data on the initial front this provides a simple mechanism to include a variety of optimization processes including error minimization and beam width minimization for a posteriori selection of optimal beam initial parameters the performance of the method is illustrated with two numerical examples

properties of neural networks storing spatially correlated patterns the author studies the behaviour of a feedforward neural network supplied with spatially organized data this inner structure is taken into account by a matrix cij whose coefficients equal the average correlation between two pixels i and j of the input patterns the storage capacity alpha is computed as a function of the required stability and of the eigenvalues of c the author proposes a geometrical transformation allowing an intuitive interpretation of these results numerical simulations using real and binary patterns show a very good agreement with the theory finally the author analyses the synaptic couplings correlations resulting from the training of the network with structured patterns focusing on exponentially decreasing correlations one and two dimensions the author finds that they exhibit a mexican hat profile the excitatory centre size of which depends on alpha

numerical microlocal analysis of harmonic wavefields we present and test a numerical method which given an analytical or numerical solution of the helmholtz equation in a neighborhood of a fixed observation point and assuming that the geometrical optics approximation is relevant determines at this point the number of crossing rays and computes their directions and associated complex amplitudes

learning and retrieval in attractor neural networks above saturation the authors investigate neural networks in the range of parameters when the ground state energy is positive namely when a synaptic matrix which satisfies all the desired constraints cannot be found by the learning algorithm in particular they calculate the typical distribution functions of local stabilities obtained for a number of algorithms in this region these functions are used to investigate the retrieval properties as reflected by the size of the basins of attraction this is done analytically in sparsely connected networks and numerically in fully connected networks the main conclusion is that the retrieval behaviour of attractor neural networks can be improved by learning above saturation

retrieval phase diagrams for attractor neural networks with optimal interactions the authors consider the retrieval properties of attractor neural networks whose synaptic matrices have been constructed to maximise the number of patterns which can be stored in a perceptron satisfying certain constraints retrieval is studied in the absence as well as in the presence of fast noise temperature the discussion is restricted to dilute networks for which dynamical equations for the overlaps are available when the patterns are stored with a prescribed lower limit on the stability parameter on every site the full stability of the perceptron ensures the existence of an attractor with perfect retrieval it is found that the curve of critical storage capacity alpha against temperature t as a line of first order transitions for high values of alpha and becomes second order for low alpha at a point of a tricritical nature the phase diagram is compared with the dilute hopfield model it is found that at high synaptic noise levels the diluted hopfield net stores more effectively than the network trained for optimal perceptron storage when a given fraction of sites is allowed to violate the stability bound the solution of the perceptron learning problem does not ensure the existence of an attractor of finite overlap even in the absence of noise this case is studied separately for t 0 and for finite t

learning to control dynamical behaviour abstract control of dynamical behaviour requires the transformation of one dynamical sequence into another the ability of a neural network to perform this task depends on the complexity of a time series involved and on the connectivity of the network used scaling estimates of the information extracted during learning and the resulting control error achieved are compared with simulations of period doubling maps

from statistical mechanics to neural networks and back abstract the neural network models which have attracted so much attention in the physics community consist of a large assembly of two state elements highly interconnected by ferromagnetic and antiferromagnetic interactions methods of statistical mechanics have been applied to these models bringing new discoveries and surprises to neural network science at the same time neural network models have posed new problems and incentives to statistical mechanics one of the basic problems is that of learning which in the context of these models means a choice of the interactions for which the network dynamics leads to specific attractors the main emphasis in this paper is on the essential developments and results on this problem derived from methods of statistical mechanics

categorization in the pseudo inverse neural network we investigate analytically the emergence of the categorization ability in the pseudo inverse attractor neural network more pointedly we consider the problem of learning an extensive number of concepts by storing a finite number of examples s of each concept we find that there is a critical value beyond which the categorization error as measured by the average fraction of unstable sites in the concepts decreases monotonically with s

gaussian beam methods for the schrodinger equation with discontinuous potentials we propose eulerian and lagrangian gaussian beam methods for the schrodinger equation with discontinuous potentials at the quantum barriers where the potential is discontinuous we derive suitable interface conditions to account for quantum scattering information these scattering interface conditions are then built into the numerical fluxes in the eulerian level set formulation of the gaussian beam methods and are also used in the lagrangian formulation including an interface condition for the hessian matrix we carry both 1d and 2d numerical examples to verify the accuracy of the method

large systems of random linear equations with nonnegative solutions characterizing the solvable and the unsolvable phase large systems of linear equations are ubiquitous in science quite often e g when considering population dynamics or chemical networks the solutions must be nonnegative recently it has been shown that large systems of random linear equations exhibit a sharp transition from a phase where a nonnegative solution exists with probability one to one where typically no such solution may be found the critical line separating the two phases was determined by combining farkas lemma with the replica method here we show that the same methods remain viable to characterize the two phases away from criticality to this end we analytically determine the residual norm of the system in the unsolvable phase and a suitable measure of robustness of solutions in the solvable one our results are in very good agreement with numerical simulations

depth migration by the gaussian beam summation method seismic depth migration aims to produce an image of seismic reflection interfaces ray methods are suitable for subsurface target oriented imaging and are less costly compared to two way wave equation based migration but break down in cases when a complex velocity structure gives rise to the appearance of caustics ray methods also have difficulties in correctly handling the different branches of the wavefront that result from wave propagation through a caustic on the other hand migration methods based on the two way wave equation referred to as reverse time migration are known to be capable of dealing with these problems however they are very expensive especially in the 3d case it can be prohibitive if many iterations are needed such as for velocity model building our method relies on the calculation of the green functions for the classical wave equation by per forming a summation of gaussian beams for the direct and back propagated wavefields the subsurface image is obtained by cal culating

the failure distribution in percolation models of breakdown the probability of finding a largest defect cluster of size n in a percolation network is calculated analytically using a new distribution function scaling equation from this result and the stress voltage enhancement at the tip of the most critical defect in the network the probability of failure in percolation models of breakdown is calculated for defect fractions less than the percolation point this distribution is found to be of the form exponential of an exponential numerical simulations on the two dimensional random fuse network confirm the new distribution function and convincingly distinguish between it and the weibull 1951 form most often used in the fitting of breakdown data

on the capacity per synapse the optimal storage capacity of a perceptron with a finite fraction of sign constrained weights which are prescribed a priori is examined the storage capacity is calculated by considering the fractional volume of weights which can store a set of alpha n random patterns where n is the size of the input it is found that in the case where 1 s n weights are sign constrained the capacity is 1 s 2 alpha gc k where alpha gc k is the maximal storage capacity in gardner s case and k is a stability parameter

on line learning in parity machines the optimal algorithm for on line learning in the tree k parity machine is studied we introduce a set of recursion relations for the relevant probability distributions which permit study of the general k case the generalization error curve is determined and shown to decay to zero for large as even in the presence of noise there is no critical noise level the dynamics of on line learning is studied analytically near the origin in the absence of previous knowledge the learning dynamics has a fixed point at previous knowledge is needed in at least k 1 branches for the learning to take place

the segment projection method for interface tracking there has recently been important progress in the development of front tracking and level set methods for the numerical simulation of moving interfaces the segment projection method is a new technique for computational geometry it can be seen as a compromise between front tracking and level set methods it is based on the regular mathematical representation of a manifold as an atlas of charts each chart or segment is evolved independently by a partial differential equation that is discretized on an eulerian grid the connectivity of the segments is handled by an appropriate data structure and by numerical interpolation the method is presented and its properties are analyzed applications to multiphase flow epitaxial growth and high frequency wave propagation are given 2002 wiley periodicals inc

phase diagram of restricted boltzmann machines and generalized hopfield networks with arbitrary priors restricted boltzmann machines are described by the gibbs measure of a bipartite spin glass which in turn can be seen as a generalized hopfield network this equivalence allows us to characterize the state of these systems in terms of their retrieval capabilities both at low and high load of pure states we study the paramagnetic spin glass and the spin glass retrieval phase transitions as the pattern i e weight distribution and spin i e unit priors vary smoothly from gaussian real variables to boolean discrete variables our analysis shows that the presence of a retrieval phase is robust and not peculiar to the standard hopfield model with boolean patterns the retrieval region becomes larger when the pattern entries and retrieval units get more peaked and conversely when the hidden units acquire a broader prior and therefore have a stronger response to high fields moreover at low load retrieval always exists below some critical temperature for every pattern distribution ranging from the boolean to the gaussian case

finite state neural networks a step toward the simulation of very large systems neural networks composed of neurons withq n states and synapses withq states are studied analytically and numerically analytically it is shown that these finite state networks are much more efficient at information storage than networks with continuous synapses in order to take the utmost advantage of networks with finite state elements a multineuron and multisynapse coding scheme is introduced which allows the simulation of networks having 1 0 109 couplings at a speed of 7 1 109 coupling evaluations per second on asingle processor of the cray ymp a local learning algorithm is also introduced which allows for the efficient training of large networks with finite state elements

replica calculation of the vapnik chervonenkis bound for the perceptron abstract using the replica technique we calculate the maximum difference between test and generalization error for the ensemble of all perceptrons trained by a teacher perceptron the result is compared with the bound provided by the vapnik chervonenkis theorem

on the computational capability of a perceptron the minimal fraction of errors a perceptron can make on p random dichotomies is investigated an exact upper bound is presented as well as comparisons between the replica theory estimates and a rigorous lower bound

structure of metastable states in the hopfield model an upper bound for the number of metastable states in the hopfield model is calculated as a function of the hamming fraction from an input pattern for all finite values of alpha the ratio of number of patterns to nodes the hamming fraction from the input pattern to the nearest metastable state is infinite when alpha 0 113 the bound also implies that there is a gap between a set of states close to the input pattern and another set centred around the hamming fraction 0 5 from it

the statistical mechanics of constructive algorithms the storage capacity of multilayer networks with overlapping receptive fields is studied for constructive algorithms using boolean perceptrons as their basic building block which have been investigated within a replica framework the assumption of weak coupling between subsequently constructed perceptrons is verified within a replica symmetric rs ansatz and shown to be negligible in most cases in comparison with correction due to replica symmetry breaking rsb in individual perceptrons the capacities of a tiling like and variants of the upstart algorithm are then calculated within rs and one step rsb with the quenched average taken over the individual units separately for networks with up to k 4000 and k 600 units respectively within this treatment the storage capacity seems to exhibit a power law behaviour in with an exponent n that may depend on the algorithm and the stability however due to finite size effects in k reliable estimates of n could not be extracted nevertheless the results strongly indicate that n should be strictly smaller than 1 within one step rsb whereas within rs the mitchison durbin bound is violated for finite k and n 1 may hold asymptotically

optimal storage capacity of neural networks at finite temperatures gardner s analysis 1989 of the optimal storage capacity of neural networks is extended to study finite temperature effects the typical volume of the space of interactions is calculated for strongly diluted networks as a function of the storage ratio alpha temperature t and the tolerance parameter m from which the optimal storage capacity alpha c is obtained as a function of t and m at zero temperature it is found that alpha c 2 regardless of m while alpha c in general increases with the tolerance at finite temperatures the authors show how the best performance for given alpha and t is obtained which reveals a first order transition from high quality performance to a low quality one at low temperatures an approximate criterion for recalling which is valid near m 1 is also discussed

learning and retrieval in attractor neural networks with noise abstract a recent study on noiseless learning and retrieval in attractor neural networks above saturation by griniasty and gutfreund is extended to take account for imperfect learning by means of a temperature 1 t violations of the constraint imposed on the local stabilities are taken into account by various cost functions the distribution of local stabilities and the fraction of errors during the learning stage are analysed the retrieval dynamics for the sparsely connected network is studied showing high retrieval overlap in reduced retrieval regions for finite t

fuzzy clustering with weighted medoids for relational data the well known k medoids clustering approach groups objects through finding k representative objects based on the pairwise dis similarities of objects in the data set in real applications using only one object to capture or interpret each cluster may not be sufficient enough which in turn could affect the accuracy of the data analysis in this paper we propose a new fuzzy clustering approach called pfc for dis similarity based data or relational data analysis in pfc objects in each fuzzy cluster carry various weights called prototype weights to represent their degrees of representativeness in that cluster this mechanism enables each cluster to be represented by more than one objects compared with existing clustering approaches for relational data pfc is able to capture the underlying structures of the data more accurately and provide richer information for the description of the resulting clusters we introduce the detailed formulation of pfc and provide the analytical as well as experimental studies to demonstrate the merits of the proposed approach

optimal storage of a neural network model a replica symmetry breaking solution a break in replica symmetry is found above the critical line alpha c kappa for storage with a finite minimal fraction of errors within the first stage of the parisi scheme for noiseless networks with continuous synapses the replica symmetry breaking solution yields the highest most stable minimal fraction of errors everywhere above alpha c kappa

storage capacity of a constructive learning algorithm upper and lower bounds for the typical storage capacity of a constructive algorithm the tilinglike learning algorithm for the parity machine biehl m and opper m 1991 phys rev a 44 6888 are determined in the asymptotic limit of large training set sizes the properties of a perceptron with threshold learning a training set of patterns having a biased distribution of targets needed as an intermediate step in the capacity calculation are determined analytically the lower bound for the capacity determined with a cavity method is proportional to the number of hidden units the upper bound obtained with the hypothesis of replica symmetry is close to the one predicted by mitchinson and durbin 1989 biol cybern 60 345

perceptrons above saturation we study the storage of random patterns by a perceptron above its storage capacity alpha c i e in the region where perfect storage becomes impossible we determine the minimal fraction of learning errors and the distribution of stabilities for different learning rules in one step replica symmetry breaking thereby we not only extend the known replica symmetric results to values of the storage capacity beyond the at line but also show that depending on the learning rule replica symmetry may be globally unstable already well below the at line as an example for possible implications we compare the results for the typical basins of attraction of an extremely diluted attractor neural network as given by replica symmetry and one step replica symmetry breaking

fuzzy relational clustering around medoids a unified view medoid based fuzzy clustering generates clusters of objects based on relational data which records pairwise similarities or dissimilarities among objects compared with single medoid based approaches our recently proposed fuzzy clustering with multiple weighted medoids has shown superior performance in clustering via experimental study in this paper we present a new version of fuzzy relational clustering in this family called fuzzy clustering with multi medoids fmmdd based on the new objective function of fmmdd update equations can be derived more conveniently moreover a unified view of fmmdd and two existing fuzzy relational approaches fuzzy c medoids fcmdd and assignment prototype a p can be established which allows us to conduct further analytical study to investigate the effectiveness and feasibility of the proposed approach as well as the limitations of existing ones the robustness of fmmdd is also investigated our theoretical and numerical studies show that the proposed approach produces good quality of clusters with rich cluster based information and it is less sensitive to noise

a simple derivation of a bound on the perceptron margin using singular value decomposition the perceptron is a simple supervised algorithm to train a linear classifier that has been analyzed and used extensively the classifier separates the data into two groups using a decision hyperplane with the margin between the data and the hyperplane determining the classifier s ability to generalize and its robustness to input noise exact results for the maximal size of the separating margin are known for specific input distributions and bounds exist for arbitrary distributions but both rely on lengthy statistical mechanics calculations carried out in the limit of infinite input size here we present a short analysis of perceptron classification using singular value decomposition we provide a simple derivation of a lower bound on the margin and an explicit formula for the perceptron weights that converges to the optimal result for large separating margins

dynamics of learning and generalization in perceptrons with constraints abstract depending on the kind of constraints imposed on the weights of a perceptron learning can be a combinatorially hard problem as an example of this type i discuss a perception with binary weights comparing results obtained from replica theory dynamic mean field theory and simulated annealing contrary to the replica calculation dynamics yields information about the performance of a polynomial algorithm in a situation where the best solution cannot be found in polynomial time i also discuss improved learning algorithms and results for finite size perceptrons

on the ability of the optimal perceptron to generalise a linearly separable boolean function is derived from a set of examples by a perceptron with optimal stability the probability to reconstruct a pattern which is not learnt is calculated analytically using the replica method

gaussian beam decomposition of high frequency wave fields using expectation maximization a new numerical method for approximating highly oscillatory wave fields as a superposition of gaussian beams is presented the method estimates the number of beams and their parameters automatically this is achieved by an expectation maximization algorithm that fits real positive gaussians to the energy of the highly oscillatory wave fields and its fourier transform beam parameters are further refined by an optimization procedure that minimizes the difference between the gaussian beam superposition and the highly oscillatory wave field in the energy norm

capacity of neural networks with discrete synaptic couplings the authors study the optimal storage capacity of neural networks with discrete local constraints on the synaptic couplings jij models with such constraints include those with binary couplings jij or 1 or jij 0 1 quantised couplings with larger synaptic range e g jij or 1 l or 2 l or 1 and in the limit continuous couplings confined to the hypercube mod jij mod or 1 box confinement they find that the optimal storage capacity alpha kappa is best determined by the vanishing of a suitably defined entropy as calculated in the replica symmetric approximation they also extend their results to cases with biased memories and make contact with sparse coding models

generalization performance of bayes optimal classification algorithm for learning a perceptron the generalization error of the bayes optimal classification algorithm when learning a perceptron from noise free random training examples is calculated exactly using methods of statistical mechanics it is shown that if an assumption of replica symmetry is made then in the thermodynamic limit the error of the bayes optimal algorithm is less than the error of a canonical stochastic learning algorithm by a factor approaching ensuremath surd 2 as the ratio of the number of training examples to perceptron weights grows in addition it is shown that approximations to the generalization error of the bayes optimal algorithm can be achieved by learning algorithms that use a two layer neutral net to learn a perceptron

bayesian learning versus optimal learning abstract we consider the optimal performance that may be reached in the problem of learning the symmetry breaking direction of a cloud of p n points in a n dimensional space the performance is measured through the overlap ropt between the true symmetry breaking direction and the learnt one depending on the problem the learning curves ropt may present discontinuities we show that close to these bayesian learning is not optimal