storage of sets of correlated data in neural network memories in most applications of neural network models one has to consider sets of correlated data the authors study the problem of the storage of associated patterns in neural network memories they consider two basic types of correlation semantic ones among the different patterns and spatial ones among the different sites of the network they apply gardner s program to evaluate optimal storage conditions for both kinds of correlation the spatial ones worsen generally speaking the storage properties of a simple perceptron while they improve them for the hopfield network in the case of semantic correlations they obtain bounds of the critical capacity for both kinds of networks the storage ratio of the perceptron may be significantly increased in this case

an algebraic proof of rabin s tree theorem we continue the development of a theory of recognisability for infinite trees by introducing the equivalent of a wilke algebra as an application we give a new proof of the decidability of the monadic second order theory of the infinite binary tree a proof that does not use automata or games

on labeled birooted tree languages with an aim to developing expressive language theoretical tools applicable to inverse semigroup languages that is subsets of inverse semigroups this paper explores the language theory of finite labeled birooted trees munn s birooted trees extended with vertex labeling to this purpose we define a notion of finite state birooted tree automata that simply extends finite state word automata semantics this notion is shown to capture the class of languages that are definable in monadic second order logic and upward closed with respect to the natural order then we derive from these automata the notion of quasi recognizable languages that is languages recognizable by means of adequate premorphisms into finite adequately ordered monoids this notion is shown to capture finite boolean combinations of languages as above a class that contains classical regular languages of finite mono rooted trees this contrasts with the known collapse of classical algebraic tools when applied to inverse semigroups

positive varieties of tree languages pin s variety theorem for positive varieties of string languages and varieties of finite ordered semigroups is proved for trees i e a bijective correspondence between positive varieties of tree languages and varieties of finite ordered algebras is established this in turn is extended to generalized varieties of finite ordered algebras which corresponds to steinby s generalized variety theorem also families of tree languages and classes of ordered algebras that are definable by ordered syntactic or translation monoids are characterized

robustness against random dilution in attractor neural networks the authors study the robustness of attractor neural networks against random disruption of a fraction of the synaptic couplings for the maximally stable network msn the authors determine the effect of different degrees of dilution on the overall storage capacity the size of the basins of attraction and the attractor overlap comparison with corresponding results for the hopfield model indicates that although msn is more robust at low degrees of dilution the hopfield network becomes more robust at high dilution

probing the basins of attraction of a recurrent neural network a recurrent neural network is considered that can retrieve a collection of patterns as well as slightly perturbed versions of this pure set of patterns via fixed points of its dynamics by replacing the set of dynamical constraints i e the fixed point equations by an extended collection of fixed point like equations analytical expressions are found for the weights wij b of the net which depend on a certain parameter b this so called basin parameter b is such that for b 0 there are a priori no perturbed patterns to be recognized by the net it is shown by a numerical study via probing sets that a net constructed to recognize perturbed patterns i e with values of the connections wij b with b 0 possesses larger basins of attraction than a net made with the help of a pure set of patterns i e with connections wij b 0 the mathematical results obtained can in principle be realized by an actual biological neural net

supervised learning with restricted training sets a generating functional analysis we study the dynamics of supervised on line learning of realizable tasks in feed forward neural networks we focus on the regime where the number of examples used for training is proportional to the number of input channels n using generating functional techniques from spin glass theory we are able to average over the composition of the training set and transform the problem for n to an effective single pattern system described completely by the student autocovariance the student teacher overlap and the student response function with exact closed equations our method applies to arbitrary learning rules i e not necessarily of a gradient descent type the resulting exact macroscopic dynamical equations can be integrated without finite size effects up to any degree of accuracy but their main value is in providing an exact and simple starting point for analytical approximation schemes finally we show how in the region of absent anomalous response and using the hypothesis that as in detailed balance systems the short time part of the various operators can be transformed away one can describe the stationary state of the network succesfully by a set of coupled equations involving only four scalar order parameters

from quenched to annealed a study of the intermediate dynamics of disorder a model spin system with disorder is examined the disorder is not necessarily quenched but it may evolve on a time scale that can be tuned the annealed and quenched cases obtained as limiting cases are respectively the infinite range ashkin teller p 2 colours model and the hopfield neural net finite number of patterns p 2 the intermediate dynamic model behaves like a hopfield associative memory on short time scales and like an ashkin teller system in the long run the time evolution of the order parameters is obtained from the master equations in the mean field approximation

learning and retrieval in attractor neural networks above saturation the authors investigate neural networks in the range of parameters when the ground state energy is positive namely when a synaptic matrix which satisfies all the desired constraints cannot be found by the learning algorithm in particular they calculate the typical distribution functions of local stabilities obtained for a number of algorithms in this region these functions are used to investigate the retrieval properties as reflected by the size of the basins of attraction this is done analytically in sparsely connected networks and numerically in fully connected networks the main conclusion is that the retrieval behaviour of attractor neural networks can be improved by learning above saturation

robustness and approximation of escape times and large deviations estimates for systems with small noise effects for the purposes of estimating escape time from a given set or other statistical properties of systems with small noise effects it is generally assumed in applications that the system noise is white gaussian the gaussian assumption greatly simplifies the computation but is not adequate for many important classes of applications in control and communication theory for example when the noise is small the mean escape time from a set can be quite sensitive to the underlying statistics even though in the study of the effects of the noise over any fixed finite time interval the gaussian approximation might be a good one this paper is concerned with the sensitivity of these statistical quantities to the underlying statistical structure when the noise effects are small and also with the question of when the gaussian assumption makes sense consider a sequence of systems with small noise effects whose statistics converge in some sense to those of a limit system the techniques developed involve approx

hierarchical self programming in recurrent neural networks we study self programming in recurrent neural networks where both neurons the processors and synaptic interactions the programme evolve in time simultaneously according to specific coupled stochastic equations the interactions are divided into a hierarchy of l groups with adiabatically separated and monotonically increasing time scales representing sub routines of the system programme of decreasing volatility we solve this model in equilibrium assuming ergodicity at every level and find as our replica symmetric solution a formalism with a structure similar but not identical to parisi s l step replica symmetry breaking scheme apart from differences in details of the equations due to the fact that here interactions rather than spins are grouped into clusters with different time scales in the present model the block sizes mi of the emerging ultrametric solution are not restricted to the interval 0 1 but are independent control parameters defined in terms of the noise strengths of the various levels in the hierarchy which can take any value in 0 t his is shown to lead to extremely rich phase diagrams with an abundance of firstorder transitions especially when the level of stochasticity in the interaction dynamics is chosen to be low

optimal storage capacity of neural networks at finite temperatures gardner s analysis 1989 of the optimal storage capacity of neural networks is extended to study finite temperature effects the typical volume of the space of interactions is calculated for strongly diluted networks as a function of the storage ratio alpha temperature t and the tolerance parameter m from which the optimal storage capacity alpha c is obtained as a function of t and m at zero temperature it is found that alpha c 2 regardless of m while alpha c in general increases with the tolerance at finite temperatures the authors show how the best performance for given alpha and t is obtained which reveals a first order transition from high quality performance to a low quality one at low temperatures an approximate criterion for recalling which is valid near m 1 is also discussed

learning and retrieval in attractor neural networks with noise abstract a recent study on noiseless learning and retrieval in attractor neural networks above saturation by griniasty and gutfreund is extended to take account for imperfect learning by means of a temperature 1 t violations of the constraint imposed on the local stabilities are taken into account by various cost functions the distribution of local stabilities and the fraction of errors during the learning stage are analysed the retrieval dynamics for the sparsely connected network is studied showing high retrieval overlap in reduced retrieval regions for finite t

characterizing ctl like logics on finite trees we associate a modal operator with each language belonging to a given class of regular tree languages and use the cascade product of tree automata to give an algebraic characterization of the expressive power of the resulting logic

retrieval phase diagrams for attractor neural networks with optimal interactions the authors consider the retrieval properties of attractor neural networks whose synaptic matrices have been constructed to maximise the number of patterns which can be stored in a perceptron satisfying certain constraints retrieval is studied in the absence as well as in the presence of fast noise temperature the discussion is restricted to dilute networks for which dynamical equations for the overlaps are available when the patterns are stored with a prescribed lower limit on the stability parameter on every site the full stability of the perceptron ensures the existence of an attractor with perfect retrieval it is found that the curve of critical storage capacity alpha against temperature t as a line of first order transitions for high values of alpha and becomes second order for low alpha at a point of a tricritical nature the phase diagram is compared with the dilute hopfield model it is found that at high synaptic noise levels the diluted hopfield net stores more effectively than the network trained for optimal perceptron storage when a given fraction of sites is allowed to violate the stability bound the solution of the perceptron learning problem does not ensure the existence of an attractor of finite overlap even in the absence of noise this case is studied separately for t 0 and for finite t

attractor properties of dynamical systems neural network models we propose tools to probe the nature of attractors in dynamical systems these include the activity distribution the evolutions of the state damage activity damage and temporal correlation damage when they are used to study the retrieval attractors in dilute asymmetric neural networks a transition from a partially frozen phase to an unfrozen phase is found for networks trained with sufficiently noisy data near storage saturation and this confirms that the retrieval attractors are more chaotic in this case we are also able to demonstrate that the retrieval attractors in dilute asymmetric neural networks are not clouds of attractors but consist of a single chaotic attractor for each stored pattern furthermore they facilitate the device of effective freezing procedures which significantly improve the quality of retrieval in neural networks

a nonlinear shooting method for two point boundary value problems abstract we study a new nonlinear shooting method for solving two point boundary value problems and show numerical experiments with various initial velocity conditions we discuss and analyze the numerical solutions which are obtained by the shooting method

coupled dynamics of sequence selection and compactification in mean field hetero polymers we study a simple solvable model describing the genesis of monomer sequences fo rh etero polymers such as proteins as the result of the equilibration of a slow stochastic genetic selection process which is assumed to be driven by the competing demands of functionality and reproducibility of the polymer s folded structure since reproducibility is defined in terms of properties of the folding process one is led to the analysis of the coupled dynamics of fast polymer folding and slow genetic sequence selection for the present meanfield model this analysis can be carried out using the finite dimensional replica method leading to exact results for first a nd second order transitions and to rich phase diagrams

optimally adapted attractor neural networks in the presence of noise by adapting an attractor neural network to an appropriate training overlap the authors optimize its attractor overlap and subsequently the storage capacity when retrieval noise temperature is present in the system the training overlap is determined self consistently by the optimal attractor overlap the phase diagram of the optimal attractor overlap in the temperature storage space is found a novel co existence phase of strong and weak retrievers is present the maximum storage capacity deviates from the storage capacity of the maximally stable network on increasing temperature and in the high temperature regime t or 0 38 for gaussian noise the hopfield network yields the maximum storage capacity this analysis demonstrates the principles of specialization and adaptation in neural networks

the roles of stability and symmetry in the dynamics of neural networks the authors study the retrieval phase of spin glass like neural networks considering that the dynamics should depend only on gauge invariant quantities they propose that two such parameters characterising the symmetry of the neural net s connections and the stabilities of the patterns are responsible for most of the dynamical effects this is supported by a numerical study of the shape of the basins of attraction for a one pattern neural network model the effects of stability and symmetry on the short time dynamics of this model are studied analytically and the full dynamics for vanishing symmetry is shown to be exactly solvable

diagonalization of replicated transfer matrices for disordered ising spin systems we present an alternative procedure for solving the eigenvalue problem of replicated transfer matrices describing disordered spin systems with random 1d nearest neighbour bonds and or random fields possibly in combination with random long range bonds our method is based on transforming the original eigenvalue problem for a 2n 2n matrix where n 0 into an eigenvalue problem for integral operators we first develop our formalism for the ising chain with random bonds and fields where we recover known results we then apply our methods to models of spins which interact simultaneously via a one dimensional ring and via more complex long range connectivity structures e g 1 dimensional neural networks and small world magnets numerical simulations confirm our predictions satisfactorily

generalized quasilinearization method for mixed boundary value problems a generalized quasilinearization method for a nonlinear mixed bvp is developed and a sequence of approximate solutions converging monotonically and quadratically to a solution of the given problem is presented

stochastic dynamics of supervised learning the stochastic evolution of adiabatic slow backpropagation training of a neural network is discussed and a fokker planck equation for the post training distribution function in the network space is derived the distribution obtained differs from the one given by radons et al 1990 studying the character of the post training distribution the authors find that except under very special circumstances the distribution will be non gibbsian the validity of the present approach is tested on a simple backpropagation learning system in one dimension which can be solved analytically as well implications of the fokker planck approach for general situations are examined in the local linear approximation surprisingly they find that the post training distribution is isotropic close to its peak hence simpler than the corresponding gibbs distribution

stochastic perturbation methods for spike timing dependent plasticity online machine learning rules and many biological spike timing dependent plasticity stdp learning rules generate jump process markov chains for the synaptic weights we give a perturbation expansion for the dynamics that unlike the usual approximation by a fokker planck equation fpe is well justified our approach extends the related system size expansion by giving an expansion for the probability density as well as its moments we apply the approach to two observed stdp learning rules and show that in regimes where the fpe breaks down the new perturbation expansion agrees well with monte carlo simulations the methods are also applicable to the dynamics of stochastic neural activity like previous ensemble analyses of stdp we focus on equilibrium solutions although the methods can in principle be applied to transients as well

finite size effects and bounds for perceptron models ablncl in lhis paper we consider two main aspsln of the binary perccptron problem the maximal capacity when random paltem are stored model a and its generalhation ability model b we have extended previous numerical estimates of critical capacilies and studied thermal properties of systems of small sizes to lest recent replica predictions we have also considered some simpler versions of thest models the discrete spherical vemions can be salved exactly using gardner s replica calculation for the spherical model and are shown to give a rigorous upper bound and lower bound on the capacities of models a and b respectively iby vemions of models a and b are soivcd in detail and provide information which is useful for interpreting the bnile size eeeciects present in the numerical studies of models a and b

high order finite difference schemes for the solution of second order bvps we introduce new methods in the class of boundary value methods bvms to solve boundary value problems bvps for a second order ode these formulae correspond to the high order generalizations of classical finite difference schemes for the first and second derivatives in this research we carry out the analysis of the conditioning and of the time reversal symmetry of the discrete solution for a linear convection diffusion ode problem we present numerical examples emphasizing the good convergence behavior of the new schemes finally we show how these methods can be applied in several space dimensions on a uniform mesh

external fields in attractor neural networks with different learning rules the effects of external fields on the retrieval properties of highly dilute attractor neural networks with general classes of learning rules are examined it can be shown that external fields increase basins of attraction making even perfect retrieval possible for relatively high loading the application of different classes of noise distributions on the stimulus field indicates that certain first order transitions occurring are peculiarities of the type of noise optimally adapted networks in the presence of external fields extend the critical loading above which perfect retrieval is impossible in the presence of external fields in the high temperature regime hebb networks also retrieve better than rules with more optimal performances at low temperatures

a constructive monotone iterative method for second order bvp in the presence of lower and upper solutions this paper concerns the monotone approximations of solutions of boundary value problems such as u f t u u 0 u 0 u 1 0 we consider linear iterative scheme in case f is lipschitz in u and satisfies a one sided lipschitz condition in u the initial approximations are lower and upper solutions which can be ordered one way a email protected we also consider the periodic and the dirichlet problems

optimal storage of a neural network model a replica symmetry breaking solution a break in replica symmetry is found above the critical line alpha c kappa for storage with a finite minimal fraction of errors within the first stage of the parisi scheme for noiseless networks with continuous synapses the replica symmetry breaking solution yields the highest most stable minimal fraction of errors everywhere above alpha c kappa

gardner derrida neural networks with correlated patterns the storage properties of an optimal neural network with correlated patterns is studied allowing for a finite fraction of errors determined by the gardner derrida cost function a discontinuity in the probability distribution of the local stabilities is seen as a drastic decrease in the minimal fraction of errors there is also an enlargement of the domain in which the replica symmetric solution is stable allowing for higher storage capacities

universality in the space of interactions for network models by modifying the measure used to sum over coupling matrices the authors generalise gardner s 1988 calculation of the fractional interaction space volume and storage capacity of neural network models they also compute the local field distribution for the network the generalised measure allows one to consider networks with a wide variety of properties away from saturation but they find that the original results for saturated networks are universal for all well behaved measures other universality classes including those containing hebb matrices and pseudo inverse matrices are obtained by considering singular measures

perceptrons above saturation we study the storage of random patterns by a perceptron above its storage capacity alpha c i e in the region where perfect storage becomes impossible we determine the minimal fraction of learning errors and the distribution of stabilities for different learning rules in one step replica symmetry breaking thereby we not only extend the known replica symmetric results to values of the storage capacity beyond the at line but also show that depending on the learning rule replica symmetry may be globally unstable already well below the at line as an example for possible implications we compare the results for the typical basins of attraction of an extremely diluted attractor neural network as given by replica symmetry and one step replica symmetry breaking

estimates of optimal storage conditions in neural network memories based on random matrix theory the authors formulate a method for estimating the critical conditions for storage of sets of data in neural network memory this variational method is based on random matrix theory and calculating the average spectrum of a matrix whose elements are given by overlaps of the stored patterns several generic cases of random overlap matrices are considered they investigate the cases of simply uncorrelated random patterns and spatially and semantically correlated ones they obtain bounds of the critical curve in the control parameters space which determine the stability of the stored data sets

derivation of hebb s rule on the basis of the general form for the energy needed to adapt the connection strengths of a network in which learning takes place a local learning rule is found for the changes this biologically realizable learning rule turns out to comply with hebb s neuro physiological postulate but is not of the form of any of the learning rules proposed in the literature the learning rule possesses the property that the energy needed in each learning step is minimal and is as such evolutionary attractive moreover the pre and post synaptic neurons are found to influence the synaptic changes differently resulting in an asymmetric connection matrix a fact which is in agreement with biological observation it is shown that if a finite set of the same patterns is presented over and over again to the network the weights of the synapses converge to finite values furthermore it is proved that the final values found in this biologically realizable limit are the same as those found via a mathematical approach to the problem of finding the weights of a partially connected neural network that can store a collection of patterns the mathematical solution is obtained via a modified version of the so called method of the pseudo inverse and has the inverse of a reduced correlation matrix rather than the usual correlation matrix as its basic ingredient thus a biological network might realize the final results of the mathematician by the energetically economic rule for the adaption of the synapses found in this article

retrieval properties of diluted attractor neural networks we investigate the dependence of the retrieval properties of the pseudo inverse and optimal attractor neural networks on the fraction of stored patterns the temperature t and the margin parameter phase diagrams in the full space of parameters are presented in the regime of extreme dilution i e when the connectivity c satisfies the condition where n is the number of neurons furthermore we study analytically the neighbourhood of a stored pattern for both models by calculating the average fraction of unstable sites in a pattern that differs by d sites from a given stored pattern this analysis may shed light on the properties of the basins of attraction of the stored patterns

variational iteration method for solving nonlinear boundary value problems abstract in this paper he s variational iteration method is implemented to give approximate and analytical solutions for a class of boundary value problems the variational iteration method which produces the solutions in terms of convergent series requiring no linearization or small perturbation numerical examples are given and comparisons are made with the adomian decomposition method the fact that this method solves nonlinear equations without using adomian polynomials can be considered as an advantage of this method over adomian decomposition method

statistical mechanics of hopfield like neural networks with modified interactions hopfield like neural networks 1982 with modified interactions are studied by a mean field theory the modification of interactions is achieved during a special thermally noised iterative procedure the resulting couplings have an intermediate form between the hebb like learning rule and the pseudo inverse one replica symmetric free energy of the model is obtained statistical properties of the model depend on three parameters reduced number of the stored patterns alpha reduced number of iteration steps of the modification procedure lambda and the temperature t the phase diagram in the space of these parameters is obtained the network can retrieve patterns at t 0 for alpha alphac lambda where alphac 0 approximately 0 14 and alphac lambda to infinity approximately 1 07 as alpha decreases below alpha0 lambda the fm retrieval states become ground states of the system where alpha0 0 0 05 and alpha0 lambda to infinity 2 pi

on stochastic dynamics of supervised learning recently hansen et al 1993 derived a fokker planck equation fpe for the learning dynamics of neural networks which differs from a previously given version by radons et al 1993 it is shown that the discrepancies are due to different implicit assumptions for the distribution of time intervals between the discrete learning events both approximations are therefore equally justified from a general point of view the long time properties however are independent of this distribution and are in general more accurately described in the original fpe of radons et al especially mean and variance of the synaptic parameter distributions are exact only in the latter approach

slowly evolving random graphs ii adaptive geometry in finite connectivity hopfield models we present an analytically solvable random graph model in which the connections between the nodes can evolve in time adiabatically slowly compared to the dynamics of the nodes we apply the formalism to finite connectivity attractor neural network hopfield models and show that due to the minimization of the frustration effects the retrieval region of the phase diagram can be significantly enlarged moreover the fraction of misaligned spins is reduced by this effect and is smaller than that in the infinite connectivity regime the main cause of this difference is found to be the non zero fraction of sites with vanishing local field when the connectivity is finite

variational iteration method for solving two point boundary value problems variational iteration method is introduced to solve two point boundary value problems numerical results demonstrate that the method is promising and may overcome the difficulty arising in adomian decomposition method

algebraic tools for the concatenation product this paper is a contribution to the algebraic study of the concatenation product in the first part of the paper we extend to the ordered case standard algebraic tools related to the concatenation product like the schutzenberger product and the relational morphisms we show in a precise way how the ordered schutzenberger product corresponds to polynomial operations on languages in the second part of the paper we apply these results to establish a bridge between the three standard concatenation hierarchies namely the straubing therien hierarchy the brzozowski s or dot depth hierarchy and the group hierarchy

learning by a neural net in a noisy environment the pseudo inverse solution revisited abstract a recurrent neural net is studied that learns a set of patterns in the presence of noise the learning rule is of a hebbian type it is well known that if noise is absent during the learning process the resulting final values of the weights wij correspond to what is usually referred to as the pseudo inverse solution of the fixed point equation associated with the learning rule in the limit of vanishing noise the expressions derived in this article for the expectation value of the weights do not converge to the usual pseudo inverse solution in contrast to what one might expect since biological systems in general are noisy the usual pseudo inverse solution is less realistic in principle than the solution found in this article

slowly evolving connectivity in recurrent neural networks i the extreme dilution regime we study extremely diluted spin models of neural networks in which the connectivity evolves in time although adiabatically slowly compared to the neurons according to stochastic equations which on average aim to reduce frustration the fast neurons and slow connectivity variables equilibrate separately but at different temperatures our model is exactly solvable in equilibrium we obtain phase diagrams upon making the condensed ansatz i e recall of one pattern these show that as the connectivity temperature is lowered the volume of the retrieval phase diverges and the fraction of mis aligned spins is reduced still one always retains a region in the retrieval phase where recall states other than the one corresponding to the condensed pattern are locally stable so the associative memory character of our model is preserved

varieties of recognizable tree series over fields we introduce varieties of recognizable s tree series k s vts for short over a field k and a ranked alphabet s our variety theorem establishes a bijective correspondence between these k s vtss and the varieties of finite dimensional k s algebras k s vfda for short a k s algebra is a k vector space equipped with multilinear s operations the link between k s vtss and k s vfdas is provided by the syntactic k s algebras of tree series the most immediate predecessors of this study are berstel s and reutenauer s 1982 2 work on tree series over fields reutenauer s 1980 27 theory of varieties of string series bozapalidis and his associates 1983 1989 1991 8 5 4 work on syntactic k s algebras steinby s 1979 1992 30 31 theory of varieties of tree languages and our previous work 2009 on series of general algebras and their syntactic k s algebras

optimal basins of attraction in randomly sparse neural network models the size of the basin of attraction for randomly sparse neural networks with optimal interactions is calculated for all values of the storage ratio alpha p c 2 where p is the number of random uncorrelated patterns and c is the connectivity the basin of attraction is finite while for alpha 0 42 the basin of attraction is almost 100

learning and generalization with minimerror a temperature dependent learning algorithm we study the numerical performances of minimerror a recently introduced learning algorithm for the perceptron that has analytically been shown to be optimal both on learning linearly and nonlinearly separable functions we present its implementation on learning linearly separable boolean functions numerical results are in excellent agreement with the theoretical predictions

multivalley structure of attractor neural networks we numerically investigate the structure of the basins of attraction and the nature of the spurious attractors of the pseudo inverse and the optimal weights attractor neural networks we show that the number of attractors in the optimal weights model increases as the margin parameter increases and that the basins of attraction of the stored patterns are not significantly enlarged by moreover the number of attractors is smaller than in the pseudo inverse model

shuffle on positive varieties of languages we show there is a unique maximal positive variety of languages which does not contain the language ab this variety is the unique maximal positive variety satisfying the two following conditions it is strictly included in the class of rational languages and is closed under the shuffle operation it is also the largest proper positive variety closed under length preserving morphisms the ordered monoids of the corresponding variety of ordered monoids are characterized as follows for every pair a b of mutually inverse elements and for every element z of the minimal ideal of the submonoid generated by a and b abzab ab in particular this variety is decidable

optimal storage properties of neural network models the authors calculate the number p alpha n of random n bit patterns that an optimal neural network can store allowing a given fraction f of bit errors and with the condition that each right bit is stabilised by a local field at least equal to a parameter k for each value of alpha and k there is a minimum fraction fmin of wrong bits they find a critical line alpha c k with alpha c 0 2 the minimum fraction of wrong bits vanishes for alpha alpha c k the calculations are done using a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is locally stable in a finite region of the k alpha plane including the line alpha c k but there is a line above which the solution becomes unstable and replica symmetry must be broken

learning algorithms with optimal stability in neural networks to ensure large basins of attraction in spin glass like neural networks of two state elements xi imu or 1 the authors propose to study learning rules with optimal stability delta where delta is the largest number satisfying delta or sigma j jij xi jmu xi imu mu 1 p i 1 n where n is the number of neurons and p is the number of patterns they motivate this proposal and provide optimal stability learning rules for two different choices of normalisation for the synaptic matrix jij in addition numerical work is presented which gives the value of the optimal stability for random uncorrelated patterns

the space of interactions in neural network models the typical fraction of the space of interactions between each pair of n ising spins which solve the problem of storing a given set of p random patterns as n bit spin configurations is considered the volume is calculated explicitly as a function of the storage ratio alpha p n of the value kappa 0 of the product of the spin and the magnetic field at each site and of the magnetisation m here m may vary between 0 no correlation and 1 completely correlated the capacity increases with the correlation between patterns from alpha 2 for correlated patterns with kappa 0 and tends to infinity as m tends to 1 the calculations use a saddle point method and the order parameters at the saddle point are assumed to be replica symmetric this solution is shown to be locally stable a local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist

fundamental study tree algebras and varieties of tree languages we consider several aspects of wilke s t wilke an algebraic characterization of frontier testable tree languages theoret comput sci 154 1996 85 106 tree algebra formalism for representing binary labelled trees and compare it with approaches that represent trees as terms in the traditional way a convergent term rewriting system yields normal form representations of binary trees and contexts as well as a new completeness proof and a computational decision method for the axiomatization of tree algebras varieties of binary tree languages are compared with varieties of tree languages studied earlier in the literature we also prove a variety theorem thus solving a problem noted by several authors syntactic tree algebras are studied and compared with ordinary syntactic algebras the expressive power of the language of tree algebras is demonstrated by giving equational definitions for some well known varieties of binary tree languages

recognisability for algebras of infinite trees we develop an algebraic language theory for languages of infinite trees we define a class of algebras called w hyperclones and we show that a language of infinite trees is regular if and only if it is recognised by a finitary path continuous w hyperclone

statistical mechanics of neural networks what are the differences between wide and narrow basins abstract we consider training noise in neural networks as a means of tuning the structure of retrieval basins and study how learning and retrieving properties depend on it the stability of the replica symmetric solution and the correlation in the weight space indicate that neural networks can be roughly classified into hebbian like and msn like msn meaning the maximally stable network re entrant retrieval noise robustness selectivity damage spreading and activity distribution all illustrate the differences in retrieval behaviours arising from the different basin structures

cluster derivation of parisi s rsb solution for disordered systems we propose a general scheme in which disordered systems are allowed to sacrifice energy equi partitioning and separate into a hierarchy of ergodic sub systems clusters with different characteristic timescales and temperatures the details of the break up follow from the requirement of stationarity of the entropy of the slower cluster at every level in the hierarchy we apply our ideas to the sherrington kirkpatrick model and show how the parisi solution can be derived quantitatively from plausible physical principles our approach gives new insight into the physics behind parisi s solution and its relations with other theories numerical experiments and short range models

the validity of nonlinear langevin equations in the presence of internal noise the variables describing a system are intrinsically stochastic if they constitute a markov process the expansion enables one to extract a deterministic macroscopic equation and to compute the fluctuations in successive approximations in the lowest or linear noise approximation the fluctuations can be represented by a langevin equation provided it is handled appropriately higher orders cannot be described by any white noise langevin equation the question whether the equation has to be interpreted according to ito or stratonovich concerns these higher orders for which the equation is not valid anyway

solving boundary value problems of ordinary differential equations with non separated boundary conditions abstract this paper focuses on solving the two point boundary value problem in which boundary conditions are systems of nonlinear equations the shooting method was used together with a combination of newton s method and broyden s method to update the initial values of the differential equations the experiments showed that the proposed method performed well in the sense that the overall amount of work was less than that of the newton shooting method

general varieties of tree languages we consider varieties of tree languages which are not restricted to a fixed ranked alphabet varieties of finite algebras that contain algebras of all finite types and a matching notion of varieties of congruences a variety theorem that yields isomorphisms between the lattices formed by these three types of varieties is proved to achieve this some basic universal algebra is suitably generalized and we define syntactic algebras so that two tree languages over any alphabets belong to the same varieties exactly in case their syntactic algebras are isomorphic many families of regular tree languages are shown to be varieties in our sense in particular we prove that every family of regular tree languages which can be characterized by syntactic monoids is such a variety but that the converse does not hold

the rank of a formal tree power series we prove that a tree series s t k k a field is recognizable iff dim vs iff dim fs where vs resp fs is the subspace if kp resp kt generated by the vectors t 1 s p s t t t p and where t is the set of all trees over and p is the free monoid of all pruned trees over we finally give a myhill like criterion for tree recognizability

constructive proof for existence of nonlinear two point boundary value problems owing to the importance of differential equations in physics the existence of solutions for differential equations has been paid much attention in this paper the existence of solution are obtained for the nonlinear second order two point boundary value problem in the reproducing kernel space under certain assumptions on right hand side we propose constructive proof for the existence result and a method is presented to obtain the exact solution expressed by the form of series this paper is a extension of previous paper wei jiang minggen cui the exact solution and stability analysis for integral equation of third or first kind with singular kernel appl math comput 202 2 2008 666 674 which extends a method of solving linear problems to present method for solving nonlinear problems

dynamic boundary value problems of the second order bernstein nagumo conditions and solvability abstract this article analyzes the solvability of second order nonlinear dynamic boundary value problems bvps on time scales new bernstein nagumo conditions are developed that guarantee an a priori bound on the delta derivative of potential solutions to the bvps under consideration topological methods are then employed to gain solvability

stochastic dynamics of learning with momentum in neural networks we study on line learning with a momentum term for nonlinear learning rules through introduction of auxiliary variables we show that the learning process can be described by a markov process for small learning parameters eta and momentum parameters alpha close to 1 such that gamma eta 1 alpha 2 is finite the time scales for the evolution of the weights and the auxiliary variables are the same in this case van kampen s expansion can be applied in a straightforward manner we obtain evolution equations for the average network state and the fluctuations around this average these evolution equations depend after rescaling a of the time and fluctuations only on gamma all combinations eta alpha with the same value of gamma give rise to similar behaviour the case with alpha constant and eta small requires a completely different analysis there are two different time scales a fast time scale on which the auxiliary variables equilibrate and a slow time scale for the change of the weights by projection on the space of slow variables the fast variables can be eliminated we find that for small learning parameters eta and finite momentum parameters alpha learning with momentum is equivalent to learning without a momentum term with a rescaled learning parameter eta eta 1 alpha simulations with the nonlinear oja learning rule confirm the theoretical results

algebraic recognizability of regular tree languages we propose a new algebraic framework to discuss and classify recognizable tree languages and to characterize interesting classes of such languages our algebraic tool called preclones encompasses the classical notion of syntactic algebra or minimal tree automaton but adds new expressivity to it the main result in this paper is a variety theorem a la eilenberg but we also discuss important examples of logically defined classes of recognizable tree languages whose characterization and decidability was established in recent papers by benedikt and segoufin and by bojanczyk and walukiewicz and can be naturally formulated in terms of pseudovarieties of preclones finally this paper constitutes the foundation for another paper by the same authors where first order definable tree languages receive an algebraic characterization

on fokker planck approximations of on line learning processes there are several ways to describe on line learning in neural networks the two major ones are a continuous time master equation and a discrete time random walk equation the random walk equation is obtained in the case of fixed time intervals between subsequent learning steps the master equation results when the time intervals are drawn from a poisson distribution following van kampen 1992 we give a rigorous expansion of both the master and the random walk equation in the limit of small learning parameters the results explain the difference between the fokker planck approaches proposed by radons et al 1990 and hansen et al 1993 furthermore we find that the mathematical validity of these approaches is restricted to local properties of the learning process yet fokker planck approaches are often suggested as models to study global properties such as mean first passage times and stationary solutions to check their accuracy and usefulness in these situations we compare simulations of two learning procedures with exactly the same drift vector and diffusion matrix the only moments that are considered in fokker planck approximation the simulations show that the mean first passage times for these two learning procedures diverge rather than converge for small learning parameters we reach the conclusion that fokker planck approaches are not accurate enough to compute global properties of on line learning processes

some results on the generalized star height problem abstract we prove some results related to the generalized star height problem in this problem as opposed to the restricted star height problem complementation is considered as a basic operator we first show that the class of languages of star height n is closed under certain operations left and right quotients inverse alphabetic morphisms injective star free substitutions it is known that languages recognized by a commutative group are of star height 1 we extend this result to nilpotent groups of class 2 and to the groups that divide a semidirect product of a commutative group by z 2 z n in the same direction we show that one of the languages that were conjectured to be of star height 2 during the past ten years is in fact of star height 1 next we show that if a rational language l is recognized by a monoid of the variety generated by wreath products of the form m g n where m and n are aperiodic monoids and g is a commutative group the l is of star height 1 finally we show that every rational language is the inverse image under some morphism between free monoids of a language of resticted star height 1

regular languages and partial commutations the closure of a regular language under a partial commutation i has been extensively studied we present new advances on two problems of this area 1 when is the closure of a regular language under partial commutation still regular 2 are there any robust classes of languages closed under partial commutation we show that the class pol g of polynomials of group languages is closed under commutation and under partial commutation when the complement of i in a 2 is a transitive relation we also give a sufficient graph theoretic condition on i to ensure that the closure of a language of pol g under i commutation is regular we exhibit a very robust class of languages w which is closed under commutation this class contains pol g is decidable and can be defined as the largest positive variety of languages not containing ab it is also closed under intersection union shuffle concatenation quotients length decreasing morphisms and inverses of morphisms if i is transitive we show that the closure of a language of w under i commutation is regular the proofs are nontrivial and combine several advanced techniques including combinatorial ramsey type arguments algebraic properties of the syntactic monoid finiteness conditions on semigroups and properties of insertion systems

a recurrent neural network with ever changing synapses a recurrent neural network with noisy input is studied analytically on the basis of a discrete time master equation the latter is derived from a biologically realizable learning rule for the weights of the connections in a numerical study it is found that the fixed points of the dynamics of the net are time dependent implying that the representation in the brain of a fixed piece of information e g a word to be recognized is not fixed in time

enlarging the attractor basins of neural networks with noisy external fields a neural network model with optimal connections trained with ensembles of external discrete noisy fields is studied allowing for non zero errors in the storage novel behaviour is observed which is reflected in the model s retrieval map improvement in the model s content addressability is determined by comparing the maximum storage level at which there is a near 100 basin of attraction the cases presented have the external field applied during training during retrieval and during both with statistically equal parameters in all three the content addressability is improved over the zero external field network with the equal training and retrieval fields case having the largest improvement however the apparent domination of the retrieval over the training field suggests this simple equality is perhaps not the optimal relationship

a theoretical comparison of batch mode on line cyclic and almost cyclic learning we study and compare different neural network learning strategies batch mode learning online learning cyclic learning and almost cyclic learning incremental learning strategies require less storage capacity than batch mode learning however due to the arbitrariness in the presentation order of the training patterns incremental learning is a stochastic process whereas batch mode learning is deterministic in zeroth order i e as the learning parameter spl eta tends to zero all learning strategies approximate the same ordinary differential equation for convenience referred to as the ideal behavior using stochastic methods valid for small learning parameters spl eta we derive differential equations describing the evolution of the lowest order deviations from this ideal behavior we compute how the asymptotic misadjustment measuring the average asymptotic distance from a stable fixed point of the ideal behavior scales as a function of the learning parameter and the number of training patterns knowing the asymptotic misadjustment we calculate the typical number of learning steps necessary to generate a weight within order spl epsiv of this fixed point both with fixed and time dependent learning parameters we conclude that almost cyclic learning learning with random cycles is a better alternative for batch mode learning than cyclic learning learning with a fixed cycle