an iterative method for nonsymmetric systems with multiple right hand sides we propose a method for the solution of linear systems ax b where a is a large possibly sparse nonsymmetric matrix of order n and b is an arbitrary rectangular matrix of order n times s with s of moderate size the method uses a single krylov subspace per step as a generator of approximations a projection process and a richardson acceleration technique it thus combines the advantages of recent hybrid methods with those for solving symmetric systems with multiple right hand sides numerical experiments indicate that in several cases the method has better practical performance and significantly lower memory requirements than block versions of nonsymmetric solvers and other proposed methods for the solution of systems with multiple right hand sides

main overlap dynamics for multistate neural networks the authors present explicit formulae for one step dynamics of retrieval pattern errors main overlap for the q state potts and q or 4 clock neural networks solution of the fixed point equations and critical values of the saturation parameters alpha c q in the one step approximation are considered in the potts case

stability properties of potts neural networks with biased patterns and low loading the q state potts glass model of neural networks is extended to include biased patterns for a finite number of such patterns the existence and stability properties of the mattis states and symmetric states are discussed in detail as a function of the bias analytic results are presented for all q at zero temperature for finite temperatures numerical results are obtained for q 3 and two classes of representative bias parameters a comparison is made with the hopfield model

generalization in a multi state neural network the generalization ability of an extremely dilute feedback neural network with multi state neurons is studied by means of a deterministic noiseless parallel dynamics the overlap with any one of a macroscopic number of binary full activity concepts is determined when the network is trained with examples of variable activity according to a hebbian learning algorithm that favours stable symmetric mixture states explicit results about the phase diagram and the generalization error are obtained for a network with three state neurons which remain inactive below a threshold it is shown that the generalization ability can be considerably enhanced either by training the network with low activity examples or by means of a moderate increase in

comparison of linear and classical velocity update rules in particle swarm optimization notes on scale and frame invariance in this paper we investigate whether the particle swarm optimization pso algorithm is invariant of the scale and frame i e translation and rotation in which an objective function is posed to do so we study the linear and classical velocity update rules we will show that the linear velocity update rule is scale and frame invariant but that the classical velocity update rule lacks rotational invariance it is known that the linear velocity update rule lacks diversity resulting in particle trajectories that collapse to line searches in contrast the classical velocity update rule maintains diverse space filling particle trajectories to illustrate that diversity and invariance are not necessarily exclusive we propose a new velocity update rule this update rule which is just one example of many possible formulations is rotationally invariant and at the same time directionally diverse this is achieved through consistent perturbation of the search directions we quantify the in variance and performance of the three different implementations using a popular test set the test problems are evaluated in both unrotated and arbitrarily rotated reference frames copyright 2006 john wiley sons ltd

thermodynamic properties of fully connected q ising neural networks the thermodynamic and retrieval properties of fully connected q ising networks are studied in the replica symmetric mean field approximation in particular capacity gain parameter and capacity temperature phase diagrams are derived for q 3 4 and q infinity and different distributions of the stored patterns furthermore the optimal gain function is determined in order to obtain the best performance where appropriate the results are compared with the diluted and layered versions of these models

mean field theory for the q state potts glass neural network with biased patterns a systematic study of the q state potts model of neural networks extended to include biased patterns is made for extensive loading alpha mean field equations are written down within the replica symmetric approximation for general q and arbitrary temperature t for the q 3 model and two classes of representative bias parameters the storage capacity and retrieval quality at zero temperature are discussed as functions of the bias taking into account the mattis retrieval state and the lowest symmetric states the t alpha diagram is obtained and the stability properties of the retrieval state are analysed at finite temperatures a comparison is made with the biased hopfield model

parallel dynamics of the fully connected blume emery griffiths neural network abstract the parallel dynamics of the fully connected blume emery griffiths neural network model is studied at zero temperature using a probabilistic approach a recursive scheme is found determining the complete time evolution of the order parameters taking into account all feedback correlations it is based upon the evolution of the distribution of the local field the structure of which is determined in detail as an illustrative example explicit analytic formula are given for the first few time steps of the dynamics furthermore equilibrium fixed point equations are derived and compared with the thermodynamic approach the analytic results find excellent confirmation in extensive numerical simulations

global simpler gmres for nonsymmetric systems with multiple right hand sides a new version of the simpler gmres for solving nonsymmetric linear system of equations with multiple right hand sides is presented the main advantage over the simpler block gmres is that the new method requires less storage and computational work per cycle preconditioning techniques can also be used to further accelerate the convergence of the new method finally numerical examples are given to illustrate the effectiveness of the proposed method and its some preconditioned versions

on the parallel dynamics for the little hopfield model we propose a method algorithm for calculation of the explicit formulas for evolution of the main and the residual overlaps it allows us to confirm the gardner derrida mottishaw second step formula for the main overlap and to go beyond to the next steps we discuss the dynamical status of the amit gutfreund sompolinsky formula for the main overlap and some computersimulation results

dynamical properties of a randomly diluted neural network with variable activity the subject of this paper is a neural network with binary neurons randomly diluted synapses and variable pattern activity we look at the system with parallel updating using a probabilistic approach to solve the one step dynamics with one condensed pattern we derive restrictions on the storage capacity and the mutual information content occuring during the retrieval process special focus is placed on the constraints on the threshold for optimal performance we also look at the effect of noisy updating giving a dynamical version of the critical temperature the corresponding threshold and an approximation for the time evolution for small temperatures the description is applicable to the whole retrieval process in the limit of strong dilution the analysis is carried out as exactly as possible and over the whole parameter ranges generalizing some former results

block triangular and skew hermitian splitting methods for positive definite linear systems by further generalizing the concept of hermitian or normal and skew hermitian splitting for a non hermitian and positive definite matrix we introduce a new splitting called positive definite and skew hermitian splitting pss and then establish a class of pss methods similar to the hermitian or normal and skew hermitian splitting hss or nss method for iteratively solving the positive definite systems of linear equations theoretical analysis shows that the pss method converges unconditionally to the exact solution of the linear system with the upper bound of its convergence factor dependent only on the spectrum of the positive definite splitting matrix and independent of the spectrum of the skew hermitian splitting matrix as well as the eigenvectors of all matrices involved when we specialize the pss to block triangular or triangular and skew hermitian splitting btss or tss the pss method naturally leads to a btss or tss iteration method which may be more practical and efficient than the hss and nss iteration methods applications of the btss method to the linear systems of block two by two structures are discussed in detail numerical experiments further show the effectiveness of our new methods

invariant recognition in potts glass neural networks the authors show that a potts glass neural network can be used for invariant recognition each neuron is modelled as a q state potts variable and p potts configurations are stored in the network the learning rule is a generalized hebb rule so that all pq permutations of the patterns are stabilized simultaneously they analyse the model for an extensive number of patterns and present results of numerical simulations to confirm the analytical results and to illustrate the relaxation to equilibrium the model is applied to the recognition of isomorphic graphs finally they show how to generalize the model such that its stable states are patterns were the permutational symmetry is broken in a hierarchical manner

two state representations of three state neural networks the authors investigate to what extent the dynamical behaviour of nets consisting of three state neurons can be realised in nets using two state neurons they find that the generalisation from a two state neuron to a three state neuron cannot introduce any new behaviour in the deterministic case but can do so for noisy nets even in the zero noise limit for those noisy three state nets which have a hamiltonian description and a two state representation they investigate the relationship between the three state hamiltonian and the two state hamiltonian and find an interesting level splitting phenomenon

retrieval and chaos in layered q ising neural networks using a probabilistic approach we study the parallel dynamics of theq ising layered networks for arbitraryq by introducing auxiliary thermal fields we express the stochastic dynamics within the gain function formulation of the deterministic dynamics evolution equations are derived for arbitraryq at both zero and finite temperatures an explicit analysis of the fixed point equations is carried out for bothq 3 andq the retrieval properties are discussed in terms of the gain parameter the storage capacity and the temperature using the time evolution of the distance between two network configurations we investigate the possibility of microscopic chaos chaotic behavior is always present for arbitrary finiteq however in the limitq the existence of chaos depends on the parameters of the system

on the overlap dynamics of multi state neural networks with a finite number of patterns neural networks with multi state neurons are studied in the case of low loading for symmetric couplings satisfying a certain positivity condition a lyapunov function is shown to exist in the space of overlaps between the instantaneous microscopic state of the system and the learned patterns furthermore an algorithm is derived for zero temperature to determine all the fixed points as an illustration the three state model is worked out explicitly for hebbian couplings for finite temperature the time evolution of the overlap is studied for couplings which need not be symmetric the stability properties are discussed in detail for the three state model for asymmetric couplings limit cycle behaviour is shown to be possible

implicitly restarted global fom and gmres for nonsymmetric matrix equations and sylvester equations the global gmres and global fom algorithm are recently proposed by jbilou et al for solving the linear equations with multiple right hand sides k jbilou a messaoudi h sados global fom and gmres algorithms for matrix equations appl numer math 31 1999 49 63 like gmres for the linear equations they generally uses restarting which slows the convergence however some information can be retained at the time of the restart and used in the next cycle we present algorithms that use implicit restarting in order to retain this information as morgan have proposed recently r b morgan implicitly restarted gmres and arnoldi methods for nonsymmetric systems of equations siam j matrix anal appl 21 2000 1112 1135 at the same time we prove that global gmres and global fom methods for matrix equations are equivalent with the corresponding methods for linear equations and propose implicitly restarted global fom and gmres numerical examples show that our methods are efficient

learning grey toned patterns in neural networks the problem of learning multi state patterns in neural networks is investigated an analysis of the space of couplings gardner approach yields the distribution of local fields the critical storage capacity alpha c and the minimum number of errors for an overloaded network for noisy local fields the classification error is minimized if the local fields of the patterns are allowed to lie in intervals of finite width a fast converging adaptive learning algorithm is presented which finds the coupling matrix of optimal stability

the mean field theory of a q state neural network model the mean field equations of a q state clock neural network model are derived in the replica symmetric approximation using the replica method these equations are studied for the cases q 2 3 4 and q to infinity it is shown that an infinite number of patterns can be stored in the network even in the limit q to infinity the phase diagram and storage capacity of the network are calculated and the information content considered although the overlap achieved with a nominated pattern is shown to decrease as q increases the information stored in the network is shown to increase

a lower bound for the memory capacity in the potts hopfield model we consider potts hopfield networks of sizen we prove the result c 0 such that for all 0 0 in such a way that whenn we can store n patterns all of them being sorrounded by energy barriers at distance

storing an extensive number of grey toned patterns in a neural network using multistate neurons grey toned patterns are pictures composed of pixels of several shades of grey the ability of neural networks using multistate neurons to store such patterns is systematically investigated if conventional generalizations of hopfield networks using analogue or soft neurons are considered it is impossible to stabilize these grey tones nevertheless it is shown that it can be done with networks that use neurons which have only a discrete set of possible activities this is demonstrated for the pseudo inverse rule for the synaptic couplings where only the stability of the patterns shrinks with increasing number q of grey tones one wants to store if the patterns are uncorrelated one can use the hebb rule and in this case the mean field theory is presented applying this rule the storage capacity decreases as q 2 with the number of grey tones

block minimum perturbation algorithm based on block arnoldi process for nonsymmetric linear systems with multiple right hand sides abstract in many applications in scientific computing and engineering one has to solve huge sparse linear systems of equations with several right hand sides customarily one uses the residual error as a stopping condition but small residue does not imply accurate approximate solutions therefore minimizing quantities other than the residual norm may be more adequate in this context based on the above consideration a block minimum perturbation krylov subspace method bminpet for the solution xm satisfying a a x m b b at each step of nonsymmetric linear systems a x b is shown such that the normwise backward error meets some optimality condition with minimizing the block joint backward perturbation norm a b f of the matrix a b and a stopping criterion is given for consistent system with multiple right hand sides to get a more robust way to stop the iterations this process is a generalization of minimum perturbation algorithm minpert by kasenally and simoncini 1997 analysis of a minimum perturbation algorithm for nonsymmetric linear systems siam j numer anal 34 48 66 to multiple right hand sides for the benefit of the amount of calculation we give lower and upper bounds about min a b f that are easier to compute as a by product we come up with a conception of function f g which is a generalization of function that is a symmetric gauge sg function the algorithm is used to analyze the performance of bgmres method by evaluating the proximity of their solutions to block joint backward perturbation optimality the relationships between bminpet method and relative methods are investigated numerical examples show us that bminpert shows its superiority compared with bfgmres s m pf gsgmres bl bicg rq bgmres and barnoldi in solving the large sparse ill conditioned problems

on the parallel dynamics of the q state potts and q ising neural networks using a probabilistic approach the parallel dynamics of theq state potts andq ising neural networks are studied at zero and at nonzero temperatures evolution equations are derived for the first time step and arbitraryq these formulas constitute recursion relations for the exact parallel dynamics of the extremely diluted asymmetric versions of these networks an explicit analysis including dynamical capacity temperature diagrams and the temperature dependence of the overlap is carried out forq 3 both types of models are compared

on the parallel dynamics of the diluted clock neural network abstract we present explicit formulae for one step evolution of the overlaps in a clock neural network for the parallel dynamics these formulae constitute recursion relations for the exact dynamics in the case of extreme dilution the fixed point equation for this exactly soluble model is investigated for varying temperature t capacity parameter and the number q of the clock states

comparison of linear and classical velocity update rules in particle swarm optimization notes on diversity in this study we investigate the significance of diversity in the particle swarm optimization pso algorithm to do so we study two different implementations of the pso being the so called linear and classical pso formulations while the behaviour of these two implementations is markedly different they only differ in the formulation of the velocity update rule in fact the differences are merely due to subtle differences in the introduction of randomness into the algorithm in this paper we show that in algorithms employing the linear pso velocity update rule particle trajectories collapse to line searches in n dimensional space the classical formulation does not suffer this drawback instead directional diverse stochastic search trajectories are retained which in turn helps to alleviate premature convergence the performance of the classical implementation is therefore superior for all test problems considered due to the presence of adequate diversity copyright 2006 john wiley sons ltd

information processing in three state neural networks we introduce networks of three state neurons where the additional state embodies the absence of information their dynamical behavior is studied from the standpoint of information processing these networks display strong pattern completion capabilities moreover inference naturally occurs between coherent patterns

parallel dynamics of fully connected q ising neural networks using a probabilistic approach the parallel dynamics of fully connected q ising neural networks is studied for arbitrary q a novel recursive scheme is set up to determine the time evolution of the order parameters through the evolution of the distribution of the local field taking into account all feedback correlations in contrast to extremely diluted and layered network architectures the local field is no longer normally distributed but contains a discrete part as an illustrative example an explicit analysis is carried out for the first four time steps for the case of the q 2 and q 3 model the results are compared with extensive numerical simulations and excellent agreement is found finally equilibrium fixed point equations are derived and compared with the thermodynamic approach based upon the replica symmetric mean field approximation

retrieval and chaos in extremely diluted q ising neural networks using a probabilistic approach the deterministic and the stochastic parallel dynamics of aq ising neural network are studied at finiteq and in the limitq exact evolution equations are presented for the first time step these formulas constitute recursion relations for the parallel dynamics of the extremely diluted asymmetric versions of these networks an explicit analysis of the retrieval properties is carried out in terms of the gain parameter the loading capacity and the temperature the results for theq network are compared with those for theq 3 andq 4 models possible chaotic microscopic behavior is studied using the time evolution of the distance between two network configurations for arbitrary finiteq the retrieval regime is always chaotic in the limitq the network exhibits a dynamical transition toward chaos

modified fuzzy c means for ordinal valued attributes with particle swarm for optimization there are well established methods for fuzzy clustering especially for the cases where the feature values are numerical of ratio or interval scale not so well established are methods to be applied when the feature values are ordinal or nominal in that case there is no satisfactory method it seems this paper discusses a modified fuzzy c means clustering method where an ordinal to numeric mapping for the ordinal features is obtained as part of the clustering process part of minimizing the objective function for the clustering is to find this ordinal to numeric mapping having this mapping allows standard methods of fuzzy c means clustering to be used since then if there are no categorical features all the features are numeric the mapping is not of interest in itself and to obtain it is only a subsidiary objective of the clustering process the mapping allows for the degrees of freedom that is characteristic of the data the method involves solving a rather challenging optimization problem since the objective function has many local minima this makes the use of a global optimization method such as particle swarm optimization pso attractive for determining the membership matrix for the clustering to minimize computational effort a bayesian stopping criterion may be used in combination with a multi start strategy for the pso other clustering methods generally find local optimum of their objective function through simulations and experiments with real and artificial data the method proposed here is shown to be quite effective

multiplicity of metastable retrieval phases in networks of multistate neurons it is demonstrated that networks of multistate neurons storing an ensemble of multistate patterns exhibit a multiplicity of metastable retrieval phases these phases are described by solutions of fixed point equations with characteristic retrieval errors they emerge if the gain of the neural input output relation is varied the number of these phases increases asq2 with the numberq of gray levels available to each neuron implications for the optimal gain function and basins of attraction are briefly discussed again networks endowed with pseudoinverse couplings are found to perform better than networks with hebbian couplings at moderate loading levels phases with retrieval errors are destabilized whereas the error free phase remains stable up to the theoretically possible maximum if the gain parameter is properly chosen

capacity of diluted multi state neural networks the optimal storage capacity is studied for diluted networks with multi state neurons and continuous respectively discrete couplings within the replica symmetric gardner approach the gardner derrida line the de almeida thouless line and the zero entropy line are compared and the validity of the replica symmetric approximation is discussed in detail the distribution of the synaptic couplings is determined the results are analysed in terms of the number of states of the neuron the distribution of the stored patterns the amount of dilution and the number of discrete values for the couplings

a new family of global methods for linear systems with multiple right hand sides the global bi conjugate gradient gl bcg method is an attractive matrix krylov subspace method for solving nonsymmetric linear systems with multiple right hand sides but it often show irregular convergence behavior in many applications in this paper we present a new family of global a biorthogonal methods by using short two term recurrences and formal orthogonal polynomials which contain the global bi conjugate residual gl bcr algorithm and its improved version finally numerical experiments illustrate that the proposed methods are highly competitive and often superior to originals

neural networks that use three state neurons the dynamics of a neural network which uses three state neurons 1 0 and 1 is solved exactly in the limit of non symmetric and highly dilute synapses recursion relations for the activity the fraction of non zero neurons and overlap of the network with a given pattern are derived which have three generic kinds of fixed points a retrieval fixed point a chaotic fixed point which corresponds to non zero activity but no overlap and a zero fixed point where all the neurons go to the 0 state as the non retrieval fixed points both have activities different from the retrieval fixed point one can easily tell whether a pattern has been recovered an analysis of which fixed points occur as a function of the thresholds and the storage ratio of the system yields remarkably rich phase diagrams optimising the threshold level can be very important especially when low activity patterns are stored a similar analysis can be applied to biased networks using two state 1 0 neurons finally one finds that mixture states which have an overlap with two patterns can be stabilised by a threshold in the networks using three state neurons this property allows larger higher activity memories to be naturally constructed out of smaller ones

thermodynamic limit of the q state potts hopfield model with infinitely many patterns we prove the almost sure convergence of the free energy and of the overlap order parameters in aq state version of the hopfield neural network model we compute explicitly these limits for all temperatures different from some critical value the number of stored patterns is allowed to grow with the size of the systemn like lnq lnn we study the limiting behavior of the extremal states of the model that are the measures induced on the gibbs measures by the overlap parameters

finite state neural networks a step toward the simulation of very large systems neural networks composed of neurons withq n states and synapses withq states are studied analytically and numerically analytically it is shown that these finite state networks are much more efficient at information storage than networks with continuous synapses in order to take the utmost advantage of networks with finite state elements a multineuron and multisynapse coding scheme is introduced which allows the simulation of networks having 1 0 109 couplings at a speed of 7 1 109 coupling evaluations per second on asingle processor of the cray ymp a local learning algorithm is also introduced which allows for the efficient training of large networks with finite state elements

parallel dynamics of extremely diluted symmetric q ising neural networks the parallel dynamics of extremely diluted symmetric q ising neural networks is studied for arbitrary q using a probabilistic approach in spite of the extremely diluted architecture the feedback correlations arising from the symmetry prevent a closed form solution in contrast with the extremely diluted asymmetric model a recursive scheme is found determining the complete time evolution of the order parameters taking into account all feedback it is based upon the evolution of the distribution of the local field as in the fully connected model as an illustrative example an explicit analysis is carried out for the q 2 and q 3 model these results agree with and extend the partial results existing for q 2 for q 2 the analysis is entirely new finally equilibrium fixed point equations are derived and a capacity gain function diagram is obtained

on the dynamics of analogue neurons with nonsigmoidal gain functions abstract the characteristic properties of the macroscopic retrieval dynamics of analogue neurons with hebbian coupling strengths are studied usign the shape of the gain function as a modeling parameter already at low loading a rich diversity in dynamical behaviour is observed covering the full range from point attractors to chaotic dynamics the attractors which are not of the fixed point type are interpreted as an intermediate phase between the well known retrieval states and the zero state enabling a waiting mode in the system using a probabilistic approach it is shown that these features persist in extremely and asymmetrically diluted systems at sufficiently low loading a number of generic examples are worked out in detail illustrating some properties of networks governed by nonmonotonic piecewise linear gain functions the critical storage level above which the chaotic behaviour is absent is numerically determined

on the phase diagram of the q state potts glass neural network the q state potts model of neural networks extended to include biased patterns is studied for extensive loading a mean field equations are written down for general q and arbitrary temperature t within the replica symmetric approximation the complete t phase diagram is discussed especially for q 3 a study of the bifurcation diagrams for the overlap and the free energy clarifies the results obtained

minimization by random search techniques we give two general convergence proofs for random search algorithms we review the literature and show how our results extend those available for specific variants of the conceptual algorithm studied here we then exploit the convergence results to examine convergence rates and to actually design implementable methods finally we report on some computational experience

convergence properties of block gmres and matrix polynomials abstract this paper studies convergence properties of the block gmres algorithm when applied to nonsymmetric systems with multiple right hand sides a convergence theory is developed based on a representation of the method using matrix valued polynomials relations between the roots of the residual polynomial for block gmres and the matrix e pseudospectrum are derived and illustrated with numerical experiments the role of invariant subspaces in the effectiveness of block methods is also discussed

image evolution in potts glass neural networks neural networks of the potts type where neurons can occupy q or 2 discrete states are considered for couplings which need not be symmetric the time evolution of the macroscopic overlap between the instantaneous microscopic state of the system and the learned patterns is studied for small values of q and for a finite small number of patterns retrieval and limit cycle behaviour in this type of model is discussed in some detail

numerical study on incomplete orthogonal factorization preconditioners we design analyse and test a class of incomplete orthogonal factorization preconditioners constructed from givens rotations incorporating some dropping strategies and updating tricks for the solution of large sparse systems of linear equations comprehensive accounts about how the preconditioners are coded what storage is required and how the computation is executed for a given accuracy are presented a number of numerical experiments show that these preconditioners are competitive with standard incomplete triangular factorization preconditioners when they are applied to accelerate krylov subspace iteration methods such as gmres and bicgstab