Connectivity in Real and Simulated Associative Memories

Connectivity in Real and Simulated Associative Memories Neil Davey (N.Davey@herts.ac.uk) Lee Calcraft (L.Calcraft@herts.ac.uk) Rod Adams (R.G.Adams @herts.ac.uk) School of Computer Science University of Hertfordshire Hatfield, UK Abstract In such sparse networks the connection strategy employed has to balance two competing goals. Firstly the total amount of neuronal fiber should be minimized, both because it is biologically expensive and because increasing length implies increasing difficulty in finding physical paths for the fiber, in the brain. Secondly, however, information needs to travel efficiently throughout the system for fast global computation to take place. The connectivity satisfying both goals is therefore highly likely to be have been optimized by evolution and will be far from random. Finding efficient patterns of connectivity in sparse associative memories is a difficult problem. It is, however, one that real neuronal networks, such as the mammalian cortex, must have solved. We have investigated computational models of sparsely connected associative memories and found that some patterns of connectivity produce both good performance and efficient use of resources. This could illuminate how real biological systems solve the problem. Our major finding is that a Gaussian distribution of connection lengths, with a relatively small variance, gives networks with high performance and minimal wiring. Non-Random Graphs Keywords: Associative Memory; Connectivity; Sparse Networks. Before looking at some of what is currently known about the actual connectivity strategy employed in real systems, we examine what is known about connection graphs in general. There is a long history of research into the properties of random graphs, graphs in which the connectivity matrix is randomly configured, often with a specific probability of connectivity (Bollobas, 2001). Recently, however, there has been an explosion of interest in networks with non-random connectivity graphs, such as small-world and scale-free networks. The seminal paper of Watts and Strogatz (Watts & Strogatz, 1998) formalised the notion of a small-world network. The idea was inspired by work in the Social Sciences showing that there appeared to be only roughly 6 degrees of separation (by acquaintance) between any two people in North America (Milgram, 1967); this despite the fact that most people have a cliquish group of acquaintances, in the sense that any two of their acquaintances are also likely to be acquaintances. The small-world effect is therefore characterised as a network with short path lengths (the minimum number of arc traversals to get from one node to another), between any pair of nodes. The simplest sort of network that displays this characteristic is a random network. In a regular random network of N nodes, with each node having k connections, the number of first order acquaintances is k, second order is about k 2 , third order k 3 and so on. So in general the number of degrees of separation, D, to reach all N nodes in the ln N D network is given by setting k = N , which gives D = ln k so that D increases logarithmically with the size of the network – the small-world effect. However, random Introduction In recent years much has been discovered about the pattern of connectivity of the neurons in real neuronal networks. A good deal of this research has explored the way that specific neuronal circuits operate and how functional areas in the brain are connected. Another approach, however, has been to examine the connectivity matrix at the level of individual neurons or of functional areas, and to look for patterns of connectivity familiar in other contexts. This ties in with the large amount of current interest in small-world and scale-free networks. In this paper we look at this second issue in the context of associative memory networks. We summarize some of what is currently known about the connectivity matrix in real neuronal systems and show how the search for optimal patterns of connectivity in artificial associative memory networks may shed light on some of the issues that real neuronal systems must deal with. Our major finding is that a Gaussian distribution of connection lengths, with a relatively small variance, gives networks with high performance and minimal wiring. Connection Strategies The connectivity in real neuronal systems, such as the mammalian cortex, is quite different from that found in most artificial neural networks. Real neuronal systems have vast numbers of neurons connected to only a fraction of the other neurons. For example the human cortex has about 10 11 neurons with each connected to roughly, on average, 10,000 other neurons (Braitenberg & Schuz, 1998).