Implementation Of Neural Networks On Parallel Architectures

Computations in Arti cial Neural Network (ANN) models are based on the same organizational principles as the brain. Just as in the brain, ANN models employ many simple computational elements that work concurrently to achieve `brain-like' tasks. Although ANN's have shown great promise, their full potential has yet to be realized as most implementations have been on sequential machines that are unable to exploit the inherent parallelism in these networks. In this dissertation, we describe how ANN models can be implemented on programmable parallel digital machines that provide an attractive way to exploit parallelism in a exible manner. We consider two broad classes of neural models{networks with Static Links and those with Dynamic Links{and show how they can be implemented on parallel architectures. For the case of static link models, we present implementation algorithms on 2D array based architectures. A general algorithm is designed to implement one iteration of a fully connected single layer network on an array with n2 modules in O(n) time, where n is the number of neurons in the network. We present an algorithm to implement sparsely connected single layer networks on an (n+ e) module 2-D array in O(pn+ e) time, where e is the number of non-zero connections in the network. This algorithm uses preprocessing on the connection weight matrix and ideas from Interconnection Networks to yield the above optimal asymptotic time complexity with a small constant factor. Finally, we present an algorithm to implement multi-layer networks running the Back-propagation learning algorithm. We show how the above algorithms can be adapted for two members of the class of 2D arrays{the Reduced Mesh of Trees (RMOT) and theMesh Connected Computer (MCC). Most real machines have a xed number of modules and provide a coarse grain of parallelism for the problems we are considering. We show how the above algorithms can be run on xed sized arrays. Any computation that can be performed on an RMOT with n processing elements (PE's) in T (n) time, can be performed on a xed-size RMOT with p n PE's in O(nT (n)=p) time. Sparse ANN's of the type described above can be implemented on a p p array of processors in O((n + e)=p) time. Each PE of the array contains a multiplier, an adder and O((n + e)=p2) local storage. Within the class of networks with dynamic links, we concentrate on the xi Dynamic Link Architecture [vdM88]. This is a model that is capable of translation invariant object recognition. Objects in the framework of the model are represented as labeled graphs and object recognition is achieved by labeled graph matching. We show how the choice of feature types to label the nodes of the graph a ects the granularity of parallel implementation. We rst consider graphs in which nodes are labeled by discrete features and present an algorithm to implement the graph matching on a linear array of processors in O(NA logNA + NANB=p) time where NA is the number of nodes in layer A of the model, NB is the number of nodes in layer B, and p is the number of processors in the array. We then describe a Face Recognition System inspired by the Dynamic Link Architecture where continuous features are used to label the nodes of the labeled graph. Salient points on the face are chosen automatically by the system and form the nodes in the labeled graph representation of the face. The system improves upon existing face recognition systems by using phase information to match features. Face-on pictures of 88 people form the database of the system. In a recognition run for pictures of the same people with a 15 rotation in depth, the system yielded a 91% recognition rate. We show how this system can be implemented e ciently in parallel on a coarse grained network of transputers. xii Chapter