Overview of neural hardware

Neural hardware has undergone rapid development during the last few years. This paper presents an overview of neural hardware projects within industries and academia. It describes digital, analog, and hybrid neurochips and accelerator boards as well as large-scale neurocomputers built from general purpose processors and communication elements. Special attention is given to multiprocessor projects that focus on scalability, flexibility, and adaptivity of the design and thus seem suitable for brain-style (cognitive) processing. The sources used for this overview are taken from journal papers, conference proceedings, data sheets, and ftp-sites and present an up-to-date overview of current state-of-the-art neural hardware implementations. 1 Categorization of neural hardware This paper presents an overview of time-multiplexed hardware designs, some of which are already commercially available, others representing design studies being carried out by research groups. A large number of design studies are being carried out in the US, Japan, and Europe. In many cases these studies concern design concepts of neurocomputers that will never be built in full. Neurocomputer building is expensive in terms of development time and resources, and little is known about the real commercial prospects for working implementations. Moreover, there is no clear consensus on how to exploit the currently available VLSI and even ultra large-scale integration (ULSI) technological capabilities for massively parallel neural network hardware implementations. Another reason for not actually building neurocomputers might lie in the fact that the number and variety of (novel) neural network paradigms is still increasing rapidly. For many paradigms the capabilities are hardly known yet. Paradoxically, these capabilities can only be tested in full when dedicated hardware is available. Commercially available products mostly consist of dedicated implementations of well known and successful paradigms like multi-layer perceptrons with backpropagation learning (e.g., Rumelhart & McClelland, 1986), the Hopfield (e.g., Hopfield, 1982), or the Kohonen models (e.g., Kohonen, 1989). These dedicated implementations in general do not offer much flexibility for the simulation of alternative paradigms. More interesting implementations, from a scientific as opposed to an application viewpoint, can mainly be found only in research laboratories. Dedicated neural hardware forms the sixth computer generation. The first four computer generations are mainly distinguished by the implementation technology used. They were built up respectively from 1 This is a draft version based on Chapter 3 in: Heemskerk,  (1995). Neurocomputers for BrainStyle Processing. Design, Implementation and Application . PhD thesis, Unit of Experimental and Theoretical Psychology Leiden University, The Netherlands. This draft is ftp-able from: ftp.mrc-apu.cam.ac.uk/pub/nn. 2 Jan N.H. Heemskerk vacuum tubes, transistors, integrated circuitry, and VLSI. Third and fourth generation machines were the first parallel machines. Fifth generation computer systems were defined as knowledge based systems, originally designed to accept sensory inputs (Carling, 1992). They formed a combination of AI software and parallel hardware for operations like logical inference and data retrieval. The sixth generation can be seen as an integration of insights into computer design and programming from cognitive science and neuroscience. The machine of the future must perform a cooperative computation system that integrates different subsystems, each quite specialized in structure and some supplied with sensors and motors. A number of these subsystems should implement huge neural networks. Some of the neural hardware implementations described in this overview allow for fast real-world interaction via sensors and effectors and can thus function as action oriented systems (Arbib, 1989). A large number of parallel neural network implementation studies have been carried out on existing massively parallel machines, simply because neural hardware was not available. Although these machines were not specially designed for neural implementations, in many cases very high performance rates have been obtained. Many examples can be found in the literature, for instance: implementation of backpropagation networks has been performed on the Connection Machine (Singer, 1990), Warp (Pomerleau et al., 1988), MasPar (Chinn et al., 1990; Grasjki, 1992), Hughes (Shams & Gaudiot, 1990), GF11 (Witbrock & Zagha, 1989), AAP-2 (Watanabe et al., 1989), transputer based machines (e.g., Vuurpijl, 1992), and the CRAY YM-P supercomputer (Leung & Setiono, 1993). Much can be learned from these studies about programming neural functions and mapping networks onto pre-specified architectures. Several successful architectural issues have been re-applied in the newer dedicated neurocomputer designs. This paper is limited to an overview of electronic learning neural hardware and will not outline the architectures of general purpose massively parallel computers (supercomputers), or neurocomputer designs based on other (although very promising) implementation techniques such as opto-electronics, electro-chemical, and molecular techniques. Other neural hardware overviews have been written by Treleaven (1989), Nordström et al. (1992), Vellasco (1992), Ienne (1993b), Glesner and Pöchmüller (1994), and Ramacher and Rückert (1991). This neural hardware overview will be given by grouping the different approaches into four main categories, according to a scheme given by Rückert (1993) Speed performance increases with (bottom) category from left to right. The first two main categories consist of neurocomputers based on standard ICs. They consist of Accelerator boards which speed up a conventional computer like a PC or workstation, and parallel multiprocessor systems, which mostly run stand alone and can be monitored by a host computer. In these approaches, where standard parts are used, designers can concentrate fully on developing one technology at a time. The other main categories consist of neurocomputers built from dedicated neural ASICs (application specific integrated circuits). These neurochips can be digital, analog, or hybrid. The number of world wide ongoing projects is already too large to allow a complete overview. Special attention will be given to projects which enable the implementation of action oriented systems and supports the following issues: speed, adaptivity, flexibility, and scalability. Reported performance rates are in (million) CPS and CUPS and are taken from publications. They only serve as indications and have to be compared with care since the implementations differ in precision, and size. Furthermore, adequate benchmarks have not yet been generally developed. An often used benchmark for the learning and recall phase of backpropagation networks is NETtalk, which translates text to phonemes (Sejnowksi and Rosenberg, 1987). Other hardware benchmark proposals have for instance been made by Ienne (1993a) and Van Keulen et al. (1994). 2 Neurocomputers consisting of a conventional computer + accelerator board Accelerator boards are the most frequently used neural commercial hardware, because they are relatively cheap, widely available, simple to connect to the PC or workstation, and typically provided with user 3 Overview of neural hardware friendly software tools. In some cases users are already familiar with the software tools, having used them on Fig.1. Neurocomputer categories. their standard, non-accelerated systems. (An overview of neural network simulators can be found in Murre, 1995) Such users will only notice the addition of an accelerator board by the reduced waiting times when training or running a neural model. The speed-up that can be achieved is at about one order of magnitude compared to sequential implementations. Some accelerators come with up to 20 different versions of paradigms which allow for a lot of experimenting. A drawback of this kind of neurocomputer and commercially available software simulators is that they lack flexibility and thus do not offer many possibilities for setting up novel paradigms. Some commercially available accelerator systems will now be briefly reviewed. ANZA plus (HNC [Hecht-Nielsen Cooperation] CA, USA) The ANZA coprocessor boards plug in to the backplane of a PC. They contain a Motorola NC68020 processor and a MC68881 floating point coprocessor. Performance rates are given in Treleaven (1989): 1M virtual PEs, 1.5 M Connections, 1.500 CUPS, 6 MCPS.  User Interface Subroutine Library. SAIC SIGMA-1 (Science Applications International Corporation) This is a Delta floating point processor based system to be plugged into a PC. Performance: 3.1 M virtual PEs and interconnections, 11 MCUPS. Software: neural net library ANSim and ANSpec, an object oriented language (from: Treleaven, 1989). NT6000 (Neural Technologies Limited, HNC, and California Scientific Software) Neural Technologies Limited is an organization which offers a wide range of commercially available neural products. The NT6000st and NT6000hs, for instance, are neural network plug-in PC-cards that fulfill the need for intelligent data acquisition. They are equipped with a (TMS320) DSP and an NISP (Neural Instruction Set Processor), which speeds up neural processing to 2 MCPS. Products like these are supplied by well developed software packages that allow for interfacing to other commercially available neural network simulators like BrainMaker and NeuralWorks (see Murre, 1995). The NT6000 series are well suited 4 Jan N.H. Heemskerk to be connected to both analog and digital input/output systems (from: Neural Technologies data sheet, 1993). All this together makes a standard (386 or 486) based PC an interesting neural network implementation tool. The implemented neural network paradigms are, however, limited to back