SCALABLE FLOW VIRTUAL LEARNING NEUROCOMPUTER

Abstract: A scalable flow virtual learning neurocomputer system and apparatus (Fig. 2) with a scalable hybrid control flow/data flow employing a group partitioning algorithm, and a scalable virtual learning architecture, synapse processor architecture mapping, inner square folding and array separation, with capability of back propagation for virtual learning. The group partitioning algorithm creates a common building block of synapse processors containing their own external memory. The processor groups are used to create a general purpose virtual learning machine which maintains complete connectivity with high performance. The synapse processor group allows a system to be scalable in virtual size and direct execution capabilities. Internal to the processor group, the synapse processors are designed as a hybrid control flow/data flow architecture with external memory access and reduced synchronization problems.

Description: SCALABLE FLOW VIRTUAL LEARNING NEUROCOMPUTER FIELD OF THE INVENTION The inventions relate to a scalable flow virtual learning neurocomputer system and apparatus and particularly to a scalable hybrid control flow/data flow virtual learning neurocomputer which employs a new group partitioning algorithm, and a scalable virtual learning architecture, synapse processor architecture (SPA) mapping, inner square folding and array separation, with capability of back propagation for virtual learning. CROSS REFERENCE TO RELATED APPLICATIONS The present application claims priority and is a continuation-in- part of the following related co-pending Patent Applications: APPARATUS AND METHOD FOR NEURAL PROCESSOR, S. Vassiliadis, and G.G. Pechanek, U.S. Serial No. 07/526,866, filed May 18, 1990, (IBM Docket EN9-90-045), sometimes referred to as "SNAP" A TRIANGULAR SCALABLE NEURAL ARRAY PROCESSOR, G.G. Pechanek, and S. Vassiliadis, U.S. Serial No. 07/682,785, filed April 8, 1991 (IBM Docket EN9-91-018) sometimes referred to as "T-SNAP". SPIN: A SEQUENTIAL PIPELINED NEURO COMPUTER, S. Vassiliadis, G.G. Pechanek.and J.G. Delgado-Frias, U.S. Serial No. 07/682, , filed April 8, 1991 (IBM Docket EN9-91-026) sometimes referred to as "SPIN". In addition, filed concurrently are related applications: A LEARNING MACHINE SYNAPSE PROCESSOR SYSTEM APPARATUS G.G. Pechanek, S. Vassiliadis, and J.G. Delgado-Frias, U.S. Serial No. 07/ , filed May, 1991 (IBM Docket EN9-91-051), some¬ times referred to as "LEARNING MACHINE", or Pechanek LEARN 91. VIRTUAL NERUOCOMPUTER ARCHITECTURES FOR NEURAL NETWORKS, G.G. Pechanek, J.G. Delgado-Frias, and S. Vassiliadis, U.S. Serial No. 07/ , filed May, , 1991 (IBM Docket EN9-91-053) some¬ times referred to as "VIRTUAL", or Pechanek VIRTUAL 91. PLAN: PYRAMID LEARNING ARCHITECTURE NEUROCOMPUTER, G.G. Pechanek, S. Vassiliadis, and J.G. Delgado-Frias, U.S. Serial No. 07/ , filed May , 1991 (IBM Docket EN9-91-055) some¬ times referred to as "PLAN". These co-pending applications and the present application are owned by one and the same assignee, namely, International Business Machines Corpo¬ ration of Armonk, New York. The descriptions set forth in these co-pending applications are hereby incor¬ porated into the present application by this reference. REFERENCES USED IN THE DISCUSSION OF THE INVENTIONS During the detailed discussion of our inventions, we will reference other work including our own unpublished works which are not Prior Art but which will aid the reader in following our discussion. These additional references are: D.E. Rumelhart, J.L. McClelland, and the PDP Research Group, Parallel Dis¬ tributed Processing Vol. 1: Foundations Cambridge, Massachusetts: MIT Press 1986. (Herein referred to as "Rumelhart 86".) J.J. Hopfield, "Neurons With Graded Response Have Collective Computa¬ tional Properties Like Those of Two-State Neurons," Proceedings of the National Academy of Sciences 81, pp. 3088-3092, May, 1984. (Herein referred to as "Hopfield 84".) A LEARNING MACHINE SYNAPSE PROCESSOR SYSTEM APPARATUS G.G. Pechanek, S. Vassiliadis, and J.G. Delgado-Frias, U.S. Serial No. 07/ , filed May, 1991 (IBM Docket EN9-91-051 ), some¬ times referred to as "LEARNING MACHINE" or Pechanek LEARN 91. VIRTUAL NERUOCOMPUTER ARCHITECTURES FOR NEURAL NETWORKS, G.G. Pechanek, J.G. Delgado-Frias, and S. Vassiliadis, U.S. Serial No. 07/ , filed May, , 1991 (IBM Docket EN9-91-053) some¬ times referred to as "VIRTUAL" or Pechanek VIRTUAL 91. H. Hellerman, Digital Computer System Principles McGraw-Hill Book Company, pp. 346-348. (Herein referred to as "Hellerman 67".) BACKGROUND OF THE INVENTION Virtual learning implies the ability to modify the virtual connection weights and the virtual neural state values stored in an external memory. This func¬ tion would be desireable for an environment which demands the complete scalability of other functions developed in co-pending applications. This func¬ tion and an apparatus for achieving it has not been developed. SUMMARY OF OUR INVENTIONS. We have provided a virtural learning neurocomputer apparatus which is termed the Scalable Virtual Learning Machine (SVLM) providing high per¬ formance direct emulation and virtual learning capability in a completely con¬ nected scalable architecture.Our SVLM provides the ability to modify the virtual connection weights and the virtual neural state values stored in the external memory to which the apparatus can be connected. In accordance with our inventions we have provided a scalable group partitioned virtual neural synapse processor architecture apparatus with an N neuron structure partitioned into groups of synapse processing units. Each group has multiple synapse processing units that contain instruction and data storage units, receive instructions and data, and execute instructions, and interface with external data storage for weights and neuron output values for V neurons, where V > N and V is the number of neurons in the neural network to be emulated on the group partitioned N neuron structure. The groups execute group instruction and have communicating adder trees. The N neuron struc¬ ture has neuron activation function units in the preferred form of a sigmoid generator, and the structure provides for interconnecting the groups into an N neuron structure which supports V neurons and for communicating both instructions, data, and the outputs of the neuron activation function units back to the input synapse processing units by way of the communicating adder trees. The scalable flow virtual learning neurocomputer system appratus has a scalable hybrid control flow/data flow and employs a group partitioning algo¬ rithm for the scalable virtual learning architecture. We provide synapse processor architecture mapping, inner square folding and array separation, with capability of back propagation for virtual learning. The group partitioning algorithm creates a common building block of synapse processors containing their own external memory. The processor groups are used to create a general purpose virtual learning machine which maintains complete connectivity with high performance. The synapse processor group allows a system to be scalable in virtual size and allows direct execution capabilities. Internal to the processor group, the synapse processors are designed as a hybrid control flow/data flow architecture with external memory access and reduced synchronization problems. The general computer architecture supporting neural networks which is developed utilizes a method of folding and separating an array of "synapse" processors creating a cluster of processors, or a processor group, which represents a common building block for neurocomputer implementations. A neurocomputer made up of these processor groups directly executes a com¬ pletely connected N neuron network model and supports a virtual mode of execution for network models containing greater than N neurons. The synapse processor architecture (SPA) of each processor in the group sup¬ ports the direct emulation scalability resulting from the group partitioning algorithm and supports virtual synapse scalability through the use of external memory. One of the features of the unique SPA is that it allows both control flow operation with sequential, instruction execution and data flow operation where the execution is dependent upon the receipt of valid tagged data. Another unique feature of the architecture is the method of interfacing to processor groups which supports complete connectivity in a neural network model even though multiple processor groups are utilized in an implementa¬ tion. Other unique features of the neurocomputer are the group notification of the completion of operations local to the synapse processors and support of learning algorithms. These and other improvements are set fprth in the following detailed description. For a better understanding of the inventions, together with advantages and features, reference may be had to the co-pending applica¬ tions for other developments we have made in the field. However, specifically as to the improvements, advantages and features described herein, reference will be made in the description which follows to the below-described drawings. BRIEF DESCRIPTION OF THE DRAWINGS. FIGURE 1 shows a 4-neuron structure supporting 16 virtual neurons (256 Weights) weights. FIGURE 2 illustrates an 8 neuron array model. FIGURE 3 shows an 8 neuron array model split into 1 square and 2 smaller triangular processor arrays. FIGURE 4 shows the folded inner square of the 8 neuron array model. FIGURE 5 shows the folded inner square separated into two triangular arrays. FIGURE 6 shows a preferred external summation, sigmoid generator, and reverse communications path. FIGURE 7 shows a 16 neuron array model weight matrix with neuron Y values. FIGURE 8 shows triangular arrays 1 and 2. FIGURE 9 shows triangular arrays 3 and 4. FIGURE 10 shows a folded inner square 5 separated into two triangular arrays. FIGURE 11 shows a folded inner square 6 separated into two triangular arrays. FIGURE 12 shows a folded inner square 7 separated into two triangular arrays. FIGURE 13 shows a folded inner square 8 separated into two triangular arrays. FIGURE 14 shows a folded inner square 9 separated into two triangular arrays. FIGURE 15 shows a folded inner square 10 separated into two triangular arrays. FIGURE 16 illustrates a preferred external summation, sigmoid generator, with a reverse communication path. FIGURE 17 illustrates a switching mechanism. FIGURE 18 illustrates alternatively a modified 8 neuron switching mechanism to support learning. FIGURE 19 illustrates alternatively a modified 16 neuron switching mech¬ anism to support learning. FIGURE 20 illustrates another part of a modified 16 neuron switching mech¬ anism to support learning. FIGURE 21 is a table illustrating the V and N relationship to storage and oper¬ ating cycles. FIGURE 22 illustrates a 4 neuron scalable virtual learning machine with V = 16, N * = 4, G = 4. FIGURE 23 shows an external memory structure. FIGURE 24 illustrates a scalable virtual learning machine's external memory structure for V= 16, N =4, H = 16, with 1 weight, 1 Y value bit per processor element, and H weight elements per processor group. FIGURE 25 illustrates a synapse processor architecture with external memory, with a diagnoal synapse processor (DSYP) and a general synapse processor (GSYP). FIGURE 26 shows the 4-neuron SVLM timing for a 16 neuron neural network. FIGURE 27 shows the cycle time expanded to show detailed timing. FIGURE 28 illustrates a memory subsystem associate with each synapse processor group and the external memory addresssing structure. FIGURE 29 illustrates a neuron processor tagged instruction/data format. FIGURE 30 illustrates a neural network for solving the input/output problem. FIGURE 31 shows an 11 neuron I/O encoder problem mapped onto a 4 neuron scalable SVLM with V= 16, N = 4, G = 4. FIGURE 32 illustrates the initializiation. FIGURE 33 illustrates first layer execution. FIGURE 34 illustrates set SO count to advance mode and reverse communi¬ cate Y1' and Y2'. FIGURE 35 illustrates second layer execution. FIGURE 36 illustrates reverse communicate Y3'. FIGURE 37 illustrates third layer execution. FIGURE 38 illustrates reverse communicate Y4', Y5', Y6', and Y7'. FIGURE 39 illistrates fourth layer execution. FIGURE 40 shows the learning mode of Load S & I bits and SO and IA Counts, change data path to R4, and reverse communicate E8, E9, E10 and E11. FIGURE 41 shows the learning mode of issue MPY R6 * R3 T to start multiply for weighted error summations: ER4, ER5, ER6 and ER7. FIGURE 42 shows the learning mode of create weighted error summations: ER4, ER5, ER6 and ER7. FIGURE 43 shows the learning mode of step 1 weight updating MPY R5 * R3 →R4. FIGURE 44 shows the learning mode of step 2 weight updating MPY R4 * IMD →R4. FIGURE 45 shows the learning mode of weight updating in external memory, ADD R6 + R4 →EXTMEM2. (Note: For convenience of illustration, FIGURES may be separated in parts and as a convention we place the top of the FIGURE as the first sheet, with subsequent sheets proceeding down when viewing the FIGURE, in the event that multiple sheets are used.) Our detailed description follows as parts explaining our preferred embod¬ iments of our inventions provided by way of example. DETAILED DESCRIPTION OF THE INVENTIONS. Before considering our preferred embodiments, it may be worthwhile to intro¬ duce some common features which are used in illustrating our preferred embodiments. For the purposes of this description, the number V will be used to denote the number of neurons contained in the network to be modeled on the neurocomputer and the number N will denote the number physical neurons available in the physical implementation. For virtual processing V > N. The computational tasks to be implemented by the neurocomputer architectures described in this description are given by equation 1 and 2 which are based on a subset of the full Parallel Distributed Processing model of Rumelhart 86 and the Hopfield network, Hopfield 84. Where: • V is the number of neurons in the neural network. • The subscripts on the weights W (such as IV )3 ) are to be interpreted as meaning the weight of the connection from Neuron 3 to Neuron 1. • Y. is the " neuron output value connected to the /'" neuron input through a connection weight of W,,. • Ex, is an external input to the /'" neuron. • — A ≤ Ex, < +A where -A equates to Y, -= 0 and + A equates to Y, == 1 (± an acceptable range of error) for the case where £x, is the only input to the neuron. • F(z.) is the neuron activation function which many times is set equal to a sigmoid activation function whose form, for example, is: ?(*) -= 2z_ (2) 1 + e " T Where: Y j W 9 + Ex, • T is a global control parameter used to modify the slope of the sigmoid function for a given set of z, values. • e = Natural log (2.71828...) Equations 1 and 2, for a completely connected V neuron network, contain four basic operations: 1. V 2 Multiplications 2. V Product Summations 3. V Activation Functions 4. V x V Communications Neurocomputer implementations have many difficult problems to surmount requiring a high degree of parallelism to achieve high performance. In relationship to the four basic operations for neural execution, it would be desireable to have V 2 multiplication engines, V product summation and acti¬ vation mechanisms, and a V x V communications medium with minimum delay. Typically V is large compared to N, where N represents the physically realizable neurocomputer. Since V > N, a virtual emulation capability in the neurocomputer is desireable in order to support numerous sizes of neural network models. Learning imposes further requirements in that more general purpose execution capabilities are needed in the neurocomputer architecture. Complete connectivity along with high performance in conjunction with virtual and learning capability, while providing an architecture which is scalable in both virtual capacity and in direct emulation capability, has been a difficult task. These issues are addressed with the Scalable Virtual Learning Machine (SVLM). While our other co-pending applications are basic and applicable to many sit¬ uations where they provide the best way of delivering a solution to needed problems, there are, we now recognize in certain situations, certain limita¬ tions which can be found in specific environments. For example, the General Purpose Learning Machine (GPLM) Synapse Processing Architecture (SPA) of our Pechanek LEARN 91. has the storage capacity of only a single connection weight and neuron value per synapse processing element as is appropriate for the direct execution of a neural network model. This direct emulation capability limits V to be less than or equal to N. Further the GPLM utilized a simple state machine control mechanism not requiring the utilization of tagged data for synchronizational control of the automatic mode (see Pechanek LEARN 91 ). As V becomes large and virtual capability is added, the state machine control becomes complex. The SPA also contained local processing instructions which did not provide notification of completion to the instruction-issuing learning processor. A state machine control operation in the learning processor was assumed such that time outs, dependent upon instruction type, determined when a new instruction could be sent to the synapse processors. The virtual TSNAP of Pechanek VIRTUAL 91 included the storage of the virtual connection weights and neuron values within the synapse processing elements, allowing the emulation of networks of V neurons where V > N but thereby setting V to a fixed number and linking V and N to the technology capability. In addition, the virtual TSNAP did not support learning capability, using only a simple multiplication functional unit and the necessary weight and Y value storage instead. Multiple difficulties arise with both approaches due to the above characteristics as the number of neurons V to be emulated increases. One difficulty is the limitation on the size of a neural network which can be emulated on the structures. The GPLM will be limited to a physical implementation of N neurons, allowing only net¬ works of V neurons where V < N can be emulated. Whereas the GPLM limited N, the virtual TSNAP limits V, also due to technology consideration. Once V is chosen and the chips built for the virtual TSNAP, V cannot be increased. Another difficulty is the practical consideration of the number of synapse processors which can be placed on a chip due to a chosen technolo¬ gy's cell and I/O capacity. Assuming a large number of synapse processing elements on a chip and a large number of chips utilized in a desired system, the synchronization control becomes very difficult. A state machine control approach for all operations is not feasible for large systems; once a state machine controlled system is built, it will not be easily extendable. In order to solve the synchronization of an instruction's automatic mode of operation, a new architecture is developed to support both control flow (sequential instruction execution) and data flow (data dependent execution) modes of operation. An instruction's automatic mode of operation becomes defined as a data flow controlled operation which requires valid tagged data to be received prior to "firing" the function. The local processing of instructions without notification also becomes problematic for large systems. A new method of notification of local processing completion is defined whereby the group summation trees are utilized, providing not only successful operation completion but also communication of errors which may have occurred in an operation. In this manner, the completion of all operations are notified to the instruction/data issuing processor, thus ensuring proper control and further allowing scalability. Given a limited number of synapse processors on a chip, a special situation which may occur, the question becomes how can one build a large N neurocomputer with learning capability made up of multiple synapse proc¬ essing chips which supports V neurons, where V ≥ N, allowing future growth through expansion of not only the weight and neuron value storage thereby increasing the virtual capacity, but also expansion of the synapse processing elements, thereby increasing the direct execution capacity improving per¬ formance? Multiple new concepts will be presented in this description, such as a group partitioning algorithm, the hybrid control flow/data flow architec¬ ture, and notification of local synapse processing completion in order to solve this scalability problem while still preserving learning and complete connectivity among the modeled neurons and providing high emulation per¬ formance. A method of breaking a large triangular array of processors into smaller triangular processor array groups via a novel partitioning algorithm is presented which allows a practical scaling mechanism. An external memory architecture is developed, allowing the growth of the connection weight and neuron value storage only dependent upon the amount of external memory utilized. Both the GPLM of the co-pending application and the virtual TSNAP of the co-pending application may be inadequate to accom¬ plish the scalability in both capacity and performance. A new neurocomputer architecture is developed maintaining its support of learning and repres¬ enting a new type of neurocomputer to be referred to as the Scalable Virtual Learning Machine (SVLM). To illustrate the best way of developing this kind of machine, we will present examples of our preferred embodiments. Pre¬ sented first is the synapse processor architecture mapping onto a virtual TSNAP structure, next the inner square folding and array separation group partitioning algorithm, followed by the description of the new synapse processor architecture and group architecture supporting external memory, hybrid control flow/data flow operation, and notification of local synapse processing completion, with or without errors. An example of back- propagation learning will also be presented demonstrating the utilization of the new architecture. SCALABLE VIRTUAL LEARNING ARCHITECTURE SPA MAPPING ONTO A VIRTUAL TSNAP The virtual TSNAP of Pechanek VIRTUAL 91 includes the weights and neuron values in the synapse processor cells and shares the multipliers and adder trees in a cyclic fashion with the use of an iterative adder. Each iterative add cycle generates a partial sum, stored in the iterative adder (IA), which accu¬ mulates the network summation each iterative add cycle. The register multiplier architecture of the virtual TSNAP can be mapped onto the Synapse Processor Architecture of Pechanek LEARN 91 . There are two types of processor "cell" structures used for learning in the synapse processor architecture , the DSYP, diagonal synapse processing "cell" and the GSYP, general synapse processing "cell". The DSYP contains one weight/Y value multiplication element and the GSYP contains two weight/Y value multiplication elements. The connection weight and neuron value storage requirements for a virtual 16 neuron network mapped onto a four physical neuron GPLM is shown in FIGURE 1. The DSYPs are outlined with dots and the GSYPs are outlined with dashes. The weights within the DSYPs and GSYPs are grouped into four columns where each column is made up of the connection weights associated with four iterative add cycles. In this example, four iterative add cycles are necessary to produce a neuron output. As depicted, the first set of four iterative add cycles will produce Y1', Y2', Y3\ and Y4'. The second set of four iterative add cycles will produce Y5', Y6', Y7', and Y8'. This continues until all 16 neuron outputs are produced. The merged structure represents the starting point for describing the new SVLM architecture. INNER SQUARE FOLDING AND ARRAY SEPARATION A GPLM structure of Pechanek LEARN 91 for eight neurons is shown in FIGURE 2 where: It can be noticed that there are one square and two smaller triangular arrangements of synapse processor cells within the larger triangular struc¬ ture of FIGURE 2. Separating these out from the main structure and allowing the final stage of the summation tree to be external to the arrangement of synapse processors creates the structure shown in FIGURE 3. FIGURE 3 con¬ tains three structures ie. the two triangular arrays and one square array. In order to support scalability, a common replicatable design is desired. Since there are two types of structures shown in FIGURE 3, a square and two trian¬ gular arrangements of synapse processors, scalability via a single design is not achievable. It will now be shown how the square organization can be restructured into two triangular arrays the same size as the other two trian¬ gular arrays making up the original 8 neuron model, thereby creating a common replicatable design. In order to accomplish this the square structure is first folded along its diagonal as is shown in FIGURE 4 with all synapse processors kept intact and only the summation trees folded. By rearranging the weight and Y value storage within each synapse processor, the folded structure of FIGURE 4 can be redrawn as two separate triangular processing arrays as shown in FIGURE 5. By following this procedure the large trian¬ gular structure of FIGURE 2 has been shown to be made up of 4 smaller tri¬ angular arrays. A new notation relating to groups of processors will now be introduced where G represents the number of groups and H represents the number of synapse processor elements within a group G. Each smaller triangular structure represents a group G of size H **** 16 in this case. FIGURE 3 depicts two groups one producing A1 , A2, A3, and A4 and the second producing B5, B6, B7, and B8. FIGURE 5 depicts the separated triangular array groups one producing B1 , B2, B3, and B4 and the other producing A5, A6, A7, and A8. To make the N = 8 neuron triangular array, FIGURE 2, four groups, G * = 4, are required. The external summation and sigmoid generation functions, and reverse communication paths, completing the 8 neuron model, are shown in FIGURE 6 where the (- ) indicates the forward summation path and the (*-) indicates the reverse communications path. This partitioning concept is next presented for a 16 neuron neurocomputer. The first step is to partition the 16 neuron structure into groups of triangular and square arrays as shown in FIGURE 7. For a 16 neuron model using a group size H of 16, there are 4 triangular arrays, labeled 1, 2, 3, and 4 and 6 square arrays, labeled 5, 6, 7, 8, 9, and 10. The square arrays are next folded and separated into the desired triangular arrays. The resulting 16 tri¬ angular arrays, in this case, are shown in FIGURES 8 through 15, each marked with a group label referring to FIGURE 7. The external summation and sigmoid generation functions, and reverse communications paths, com¬ pleting the 16 neuron model, are shown in FIGURE 16, where (→) indicates the forward summation path and the (<— ) indicates the reverse communi¬ cations path. For ease of representation in FIGURES 8 - 17 please note: • A = 10 • B = 11 • C = 12 • D = 13 • E = 14 • F = 15 • G = 16 In general, as the group partitioning algorithm is applied to larger structures with a given size H, more summation stages are added to the external sum¬ mation tree. Assuming an implementation where a processor group is placed on a single chip, ie. H synapse processing elements on a chip, then a scalable neurocomputer design can be built where only the external summa¬ tion tree needs to be modified as additional group chips are added. This can be seen in the 8 and 16 neuron model examples. The rearrangement of Y values, which allows the square array to be restruc¬ tured as two triangular arrays, affects the reverse communications path. The switching path mechanism is shown in more detail in FIGURE 17. FIGURE 17A presents the switches for the 8 neuron example for Y1 and Y5 while 17B presents the switches for the 16 neuron example for Y1 , Y5, Y9, and YD. The same switching mechanism is used for the other neuron outputs in each example. It should be noticed that a path switch in the reverse communi¬ cations path is implemented by the switching mechanism. For example in' FIGURE 17A Switch Position For Reverse Communications, Y1 is reverse communicated to the A1 and A5 summation path while Y5 is reverse commu- - Ii nicated to B1 and B5. Looking back to FIGURES 3 and 5, triangular struc¬ tures only, it will be seen that within the DSYPs and GSYPs the received Y values are serialized into the correct register, opposite from the respective summation tree's source point. The same switch in the reverse communi¬ cations path is followed in the 16 neuron example, FIGURE 17B. The external switching mechanism, FIGURE 17, requires further modification for the support of back-propagation learning. There are four different oper¬ ating modes to be supported by the external processing switching mech¬ anism, two due to the neuron execution, as presented in FIGURE 17A and 17B, and two due to back-propagation learning: 1. External summation of the neuron input summation function. 2. Reverse communication of neuron output Y values. 3. Reverse communication of error signals E. in learning. 4. External summation of the weighted error summations ER, in learning. Operating modes 1 and 4 relate to the forward summation paths, and oper¬ ating modes 2 and 3 relate to the reverse communications path. The external summation and reverse communication requirements for learning and neuron execution are different. Therefore, different values are externally summed and different values are reverse communicated depending upon whether the hardware is in a neuron execution mode or in a learning mode. In more detail, consider the reverse communication paths for neuron execution as compared to learning. In neuron execution mode, a reverse communicated Y. must be multiplied with its appropriate W,,, while in learning, a reverse com¬ municated error signal, £,, must be multiplied with its appropriate W tl . This seemingly small change in subscripts of the reverse communicated values, Y. and E„ indicates that different reverse communications paths are required for the Y, value as compared to the E, value. The different paths are due to the separated triangles of the folded square. The external switch, FIGURE 17 of the examples, must be modified to support both different path requirements. An additional change to the switching function of FIGURE 17 is made to accommodate the forward summation requirements. FIGURE 18 presents the modified external switching mechanism for the 8 neuron example where the labeled figures 1-4 correspond to the four switch operating modes described above. FIGURES 19 and 20 present the modified external switching mech¬ anism for the 16 neuron example. FIGURE 19 corresponds to the switch operating modes 1 and 2 while FIGURE 20 corresponds to switch operating modes 3 and 4. It will be noted that not all paths change depending upon the operating mode. Using FIGURE 18 as an example, the neuron input forward summation paths (1 ) for A1 and B5 are the same paths required for the weighted error summations (4) for A1 and B5. In addition, the reverse com¬ munication's paths for A1 and B5 are the same in both neuron execution and in learning. It will be noticed from FIGURE 3, that A1 and B5 are associated with the diagonal triangular arrays not requiring use of the folding algorithm. The paths which must change depending upon the operating mode are those that are associated with the separated triangular structures from the folded inner squares. It should be noted that the folded inner square prior to separating, FIGURE 4, could be used as a common replicatable building block. By zeroing out the weights and Y values for half of the folded array the folded array can model the functions of the edge triangular arrays using twice as many processors. With this arrangement of groups there is still a requirement for an external switching function. Instead of requiring the switching function due to the sep¬ arated folded inner square triangular arrays, the switching function is needed for the edge triangular arrays modeled on the folded square building block. Since the folded square building block approach requires twice as many processors in a processor group, for a given size diagonal triangular array, and there is still a requirement for an external switching function, the folded square building block approach will not be considered further in this description. Neither the 8 nor the 16 neuron example, FIGURES 2 through 17, requires virtual capability as V = N in both cases. The virtual neural processing capability requires, in addition to the storage of more than one weight and Y value per synapse processing element, a method of processing a portion of the neuron input sum of products function. An iterative adder with a Partial Sum (PS) storage element allows the total neural processing to be broken up into multiple cycles, adding a previous partial sum to the newest partial sum¬ mation generated. An example which adds virtual capability to the processor group concept is described next for the case where V * = 16, N = 4, H = 4, and G = 4, FIGURE 22, which will be used later in this description in the virtual learning example. FIGURE 22 includes the modified switching mech¬ anism, an iterative adder with Partial Sum (PS) storage, and the Initialization and Control Learning Processor which interfaces with the Host computer. The HOST computer assumes the responsibilities of initializing the processor groups through the learning processor. For example, the loading of the number of neurons in the network to be emulated, all the connection weights, (the initial neuron values, if required,) "teacher" values in support of a learning algorithm, the number of network update cycles to be run, as well as starting the model into execution are requirements which must be met prior to beginning an emulation cycle. In addition, the learning processor can contain a tracing facility for monitoring a network's execution. There are G = 4 groups where each group contains H = 4 processing ele¬ ments labeled in FIGURE 22 as G1 , G2, G3, and G4 where: • G1 generates A1 and A2 • G2 generates A3 and A4 • G3 generates B1 and B2 • G4 generates B3 and B4 Since V = 16, there are V 2 connection weights to be stored in memory. For this example, FIGURE 22, there are four groups of four processing elements each resulting in sixteen total processing elements in the completed system. Therefore, there are required sixteen connection weights in each processing element to support a completely connected network of sixteen neurons. Let a SO denote a Structure Output cycle and IA denote a partial sum Itera¬ tive Add cycle. In general, given V virtual neurons and N physical neurons, assuming, for simplicity, both are powers of 2, and a connection weight and neuron value storage associated with each synapse processing element, a virtual learning structure is formed: • {V/N) Y values to be stored at each synapse processor element (1 element per DSYP and 2 elements per GSYP). • (V 2 /N 2 ) connection weights to be stored at each synapse processor element. • (V/N) Iterative Add (IA) cycles of N 2 weight * Y value multiplications each per Structure Output (SO) cycle. • (V/N) SO cycles of N neuron values per SO cycle. For example, FIGURE 21 shows a V, N relationship to storage and operating cycles. In FIGURE 22 each connection weight memory of 16 weight values is organ¬ ized into four SO sets of four values each, labeled S01 , S02, S03, and S04, on the connection weight storage blocks. Each set requires four partial sum iterative add cycles to generate one set of neuron N = 4, values. A few other general comments are in order for FIGURE 22 regarding the switching mechanism and the iterative adder. The switching mechanism shown is a composite picture of all paths required to support the four switch operating modes of the structure. The iterative adder concept was presented in co-pending application Pechanek VIRTUAL 91 where an additional delay in the summation path was required due to the iterative adder. In the present architecture it will be noted that the iterative adder delay is not necessary as this function is incorporated in the last stage of the summation tree utilizing a 3 to 1 adder, FIGURE 22. Additionally, an external input, Ex, path, as pre- sented in co-pending application Pechanek LEARN 91 is provided prior to the neuron activation function input. EXTERNAL MEMORY EXPANSION The synapse processing architecture (SPA) provided by Pechanek LEARN 91 had the storage capacity of only a single connection weight per synapse processing element as appropriate for the direct execution of a neural network model. For larger network models, which contain V neurons where V > N, a general method of expanding the weight and neuron value storage is required without greatly impacting performance or requiring a new chip design containing expanded internal storage. For data parallel system proc¬ essing provided in Pechanek LEARN 91 the same scaling issues apply where matrix elements must be easily accessed from the synapse processor level. The memory can be thought of as being split among groups of synapse processors where each group has independent read/write and addressing capabilities. This concept is shown in FIGURE 23 illustrating a preferred external memory structure. Assuming an external connection weight memory configuration as shown in FIGURE 24, the memory is arranged in G groups where, given N physical neurons and H processor elements within a processor group, G = N 2 jH. For example, assuming N *** _ 256 and H ** __ 256 then G = 256 2 /256 = 256. If one assumes that a ch'p contains a single processor group, then a 256 neuron neurocomputer is made up of 256 group chips containing a total of 65,536 synapse processing elements. The external memory word length is made up of two pieces, one associated with the connection weight interface and the other associated with the neuron Y value interface. Both the weight and Y values are in bit-serial format. One weight memory address accesses H weight bits where there is one weight bit per processor elements in the processor group, (1 synapse processor element per Diagonal SYnapse Processor, DSYP, and 2 synapse processor elements per General SYnapse Processor, GSYP, FIGURE 25). For simplicity, assume an even power of two criteria for specifying H and N, then the _β represents the number of Y values utilized within the processor group. The external memory interface then has a composite word length of (H + ECC1 + * jW + ECC2) bits, the ECC1 bits correspond to error correction code bits on each group of H bits, and the ECC2 bits correspond to error correction code bits on each H bits. Both ECC1 and ECC2 bits are determined using an error detection and error correction coding scheme such as the Hamming single-error correction code illustrated in Hellerman 67 which requires ECC additional bits based on equations 4 and 5: 2 -.ccι ≥ H + £CC 1 + *■ ( 4 ) 2 £CC2 > > /H + ECC2 -M (5) For example, with H = 64, corresponding to 64 synapse processing ele¬ ments, then ECC1 = 7 additional bits must be used. The ECC interface will allow the fix up of data detected as in error. The weight array interface is bidirectional, allowing each synapse processor write access to its external weight value storage as is required to support learning. In addition to the weight memory, two other external memory arrays are utilized, one for the present neuron state value (Y) and one for the updated neuron state value (Y'). There are JW Y values output for each processor group. The present and new Y value arrays may be toggled back and forth every network update cycle. For example, the Present Y Value Array will output Y values for a whole network update cycle while updated Y values are being loaded into the New Y Value Array. For the next network update cycle, the arrays switch meaning where the newly updated Y values (New Y Value Array) are used as the present values in the next network update cycle and the previous Present Y Value Array is overwritten with new Y values. An array selector signal, which selects one array for output and the other array for loading Y values, allows the arrays to toggle back and forth. In another mode of operation, only the present Y value array is used and new Y values overwrite previously stored Y values. The synapse processor internal register length, which corre¬ sponds to the bit length of the parameters used, is D bits long which includes within the D bits the Conditional Execution Bit (CEB), FIGURE 29. D external memory addresses correspond to H separate values of D bits each, which can be loaded 1 value per processor element in a bit-serial fashion. The external weight memory holds Q * H values or equivalently Q * D memory locations of H + ECC1 bits per memory location. Memory values are accessed for groups of processors and memory values can be read from or written to via group processor controls. The memory connection to the synapse processor cells is depicted in FIGURE 25. A memory connects to a diagonal synapse processor cell, FIGURE 25A, with two external memory I/O line going to the distributor. The weight and Y value path from external memory is controlled by a stored PATH address. There is one address for the weight memory with bidirectional I/O lines, ie. weight values can be either read from or written to memory separately but not simultaneously. There are two addresses for the Y value arrays, one set of addresses for each array and each independently controlled. For the G-cell synapse processors, FIGURE 25B, a bidirectional weight bit and an output Y value bit for each synapse processor element are connected to the distributor allowing separate paths to the internal two processor elements in a GSYP. All memory accesses are loaded into the internal processor regis¬ ters. The updated Y values, Y', are sourced to external memory directly from their entry point to the processor group chip. The Y values coming from the external memory are fanned out on the processor group chip to the synapse processor elements. The source and destination addresses are relative to the command register. The relative addressing is shown in FIGURE 25A and 25B where the top command register CMD TREG relative addresses are shown in columnar fashion located to the right of the register blocks, while the relative addressing for the bottom command register CMD BREG is shown in columnar fashion located to the left of the register blocks. It should be noted, for example, that the bottom command register R2 is the same as the top command register R5. A bit string received from the Add tree, if it is a command, is serialized into one of the two CMD registers in each G-cell and the single CMD register of a diagonal cell. Alternatively, a data bit string received from the Add tree, is serialized to one of the six other registers available in a G-cell and one of the 3 other registers available in a diagonal cell. In the DSYPs a source or destination address of R4 through R6 and CR2 are mapped as follows: • CR2 → CR1 An example timing diagram for a 4 neuron SVLM with V =_ 16 neuron capacity for which there are 16/4 = 4 SO cycles of 4 IA cycles each, is shown in FIGURE 26. Each SO cycle is made up of V/N IA cycles. An IA cycle consists of the weight times Y value multiplication followed by the summation of the V/N multiplied values and storing the partial summation result in the iterative adder. To ensure that good performance, the weight and Y values for the next IA cycle need to have been accessed during the first lA's multiply time. For this to occur, a requirement is placed upon the EXU's multiplier (or any EXU function) such that all operands entering the EXU are not held fixed, external to the EXU, during a function's execution. For the multiplier, this implies that it must contain its own operand register, if required, and for good performance, a result bit is generated one cycle later from when its cor¬ responding operand bits enter the multiplier. To ensure no hazards occur the following relationship, equation 6, for the Execution Unit (EXU) multiplier function must be met. This is applicable to other EXU functions also, equation Dδ extmem ≤ δ M FOR MULTIPLICATION D U δ ϋ ext .mem < — ,N GENERAL (7) Where: • D bits per read/write access • δ extmem is the external memory access delay . δ M is the multiplier delay, typically 2 * (D-1) clock cycles long • δ Eχ u the delay of an EXU specified function All functions of the EXU can be made to execute in the same length of time by padding shorter executing functions. In this fashion the longest executing function determines the criteria for the external memory access timing. For example, assuming the multiplication function is the longest EXU executing function and the multiplication of two 32 bit operands takes 64 clock cycles to accomplish, the maxmimum external memory access delay of a 32 bit operand is 64 clock cycles. By forcing all EXU functions to be of the same delay no stricter requirements are placed on the external memory access delay. FIGURE 27 depicts a timing diagram for one SO cycle which generates one set of N neuron values. The numbered items are explained in detail: 1. Begin IA1 cycle with first bit serialized into the EXU multiplier. 2. The first bit of the multiplied result exits the multiplier and enters the summation tree. 3. The first bit of the multiplied result plus a partial sum of 0 exits the 3 to 1 adder and enters the partial sum register. 4. The last bit of the multiplied result exits the multiplier and enters the sum¬ mation tree. 5. The last bit of the multiplied result plus a partial sum of 0 exits the 3 to 1 adder and enters the partial sum register. 6. The last bit of the second IA cycle's partial sum is stored into the partial sum register. 7. The last bit of the third IA cycle's partial sum is stored into the partial sum register. 8. The last bit of the fourth IA cycle's partial sum enters the sigmoid gener¬ ator. 9. The first bit of the sigmoid result exits the sigmoid generator and enters the summation tree for the reverse communications back to either external memory or a register in the synapse processor. 10. The last bit of the sigmoid result exits the sigmoid generator and enters the summation tree for reverse communications. 1 1. The last bit of the sigmoid result is received in either external memory or in a register (first set of N Y values stored). The time from 1 to 4 represents the time available to load a new Y value or weight value from external memory, as illustrated by FIGURE 27, where the time scale is expanded to show detailed timing. The processor group size depends upon the chip technology cell and I/O specifications. There will be one memory subsystem associated with each synapse processor group chip as depicted in FIGURE 28. There must be two types of commands required to support the processor group concept, one dedicated for group only commands and the other asso¬ ciated with the synapse processors. The command format also must support the external memory as a new source and destination point. Addressing for external memory can be obtained by using the immediate data field of a synapse command. Further, the Conditional Execution Bit, CEB, must be associated with each weight and Y value stored in the external memory. For example with N = 256 neurons, H = 256, and D = 33 bits the command format L is defined as: B BROADCAST BIT 1 BIT TAG 8 BITS (1og 2 Λ/ 2 ///) TAG PARITY 1 BIT CMD BIT — 1 BIT GROUP BIT - 1 BIT AUTO BIT - 1 BIT CMD 4 BITS (16 COMMANDS) SOURCE 1 4 BITS (16 SOURCE LOCATIONS) SOURCE 2 4 BITS (16 SOURCE LOCATIONS) DESTINATION - 4 BITS (16 DESTINATION LOCATIONS) IMMED.DATA / EXT.MEM.ADR. 15 BITS (15 BIT DATA / 32K D-BIT LOCATIONS) ECC 6 BITS TOTAL - **• 50 BITS A data word example with N = 256 neurons, H = 256, and D - 33 bits (CEB + 32 bits Data) the data format L is defined as: B BROADCAST BIT **• 1 BIT TAG «_ 8 BITS (log 2 /V 2 / ) TAG PARITY ► 1 BIT CMD BIT ► 1 BIT CEB BIT >_ 1 BIT DATA ► 32 BITS ECC >- 6 BITS TOTAL -> 50 BITS Since the internal instructions and data are D bits long, FIGURE 29, and each must be received in their entirety, the external memory address is formed by appending ceiling ( log 2 D) bits to the memory address specified in the instruction field. The ceiling function takes the next largest integer of the function in parenthesis result. For example, 6 bits are appended to the address for a 33 bit data word stored in memory and 5 bits for a 32 bit word. The additional bits are generated automatically from a counter which is incre¬ mented each bit-slice access from the memory. After D bits have been accessed per instruction specified memory address, the counter returns to zero in preparation for the next access. In a D **** 32 bit example, a 15 bit external memory address field in the instruction translates to a 15 + 5 bit external memory address allowing a 20 bit field or Q * D = 2 20 or 1 MEG locations of H + ECC bits each or 2' 5 * H (32K * H) weight values. For an H = 256 processor group with D = 32 bits there are addressable up to 8 MEG connection weights per processor group chip. BROADCAST BIT The Broadcast bit indicates the following: • B = 0: Command/Data is for a specific group. • B = 1 : Command/Data is for ail groups. GROUP COMMANDS There is a single group command register within each processor group chip. An alternative exists for utilization of the group command register for acting as the command source for all synapse processing elements. By utilizing one command register per processor group, the individual processing element command registers could be eliminated by combining the synapse processor commands with the group commands. One group command reg¬ ister would save H command registers, representing a considerable savings in chip real estate. The savings in chip real estate, however, does not come free, as the group command decode must now be distributed to each of the processing elements to control the H-EXUs, H-distributors, and H-selectors. In addition to the wiring problem this may cause, there is also a timing penalty due to the potentially large fanout of signals from the centralized source. Consequently, the synapse processing element command registers are left intact and only group oriented commands, typically associated with the external memory control, are implemented. There is a single group tag which is loaded into all H tag registers at initialization time. There are two types of commands specified, both with the Command Bit set to a "1 " to dif¬ ferentiate the commands from the data (where the Command Bit is set to a "0"). A Group Bit (G) specifies whether the command is for the synapse processors (G = 0) or for the group of synapse processors (G = 1 ). Table 1 lists the presently defined group commands. The SO counter can be set to advance or decrement depending upon the requirements, advancing in neural execution mode and decrementing in back-propagation learning mode, for example. For advancing, the counter begins at 1 and advances to the specified SO COUNT and then wraps around to 1. For decrementing, the counter begins at the SO COUNT and decrements to 1 and then wraps around to the SO COUNT. For sparsely connected networks, the SO count and IA count for each group can be specified by a group command, GLDCT with S = 00 & l = 0, thereby not requiring the SO or IA counters to be clocked, as will be shown in the back-propagation learning example. When specified to count, the IA counter is always advanced at the completion of the instruc¬ tion execution. The SO counter is advanced or decremented at the com¬ pletion of all IA count cycles. For any neural network emulation, the group level commands must be issued to specify the number of IA cycles per SO cycle and the total number of SO cycles required for emulation of V neurons in the network model. In addition, the "S", "I", and "U" bits specify how the processor group accesses the external memory, providing control on the addressing via the "S" and "I" bits and control on how the external Y value arrays are utilized via the "U" bit. With the "S" bits set to a "00", the external memory address will not advance beyond the presently defined SO cycle, though the memory address will advance for each IA cycle, if "\" = 1 , allowing full execution of the IA cycles for the presently defined SO cycle. With "S" set to a "01" the SO count will advance after the next instruction is executed, which requires the specified number of IA cycles to be completed. Advancing the SO and IA counters allows execution to proceed through the V neurons where V > N. Holding the SO count fixed allows repeated execution of the N neuron values being exe¬ cuted, a facility used when multiple layers of a multilayer neural network are mapped onto a single SO cycle set of N neurons. With the "U" bit set to a "0", only one Y value array is utilized, for example the Present Y Value Array which causes newly updated Y values to overwrite the old Y values stored in the array. With the "U" bit set to a "1", both Y value arrays are used such that new Y values are only written to the opposite array which is being used for the present network emulation, and the arrays toggle their meaning after V neurons have been updated. A Y value can be read from external memory while a Y value is being written to the external memory. If "U" * = "0", this simultaneous operation must occur from the same external memory array, requiring an array two port operation. Since the group command is common to all synapse processors on a chip, only one input path is necessary to communicate to the group. To accom¬ plish this, one of the processor summation tree paths is designated as unique in that it also serves as the group communications path. Once specified, this path is uniquely identified and used for group communications from the initialization and control learning processor. TABLE 1 PROCESSOR GROUP INSTRUCTION SET - 32 SYNAPSE PROCESSOR COMMANDS With the Command Bit set to a "\" and the Group Bit set to a "0", the command received is specified for the synapse processors. Table 2 lists the synapse processor's instruction set which supports external memory. An external memory address may be specified in the immediate data field of an instruction or may be specified to use the present SO and IA count values for the address ie. for local address control of the external memory. The source or destination code specifies which addressing mode to use. *AUTO* * 1 -» AUTOMATIC REPEAT OF FUNCTION AFTER RECEIPT OF UPDATED DATA FROM SOURCE EXTERNAL TO SYNAPSE PROCESSOR *IHD1,1MD2» = IMMEDIATE DATA IN COMMAND REGISTERS 1 OR 2. AS SPECIFIED -EXT.HEM.l =- USE EXT.HEM. ADDRESS IN IHHED. DATA FIELD *EXT.MEH.2 - USE PRESEHT SO AND IA COUNT VALUES AS ADDRESS, i.e. LOCAL ADDRESSING TABLE 2 SYNAPSE PROCESSOR INSTRUCTION SET *AUTO« = 1 ■ » AUTOMATIC REPEAT OF FUNCTION AFTER RECEIPT OF UPDATED DATA FROM SOURCE EXTERNAL TO SYNAPSE PROCESSOR ** !MD1.IMD2'- = IMMEDIATE DATA IN COMHAND REGISTERS 1 OR 2, AS SPECIFIED •EXT. HEM.1 = USE EXT. HEM. ADDRESS IN IMMED. DATA FIELD *EXT.MEM.2 = USE PRESENT SO AND IA COUNT VALUES AS ADDRESS, i.e. LOCAL ADDRESSING TABLE 2/2SYNAPSE PROCESSOR INSTRUCTION SET O D O •AUTO* = 1 -» AUTOMATIC REPEAT OF FUNCTION AFTER RECEIPT OF UPDATED DATA FROM SOURCE EXTERNAL TO SYNAPSE PROCESSOR i •IMD1.IMD2* = IMMEDIATE DATA IH COMMANO REGISTERS 1 OR 2. AS SPECIFIED •EXT.MEM.1 = USE EXT.HEM. ADDRESS IN IMMED. DATA FIELD •EXT.MEM.2 = USE PRESENT SO AHD IA COUNT VALUES AS ADDRESS, i.e. LOCAL ADDRESSING O TABLE 2/3SYNAPSE PROCESSOR INSTRUCTION SET 3 34 - The definition of the Automatic bit depends upon the setting of the S bits which are loaded by a group command. The definition is as follows: • A = 0, S = 00: Single SO cycle update; with "I" = 1 all lAs at present SO cycle are executed and then wait for next instruction. The SO count is not advanced. • A = 0, S = 01 or 10: Execute present SO cycle and advance or decrement the SO count and then wait for a new command. With "I" = 1 all lAs at present SO cycle are executed. • A = 1 , S = 00: Execute present SO cycle then wait for new data. Upon receipt of valid tagged data execute the same SO cycle and wait .for new data. This procedure is repeated upon every receipt of valid tagged data, continuing until a new command is received which stops it. • A = 1 , S = 01 or 10: Execute present SO cycle and advance or decrement the SO count and then wait for new data. Upon receipt of valid tagged data, execute a SO cycle and advance or decrement the SO count and then wait for new data. This procedure is repeated upon every receipt of valid tagged data, continuing until a new command is received which stops it. The automatic bit essentially specifies when an execution cycle begins; For A = 0 the execution cycle begins only on receipt of an instruction and for A = 1 the execution cycles begin, after the first execution of the automatic instruction, on every receipt of valid tagged data. Before continuing with the synapse processor instruction set discussion, a brief description of the utilization of the "A", "S", "I", and "U" bits will be reviewed in their application to two "classic" types of neural networks. Beginning with the Hopfield network, the automatic bit will be set to a "1", the "S" bits set to a "01", the "I" and the "U" bits each set to a "1". Each set of N Y values will be written each SO cycle into the Y array opposite to the one being used for the present V neuron network emulation. This is accom¬ plished by overiapping the write operation with the fetch of a set of present Y values by utilizing separate address and control lines for each Y value array. - 35 - After the receipt of the last set of N values making up the V neuron emu¬ lation, the selection of the Y array will toggle such that the recently updated Y values will be treated as the present Y values for the next network update cycle. This process will continue until a new command is received which stops the operation, such as a NOP instruction. In a multilayer network emu¬ lation, the Automatic bit will probably be set to a "1", the "S" and "\" bits will be set dependent upon the network execution sequence, and the "U" bit will be set to a "0". If multiple layers are mapped onto one set of N neurons in SVLM, the "S" bits will be set to a "00" for the execution of a subset of those layers. For example with 3 layers mapped onto one set of N neurons, "S" will be kept set at a "00" for the execution of the first 2 layers. Then a Group Command would be issued, changing "S" to a "01" so that the network emu¬ lation can continue with advancing the SO count after the execution of the next layer (ie. the third layer of the three mapped onto one SO cycle set of N neurons). Since "U" is set to a "0", the new Y data overwrites the Y data in the present Y value array such that each layer's execution will utilize the pre¬ vious layers calculated Y value data, which is the normal operation in a feedforward multilayer neural network. In order to specify the "U", "I", and "A" bits, capitals "U", "I", and "A" will be appended to an instruction mne¬ monic to indicate whether the specified bits are set to a "1". If "U", "I", or "A" are not appended to an instruction, the bits are set to a "0". The "S" bits value is specified as either "1" or "2" as no specification indicates the"S" bits are "00" and "1 1" is not presently defined. The synapse processor instruction set contains an external memory address field (immediate data field) which, through the use of the PATH (EXT.MEM.) command, allows external memory to be specified as a data source. The utilization of the PATH command with external memory will be described first, followed by the description of the common method the other instructions used in writing to external memory. The PATH command sets up the data and cmd (command) path within a synapse processor, for information either reverse communicated up the - 36 - Adder tree or accessed from external memory, by loading the Add tree desti¬ nation path in one path register and the EXT. MEM. register destination path in another separate register, FIGURE 25. The PATH command's Source 1 field determines which register is to be used and can be viewed as a command field extension further defining the PATH command. The PATH command specifies only the destination point for data received from the Add tree or from EXT. MEM.. Because of this, a PATH command specifying data received from EXT.MEM. with a destination of EXT.MEM. would not accom¬ plish anything, as the same data would be received and sent to the same location without any processing, ie. a NOP action. The path once specified remains the same unless a new PATH command is received which changes the path. For example, in a Hopfield network model on SVLM, if the default paths had been changed due to previous processing, the PATH commands for the reverse communicated Y values and the virtual weights would be issued once allowing, in reverse communication mode within a GSYP synapse processing cell, the top Adder tree to connect its Y value to the bottom Y value register and the bottom Adder tree to connect its Y value to the top Y value register. For the Hopfield model, this path command remains in effect through out the network execution, assuming a fixed weight mode of execution. The PATH command also allows the specification of the path for the next commands. In a virtual mode example of the Hopfield network, the fetch of the weight values from the external memory will be done during the previously fetched weight multiplication time. In data flow computing, an instruction execution waits until all operands have been received, at which point the instruction is "fired". The data dependent controlling mechanism can be viewed as using tagged data such that when the valid data operands are received with matching tags the instruction is executed. Given the operands may not be received at the same time, this data dependent mechanism offers great performance potential and resolution of synchronization problems. The automatic mode is a data dependent oper¬ ation in that the next execution of the instruction will not occur until valid data with a matching tag are received. Reverse communicated Y values, for - 37 - example, sent back to a processor group will contain a common group tag. The receiving processor group will compare each Y value's tag in the synapse processing elements with the group tag and if they match the auto instruction will be executed. For the present architectural example, all -/# " tags to a processor group chip must have the same group tag or a group error will occur. In a similar manner, data from external memory can be tagged. The tagging control may not be done in the same manner as the reverse communicated data from an external source since each processor group chip interfaces to its own set of external memory arrays. For example, using a tag does not imply that a tag is associated with each weight and Y value in memory since the tag is the same for all stored values. Rather, a tag source may be set up in external memory which is multiplexed onto the data lines at the appropriate times. Alternatively, a different type of tag can be used on the external memory interface to minimize the interfacing impact. Since the memory and group chip are tightly coupled, a smaller tag field can be used than that required from the reverse communications tree path, as identification between different group chips is not required. Equations 6 and 7 must be revisited depending upon the implementation decision. Many instructions specify a destination which is local to the individual synapse processor. This local processing can cause synchronization prob¬ lems if not handled correctly. In the GPLM of co-pending Pechanek LEARN 91 , the synchronization problem was localized to the learning processor at the neuron output point. No notification of local processing completion was generated from the synapse processors. Rather a fixed hardware mechanism was provided at the neuron output point to guarantee safety of the oper¬ ations. Any local processing command to the same synapse processor was to be separated from the next command to that same synapse processor by the specified instruction's execution time. For example, a multiply could take 2 * (D-1) clocks to ensure the completion of a MPY command before the second command can be sent. For SVLM this time out method is not suffi¬ cient to resolve the synchronization problem. A new method is used where all local instructions notify completion of execution to the issuing point. The 38 - method uses the summation tree as the communications channel. Consider a local operation done at each synapse processing element where the result is stored back into a local synapse register. At the successful completion of the local operation, assume each synapse processing element entered a value of "1" to the summation tree. Then, after a delay of log 2 /V cycles, a summed value of N would result, indicating normal completion of all N local oper¬ ations associated with a particular summation tree. Any other value would indicate an error had occurred in the operation sequence. By allowing each processor group chip to append an error value different than the other group chips, isolation to a failing chip becomes feasible. In a similar manner, dif¬ ferent summed values could indicate different types of errors detected. This communications of local operation completion incurs a small time penalty as compared to the time out approach in that the next instruction to a processor group cannot be sent until notification of completion has been received. The time to communicate instruction completion is small, being a log 2 /V delay addition to the local execution time. Non-local instructions, ie. those instructions where the destination is the ADD TREE, provide notification of operation completion when the converged tree result reaches the learning processor. The learning processor must only distinguish between local and non-local instructions for instruction issuing synchronization control, and no variable time out control is required. As a final note, a SVLM compiler would be required to ensure no destination conflicts occur in programs using the described instruction set. BACK-PROPAGATION VIRTUAL LEARNING EXAMPLE An input/output encoder problem will be used to demonstrate how back- propagation is employed on the SVLM structure. Table 3 presents the input to output pattern encoding problem recognized by Rumelhart 86. - 39 INPUT OUTPUT PATTERNS PATTERNS 00 1000 01 0100 10 -* > 0010 11 0001 TABLE 3 INPUT/OUTPUT ENCODING PROBLEM The network structure used for this problem is shown in FIGURE 30, which illustrates an example neural network for the input/output encoding problem. Multi-layer networks are mapped onto a completely connected network struc¬ ture where unused connection paths are kept at a zero weight value by the weight registers' Conditional Execution Bit being set to zero. For the virtual machine, the size of the largest layer of neurons dictates the minimum size of N the number of physical neurons in the SVLM structure. There are 11 neurons in the I/O encoder problem but the largest layer has only 4 neurons which is the criteria for determining the number of physical neurons. A 4 neuron SVLM structure will be used in the input/output encoding problem. For example, the one already depicted in FIGURE 22 is made up of four groups, G = 4, each of size H = 4 capable of emulating a 16 neuron network. The input/output encoder problem does not require all possible connections available in the 11-neuron SVLM structure. For example, the input/output encoder network equation for neuron 9 is: Y 9 = F(W SA Y 4 + w 95 γ 5 + w 9 S γ β + ιv 9 7 y-) FIGURE 31 shows the general 11 neuron SVLM with each synapse processing external memory Weight and Y values shown for a completely connected 11 neuron structure. There are 121 weights in a completely connected 11 neuron network all of which are presented in FIGURE 31. Note that the 40 - neuron mapping is different than that presented in FIGURE 22 to allow the processing of the largest layer of N neurons to be accomplished in one update cycle of N neurons. Further note that layers ϊ and 2, neuron outputs Y1 , Y2, and Y3, are mapped to the S01 cycle of N neurons. During execution mode the Y values are calculated layer by layer, layer 1 first whose Y value results are used for layer 2, and continuing through the layers. After all K layers of neurons have been processed, the network is placed into a learning mode. The output neurons calculate the error signals from the output layer and then reverse communicate the error signal values through the tree to the synapse processing cells. The back-propagation of error signals is done layer by layer beginning at the output layer and contin¬ uing to the input layer. The error signals are used in the weight updating algorithm. By executing the proper instructions in the various synapse processors, the back-propagation algorithm can be executed. A detailed example with ordered procedures will explain how the input/output encoder learning problem is implemented with a back- propagation algorithm when mapped onto the SVLM. For some pattern p (in general the superscript p is removed for clarity) the following itemized procedure is followed in the SVLM neurocomputer: (The "no-connection" weights are blanked out for clarity. The positions of the reg¬ isters, as viewed relative to the command register, within each synapse proc¬ essing element are shown in FIGURE 31.) 1. SVLM is initialized by reading in the initial set of weight and Y values from external memory with the CEB bits set to a "0" for those connections not to be used in the model, Figure 32. Only 22 out of 121 possible con¬ nection weights for a completely connected 1 1 neuron model are utilized in the I/O encoder example. Three SO cycles of three IA cycles each are executed to obtain the 11 neuron outputs. The fourth SO and IA cycle are not needed in this example as there are no neuron 13, 14, 15, or 16 in the 41 I/O encoder problem. A Group command is issued to load the IA and SO counts and set the "S", "I", and "U" control bits. GLDCT 0,1 ,1 sets "S", "I", and "U" to "0"s and loads SO = 1 and IA = 1. 2. After initialization, SVLM is placed into execution mode and an input pattern applied, EX1 and EX2. The default path is assumed for the General Synapse Processors. Specifically, commands received from the top ADD tree go to CR2, the bottom command register, and commands received from the bottom ADD tree go to CR2, the top command register. Data received from the top ADD tree goes to R5 the Y, register and data received from the bottom ADD tree goes to R5 the Y, register. (Relative addressing is assumed.) 3. The four physical neurons issue MPYA R1 * R2 -→ ADD TREE (T). The auto¬ matic mode is used to improve performance by not having to reissue the multiply instruction for each layer of the network execution. Since layers 1 and 2 are mapped onto the S01 cycle of N neurons "S" * = "00" keeping the SO counter from advancing the SO count. In addition, with the "U" bit set to a "0" the newly updated Y values will be utilized in the next neuron updates. Since the Y value registers in the SVLM structure are initially zero, all weight times Y values will produce a zero product which when summed produce a zero summation value. 4. The first layer of neurons is executed on the SVLM structure, creating the first layer neuron outputs and their derivatives, Figure 33. (The learning processor ignores the other two neuron output values.) • Y, = F{Eχ.) • Y 2 = F(Ex 2 ) • Dr. = Y,(1 - Y,) • Dr 2 = Y 2 (1 - Y 2 ) 5. To emulate the 2nd layer of the network model and then continue with layers 3 and 4 the SO count must be advanced after the execution of the 2nd layer. To accomplish this a new Group instruction is issued changing the state of "S" to a "01", GLDCT 1 ,3,1. The "I" and "U" bits are not speci- - 42 - fied so they both remain set to a "0". After the GLDCT 1 ,3,1 instruction is sent, the first layer neuron outputs are fed back into SVLM, Figure 34. Since the automatic MPY instruction was specified, the second layer of neurons is executed upon receipt of Y1 and Y2, creating the second layer of neuron outputs and their derivatives, Figure 35. After the 2nd layer is executed the SO count is advanced to 2. • Reverse communicate V, and Y 2 . • Y 3 = F(W 3ι* Y, + W 2ι2 Y_) . Dr. = V 3 (1 - Y 3 ) 6. This continues until all layers have been executed and the network output and their derivatives are obtained, Figures 36, 37, 38, and 39. Upon receipt of Y_' the second SO cycle is initiated which generates the third layer outputs, namely Y„, Y 5 , Y 6 , Y 7 and then the SO count is advanced to 3. The third layer Y values are reverse communicated and the third SO cycle is initiated which generates the fourth layer outputs and advances the SO count to 1 as the network emulation is completed. • Third layer execution begins with the SO count equal to 2. • Reverse communicate Y 3) Figure 36. • Third layer execution, Figure 37. - Y* = F(W 4,3 Y 3 ) - Ys = F(W 5,3 Y 3 ) - Y t = F{W β,3 Y_) - Y ? = F(W 7,3 Y_) - Dr* = Y 4 (1 - Y 4 ) - Dr, = Y 5 (1 - Y 5 ) - Dr. = Y β (1 - Ye) - Dr 7 = Y 7 (1 - Y 7 ) - Advance SO count to 3. • Issue GLDCT 1 ,3,2 to load the IA COUNT with a 2 so that the proper weights can be accessed from external memory. The "S" bits = "01 " and the SO COUNT = 3 remain the same. • Fourth layer execution begins with the SO count equal to 3. - - • Reverse communicate Y 4 , Y 5 , Y 6 , and Y 7 , Figure 38. • Fourth layer execution, Figure 39. - y 8 = F(l 8ι4 / 4 + W t5 Y 5 + W 8 , 6 Y 6 + W B Y 7 ) - V 9 = F(W 9ι4 Y 4 + W 9 , 5 Y 5 + W 9 , 6 Y 6 + W/ 9 , 7 Y 7 ) - V10 = F(v 10 . 4 Y4 + W10.5 5 + W Wfi Y* + V 10ι7 Y 7 ) - y„ = F(W Y + w t5 γ s + Wu.tYi + M .7/7) - Dr* = y β (i - y 8 ) - Dr 10 = y, 0 (i - y, 0 ) - Advance SO count to 1. 7. After all 11 neuron values and their derivatives have been calculated and placed into a Y value and Y derivative array, SVLM is placed into the learning mode. For sparsely connected networks, as in this example, the IA and SO counts are used in a no clocking manner and changed explic¬ itly by a group command instruction whenever a new IA or SO count is desired. 8. The learning processor checks the Y,s for convergence to the minimum specified error. If reached, the learning process can be stopped for pattern p. If the convergence has not been reached, the learning for pattern p continues. • ConvergenceTestY B — |(f 8 — ≤ MinimυmError • ConvergenceTestY 9 = |(f 9 - Y 9 )| < MinimυmError • ConvergenceTestYio = |( 10 - V 10 )I < MinimumError • ConvergenceTestY - ≤ MinimumError 9. The learning processor calculates the δ? for the output neurons for use in updating the weights between the output neurons and the neurons in the previous layer. • δ s = £8 = ( - Y B )Dr_ - 44 - • δ_ = £9 = (f β - Y,)Dr 9 • <5ιo = £10 = (f 10 - Y 10 )Dr 10 • S„ = EU = (t - Y< ,)Dr_. 10. Load the SO and IA counts with no clocking mode specified. The data path is changed, and the error signals are then reverse communicated to their respective synapse processor elements' temporary registers, Figure 40. • GLDCT 0,3,2 loads SO = 3 and IA = 2 which points to where all the weights are located in external memory and ensures both the SO and IA counters are not clocked, ie. S -= 00 and l * = 0. • The four physical neurons change the data path by issuing PATH R4. The issuance of PATH R4 stops the automatic multiply mode specified by MPYA issued in step 3 above. The next data being received goes to the relative address R4 temporary register in the GSYPs and R3 in the DSYPs. The command path remains unchanged. • E8, E9, E10, and E11 are reverse communicated from the Ys, Ys, Yio, and Y,, source points. 1. Issue MPY instruction to start the multiply of the error signals times the appropriate weights, Figure 41. The products are summed, creating the error summations,£ / P -to be used in the next layer back weight modifica¬ tion process, Figure 42. Since the command path was not changed, ie. top ADD tree command to bottom command register (CR2) and bottom ADD tree command to top command register (CR2), the error signals are residing in R3 the temporary register address relative to the received commands. Where m = M, + M 2 + ... 4- M L • Neurons 8, 9, 10, and 1 1 issue MPY R6 * R3 -→ ADD TREE (T), Figure 41. • Create weighted error summations: ER _ ER 5 , ER 6 , ER 7 , Figure 42. - ER . = W 8 E B + W 9ι4 £ 9 + W.0. 4 E. 0 + W 11l4 £n 11 - ER 5 = W 8 5 £ 8 + W 9 5 £ 9 + W w .sE.o + W 11 . 5 E 11 - ER e = W 8 , 6 £ 8 + W 9 , e £ 9 + W 10,6 £ιo + W 11ιβ E„ - ER 7 = IV 8 , 7 £ 8 + W 9t7 E 9 + lV 10i7 E,o + W E 12. The weights between layers 3 and 4 are now modified by the learning ru l e. AWjj is created by pYβl in two instruction steps. a. STEP 1 - MPY R5 * R3 → R4, Figure 43. • TEMPREG A = Y 4 £8 • TEMPREG B = Y 7 £8 • TEMPREG S , 5 = Y 5 £8 • TEMPREG 9A = Y 4 f 9 • TEMPREG^ -- Y 6 £8 • Continuing until: • TEMPREG .7 = Y7EH b. STEP 2 - MPY R4-IMD → R4, Figure 44. • ΔW 8>4 = pTEMPREG_ A • ΔW 8,7 = pTEMPREG SJ • ΔW 8 , 5 = pTEMPREG 8ιS # ΔW 9,4 = pTEMPREG 9A • ΔW 8ι6 = pTEMPREG Qtβ • Continuing until: • ΔW llι7 = pTEMPREG ι7 13. One more step creates the new weights values between layers 3 and 4. New W/ j j= W j j= Old W-g+ ΔWij.The new weights are stored In external memory. ADD R6 + R4 → EXT.MEM.2 use the present SO and IA counts as the address to external memory, Figure 45. • w 8<6 W 8ι6 + ΔW 8,6 • vV' 8 , 7 W_ + ΔW 8ι7 • W' βι4 W + ΔW 9ι4 - 46 - • Continuing until: 14. By following this procedure, the weights can be updated for the other layers in the network. For layer 3, SO = 2 and I A == 1 , error signals E4, E5, E6, and E7 can be created from the general equation E; = DrjEf?; in the learning processor and then the error signals can be reverse communi¬ cated in the structure. The error signal ER 3 can then be created followed by the weight updating sequence for weights W 4 _ , W 53 , W β, , W 73 . This con¬ tinues until all weights are updated. 15. A new path command is sent in preparation for the next learning cycle, PATH R2 so that the new Y values are sent to the proper registers. 16. SVLM is placed in execution mode and the existing pattern p is applied again until the adjusted weights allow the output pattern to match the teaching pattern with a minimum specified error. Until the minimum spec¬ ified error is reached SVLM will alternate between execution mode and learning mode. After convergence to the minimum specific error is reached, a new pattern can be applied and the process repeated until all teaching patterns have been applied. SUMMARY A group partitioning algorithm has been presented which creates a common building block of synapse processors containing their own external memory. The processor groups are used to create a general purpose virtual learning machine which maintains complete connectivity with high performance. The synapse processor group allows a system to be scalable in virtual size and have direct execution capabilities. Internal to the processor group, the synapse processors are designed as a hybrid control flow/data flow architec¬ ture with external memory access and reduced synchronization problems. - 47 - While we have described our preferred embodiments of our inventions it will be understood that those skilled in the art, both now and in the future, upon the understanding of these discussions will make various improvements and enhancements thereto which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the inventions first disclosed.

Claims: - 48 - What is claimed is: 1. A computer system apparatus having a scalable group partitioned virtual neural synapse processor architecture apparatus comprising: an N neuron structure partitioned into groups of synapse processing units, said groups having multiple synapse processing units that contain instruction and data storage units, receive instructions and data, and execute instructions, and interface with external data storage for weights and neuron output values for V neurons where V > N and V is the number of neurons in the neural network to be emulated on the group partitioned N neuron structure, and said groups further containing group instruction execution means and communicating adder trees, and said N neuron structure further containing neuron activation function units, and means for interconnecting said groups into an N neuron structure supporting V neurons and for communicating both instructions, data, and the outputs of the neuron activation function units back to the input synapse processing units by means of the communicating adder trees. 2. The apparatus according to claim 1 wherein there is contained N 2 synapse processing units, each associated with a connection weight in a N neural network, placed in the form of a N by N matrix that has been folded along the diagonal and made up of diagonal cells and general cells. 3. The apparatus according to claim 2 wherein the diagonal cells, each com¬ prised of a single synapse processing unit, are associated with the diagonal connection weights of the folded N by N connection weight matrix and the general cells, each comprised of two synapse processing units merged together, are associated with the symmetric connection weights of the folded N by N connection weight matrix. - 49 4. The apparatus according to claim 3 wherein the diagonal cells' synapse processing units supply results to and receive instructions and data from an attached communicating adder tree of the N neuron structure. 5. The apparatus according to claim 3 wherein the general cells' two merged synapse processing unit structures are organized symbolically as a top synapse processing unit and a bottom synapse processing unit each of said top and bottom synapse processing unit supplying a result to and receiving instructions and data from an attached communicating adder tree of the N neuron structure. 6. The apparatus according to claim 3 wherein the folded N 2 matrix is sepa¬ rated into a single square matrix or multiple square matrices each containing 2H synapse processing units made up of all general cells and multiple trian¬ gular matrices each containing H synapse processing units in the form of diagonal cells and general cells, where H is determined by technology and design constraints for the purposes of packaging a group of H synapse proc¬ essing units on a chip and a given size of H and of N determines the number of triangular and square matrices separated out from the folded N 2 original matrix. 7. The apparatus according to claim 6 wherein each square matrix, containing 2H synapse processing units, is folded along its diagonal keeping the synapse processing units intact and folding only the communicating adder trees attached to said synapse processing units. 8. The apparatus according to claim 7 wherein by rearrangement of the weight and Y values in each synapse processing unit, the folded square structures can each be redrawn as two separate triangular arrays where each said triangular array contains H synapse processing units and where each triangular array represents a processor group. 50 9. The apparatus according to claim 2 wherein the N neuron structure con¬ taining N 2 synapse processing units is partitioned into G groups wherein each said group containing H synapse processing units is made up of diagonal cells and general cells and each said diagonal cell is attached to a communi¬ cating adder tree and each said general cell is attached to top and bottom communicating adder trees. 10. The apparatus according to claim 9 wherein the G groups are attached to external summation and switching mechanisms for the purposes of: • External summation of N partial neuron input summations of the N 2 W^Y j products as generated from the groups' communicating adder trees. • Reverse communication of N neuron activation function units' output Y, values where each said Y, value is communicated to N synapse proc¬ essing units' Y. = , storage unit wherein said synapse processing units contain the connection weight storage units W, t thereby associating, within each synapse processing unit, a j' h Y value with the connection weight with the matching /" subscript. • Reverse communication of N neuron error signal E, values where each £, value is communicated to N synapse processing units' storage unit wherein said synapse processing units contain the connection weight storage units W,, thereby associating, within each synapse processing unit, an /'" error signal value with the connection weight with the matching i' h subscript. • External summation of partial weighted error summations of W,,E, products as generated from the groups' communicating adder trees. 1 1. The apparatus according to claim 10 wherein the external summation and switching mechanisms are attached to iterative adders each containing an accumulating storage unit. 12. The apparatus according to claim 11 wherein the iterative adder is incor¬ porated into the final summation stage of the external summation tree. 51 13. The apparatus according to claim 11 wherein the N iterative adders each connect to an additional summation stage wherein an external input value is summed with the output of the iterative adder and said additional summation stage provides results to a neuron activation function unit. 14. The apparatus according to claim 3 wherein the diagonal cells' single synapse processing units are each comprised of a tag matching unit, a desti¬ nation path control mechanism for externally received instructions and data by means of an instruction/data decoding mechanism, an add tree data path storage unit, an external storage data path storage unit, and a distributor unit, instruction and data storage units comprised of one instruction storage unit, one weight value storage unit, one neuron activation function unit output value storage unit herein termed a Y value storage unit, and one temporary data storage unit, a storage unit operand selection mechanism controlled by means of an instruction decoding mechanism, a selector unit, and addressing means relative to the instruction storage unit, a result destination path control mechanism controlled by means of an instruction decoding mech¬ anism and a distributor unit, an external storage unit read only Y value data path, an external storage unit read and write weight value data path, and a programmable execution unit. 15. The apparatus according to claim 3 wherein the general cells' two synapse processing units merged together are comprised of two tag matching units, a common destination path control mechanism for externally received instructions and data by means of two instruction/data decoding mechanisms, two add tree data path storage units, two external storage data path storage units, two command path bits, and a common distributor unit, instruction and data storage units comprised of two instruction storage units, two weight value storage units, two Y value storage units, and two temporary data storage units, two storage unit operand selection mechanisms controlled by means of two instruction decoding mechanisms, a common selector unit, and two addressing means relative to the two instruction storage units, two result destination path control mechanisms controlled by means of two - 52 - instruction decoding mechanisms and a common distributor unit, two external storage unit read only Y value data paths, two external storage unit read and write weight value data paths, and two programmable execution units. 16. The apparatus according to claim 1 wherein the synapse processors' data storage units contain conditional execution bits with one bit per data storage unit which said bit controls the use of the data and whether the data may be overwritten. 17. The apparatus according to claim 9 wherein, with H an even power of two, each group interfaces with one external storage unit wherein said interface contains H read and write weight value data paths, _β read only Y value data paths, _jH write only Y value data paths, controlling means of group weight value addressing and separate addressing of the Y value write path and the Y value read path, and controlling means for the reading and writing of data at selected addresses. 18. The apparatus according to claim 17 wherein the external storage unit contains one weight array and two Y value arrays, one termed the Present Y Value Array and the other termed the New Y Value Array, 19. The apparatus according to claim 18 wherein, with V and N powers of two, the external storage unit contains V ]N 2 weight values and V/N Y values in each Y value array with all array weight and Y values containing a Condi¬ tional Execution Bit. 20. The apparatus according to claim 18 wherein the group partioned N neuron structure supporting V neurons can be scaled to a larger V by the addition of more external storage locations in each external storage unit that interfaces with a group of synapse processing units. 53 - 21. The apparatus according to claim 17 wherein under common weight value read and write controlling means H weight values are simultaneously read from or H weight values are simultaneously written to external storage. 22. The apparatus according to claim 17 wherein, assuming H is an even power of two, under separate Y value read and separate Y value write con¬ trolling means the simultaneous reading of JW Y values and the simul¬ taneous writing of JW Y values is accomplished separately or in parallel. 23. The apparatus according to claim 22 wherein controlling means are pro¬ vided for two modes of operations wherein said first mode of operation allows the storage of newly created Y values into the New Y Value Array while using Y values read from the Present Y Value Array in neural calcu¬ lations and wherein said second mode of operation allows the writing of newly created Y values in the Present Y Value Array from which Y values are read for neural calculations. 24. The apparatus according to claim 18 wherein the weight and Y values are stored in a bit-serial format in the external storage. 25. The apparatus according to claim 17 wherein a programmable group exe¬ cution unit responds to group commands received from a selected group attached communicating adder tree for the purposes of controlling the external storage interface. 26. The apparatus according to claim 1 wherein the neuron activation function unit is a non-linear sigmoid generator providing a Y value output and sigmoid derivative generator providing a Y(1-Y) value output. 27. The apparatus according to claim 1 wherein the communicating adder trees contain log 2 /V 2 to 1 communicating adder stages. 54 28. The apparatus according to claim 27 wherein each stage in the communi¬ cating adder trees contains 2 to 1 communicating adders comprised of a 2 to 1 adder, an adder bypass path for the purposes of communicating values in a reverse direction than that obtained from the addition, and means for switching between the addition function and the reverse communication's path. 29. The apparatus according to claim 1 wherein a programmable processor controlling apparatus which interfaces to an attached Host computer and to the N external summation and switching mechanisms, provides the following functions: • external summation and switching mechanism control • communicating adder tree control • iterative adder control • synapse processor initializations • synapse processor instruction issuing • algorithmic data calculations • synapse processor data issuing • starting a neural network emulation • sequence control for multi-layer networks • stopping a neural network emulation — by convergence testing to a desired result - by completing a host specified number of emulation cycles 30. The programmable processor controlling apparatus of claim 29 contains a weight array, a Y value array, a Y(1-Y) array, a teacher array, and a trace array. 31. The apparatus according to claim 1 wherein there are N 2 synapse proc¬ essing units, N communicating adder trees, and N neuron activation function units for a N neuron structure partitioned into G groups of H synapse proc¬ essing units per group and each group interfacing with external storage pro¬ viding storage capabilities for V neurons where V > N, with N external - 55 summation and switching mechanisms, and a programmable processor con¬ trolling apparatus. 32. The apparatus according to claim 31 where the group partitioned N neuron structure can be scaled to a larger N by the addition of groups to the existing structure and by modification to the external summation and switching mechanism and the programmable processor controlling apparatus to support the additional neurons. 33. The apparatus according to claim 31 wherein each communicating adder tree connects to N synapse processing units at the leaf nodes of the tree and one neuron activation function unit which connects to the root of the external summation tree's output of the additional external input summation stage providing results to a programmable processor controlling apparatus and where said communicating adder trees, external summation and switching mechanisms, iterative adders, groups of synapse processing units interfacing to external storage, and neuron activation function units constituting the N neuron structure have: • means for inputing weights and data values to each synapse processing unit, • means for communicating tagged instructions, data, and neuron output values herein termed Y values back to the input synapse processing units, • means for reading/writing data from/to external storage to/from the input synapse processing units, • means for controlling the destination of instructions, data, and Y values, received from the attached communicating adder trees, in each synapse processing unit, • means for controlling the destination of data, received from external storage, in each synapse processing unit, • means for controlling the external summation and switching mechansim depending upon operating mode, • means for the execution of the received synapse processor instructions in each synapse processing unit, - 56 - • means for the execution of the received group instructions in each group, • means for the execution of a previously received instruction when, in an auto mode, data is received to be used in the next operation, • means for operand selection and destination path control allowing results to stay locally in each synapse processing unit, to be sent to external storage, or to be sent to the attached communicating adder tree, • means for the communication of operation completion to the attached pro¬ grammable processor controlling apparatus for those operations whose results stay locally in each synapse processing unit, • means for the summation of values received from the synapse processing units, • means for the inputing of external input values to each neuron activation function unit, • means for the generation of new instructions, data, and neuron activation function unit Y values. 34. The apparatus according to claim 33 wherein the means for inputting weights and data values to each synapse processing unit comprises a host interface controlling mechanism in the form of a programmable processor controlling apparatus which has access to each weight and data value storage unit in each synapse processing unit and in external storage. 35. The apparatus according to claim 33 wherein the means for communi¬ cating tagged instructions, data, and neuron output values herein termed Y values back to the input synapse processing units is by means of the commu¬ nicating adder trees acting in a communications mode and tag matching units in each synapse processing unit wherein the received values each contain a tag header comprised of a broadcast bit and a tag field. 36. The apparatus according to claim 35 wherein the tag field is a group tag common for all synapse processing units in a group. 57 37. The apparatus according to claim 33 wherein the means for communi¬ cating data from external storage to the input synapse processing units is by a group external storage read and write controlling mechanism consisting of one weight external storage address, two Y value external storage addresses, and selective read and write controls whose addressing and controlling mechanism is set up by received group instructions and where a D bit weight value stored in external storage requires D accesses of H bits each whose addressing is accomplished by appending ceiling (log 2 D) bits to the memory address specified in the instruction immediate data field, where the ceiling function takes the next largest integer of log 2 D if fractional. 38. The apparatus according to claim 33 wherein the means for controlling the destination of instructions, data, and Y values, received from the attached communicating adder trees, in each synapse processing unit is, for instructions, by an instruction decoding mechanism, a command path bit, and distributor logic in the general cells and by an instruction decoding mech¬ anism, a storage unit mapping logic whereby R4 → R3, R5 → R2, 6 → /.1 , and CR2 → C 1 , where R1 , R2, ..., R6 represent the data storage units addressable from an instruction, and distributor logic in the diagonal cells and, for data including Y values, by a data decoding mechanism and a data path storage unit in both the diagonal cells and the general cells. 39. The apparatus according to claim 33 wherein the means for controlling the destination of data, received from external storage, in each synapse proc¬ essing unit is by an external storage data path storage unit/s in both the diagonal cells and general cells. 40. The apparatus according to claim 38 wherein in one mode of operation of the general cells, termed the YINMODE, the data path storage units and the command path bits are set up such that instructions received from the top communicating adder tree are directed to the top synapse processing unit's instruction storage unit and instructions received from the bottom communi¬ cating adder tree are directed to the bottom synapse processing unit's 58 instruction storage unit and data received from the top communicating adder tree are directed to the top synapse processing unit's specified data storage unit and data received from the bottom communicating adder tree are directed to the bottom synapse processing unit's specified data storage unit. 41. The apparatus according to claim 38 wherein in a second mode of opera¬ tion of the general cells, termed the YOUTMODE, the data path storage units and the command path bits are set up such that instructions received from the top communicating adder tree are directed to the bottom synapse proc¬ essing unit's instruction storage unit and instructions received from the bottom communicating adder tree are directed to the top synapse processing unit's instruction storage unit and data received from the top communicating adder tree are directed to the bottom synapse processing unit's specified data storage unit and data received from the bottom communicating adder tree are directed to the top synapse processing unit's specified data storage unit. 42. The apparatus according to claim 33 wherein the means for controlling the external summation and switching mechansim depending upon operating mode is by algorithmic control from the programmable processor controlling apparatus which algorithmic control varies depending upon the operating mode. 43. The apparatus according to claim 33 wherein the means for the execution of the received synapse processor instructions in each synapse processing unit is through a programmable execution unit responding to No operation (NOP), PATH (for set up of the add tree path), PATH (for set up of the external storage path), Arithmetic, Logical, Read external storage, and Write external storage instructions containing specifications of an auto operation mode, the source operands, the result destination, and specification of immediate data for the purposes of data or external storage addressing depending upon source and destination specifications. 59 44. The apparatus according to claim 43 wherein all instructions execute in the same amount of time by padding shorter executing instructions thereby allowing the longest executing instruction to determine the criteria for external storage access timing. 45. The apparatus according to claim 33 wherein the means for the execution of the received group instructions in each group is by separate group logic controlling the external storage address formation and responding to No Operation (NOP), GSET (for setting of S, I, and U parameters), and GLDCT (for setting of S, I, U, SO Count, and IA Count parameters) instructions con¬ taining specifications of; • S = 00 don't count • S = 01 Advance the count • S = 10 Decrement the count • S = 1 1 Not Used and specifications of I = 0 Don't count, I = 1 Count, U = 0 Use one Y value array, U = 1 use both Y value arrays and toggle use, the SO count, and the IA counts where SO stands for Structured Output cycle in which N neuron outputs are generated for each SO cycle and IA stands for Iterative Add cycle in which V/N iterative adds are performed for each neuron in a V neuron network. 46. The apparatus according to claim 33 wherein the means for the execution of a previously received instruction when, in an auto mode, data is received to be used in the next operation is by means of an auto mode flag as set by a received instruction having the capability of setting the auto mode, by the set¬ tings of the S bits as specified by a previously issued group instruction in the following manner; • A = 0, S = 00, Single SO cycle update; with "l" = 1 all lAs at present SO cycle are executed and then wait for next instruction. The SO count is not advanced. 60 - • A = 0, S = 01 or 10: Execute present SO cycle and advance or decrement the SO count and then wait for a new command. With "|" = 1 all lAs at present SO cycle are executed. • A = 1 , S = 00: Execute present SO cycle and then wait for new data. Upon receipt of valid tagged data execute the same SO cycle and wait for new data. This procedure is repeated upon every receipt of valid tagged data, continuing until a new command is received which stops it. • A = 1 , S = 01 or 10: Execute the present SO cycle and advance or decre¬ ment the SO count and then wait for new data. Upon receipt of valid tagged data, execute a SO cycle and advance or decrement the SO count then wait for new data. This procedure is repeated upon every receipt of valid tagged data, continuing until a new command is received which stops it. and by receipt of valid tagged data that is sent by a programmable processor controlling apparatus. 47. The apparatus according to claim 33 wherein the means for operand selection and destination path control allowing results to stay locally in each synapse processing unit, to be sent to external storage, or to be sent to the attached communicating adder tree is by means of an instruction decoding mechanism, group addressing controls and address formation for external storage destinations, and distributor logic. 48. The apparatus according to claim 33 wherein the means for the communi¬ cation of operation completion to the attached programmable processor con¬ trolling apparatus of operations whose results stay locally in each synapse processing unit is by use of the communicating adder trees in a summation mode whereby a a synapse processor successfully completing an operation injects a "1" to the tree and an unsucessful operation results in the synapse processor injecting a "0" to the tree thereby resulting in a summation value of N for N successfully completing synapse processor local operations and oth¬ erwise if any synapse processor operation was unsuccessful. - 61 - 49. The apparatus according to claim 33 wherein the means for the summa¬ tion of values received from the synapse processing units is through the attached communicating adder trees acting in a function execution mode and the external summation and switching mechanism in a summation mode. 50. The apparatus according to claim 33 wherein the means for the inputing of external input values to each neuron activation function unit is through an externally applied input, provided by the programmable processor controlling apparatus, applied to a final summation stage located at the output of the iterative adder. 51. The apparatus according to claim 33 wherein the means for the gener¬ ation of new instructions, data, and neuron activation function unit Y values is by means of neuron activation function units and a programmable processor controlling apparatus which interfaces to an attached Host computer and to the N external summation and switching mechanisms. 52. The programmable processor controlling apparatus of claim 51 wherein local and non-local synapse processor instructions issued by the program¬ mable processor controlling apparatus determine the use of the received values from the external summation path for operations and as an instruction and data issuing mechanism to avoid hazards on the structure. 53. The apparatus of claim 33 wherein the data are in a bit-serial format which for the data is, in the order the bits are received into a diagonal or general cell, first a broadcast bit, next a tag field, next an error handling bit/s, continuing with a command bit set to an inactive state to indicate data, a conditional execution bit, a data field, and ending in error handling bit/s. 54. The apparatus of claim 33 wherein the synapse processor instructions are in a bit-serial format which for instructions is, in the order the bits are received into a diagonal or general cell, first a broadcast bit, next a tag field, next an error handling bit/s, continuing with a command bit set to an active - 62 state to indicate an instruction, a group bit set to an inactive state to indicate a synapse processor instruction, an auto bit, a command field indicating the instruction type, a source-1 field indicating the first operand, a source-2 field indicating the second operand, a destination field indicating the destination of results, an immediate data field for immediate data or external memory address, and ending in error handling bit/s. 55. The apparatus of claim 33 wherein the group instructions are in a bit- serial format which for instructions is, in the order the bits are received into a group, first a broadcast bit, next a tag field, next an error handling bit/s, con¬ tinuing with a command bit set to an active state to indicate an instruction, a group bit set to an active state to indicate a group instruction, two structure output counter control bits, an iterative adder counter control bit, a Y array use bit, a command field indicating the instruction type, a SO count field, an IA count field, and ending in error handling bit/s. 56. The apparatus of claim 33 wherein there is means provided for sequen¬ tially performing to emulate back-propagation learning on a multi-layer neural network, comprised of a V equal to eleven neuron structure mapped onto a group partitioned N equal to four neuron structure with four groups, G = 4, and H = 4 synapse processing units per group wherein said multi-layer neural network is made up of four layers wherein the first layer contains two neurons with external inputs mapped to the S01 cycle, the second layer con¬ tains one neuron also mapped to the S01 cycle, the third layer contains four neurons mapped to the S02 cycle, and the fourth layer contains four neurons mapped to the S03 cycle, for an input/output encoder problem and where registers are used for storage units, MPY indicates a multiply instruction, a destination of ADD TREE sends results to the attached communicating adder tree, the programmable processor controlling apparatus is termed the learning processor, and SVLM (Scalable Virtual Learning Machine) is used to refer to the group partitioned N neuron structure supporting V neurons: 1. SVLM is initialized by reading in the initial set of weight and Y values from external memory with the CEB bits set to a "0" for those connections 63 - not to be used in the model. Only 22 out of 121 possible connection weights for a completely connected 11 neuron model are utilized in the I/O encoder problem. Three SO cycles of three IA cycles each are exe¬ cuted to obtain the 11 neuron outputs. The fourth SO and IA cycle are not needed in this example as there are no neuron 13, 14, 15, or 16 in the I/O encoder problem. A Group command is issued to load the IA and SO counts and set the "S", "I", and "U" control bits. GLDCT 0,1 ,1 sets "S", "I", and "U" to "0"s and loads SO = 1 and IA = 1. 2. After initialization SVLM is placed into execution mode and an input pattern applied, EX1 and EX2. The default path is assumed for the General Synapse Processors. Specifically, commands received from the top ADD tree go to CR2, the bottom command register, and commands received from the bottom ADD tree go to CR2, the top command register. Data received from the top ADD tree goes to R5 the Y, register and data received from the bottom ADD tree goes to R5 the Y- register. (Relative addressing is assumed.) 3. The four physical neurons issue MPYA R1 * R2 → ADD TREE (T). The auto¬ matic mode is used to improve performance by not having to reissue the multiply instruction for each layer of the network execution. Since layers 1 and 2 are mapped onto the S01 cycle of N neurons "S" = "00" keeping the SO counter from advancing the SO count. In addition with the "U" bit set to a "0" the newly updated Y values will be utilized in the next neuron updates. Since the Y value registers in the SVLM structure are initially zero, all weight times Y values will produce a zero product which when summed produce a zero summation value. 4. The first layer of neurons is executed on the SVLM structure creating the first layer neuron outputs and their derivatives. (The learning processor ignores the other two neuron output values.) • Y, = F{Ex ) • Y 2 = F(Ex 2 ) • Dn = Y,(1 - Y,) - 64 • Dr 2 = Y 2 (1 - Y 2 ) 5. To emulate the 2nd layer of the network model and then continue with layers 3 and 4 the SO count must be advanced after the execution of the 2nd layer. To accomplish this a new Group instruction is issued changing the state of "S" to a "01", GLDCT 1 ,3,1. The "I" and "U" bits are not speci¬ fied so they both remain set to a "0". After the GLDCT 1 ,3,1 instruction is sent, the first layer neuron outputs are fed back into SVLM. Since the automatic MPY instruction was specified, the second layer of neurons is executed upon receipt of Y1 and Y2, creating the second layer of neuron outputs and their derivatives. After the 2nd layer is executed, the SO count is advanced to 2. • Reverse communicate Y, and Y 2 . • Yz = F{W 3 Y, + W 3,2 Y 2 ) • Dr 3 = Y 3 (1 - Y 3 ) ' 6. This continues until all layers have been executed and the network output and their derivatives are obtained. Upon receipt of Y 3 ' the second SO cycle is initiated which generates the third layer outputs, namely YA, Y5, Ye, Y? and then the SO count is advanced to 3. The third layer Y values are reverse communicated and the third SO cycle is initiated which generates the fourth layer outputs and advances the SO count to 1 as the network emulation is completed. • Third layer execution begins with the SO count equal to 2. • Reverse communicate Y 3 . • Third layer execution. - < = F(W*, 3 Y 3 ) - Ys = F(W 5 . 3 Y 3 ) - γ 6 = F{W S _ 3 Y 3 ) - Dr, = Y<(1 - Y 4 ) - Dr 5 = Y 5 (1 - Y s ) - Dr 6 = Y 6 (1 - Y 6 ) 65 - Dr 7 = Y 7 (1 - Y 7 ) - Advance SO count to 3. • Issue GLDCT 1 ,3,2 to load the IA COUNT with a 2 so that the proper weights can be accessed from external memory. The "S" bits = "01" and the SO COUNT = 3 remain the same. • Fourth layer execution begins with the SO count equal to 3. • Reverse communicate Y 4 , Y 5 , e, and Y 7 . • Fourth layer execution. - y 8 = F(W BA Y* + W _ 5 Y 5 + W SiS Y. + W S Y 7 ) - y 9 = F{W..<Y> + W-. 5 Y5 + W . _Y G + W 9 Y 7 ) - Y, 0 = F{W_ Λ Y, + W__Y. + W_ Λ Y t + M 0.7Y7) - Y_ = F(W A Y* + Wu.sYs + W .β e + W,., 7 Y 7 ) - Dr . = Y 8 (1 - Y 8 ) - Dr. = Y 9 (1 - Y 9 ) - Dr_ = Y l0 (1 - Y, 0 ) - Dr . = Y„(1 - Y„) - Advance SO count to 1. 7. After all 11 neuron values and their derivatives have been calculated and placed into a Y value and Y derivative array, SVLM is placed into the learning mode. For sparsely connected networks, as in this example, the IA and SO counts are used in a no clocking manner and changed explic¬ itly by a group command instruction whenever a new IA or SO count is desired. 8. The learning processor checks the Y,s for convergence to the minimum specified error. If reached, the learning process can be stopped for pattern p. If the convergence has not been reached, the learning for pattern p continues. • ConvergenceTestYe = \(t a — Y 8 )| < MinimumError • ConvergenceTestY. = \(t 9 — Y 9 )| < MinimumError • ConvergenceTestY^ = |(-ιo — Yto)| < MinimumError • ConvergenceTestY^- — \{t — Y,,)| < MinimumError - 66 9. The learning processor calculates the δ . for the output neurons for use in updating the weights between the output neurons and the neurons in the previous layer. . δ. = £8 = (t 8 - Y 8 )Dr s . δ 9 = £9 = (t 9 - Y 9 )Dr 9 . δ _ = E .0 = (t_ - Y to )Dr,o • δ _ = £11 =- (t - Y )Dr 10. Load the SO and IA counts with no clocking mode specified. The data path is changed, and the error signals are then reverse communicated to their respective synapse processor elements' temporary registers. • GLDCT 0,3,2 loads SO = 3 and IA = 2 which points to where all the weights are located in external memory and ensures both the SO and IA counters are not clocked, ie. S = 00 and l = 0. • The four physical neurons change the data path by issuing PATH R4. The issuance of PATH R4 stops the automatic multiply mode specified by MPYA issued in step 3 above. The next data being received goes to the relative address R4 temporary register in the GSYPs and R3 in the DSYPs. The command path remains unchanged. • E8, E9, E10, and E1 1 are reverse communicated from the YB, s, Yo, and Y-* source points. 1. Issue MPY instruction to start the multiply of the error signals times the appropriate weights. The products are summed, creating the error sum¬ mations, ERf, to be used in the next layer back weight modification process. Since the command path was not changed, ie. top ADD tree command to bottom command register (CR2) and bottom ADD tree command to top command register (CR2), the error signals are residing in R3 the temporary register address relative to the received commands. - 67 - Where m - M, + M 2 + ... + M L • Neurons 8, 9, 10, and 11 issue MPY R6 * R3 → ADD TREE (T). • Create weighted error summations: ER*, ER 5 , ER _ ER 7 . - £R 4 = W_ ,E 8 + W 9 E 9 + Wn.tEio + W E - ER 5 = W e . 5 E β + W_. 5 E 9 + W.o.sE o + W ,5 E-, - ER. = W s _Es + W 9 .6E9 + l^ 10 , 6 £,o + W β E - ER 7 = W 8 . 7 £ 8 + W 9 . 7 E 9 + lV 10 .7£,o + W„, 7 E 12. The weights between layers 3 and 4 are now modified by the learning rule. AW,, is created by pYβi in two instruction steps. a. STEP 1 - MPY R5 * R3 → R4. • TEMPREG 8 = Y 4 £8 • TEMPREGe.s = Y 5 £8 • TEMPREG M = Y 6 £8 • TEMPREGs = Y 7 £8 • TEMPREG. A = Y 4 £9 • Continuing until: • TEMPREG .7 = 7 £11 b. STEP 2 - MPY R4 * IMD → R4. • AW 8A = pTEMPREG 8A • AW B , 5 = pTEMPREG 8.5 • AW_. e = pTEMPREG 8 . s • AW 8 = pTEMPREG 87 • AW 9 -= pTEMPREG 9A • Continuing until: • AW,., 7 = pTEMPREG . 7 13. One more step creates the new weights values between layers 3 and 4. New W,, = l/l/',- = Old H/. -f ΔH V . The new weights are stored in external - 68 - memory. ADD R6 + R4 → EXT.MEM.2 use the present SO and IA counts as the address to external memory. W' 8 i = W 5 + AW 8A W' s . 5 = W 8ι5 + ΔW-.s W' 8 . e = W 8,6 + AW 8 . 6 W' 8 . 7 = W 8 + AW BJ W' 9 = W 9A + AW 9 Continuing until: 14. By following this procedure the weights can be updated for the other layers in the network. For layer 3, SO = 2 and IA = 1 , error signals E4, E5, E6, and E7 can be created from the general equation £, = Dr,ERι in the learning processor and then the error signals can be reverse communi¬ cated in the structure. The error signal ER 3 can then be created, followed by the weight updating sequence for weights W .3 , W_ 3 , W e , , W 7ι3 . This con¬ tinues until all weights are updated. 15. A new path command is sent in preparation for the next learning cycle, PATH R2, so that the new Y values are sent to the proper registers. 16. SVLM is placed in execution mode and the existing pattern p is applied again until the adjusted weights allow the output pattern to match the teaching pattern with a minimum specified error. Until the minimum spec¬ ified error is reached, SVLM will alternate between execution mode and learning mode. After convergence to the minimum specific error is reached, a new pattern can be applied and the process repeated until all teaching patterns have been applied.

Inventors: DELGADO-FRIAS JOSE GUADALUPE, PECHANEK GERALD GEORGE, VASSILIADIS STAMATIS, DELGADO-FRIAS, JOSE, GUADALUPE, PECHANEK, GERALD, GEORGE, VASSILIADIS, STAMATIS
Applicants: IBM, INTERNATIONAL BUSINESS MACHINES CORPORATION
Application year: 1991
Main classification: G06F15
IPC codes: G06F15, G06G7, G06N1, G06N3