A method and apparatus for arbitration and serialization in a multiprocessor system

Abstract: A method and apparatus for providing arbitration between and serialization of plural processors in a multiprocessor system comprising, in each processor, a delay network, a priority circuit, a REQUEST generator, an ORDER generator, a serialization program, an ACK generator and an ACK receiver. In operation, the delay network insures that simultaneously generated REQUESTS received from plural processors are received by the priority circuit at the same time. A processor awarded priority issues an ORDER to the other processors and thereafter drops its REQUEST to allow an award of priority to another processor. An ACK is received by the ORDER issuing processor from each processor when it executes the ORDER. The ORDER issuing processor then completes the task which gave rise to the ORDER. To conserve processing time, priority awards may be made before previously issued ORDERS are completed. Alternatively, REQUEST issuing processors can simply hold their REQUEST and thereby prevent interruption of instructions or groups of instructions.

Description: The present invention relates to multiprocessor systems in general and to a method and apparatus for arbitration between and serialization of plural processors in a multiprocessor system in particular. Description of Prior Art A multiprocessor system comprises a plurality of processors which are coupled to each other directly or indirectly through one or more other processors. The coupling of processors in such a system comprises the coupling of data and/or control signal buses of one processor and corresponding data and/or control signal buses of one or more other processors in the system. For example, in a multiprocessor system comprising four processors designated first, second, third and fourth, respectively, the processors may be coupled in an arrangement such that the first processor is coupled directly to the second processor and indirectly through the second processor to the third and fourth processors; the second processor is coupled directly to the first and third processors and indirectly through the third processor to the fourth processor; the third processor is coupled directly to the second and fourth processor and indirectly through the second processor to the first processor; and the fourth processor is coupled directly to the third processor and indirectly through the third processor to the second and first processor. In operation, a central processing unit (CPU) in the system in the course of executing instructions may encounter instructions which require it to request special operations of the other CPU's in the system. For example, some instructions have a requirement that when executed on a CPU the results produced by instructions previously executed on the CPU must be available and visible to other CPUs and channels in the system before the results of the current instruction. Specific results of prior instructions which must be available to other CPU's and channels before the results of a current instruction include storage accesses and translation lookaside buffer purge orders. Such an instruction, executed on the requesting CPU, must initiate operations on all other CPU's before the unit of operation on the requesting CPU is completed. A unit of operation is defined as a predetermined amount of work that cannot be interrupted. Furthermore, the operations initiated on each of the other CPU's in the system must be initiated between units of operation on those CPU's. The operations initiated on each of the nonrequesting CPU's, which place all nonrequesting CPU's in a predetermined state, are collectively known as serialization. In general, the serialization of a processor in a multiprocessor system comprises forcing the processor to complete certain operations and forcing the processor to a state where it will not initiate new operations. For example, operations which are completed during serialization of a processor are specific to each processor, but typically include completing the current unit of operation, completing stores to system memory and local processor caches, and completing any pending updates to local processor translation lookaside buffers. At times, more than one processor in a multiprocessor system may request serialization of another processor in the system at the same time. When this occurs, the requesting processors must be awarded priority according to a predetermined scheme. The awarding of priority is called arbitration. EP-A-0032864 describes a system having a plurality of data processing units where each such unit has a priority awarding circuit. Requests for access to a common transmission bus are interchanged between all units in the system and each unit has the same priority circuit such that priority is awarded in the same manner in each unit. EP-A-0184657 describes how transmission time differences in transmitting a synchronising signal to different parts of a system may be approximately compensated for. The output of the signal is variously delayed according to which part of the system is the intended recipient. Principal objects of the present invention are a novel method and apparatus for arbitration between and serialization of plural central processing units (CPU's) in a multi-processor system. The present invention provides a control apparatus and method thereof in each distributed central processing unit (CPU) within a distributed multiprocessor system comprising a plurality of distributed CPUs which process data in response to instructions, said control apparatus comprising: request means responsive to a predetermined instruction for generating a REQUEST signal to request serialization of all said distributed CPUs; transmitting means connected to said request means for transmitting said REQUEST signal to all other said distributed CPUs; and priority means connected to receive REQUEST signals generated by all said distributed CPUs for awarding priority to one of said REQUEST signals when a plurality of REQUEST signals are simultaneously received by said priority means; characterized by further comprising receiving means having a plurality of inputs where each input receives REQUEST signals from a respective one of said distributed CPUs, a plurality of delays where each delay is connected to one of said inputs for compensating for the transmission time associated with transmitting REQUEST signals from the respective transmitting distributed CPU, and a plurality of outputs where each output is connected to a respective one of said delays and to said priority means; whereby REQUEST signals appearing at said outputs of said receiving means are in the same time relationship as that in which said REQUEST signals were generated by said distributed CPUs. The magnitude of the delay provided by each delay provided by each delay circuit is chosen to be inversely proportional to the time it takes to transmit a REQUEST from a CPU in the system to the input of the priority circuit to which the delay circuit is coupled. As described above the delay means comprises a selector circuit for selecting the amount of delay to be placed in series with an input of a priority circuit. For example, if for purposes of repair or replacement, CPU's are moved within the system or a CPU is replaced in the system, a simple setting of the selector circuit which depends on the location of the delay network in the system is all that is required for providing the required amount of delay in the delay network. Also, if the system is reconfigured from a multiprocessor system to a dual processor or uniprocessor, advantage can be taken of the decrease in the amount of delay needed. In operation, if a CPU requires serialization of another CPU in the system, it generates a REQUEST for permission to serialize the other CPU's and sends it to all of the other CPU's in the system. If a plurality of CPU's generate simultaneous REQUEST's, the priority circuit in each of the CPU's awards priority to one of the CPU's. Since the priority circuits in all of the CPU's are identical, each of them makes the same decision and therefore, there is no need to transmit the decision to the other CPU's. The CPU awarded priority then performs a local serialization operation, generates an ORDER ordering the other CPU's to serialize themselves and drops its REQUEST. The dropping of the REQUEST allows one of the other requesting CPU's to be awarded priority while it and the other CPU's are being serialized. While an ORDER is pending, however, no other CPU will issue an ORDER. As each nonrequesting CPU completes its serialization, it generates an ACK control signal indicating that it has completed its serialization. When the requesting CPU issuing an ORDER has received the ACK signals indicating that all of the other CPU's have completed their serialization and the requesting CPU has completed its serialization, a control signal is sent to the instruction processor, the requesting CPU completes the task which gave rise to its REQUEST and drops its ORDER. Thereafter, the next CPU then having priority performs a local serialization operation, issues its ORDER and the above-described process is repeated. At times a CPU executes an instruction or group of instructions, the execution of which cannot be interrupted. To prevent another CPU in the system from making a REQUEST and being awarded priority, the former CPU makes and holds a REQUEST until it completes its execution of the instruction or the group of instructions. In this manner, the method and apparatus of the present invention is able to prevent disruptive interruptions of specific instructions without resorting to more expensive hardware or software implementations heretofore required. Brief Description of the Drawing The above and other objects, features and advantages of the present invention will become apparent from the following detailed description of the accompanying drawing in which: Fig. 1 is a block diagram of a multiprocessor system comprising four central processing units according to the present invention; Fig. 2 is a more detailed block diagram of one of the central processing units of Fig. 1; Fig. 3 is a block diagram of one embodiment of fixed delay networks and priority circuits in the central processing units of Fig. 1; and Figs. 4-7 are block diagrams of an embodiment of programmable delay networks and priority circuits in the central processing units of Fig. 1. Detailed Description of the Drawing Referring to Fig. 1, there is provided in accordance with the present invention a multiprocessor system designated generally as 1. In the system 1 there is provided a plurality of central processing units (CPU's) designated generally as CPU 1, CPU 2, CPU 3 and CPU 4. All of the CPU's are interconnected by a plurality of REQUEST signal lines 20-23, ORDER signal lines 27-30, and acknowledge (ACK) signal lines 35-38. A plurality of broken lines are used to show that signals on each of the lines pass through CPU's 2 and 3 to adjacent CPU's after being latched and synchronized with system clocks located therein. In each of the CPU's 1-4, as seen more clearly in Fig. 2 which is a block diagram of CPU 1 and typical of the other CPU's, there is provided an instruction processor 6, a priority circuit 11, a request register circuit 12, an order generator 13, a serialization circuit 14, a local serialization circuit 15, an ACK receiver 16 and an arbitration/serialization (ARB/SER) complete circuit 17. In addition, there is provided in the CPU's 1-4, a plurality of delay networks 10A, 10B, 10C and 10D, respectively. As will be further described below, each of the delay networks 10A-10D comprises four inputs 1-4 which are coupled to control signal lines 20, 21, 22 and 23, respectively, and four outputs coupled to corresponding inputs of the priority circuit 11. The inputs 1-4 are designated 1-4 to conveniently identify the CPU from which they are derived. For example, input 1 is derived from CPU 1, input 2 is derived from CPU 2, and so on. This identification scheme is also used with respect to the serialization circuit 14 and ACK receiver 16. In the priority circuit 11 there is provided an output coupled to an output control signal line 24. In the request register circuit 12 there is provided an input coupled to an output of the instruction processor 6 by means of a REQUEST control signal line 45, an output coupled to an input of the local serialization circuit 15 by means of a REQUEST LOCAL SERIALIZATION control signal line 46, an output coupled to an input of the order generator 13 by means of an ARBITRATION ONLY control signal line 47, an input coupled to an output of the order generator 13 by means of a DROP REQUEST control signal line 48, an output coupled to an input of the ARB/SER complete circuit 17 by means of an input SELECT control signal line 49 and an output coupled to one of the control signal lines 20-23. In the order generator 13 there is also provided an input coupled to an output of the instruction processor 6 by means of a DROP ORDER control signal input line 50 and an output coupled to one of the ORDER control signal lines 27-30. In the serialization circuit 14 there is provided three inputs coupled to three of the ORDER control signal lines 27-30, a fourth input coupled to an output of the instruction processor 6 by means of a control signal line 51 and an output coupled to one of the SERIALIZATION COMPLETE (ACK) control signal lines 35-38. In the LOCAL SERIALIZATION circuit 15 there is also provided an input coupled to an output of the instruction processor 6 by means of a control signal line 52 and an output coupled to an input of the ARB/SER COMPLETE circuit 17 by means of a LOCAL SERIALIZATION COMPLETE control signal line 52. In the ACK receiver 16 there is provided a plurality of three inputs coupled to three of the control signal lines 35-38 and an OTHER CPU's SERIALIZED control signal output coupled to an input of the circuit 17 by means of a control signal line 53. In the ARB/SER circuit 17 there is also provided an output coupled to an input of the instruction processor 6 by means of control signal line 54. In operation, when an instruction in the instruction processor 6 in one of the CPU's 1-4 requires another one of the CPU's 1-4 in the system to perform a specific operation, it requests permission to serialize the other CPU's, i.e. requests priority, by generating in its instruction processor 6 a REQUEST PERMISSION TO SERIALIZE OTHER CPU's control signal (REQUEST). The REQUEST is then transmitted by the CPU to the REQUEST register 12 on the line 45, and via the register 12, to its own priority circuit 11 and to the priority circuit 11 in each of the other CPU's via one of the control signal lines 20-23 and the delay networks 10A-10D associated therewith. For example, CPU 1 uses line 20, CPU 2 uses line 21, CPU 3 uses line 22 and CPU 4 uses line 23. Referring to Fig. 3, the time it takes for each REQUEST to be received by a CPU depends on the relative location of the sending and receiving CPU's in the system and is characterized in terms of units of delay. If the sending and receiving CPU's are one and the same, there is obviously no delay. If the sending and receiving CPU's are coupled directly, there is 1 unit of delay. If the sending and receiving CPU's are separated by 1 CPU, there are 2units of delay. If the sending and receiving CPU's are separated by 2 CPU's, there are 3 units of delay, etc. For example, in the 4 CPU system described above with respect to Fig. 1, the units of delay are as follows: The priority circuits in all of the CPU's are identical and each of them awards priority to itself or to another CPU in the system independently of the priority circuits in the other CPU's. Consequently, when two or. more CPU's simultaneously request priority, i.e. request permission to serialize the other CPU's in the system, it is necessary that all of the priority circuits receive the REQUESTS from the other CPU's simultaneously. To make sure that all of the priority circuits in all of the CPU's receive the REQUESTS from the other CPU's simultaneously, delay circuits D are provided for delaying the REQUESTS received by the priority circuits from certain ones of the CPU's. The number of units of delay which is provided depends on the relative location of the sending and receiving CPU's in the system. Thus, if the sending and receiving CPU is one and the same, the number of units of delay provided is the maximum number of units of delay provided at any input of a priority circuit in a CPU in the system and is equal to the signal delay between a receiving CPU and the remotest sending CPU in the system. If the sending and receiving CPU's are coupled directly, the delay provided is 1 unit less than the maximum. If the sending and receiving CPU's are separated by one CPU, the delay provided is 2 units less than the maximum. If the sending and receiving CPU's are separated by 2 CPU's, the delay provided is 3 units less than the maximum, etc. For example, in the 4 CPU system described above with respect to Figs. 1 and 3, the delays provided are as follows: Each of the above-described delay circuits D comprises one unit of delay. Referring to Figs. 4-7, in a preferred embodiment of the present invention, there is provided a plurality of programmable delay networks. Each of the networks can be easily programmed for placing the necessary amount of delay in series with each of the inputs of the priority circuit to which it is connected. Once the required maximum number of units of delay for any input of a priority circuit in the system is determined, each of the inputs to each of the priority circuits is provided with the maximum number of delay circuits D and a multi-input selector circuit S which is responsive to one or more latches L for coupling the required number of said delay circuits D in series with each such input. Thus, if an input to a priority circuit requires 3 units of delay, the selector circuit S in response to a control signal from the latches L will couple 3 of said delay circuits D in series with said input. If an input to another priority circuit requires 2 units of delay, the selector circuit S in response to another control signal from the latches will couple 2 of said delay circuits D in series with said input, etc. For example, in the 4 CPU system described above with respect to Fig. 1, the inputs of the selector circuits for each receiver and sender combination are as follows: With identical priority, delay and selector circuits in each of the CPU's in the system, a CPU can be replaced by another CPU or exchanged with another CPU in the system and, from the foregoing description, it can be seen that all that is required to be done to accommodate the change is that the latches L controlling the selector circuits S therein be set to identify the location of the CPU in the system. The setting of the latches L to identify the location of the CPU in the system may be done by means of a conventional keyboard coupled to the CPU, switches on the CPU or by means of appropriate electrical signals coupled through a connector to the CPU when the CPU is inserted in the system. Continuing with the description of the operation of the system 1, upon the simultaneous receipt of a REQUEST from two or more of the CPU's in the system 1, the priority circuit 11 in all of the CPU's makes a decision awarding priority to one of the CPU's. Since all of the priority circuits 11 in the system are identical, the same decision is made by all of the priority circuits. Since the same decision is made by all of the priority circuits, the control signal awarding priority to a particular CPU does not have to be transmitted to the other CPU's, but is simultaneously available to all the CPU's in the system on the line 24. After a CPU is awarded priority, and, if the instruction or group of instructions being executed by the requesting CPU awarded priority requires that the other CPU's be serialized, the CPU which is awarded priority generates and transmits a control signal, i.e. issues an order called SERIALIZE YOURSELF (ORDER), to all the other CPU's in the system on one of the lines 27-30 and thereafter drops its REQUEST. The REQUEST is dropped by means of a DROP REQUEST control signal generated by the order generator 13 which is sent to the request register 12 on the line 48. At the same time as the ORDER is sent to the other CPU's, the request register sends a REQUEST LOCAL SERIALIZATION control signal to the circuit 15 on the line 46 to cause the requesting CPU to serialize itself. After issuing the ORDER to all of the other CPU's in the system, the ordering CPU waits until it receives a SERIALIZATION COMPLETE (ACK) signal from all of the other CPU's in the system on the lines 35-38, which is indicated by a signal on the line 53, and a LOCAL SERIALIZATION COMPLETE signal from the circuit 15 on the line 52. Both the circuit 14 and the circuit 15 receive inputs from the instruction processor 6 and the circuit 14 delivers inputs to the processor 6 in the course of performing the necessary serialization operations. Of course, the circuit 14 is used when an ORDER is received from another CPU and circuit 15 is used by the CPU issuing the ORDER. Also, the INPUT SELECT control signal on the line 49 causes the circuit 17 to select among its inputs on the lines 24, 52 and 53 depending on whether serialization of other CPU's or only arbitration is required. When the ARB/SER COMPLETE circuit 17 receives the signals on the lines 52, 53 and 24, it sends a signal to the instruction processor 6 on the line 54. Thereafter, processor 6 completes the execution of the instruction or other operation which gave rise to the REQUEST and then drops the ORDER by sending a DROP ORDER control signal to the order generator 13 on the line 50. As soon as a CPU which has been awarded priority and has issued an ORDER drops its REQUEST, a second CPU in the system with a REQUEST pending, or newly generated, and then having the highest priority will be awarded priority in the manner described above. However, the second CPU will not issue an ORDER to the other CPU's until the first CPU drops its ORDER. The processing of the REQUEST of the second CPU and the awarding of priority thereto, even though the second CPU is not permitted to issue an ORDER until the pending ORDER has been dropped, results in an overlapping of operations and a significant savings in processing time especially when multiple simultaneous REQUESTS are made. At this point, it should be noted that the making of a REQUEST by a CPU is not intended to imply that the requesting CPU always requires that the other CPU's be serialized. For example, a requesting CPU may require that the execution of a group of instructions be completed without interruption as might occur if another CPU made a REQUEST, was granted priority and issued an ORDER. In such a case, the instruction processor 6 sends a special REQUEST to the register 12 on the line 45. The register 12 then sends a REQUEST to its priority circuit 11 and the priority circuit 11 in the other CPU's and an ARBITRATION ONLY control signal to its order generator 13 for preventing the generation of an ORDER, if the CPU is awarded priority. Thereafter, if the requesting CPU is awarded priority, the instruction processor 6 causes the request generator 12 to hold the REQUEST until the group of instructions involved is executed. By holding its REQUEST, the requesting CPU effectively prevents any other CPU from obtaining priority. While two embodiments of the present invention are described above, it is contemplated that various modifications may be made to the embodiments described. For example, while a multiprocessor system comprising four CPU's is described, other systems comprising more or less than four CPU's may be made with the present invention.

Claims: A control apparatus in each distributed central processing unit (CPU) within a distributed multiprocessor system comprising a plurality of distributed CPUs which process data in response to instructions, said control apparatus comprising: request means (12) responsive to a predetermined instruction for generating a REQUEST signal to request serialization of all said distributed CPUs; transmitting means (20) connected to said request means for transmitting said REQUEST signal to all other said distributed CPUs; and priority means (11) connected to receive REQUEST signals generated by all said distributed CPUs for awarding priority to one of said REQUEST signals when a plurality of REQUEST signals are simultaneously received by said priority means; characterized by further comprising receiving means (10) having a plurality of inputs (20, 21, 22, 23) where each input receives REQUEST signals from a respective one of said distributed CPUs, delay means connected to said inputs for compensating for the transmission times associated with transmitting REQUEST signals from the respective transmitting distributed CPUs, said delay means comprising a plurality of delays each of a multiple of one unit of delay; and selector means for coupling in series with each of said inputs a predetermined number of said delays such that the number of units of delay in series with each input is inversely proportional to the amount of time for transmitting a REQUEST signal from the respective distributed CPU to that one of said plurality of said inputs; and a plurality of outputs where each output is connected to a respective one of said inputs via said delay means and to said priority means; whereby REQUEST signals appearing at said outputs of said receiving means are in the same time relationship as that in which said REQUEST signals were generated by said distributed CPUs. The control apparatus of Claim 1 further comprising: order means (13) responsive to an award of priority to the distributed CPU containing said control apparatus for generating an ORDER signal (28) to request that each other said distributed CPU perform a serialization operation; local means (14, 15) responsive to a LOCAL signal to request said distributed CPU issuing said ORDER to perform a local serialization operation and generating a COMPLETE signal when said local serialization operation has been completed; order transmitting means for transmitting said ORDER signal to all other said distributed CPUs; reset means (12) responsive to said generation of said ORDER signal for dropping said REQUEST signal generated by said distributed CPU generating said ORDER signal; order receiving means (14) for receiving ORDERS transmitted from all other said distributed CPUs, for performing a serialization operation in response to said received ORDER signal and for generating and transmitting an ACK signal (35) to all other said distributed CPUs upon completion of said serialization operation; acknowledge receiving means (16) for receiving said ACK signal from each of said other distributed CPUs and for generating a DONE signal when an ACK signal has been received from each of said other distributed CPUs in response to said ORDER signal issued by said distributed CPU that issued said ORDER signal; and completion means (17) responsive to DONE signal and said COMPLETE signal (52) for completing said predetermined instruction and for dropping said ORDER signal after completing said predetermined instruction. The control apparatus of Claim 2 wherein said request means further comprises: hold means, responsive to an instruction requiring the execution of a predetermined operation without interruption and without serializing of said other distributed CPUs, for holding said REQUEST signal once priority is granted to said distributed CPU to prevent any other said distributed CPUs from being granted priority until after completion of the execution of said predetermined operation. A method of arbitration between requests for serialization of a plurality of distributed central processing units, CPUs, in a multiprocessor system in which each of said CPUs comprises a priority means having a plurality of inputs and a request means responsive to an instruction for generating a REQUEST signal to request serialization of said distributed CPUs, said method comprising the steps of: transmitting each said REQUEST signal generated by a said distributed CPU to all other said distributed CPUs; and awarding priority by each said priority means to the same said REQUEST signal from a plurality of simultaneously received REQUEST signals; characterized by further comprising the step of delaying said transmitted REQUEST signals received in each said distributed CPU to compensate for the time of transmission of said REQUEST signal from each other said distributed CPU to the receiving said distributed CPU so as to maintain the time relationship between received said REQUEST signal to be the same as when said REQUEST signals were originally transmitted by said other distributed CPU when received at said priority means in said receiving distributed CPU by providing in each said distributed CPU for each distributed CPU delay means for delaying a said REQUEST signal prior to said REQUEST signals being received by said priority means and selectively coupling in series a predetermined number of delays each being of a multiple of one unit of delay, such that the number of units of delay in series is inversely proportional to the transmission time for transmitting said REQUEST signal from a said transmitting distributed CPU to a said receiving distributed CPU. The method of Claim 4 further comprising the steps of: generating in said distributed CPU having priority an ORDER signal requesting all other said distributed CPUs to perform a serialization operation; performing a local serialization operation in said distributed CPU generating said ORDER signal; transmitting said ORDER signal to all other said distributed CPUs; executing by each said distributed CPUs in response to said ORDER signal a serialization operation; generating an ACK signal by each said other distributed CPU upon completion of said serialization operation by said other distributed CPU; transmitting said ACK signal by each said other distributed CPU to said distributed CPU generating said ORDER signal; dropping said REQUEST signal in said distributed CPU generating said ORDER signal upon generating said ORDER signal; completing the instruction in said distributed CPU having priority; and dropping said ORDER signal in said distributed CPU having priority in response to receipt of said ACK signal from each of said other distributed CPUs and upon the completion of said local serialization operation in said distributed CPU having priority. The method of Claim 5 further comprising the step of: holding, in response to an instruction requiring execution of a predetermined operation without interruption and without serializing said other distributed CPUs, said REQUEST signal in said distributed CPU having priority to prevent any said other distributed CPU from being granted priority until after completion of execution of said predetermined operation.

Inventors: BEGLEY MICHAEL J, MAGEE KRISTIN, VAHIDSAFA ALI, BEGLEY, MICHAEL J., MAGEE, KRISTIN, VAHIDSAFA, ALI
Applicants: AMDAHL CORP, AMDAHL CORPORATION
Application year: 1987
Main classification: G06F15
IPC codes: G06F15, G06F13, G06F9