Multiprocessor bus protocol

Abstract: A bus protocol system for interprocessor communications. The protocol system polls the processors in an open loop fashion to determine which processors are ready to send. Upon completion of a simultaneous poll of all processors the system identifies which processors are ready to send by utilizing a send mask generated by the ready processors. The ready processors are sequentially selected as send processors and granted access to the bus for a complete data transfer cycle unless the selected processor indicates it is not ready to send. The system also includes a timing signal system that provides for a high data transfer rate. A send clock signal strobes words onto the bus from a send processor and a receive clock signal loads words from the bus to a receive processor. The send processor generates the receive clock signal by delaying the send clock signal by a fixed delay, DR.

Description: The present system relates generally to communication between digital processors in a multiprocessor computer system and, more particularly, to a protocol for a multiprocessor bus. Many digital computer systems are using a plurality of independent processors to perform computer operations. Examples include fault-tolerant, modular, and parallel processing systems. These independent processors need to exchange data, from time to time, to perform system operations. Typically data is exchanged between processors on a bus. The protocol of the multiprocessor system controls access by processors to the bus, facilitates the formation of a sender/receiver pair of processors, and provides a timing reference for transferring data from the sender to the bus and from the bus to the receiver. In many existing systems a dedicated bus controller is coupled to the bus and functions to poll the processors to determine which processors are ready to send data. Typically, the poll for senders sequence is instigated by a processor that is ready to send signalling the bus controller on a dedicated line common to all processors. The bus controller determines which processors are ready to send by serially polling the processors on dedicated select lines. In one example of a multiprocessor bus protocol disclosed in U. S. Patent No. 4,228,496 to Katzman et al., the bus controller then begins a data exchange by enabling the sender. This enablement is accomplished via the individual select line to the sender. The sender then acknowledges if ready. The bus controller enables the receiver via the individual select line to the receiver. If the receiver is ready to receive data, then it signals the bus controller with an acknowledge signal. In the Katzman system, the bus transfer is synchronized by the system clock. This clock signal is supplied to each processor by individual clock lines. These lines are all the same length so the clock signal arrival times at each processor are not skewed. The period of system clock signal must be long enough to allow propagation of a signal between processors positioned at opposite ends of the bus and to allow for the bus settling times. Accordingly, the maximum data transfer rate is determined by the length of the bus. A primary objective of any multiprocessor bus protocol is to provide a high rate of data transfer between processors via the bus. Several aspects of existing protocol systems prevent the attainment of high data transfer rates. First, the requirement of generating a sender request before a poll, sender acknowledges, and receive acknowledge before a transfer generates bus overhead time. Secondly, the synchronous transfer system requires that data be valid on the entire bus before being transferred to a receiver because the location of the receiver may be anywhere on the bus. Thus, the maximum data transfer rate is determined by the propagation delay of signals propagating the length of the bus. Additionally, the need to provide separate clock lines, of equal length, and individual select lines to each processor requires a large number of wires to form the bus thereby, increasing the manufacturing complexity and expense of the bus hardware. The present invention is a bus protocol system that polls the processors in a continuous, open loop fashion to identify processors ready to send and to sequentially select the ready processors as send processors. The bus system utilizes only bus lines common to all processors. No radial lines are utilized. Thus, less bus lines are utilized than in existing systems; thereby reducing fabrication cost and complexity and increasing reliability. According to the present invention, there is provided a multiprocessor computer system of the type including a bus and two or more processors coupled to said bus, where first and second processors may exchange a packet of data words during a packet transfer cycle, a bus protocol system for controlling access to the bus by the processors, establishing a sender/receiver processor pair and providing timing signals to transfer words between the sender/receiver pair at a high rate, said timing system being characterised by: first and second bus lines; means for generating a send clock timing signal comprising a plurality of send clock pulses and for asserting said send clock signal on said first bus line; means, at said send processor coupled to said first bus line, for strobing a word in said packet to said bus upon receipt of a selected send clock pulse; means, at said send processor, for generating a receive clock pulse, with said receive clock pulse being said send clock pulse delayed by a fixed delay DR and for asserting said receive clock signal on said second bus line; and means, at said receiver processor, coupled to said second bus line, for loading a word from said bus upon receipt of said receive clock pulse; and wherein the maximum skew time being of said bus the maximum difference between the propagation times of signals on different bus lines propagating from one end of the bus to the other and the timing margin of said loading means being the sum of a set up time and a hold time and wherein the period of said send clock signal is substantially equal to the sum of the skew time and said time margin. Advantageously, the magnitude of said delay DR is approximately equal to one half the period of said send clock signal. The invention is described further hereinafter, by way of example only with reference to the accompanying drawings, in which:- Fig. 1 is schematic diagram of a multiprocessor bus system. Fig. 2 is a schematic diagram of a bus controller. Fig. 3 is a schematic diagram of the shared IN queue section of the interprocessor control. Fig. 4A is a schematic diagram of the shared OUT queue section of the interprocessor control. Fig. 4B is a schematic diagram of the logic for OUT queue addressing. Fig. 4C is a schematic diagram of the physical circuitry for OUT queue addressing. Fig. 5 is a state diagram illustrating the operation of the bus controller. Fig. 6 is a state diagram illustrating the operation of the IN queue section. Fig. 6 shows the state transition diagram for the receive state machine. Fig. 7 is a state diagram illustrating the operation of the OUT queue section. Fig. 8A is a timing diagram depicting signal transitions on the E bus, the D bus, the receive command line, and the NAK line. Fig. 8B is a schematic diagram of the send mask. Fig. 8C is a schematic diagram of word 00 of a packet. Fig. 9 is a timing diagram depicting the various clock signals and data transitions. Fig. 10 is a timing diagram depicting the relative positions of RCLK and a data word at a receiver. Fig. 1 is schematic diagram of a multiprocessor bus system. Referring now to Fig. 1, the multiprocessor system 31 includes individual processor modules 33. Each processor module 33 comprises a central processing unit 105, a memory 107, and an interprocessor control 55. The individual processor modules 33 are interconnected by interprocessor buses 35 for interprocessor communications. In the specific embodiment of the multiprocessor system 31 up to sixteen processor modules 33 are interconnected by two interprocessor buses 35 (indicated as the X bus and the Y bus in Fig. 1.) Each interprocessor bus 35 has a bus controller 37 associated with that bus. The CPU 105 of each processor 33 includes a host processor 113 and an associated microprogram 115. The bus controllers 37, interprocessor buses 35, and interprocessor controls 55 together with associated host processors 113 and microprograms 115 provide an interprocessor bus system. The construction and operation of this system is described more fully below. The information transmitted over the bus is transmitted in multiword packets. In a preferred form of the present invention, each packet is a sixteen-word packet in which fifteen of the words are data words. The control logic within the bus controller 37 and the interprocessor controls 55 of the individual modules 33 follows a detailed protocol. The protocol provides for establishing a sender receiver pair and a time frame for the data packet transfer. The X bus 35 is identical in structure to the Y bus 35 so that the structure of only one bus will be described in detail. As illustrated in Fig. 1, each bus 35 comprises sixteen individual bus data lines 57, three encoded control lines plus one parity line (the EBUS) 59, a receive command (RCVCMD) line 62, a NAK line 64, and two clock lines (SCLK 66 and RCLK 68). As also illustrated in Fig. 1, the interprocessor control 55 of each processor module 33 includes two IN queue sections 65 (shown as an X IN queue section and a Y IN queue section) and a shared OUT queue section 67. In Fig. 1, each IN queue section 65 has inputs coupled to the data bus 57, RCLK line 68, RCVCMD, and an output connected to the NAK line 66 of the respective interprocessor bus 35. The shared OUT queue section 67 has outputs coupled to the data buses 57, the NAK line 64, the RCLK line 68, and the receive command line 62 and inputs coupled to the EBUS 59, DBUS 57, NAK line 64, and SCLK line 66 of the respective buses. All of the lines in the data buses are common to all the processor modules 33. There are no radial lines, i.e., dedicated lines from the bus controller 37 to a single processor module 33. The elimination of radial lines reduces the electrical noise, increases the system's reliability, and reduces the system cost. Fig. 2 is a schematic diagram of a bus controller 37. In Fig. 2, the bus controller 37 comprises a bus control state machine 70, an oscillator 72 coupled to the clock inputs of the bus controller state machine 70, to a bus controller pipeline register 74 clock input and to the SCLK line 66 via a bus controller delay element 76. The output of the pipeline register 74 is coupled to the data control lines 57 and 59 via an output transceiver 78. The inputs of the the bus controller state machine 70 are coupled to the data and control lines 57 and 59 via an input transceiver 80. Fig. 3 is a schematic diagram of the shared IN queue section 65 of the interprocessor control 55. In Fig. 3, an IN queue buffer 90 performs a storage function. The IN queue buffer 90 has a data input coupled to the D bus 57 by an IN queue pipeline register 92. A data output of the IN queue buffer 90 is coupled to an SM_SJ bus 94 which transfers data between the IN queue buffer 90, and the main memory 107 of a processor module 33. The IN queue section includes a receive state logic machine 96 and a CPU control state logic machine 98. The receive machine 96 includes a clock input coupled to the RCLK line 68 and the CPU logic machine 98 has a clock input coupled to the CPU clock. The CPU 105 operates on a different clock than the interprocessor bus clock. During the filling of an OUT queue or the emptying of an IN queue in the interprocessor control 55 by the CPU the operation takes place at the CPU clock rate. However, transmission of packets over the interprocessor bus always takes place at the bus clock rate. In the present system, there are two bus clocks, the SCLK and the RCLK. Both the SCLK and the RCLK are clocked at the same rate, however there is a relative phase difference between the two signals. The clock input to the pipe line register 92 is also coupled to the RCLK line via a transceiver and clock buffer. Accordingly, both the receive state machine 96 and the pipe line register 92 are clocked by the RCLK signal. The receive state logic 96 includes inputs coupled to the receive command line 62, the D bus 57, an SJ bus 100, a receive line 102, and a REQ_IQ line 104. The receive state logic machine 96 has outputs coupled to a clear line 106, the NAK line 64, and a CPU_OWNS_IQ 108. The CPU control state logic machine 98 has inputs coupled to the S_FUN bus 100 and CPU_OWNS_IQ line 108. The CPU control state logic machine 98 has outputs connected to the receive line 102, clear line 107, and the REQ_IQ line 104. An IN queue counter 110 has inputs coupled to the CNT_EN_CLR lines 106 and 107 via MUX 111 and an output coupled to an address bus 112. The control input of MUX 111 is coupled to the CPU_OWNS_IQ line 108. The address bus 112 is coupled to the address input of the IN queue buffer 90. Each IN queue buffer 90 is a sixteen word deep buffer for storing one packet. Addressing of each IN queue buffer 90 is straight forward using a four bit counter 110 to generate the word address. The counter is automatically incremented whenever the bus writes to the queue and whenever the processor reads from or writes to the queue. Fig. 4A is a schematic diagram of the shared OUT queue section 67 of the interprocessor control 55. Referring to Fig. 4A, an OUT queue buffer 160 has its data output coupled to the D bus 57 via an OQ pipeline register 162 and MUX 163. A send mask generator 163a is coupled to the D bus via MUX 163. The data input port of the OQ buffer 160 is coupled to the SM_SJ bus 94. A send state machine 164 has its clock input coupled to the SCLK line 66 and a CPU controlled send state machine 166 has its clock input coupled to the CPU clock. The clock input of the OQ pipeline register 162 is coupled to SCLK via a transceiver and clock buffer. The send state machine 164 has inputs coupled to a NAK in line 168, the E bus 59, the D bus 57, a send line 170, REQ_OQ line 172, an OUTQ_NOT_EMPTY line 173, and a count zero line 174. The send state machine 164 has outputs coupled to a NAK out line 176, the D bus 57, the receive command line 62, a CPU_OWNS_OQ 176, a CNT_EN_CLR line 178, a RESPOND_TO_POLL line 179, and a time out line 180. The CPU controlled send state machine 166 has inputs coupled to the SFUN bus 100 and the CPU_OWNS_OQ line 108. The CPU controlled send state machine 166 has outputs coupled to the SEND and REQ_OQ lines 170 and 172 and to a CNT_EN_CLR line 182. An OUT queue counter 184 has inputs coupled to the CNT_EN_CLR lines 174 and 182 and to SCLK and CPUCLK via MUX 186. The control input of MUX 186 is coupled to the CPU_OWNS_OQ line 176. An address bus 186 is coupled to the address input of the OUT queue buffer 160. Each OUT queue section further includes an RCLK generating circuit comprising an OUT queue delay element 190 having an input coupled to the SCLK line 66 and an output coupled to the RCLK line 68. Figs. 4B and 4C depict logical and physical implementations of the OUT queue addressing counter 184 and OUT queue buffer 160. Referring to Fig. 4B, the OUT queue buffer 160 consists of a single memory array 256 words deep by sixteen bits wide (plus two parity bits). The OUT queue 160 is capable of storing up to sixteen packets of sixteen words each. The addressing logic for this array implements a circular queue. The bus removes packets from the bottom of the queue and the processor adds packets to the top of the queue. Referring now to Fig. 4C, three counters, CPU_PACKPNT 184a, BUS_PACKPNT 184b, OQ_CNT 184c, are utilized to generate addresses for the OQ buffer 160. CPU_PACKPNT 184a points to the next free packet location within the queue. It provides the upper four bits of the address when the processor reads or writes the OUT queue 160. This counter can be cleared using a RESET OUT QUEUE micro-operation and can be incremented using a NEXT micro-operation. The BUS_PACKPNT counter 184b points to the next packet waiting to be sent on the bus. It provides the upper four address bits of the OUT queue when the bus state machine 70 reads from the queue. This counter is cleared with the RESET OUT QUEUE micro-operation. It is automatically incremented after a packet is sent. The counter OQ_CNT 184c provides the low order four address bits for the OUT queue and thus points to a word within a packet. This counter is reset using the RESET OUT QUEUE micro-operation or using a RESET COUNT micro-operation. OQ_CNT 184c is automatically incremented whenever a word is written to or read from the OUT queue 160. After sixteen reads or writes, this counter will wrap around to zero. Thus, the seventeenth word of a packet is the same as the first word. Figs. 5, 6, and 7 are state diagrams illustrating the operation of the bus controller 37, the IN queue section 65, and the OUT queue section 67. In each figure, the solid lines with arrows indicate transitions from the present state to the next state. Dashed arrows ending on the solid arrows indicate conditions that must be satisfied for the indicated transitions to take place. Fig. 8 is a timing diagram depicting signal transitions on the E bus 59, the D bus 57, the receive command line 62, and the NAK line 64. The following is a description of the system protocol for a successful transfer of packet between the processors 33 depicted in Fig. 1. For a successful transfer to take place, the IN queue buffer 90 is empty and the receive state machine 96 is in the WAIT_FOR_CMD state, the OUT queue buffer 160 is full, and the send state machine 164 is in the WAIT_FOR_CMD state, and the bus controller state machine 70 is in the POLL_PROC state. During the transfer the bus controller state machine transmits encoded signals on the E bus 59. Table 1 lists the names of these encoded signals, the signal code, and gives a brief description of the function of each coded signal (the * indicates that data is low asserting), Table 2 lists the state transitions for the receive state machine, and Table 3 lists the state transitions for the send state machine. DATA(00:15)* These sixteen lines carry the IPB data with DATA(00) than being the most significant bit. CNTL(0:2,P)* These encoded control lines are asserted only by the BC. In the following description, a "1" means that the signal is asserted, while a "0" means that the signal is not asserted. Note that the signals are low asserting. CNTL(P)* is odd parity for the three control lines. 000,1 - NULL The BC will assert this command to indicate that nothing is happening on the bus. It is useful to allow for propagation delays during polling cycles. 001,0 -- (reserved) 100,0 -- DATA XFER The BC asserts this command to indicate that a data transfer is in progress. 101,1 -- RESET The BC can assert this command to reset the sending state machines on each of the processors. The reset of the state machines is synchronous with SCLK. 110,1 -- POLL PROC When this command is asserted by the BC, each processor that has a packet to send will assert the data line corresponding to its processor number. 111,0 -- SNDCMD The BC asserts this command along with a processor number on DATA(04:07) and DATA(12:15) to select a processor to send. RECEIVE Machine State Transition Table Inputs to machine CPU_REQIQ R_SELECT = RCVCMD & RCVSEL RECEIVE IQ_CNT15 State Transition Table Name Current State Input Next State MUX_DELAY1 0001 x0xx 0000 0001 x1xx 0100 WAIT_FOR_IQ 0000 x00x 0000 0000 x01x 1000 0000 x10x 0100 0000 x11x 0100 NAK1 0100 xxxx 0101 NAK2 0101 xxxx 0000 MUX_DELAY2 1000 x0xx 1010 1000 x1xx 1100 CLEAR_CNT 1010 x0xx 1111 1010 xlxx 1100 NAK3 1100 xxxx 1101 NAK4 1101 xxxx 1010 WAIT_FOR_CMD 1111 00xx 1111 1111 01xx 1011 1111 10xx 0001 1111 11xx 1011 LOAD_IQ 1011 xxx0 1011 1011 xxx1 1001 CNT_DELAY 1001 xxxx 0001 Transitions From Unused States 0010 xxxx 0000 0011 xxxx 0010 0110 xxxx 0100 0111 xxxx 0110 1110 xxxx 1111 SEND State Machine Transition Table Inputs to Machine CPU_REQOQ S_SELECT = SNDCMD & SNDSEL EMPTY TIMEOUT POLL_PROC SEND NAK_IN CNT0 State Transition Table for SEND State Machine Name Current State Input Next State WAIT_FOR_OQ 0000 x0xxx0xx 0000 0000 x0xxx1xx 1000 0000 x1xxx0xx 0110 0000 x1xxx1xx 0110 NAK1 0110 xxxxxxxx 0100 NAK2 0100 xxxxxxxx 0000 WAIT_FOR_CND 1000 00000xxx 1000 1000 10xxxxxx 0000 1000 xx1xxxxx 0000 1000 0x01xxxx 1000 1000 1x01xxxx 0000 1000 0x00xxxx 1000 1000 1100xxxx 0000 1000 00001xxx 1100 1000 01001xxx 1000 1000 11001xxx 0000 1000 x1000xxx 1101 RESPOND_TO_POLL 1100 xxxx0xxx 1000 1100 xxxx1xxx 1100 BUS_DELAY 1101 xxxxxxxx 1110 POLL_RCVR 1110 xxxxxxxx 1111 SEND 1111 xxxxxx00 1111 1111 xxxxxx01 1001 1111 xxxxxx10 1000 1111 xxxxxx11 1000 CLR_TIMER 1001 xxxxxxxx 1000 Transitions from Unused States 0001 xxxxxxxx 1000 0011 xxxxxxxx 1000 0010 xxxxxxxx 0100 0101 xxxxxxxx 1110 0111 xxxxxxxx 1000 1011 xxxxxxxx 1000 1010 xxxxxxxx 0111 Referring now to Figs. 5-8 and Tables 1-3, the bus controller state machine asserts the poll processor signal for clock cycles 1-3. This poll processor signal is received by the SEND state machine and causes the transition from the WAIT_FOR_COMMAND state to the RESPOND_TO_POLL state. The state machine then determines whether there is a packet stored in the OUTQ RAM 160 ready to send. If the packet is ready to send then the send state machine 164 controls the state of the RESP_TO_POLL signal on line 175 to cause MUX 163 to couple the send mask generator 163a to the D bus 57. The send mask generator 163a asserts a signal on a unique lines of the D bus 57 assigned to that particular processor to indicate that the processor is ready to send. If a packet is not ready, then the send state machine 164 controls the state of the RESP_TO_POLL signal on line 175 to cause MUX 163 to couple the register 162 to the D bus 57. The register 162 asserts a signal on the unique lines of the D bus 57 indicating that the processor is not ready to send. As described above, the D bus includes sixteen individual lines with each one of these lines assigned to a given one of the sixteen processors coupled to the bus. Accordingly, a send mask is generated where the data lines assigned to processors ready to transmit are asserted and those data lines assigned to processors not ready to transmit are deasserted. This send mask is asserted on the data bus for clock cycles 2-4, as indicated on graph 202 of Fig. 8A. The send mask is depicted in Fig. 8B. The send mask is received at the bus controller. The bus controller advances from the POLL_PROC state to the NULL state for clock cycles 4-6. During the NULL state the bus control state machine reads the send mask and determines which processors are ready to send data. The bus controller then advances to the SEND state and generates the SEND COMMAND signal on the E bus. Additionally, the bus controller selects of the ready processors as a send processor by asserting the address of the selected send processor on the D bus during clock cycles 6 and 7, as depicted in line 202 of Fig. 8. The send processor state machine, identified by the address asserted on the D bus, makes transitions to the BUS_DELAY state, allowing the bus to settle for one clock cycle, and then to the POLL_RECEIVER state. In the POLL_RECEIVER state, the send processor transfers the first word in the OUT queue buffer 160 onto the D bus, as depicted on line 202 of Fig. 8. The format of word 00 of the packet is depicted in FIG. 8C. Simultaneously, the BC state machine advances from the SEND state to the DATA_XFER state and asserts the data transfer signal on the E bus, as depicted in Fig. 200. Thus, the BC 37 has granted the send processor access to the bus. If the send processor is not ready to send, it asserts a NAK signal on NAK line 64 to indicate to the BC 37 that it is not ready. The BC 37 automatically remains in the DATA_XFER state for a complete DATA_XFER cycle unless a NAK signal is received. In that event, the transfer from the send processor is aborted. The BC 37 asserts the SEND state and selects a different ready processor to be a send processor. If no NAK signal is received, the send processor asserts the receive command line 62 during this clock cycle and continues to drive the DBUS 57 with the first word of the data packet. The receive state machine 96 then advances from the WAIT_FOR_CMD state to the LOAD_IQ state. At clock cycle 9, the IN queue counter 110, the OUT queue counter 184, and a counter in the bus control state machine 70 begin to count sixteen cycles, (COUNT 00 to COUNT 15). The IN queue counter and OUT queue counter span the address space of the IN queue and OUT queue buffers 90 and 160, respectively. The words in the packet from the OUT queue buffer 160 are transferred to the bus and the words are clocked into the IN queue buffer 90 according to an SCLK/RCLK timing scheme to be described more fully below. At the end of the sixteen cycles the count 15 output from the counters causes the bus control state machine to advance from the DATA_XFER state to the NULL state, the send buffer state machine to advance from the SEND state to the CLR_TIMER state, and the receive buffer state machine to advance from the LOAD_IQ state to the CNT_DELAY state. These delay and null states allow the bus to settle before the transmission of another packet. Subsequently, the bus control state machine 70 returns to the SEND state and places the address of the next processor ready to send onto the E bus. The above-described cycles continue for each processor ready to send until all processors have transmitted their packets. The bus control state machine then makes the transition from the SEND state to the POLL_PROC state and repeats the above-described process. Thus, a data transfer consists of a first processor sending one sixteen word packet of data to a second processor. The transfer is initiated when the bus controller selects a first processor to send after the processor has responded to the general poll. The bus controller selects a processor to send by asserting the code for SEND on the E bus and asserting the address of the first processor on data lines 04 through 07 and twelve through fifteen. If the first processor is not ready to send, it will assert NAK for two cycles and ignore the selection. Upon receiving the NAK signal, the bus controller will abort the sender selection and will proceed to select the next sender in the sequence. If the first processor is ready to send, it will assert the RECEIVE COMMAND line and the first word of the packet in its IN queue onto the D bus. The first word in the packet contains the address of the second receiver processor and the combination of RECEIVE COMMAND and the first word in the packet uniquely selects a processor to receive. If the second processor is not ready to receive the packet, it will assert NAK for two cycles. When the first processor receives NAK it will abort its current data transfer and will wait in the WAIT_FOR_CMD state for the next general poll from the bus controller. The bus controller, upon receiving the NAK, will assume that the transfer was unsuccessful and will abort the current transfer. The bus controller will then select the next sender in the sequence. Note that both the first processor and the bus controller must monitor NAK. If the second processor is able to receive the packet, however, it will clock the data into its IN queue using the active edge of the RCLK, as described below. After it has read the packet, the second processor will become unavailable to receive until its IN queue buffer is emptied by the host CPU. These data transfers operate in an open loop fashion, in that if the first processor does not receive a NAK it will assume that the transfer was successful. This open loop operation allows the bus protocol to operate more efficiently than would be possible if a positive acknowledgement were required. Each receive state machine actually consists of two interlocked state machines. One of the machines is clocked by the processor clock and responds to the micro-operation commands. The other machine is clocked by RCLK and responds to commands from the bus as well as commands from the processor clocked machine. Commands between the two machines are doubly synchronized. Fig. 6 shows the state transition diagram for the receive state machine. The bus clocked machine arbitrates the ownership of the IN queue between the processor and the bus. If this machine is in the MUX_DELAY1, WAIT_FOR_IQ, NAK1, or NAK2 states, the queue is owned by the processor. If this machine is in any other state, the queue is owned by the bus. Assume the bus clocked state machine is in the WAIT_FOR_IQ state. If another processor tries to select this processor as a receiver while it is in this state, the machine will assert the NAK signal for two cycles. Asserting NAK causes the sending processor to abort its transfer. When the processor has emptied the IN queue and is ready to receive another packet, it will execute the GIVE_IQ micro-operation. This operation will cause the processor clocked machine to advance from the IDLE state to the RECEIVE state. This transition then will cause the bus clocked machine to transition from the WAIT_FOR_IQ state to the MUX_DELAY2 state. The MUX_DELAY2 state will then cause the processor clocked machine to transition back to the IDLE state to wait for the next micro-operation command. From the MUX_DELAY2 state, the bus machine transitions to the WAIT_FOR_CMD state through the CLEAR_CNT state. The receive machine will assert NAK if the processor is selected to receive while the machine is in the MUX_DELAY1 or CLEAR_CNT states. While the bus clocked machine is in the WAIT_FOR_CMD state, the IN queue is owned by the bus. The processor can regain ownership of the queue by executing the CPU_REQIQ micro-operation. This command causes the processor clocked machine to transition to its WAIT_FOR_IQ state. This state then causes the bus clocked machine to transition to its WAIT_FOR_IQ state, with the processor now owning the queue. The processor clocked machine then returns to its IDLE state. From the WAIT_FOR_CMD state, the bus clocked machine will transition to the LOAD_IQ state when it is selected to receive by another processor on the bus. It will remain in the LOAD_IQ state until it has loaded sixteen words in to the IN queue. It then transitions to its WAIT_FOR_IQ state through the CNT_DELAY state, thereby giving ownership of the queue back to the processor. The CNT_DELAY state assures that the enable inputs to the IN queue address counter are deasserted before switching its clock input. The send state machine actually consists of two interlocked state machines. One machine is clocked by the processor clock and responds to the micro-operation commands. The other machine is clocked by the SCLK and it controls the actual signals on the bus. Signals between the two machines are doubly synchronized with the appropriate clock. Ownership of the OUT queue is controlled by the bus clocked state machine. The queue is owned by the processor whenever the bus clocked machine is in the WAIT_FOR_OQ, NAK1, or NAK2 states. If it is in any other state, the OUT queue is owned by the bus. Packets can be sent only when the queue is owned by the bus and the processor can manipulate the queue only when it owns the queue. Assume the bus clocked state machine is in the WAIT_FOR_CMD state. If it receives a poll command from the bus controller, and the queue is not empty, and the time has not expired, then it will transition to the RESPOND_TO_POLL state. In this state, it will drive its poll mask on to the bus until the poll command is deasserted by the bus controller. It then will return to the WAIT_FOR_CMD state to wait for another command. From the WAIT_FOR_CMD state, the bus machine will transition to the states to send a packet only if it is selected to send by the bus controller, and the queue is not empty, and the timer has not expired. The BUS_DELAY state allows one bus cycle for propagation delays on the bus as a new processor (i.e., this one) begins to drive the bus. The receiving processor is selected in the POLL_RCVR state. Assuming that the receiver is ready, the send machine begins to send the rest of the packet. If NAK is asserted at any time during the send process, the processor aborts the packet by returning to the WAIT_FOR_CMD state. If all of the words are sent without NAK being asserted, however, the processor assumes that the send was successful. From the CLR_TIMER state, it then clears the send timer and increments BUS_PACKPNT to point to the next packet in the queue. The bus state machine will respond to an ownership request from the processor only when it is in the WAIT_FOR_CMD state. It then will transition to the WAIT_FOR_OQ state and give the ownership of the queue to the processor. If it is selected to send while it is in this state, the bus state machine will assert the NAK signal for two cycles. This NAK will cause the bus controller to abort its selection of this sender. When the processor is finished with the OUT queue, it will give the queue to the bus machine by causing it to transition to the WAIT_FOR_CMD state. From this state, the bus machine can begin sending packets. When the OUT queue becomes empty, the bus machine enters the WAIT_FOR_OQ state. Thus, ownership of the OUT queue is given to the processor when the OUT queue is empty. The bus reset command from the bus controller has a slightly different effect on the send state machine depending upon the OUT queue ownership. If the processor owns the OUT queue, a bus reset command forces the send machine into the WAIT_FOR_OQ state. However, if the bus owns the OUT queue, a bus reset command forces the send machine into the WAIT_FOR_CMD state. The above-described protocol does not require positive acknowledgement from a selected processor prior to initiating a data transfer or send cycle. Instead the protocol automatically initiates these cycles unless a negative acknowledge indicates that a selected processor is not ready to perform a desired function. Utilizing this negative acknowledge system reduces the overhead associated with the protocol and increases the data transfer rate of the system. The aspect of the system protocol relating to establishing a timing reference for transferring the words in a first processor (sender) to the bus and transferring words on the bus to a second prpcessor (receiver) will now be described. Fig. 9 is a timing diagram depicting the various clock signals and data transitions. Referring to Fig. 9, REFCLK 300 is the output of the bus controller oscillator 72. Both SCLK 302 and RCLK 304 are derived from REFCLK 300, thus both have the same frequency but different relative phases. The leading edge of each SCLK pulse is delayed relative to the leading edge of the corresponding REFCLK pulse by a time delay DS and delays induced by other circuit components. The time delay DS is produced by the bus controller delay element 76 (Fig. 2). Similarly, the leading edge of RCLK is delayed relative to the leading edge of SCLK by the quantity DR produced by the delay element 190 and the delays introduced by other circuit elements. As depicted in Fig. 4, SCLK is utilized as a strobe to clock data from the OUT queue pipeline register 162 of the sender onto the D bus. The RCLK signal is generated at the sender by transmitting SCLK through the delay element 190 onto the RCLK line 68. Referring back to Fig. 3, RCLK is utilized as a strobe to clock data from the D bus into the IN queue pipeline register 192 of the receiver. Referring again to Fig. 9, successive words in the packet are transferred from the OQ pipeline register 162 of the sender upon receipt of the leading edge of the SCLK pulses. Thus, each data word cycle time is equal to the period of the clock cycles. The leading edge of each data word cycle is aligned with the leading edge of the SCLK pulse. Each word transferred to the bus consists of a set of high or low voltage level signals asserted on the individual lines of the D bus 57. Each component in the transmission path delays the propagation of these signals. In particular, the time delay (Tbus) introduced by the D bus 57 itself is dependent on the length of the bus between the sender and the receiver. In the present system, a transfer between any two processors 33 may take place. Thus, the quantity Tbus may vary from Tbus max (sender and receiver at opposites ends of the bus) to Tbus min (sender and receiver adjacently positioned on the bus). The signal RCLK is generated and phased relative to the data word cycle at the sender. The quantity DR is selected so that the leading edge of the RCLK pulse arrives at the middle of the time that a data word is valid at the receiver. Because the data word and RCLK pulse are generated at the sender, and propagate together on the bus to the receiver, the bus propagation delay (Tbus) is approximately the same for both signals. Thus, the system compensates for and eliminates any dependence upon the variations of the bus propagation delay due to the relative positions of the sender and the receiver on the bus. In every bus system the propagation delays between two points on various lines of the bus are not exactly equal, but vary due to manufacturing imperfections and other physical effects. The maximum variations between propagation delays on any two signals is the skew time of the bus. Thus, the phase relationship established between the data word and the RCLK pulse at the sender will not be preserved as these signals propagate along the bus. Fig. 10 illustrates the relative timing of the data word and RCLK signal at the receiver IQ pipeline register for a processor to processor data transfer. Because of skew, the position of the leading edge of the RCLK signal may vary over the time interval TR max - TR min . As indicated in Fig. 10, the electrical characteristics of the IN queue pipeline register 92 require that a data word be valid for a minimum set-up time, TSU, at the inputs of the IN queue pipeline register 92 prior to the arrival of the leading edge of RCLK. Additionally, the data word must remain valid for a minimum hold time, TH, after the arrival of RCLK. The sum of TSU and TH is the time margin required by the IN queue pipeline register 92. The period of SCLK must be at least equal to the sum of the timing margin and the maximum skew time of the bus. Thus, the data transfer rate is limited by the maximum skew between RCLK and the data word at the receiver. And, because the maximum skew is generated for data transfers between processors at opposite ends of the bus, the maximum data transfer rate is dependent on the length of the bus. The data transfer system utilized in the present invention is asynchronous. The relative timing between the data words and RCLK is established by the sending processor. The data word need not be valid on the entire bus at any given time, but only need be valid with reference to RCLK as the signals propagate along the bus. Accordingly, several data words may be propagating along the bus at a given time. The bus may function as a pipeline register storing several words at a given time. In a synchronous bus protocol, a single clock signal controls all data transfers. Because the receiver may be positioned at any point along the bus, a data word must be valid on the entire bus before being strobed into the IN queue section of a receiver. The period of the synchronous system clock must be of sufficient magnitude to allow the data signals to settle over the entire length of the D bus 57 for a time period at least equal the timing margin requirement. Accordingly, the maximum frequency of the clock will depend on the propagation delay of a signal from one end of the bus to the other. The advantage of the present system over a synchronous bus system is now apparent. The maximum data transfer rate of each system is dependent on the length of the bus. However, in the present system this maximum transfer rate is dependent on the time margins at a given processor and on the maximum skew due to the length of the bus wherein in a synchronous system the transfer rate is dependent on the propagation delay between the ends of the bus. The skew time is generally lower than the propagation delay and thus the maximum transfer rate of the present system is greater than the transfer rate of existing bus systems. The invention has now been described with reference to specific embodiments. However, various modifications and substitutions will now be apparent to persons skilled in the relevant arts. For example, the number of words in a data packet may be any arbitrary length. The particular state transition tables and state machines are not critical to the invention. Accordingly, it is not intended that the invention be limited except as provided by the appended claims. Reference is hereby directed to our Divisional European Application No. 87302193.5 which describes and claims a multiprocessor computer system of the type including a bus and two or more processors coupled to said bus, where first and second processors may exchange a packet of data words during a packet transfer cycle, and a bus protocol system for controlling access to the bus by the processors and for establishing a sender/receiver processor pair, the bus protocol system being characterised by: means for simultaneously polling all processors to determine which processors are ready to transfer a packet of words to another processor over said bus; means for sequentially selecting each of said ready processors to be a send processor; means at a selected sender processor, for indicating that said selected send processor is not ready to send; means for initiating a complete data transfer cycle for each selected ready send processor to enable said selected processor to send data unless said selected send processor indicates it is not ready to send; and means for controlling said selecting means to select another ready processor to be a send processor when the data transfer cycle for the selected processor is completed or when the selected processor indicates that it is not ready to send.

Claims: A multiprocessor computer system of the type including a bus (35) and two or more processors (33) coupled to said bus, where first and second processors may exchange a packet of data words during a packet transfer cycle, a bus protocol system (37, 65, 67) for controlling access to the bus by the processors, establishing a sender/receiver processor pair and providing timing signals to transfer words between the sender/receiver pair at a high rate, said timing system being characterised by: first and second bus lines (66, 68); means (72, 76) for generating a send clock timing signal comprising a plurality of send clock pulses and for asserting said send clock signal on the first bus line (66); means (162), at said send processor coupled to the first bus line (66), for strobing a word in said packet to the bus (35) upon receipt of a selected send clock pulse; means (190), at said send processor, for generating a receive clock pulse, with said receive clock pulse being said send clock pulse delayed by a fixed delay DR and for asserting said receive clock signal on the second bus line (68); and means (92), at said receiver processor, coupled to the second bus line (68), for loading a word from said bus upon receipt of said receive clock pulse; and wherein the maximum skew time of the bus (35), being the maximum difference between the propagation times of signals on different bus lines (66, 68) propagating from one end of the bus to the other and the timing margin of said loading means (92) being the sum of a set up time and a hold time and wherein the period of said send clock signal is substantially equal to the sum of the skew time and said time margin. A system as claimed in claim 1 wherein the magnitude of said delay DR is approximately equal to one half the period of said send clock signal.

Inventors: LILJA DAVID J, WIERENGA STEVEN W, ZACHER RICHARD A, LILJA,DAVID J., WIERENGA, STEVEN W., ZACHER, RICHARD A., Lilja,David J. Dept.of Electrical Engineering
Applicants: TANDEM COMPUTERS INC, TANDEM COMPUTERS INCORPORATED
Application year: 1987
Main classification: G06F13
IPC codes: G06F13