Packet switch network protocol

Abstract: A first of a plurality of nodes in a multiprocessor system transmits a data packet to a second of the plurality of nodes over a data channel when the first node detects a predetermined sequence of bits on a control channel connected therebetween. The data packet is preceded on the data channel by a header which includes a plurality of fields informing the second node of the beginning of the data packet, the length of the data packet and the sequence number assigned to the packet. The data packet is followed by a field which enables the second node to determine whether the data packet was received correctly. If the second node determines that the data packet was received correctly, the data packet is stored in the node and an acknowledgement sequence is transmitted to the first node over the control channel. However, if an error occurs during the transmission of the data packet, the protocol enters an error state by removing all signal information from the data and control channels. Thereafter, a recovery processor, which is associated with each of the plurality of nodes, analyzes the state of the nodes at the time of the error in order to determine which corrective action needed to recover the respective node. The data link protocol also facilitates the transmission of a data packet from a first node through an intermediate node to a final node without storing the data packet in the intermediate node.

Description: This invention relates to multiprocessor systems and more particularly to a data link protocol for transmitting data packets between nodes in a multiprocessor system. Closely-coupled multiprocessor systems require high performance from the underlying communication network. It is not unusual to expect a constant level of performance from the communication network as the load generated by a plurality of attached processors within the system peaks. Traditionally, this demand has been met with backplane networks composed of buses of parallel wires controlled by high speed hardware units. While this interconnection methodology facilitates communication of parallel data at very high speeds, it is inherently unreliable because of the large number of wires and connectors involved in such a network. Moreover, the buses introduce crosstalk noise, signal skew and signal level shifting into the system which degrade its integrity and reliability. Additionally, the bus connectors introduce substantial reliability problems as a result of poor mechanical contacts between the bus and the connection target. These problems are increased by the repeated insertion and removal of connectors and movement of the system. Alternately, the use of long haul networks have proven to be highly reliable. In long haul networks, a data link, such as a wire or optical fiber, which interconnects the communicating systems, facilitates the serial transmission of data packets from one system to another system. Long haul networks generally are characterized by fewer wires or fibers which actually connect the communicating systems, the feasibility of shielding wires to minimize crosstalk noise, reliable serial connectors and the use of sophisticated protocols which facilitate error detection and recovery on serial data links. Conceptually, backplane networks modeled after long haul networks would be an attractive solution to the reliability problem. Unfortunately, the model fails to provide the performance required by closely-coupled systems. Similarly, local area networks, which are capable of comparatively high speed data transmission over short distances, fail to provide the required performance. Local area networks and long haul networks perform the major portion of data transmission or network function with software which is resident in each of a plurality of communicating processors or host. This transmission methodology while adding flexibility to the system severely degrades the effective data transmission rate. In an article "Services and Protocols of the Data Link Layer" by J.W. Conard published in Proceedings of the IEEE, vol. 71, no. 12, December 1983, a method of communicating serially and for recovering data in an asynchronous digital network system is described. This method, however, has some disadvantages with regard to transmission speed and data security, mainly when data packets have to be transmitted from a first node through a second node to a third node. In another article "DNA: The Digital Network Architecture" by S. Wecker, published in IEEE Transactions on Communications, vol. COM-28, no. 4, April 1980, pages 510 to 526, a digital network architecture and details on the interfaces and functions within that structure are described. Here again, special flushing-through are missing, which contribute to an accelerated speed rate in special cases. Therefore it is an object of the invention to provide an improved method of transmitting data packets within a multiprocessor system. It is another object of the invention to provide a method of transmitting a data packet from a sending node to a receiving node only when the receiving node is available. It is still another object of the invention to detect and correct errors during the transmission of data packets. It is a further object of the invention to provide a method of transmitting a data packet from a first node through a second node to a third node without storing the data packet in the second node. These objects features and advantages of the invention are accomplished by the features of the main claim. Further advantageous implementations of the invention are characterized in the subclaims. This invention will be described in greater detail by referring to the accompanying drawings and the description of the best mode that follows. Fig. 1 shows a block diagram of two of a plurality of nodes included in a multiprocessor system. Fig. 2 is a detail schematic of an input port of a node of Fig. 1. Fig. 3 is a detail schematic of an output port of a node of Fig. 1. Fig. 4 shows a format associated with a data acket which is transmitted within the multiprocessor system. Fig. 5 shows a format of a sequence of fields which are transmitted with each data packet transmitted within the multiprocessor system. Figs. 6, 7 and 8 show a state diagram of a data link protocol for transmitting data packets within the multiprocessor system. Figs. 9a and 9b show flow diagrams of the data link protocol for transmitting data packets within the multiprocessor system. Referring to Fig. 1, there is shown two nodes A and B of a plurality of nodes which are a part of a multiprocessor system. Node A includes a plurality of input ports 12 (one shown) which are selectively connected through a switch 14 to a plurality of output ports 16 (one shown) by a router circuit 18. One of the plurality of output ports 16 is coupled to one of a plurality of input ports 20 of node B by a data channel and a control channel. The data channel facilitates the transmission of data packets between various nodes in the system while the control channel is primarily used to acknowledge the reception of the data packets. Signals appearing on the data and the control channels are in the form of serial encoded binary data and clock information. A local processor 22 is coupled to an input and an output of the switch 14 in a parallel manner which enables the local processor to feed data packets to and receive data packets from the multiprocessor system, respectively. Node A also includes a recovery processor 24 having read and write access to all state information in the input port 12, the output port 16 and the router circuit 18. The recovery processor 24 is activated by one of the plurality of input ports 12 or one of the plurality of output ports 16 to perform error recovery and is not otherwise involved in error-free transmission of data packets. Moreover, node A is coupled to upstream and downstream nodes by a plurality of data channels and control channels which are associated with the respective input ports 12 and output ports 16. Node B, which is similar to node A, includes the plurality of input ports 20 (one shown), a plurality of output ports 26 (one shown), a switch 28, a router circuit 30, a local processor 32 and a recovery processor 34. Moreover, node B is coupled to upstream and downstream nodes by a plurality of data channels and control channels which are associated with the respective input ports 20 and output ports 26. Referring to Fig. 2, there is shown a detailed block diagram of one of the plurality of input ports 12 and 20. The input ports 12 and 20 associated with nodes A and B, respectively, are identical circuits. Therefore, only input port 12 will be described in detail. A data channel 33 and a control channel 35 are coupled to a receiver 36. The receiver 36 accepts serial data from the data channel 33, converts the serial data to data in a parallel format and forwards the converted data to a first-in-first-out (FIFO) buffer 38. The receiver 36 also performs analysis on the incoming data packets and sends control information to (1) an output port associated with an upstream node over the control channel 35 and (2) a FIFO controller 40. The FIFO controller 40 and the FIFO buffer 38 facilitate the storage of incoming data packets and the forwarding of the packets through switch 14 to one of the plurality of output ports 16 selected by the router circuit 18. The selected one of the plurality of output ports 16 is coupled to one of the plurality of input ports 12 as result of a destination address which a part of a header which accompany every data packet. Referring to Fig. 3, there is shown a block diagram of one of the plurality of output ports 16 and 26. The output ports 16 and 26 associated with nodes A and B, respectively, are identical circuits. Therefore, only the output port 16 will be described in detail. A transmitter 42 is included in each one of the plurality of output ports 16. The trans mitter 42 receives data in a parallel format from one of the plurality of input ports 12 coupled thereto through the switch 14. Transmitter 42 then converts the data to a serial format and forwards the converted data to a downstream node over a data channel 44. Trans mitter 42 also receives acknowledgement information in a serial form on control channel 46 from the downstream node. The transmitter 42 and the FIFO controller 40 follow a rigorous handshaking procedure during the transmission of a data packet to insure the proper packet transmission. This handshaking procedure is a part of a data link protocol which controls the transmission of data packets within the multiprocessor system. Referring to Fig. 4, there is shown a format associated with each data packet transmitted within the switching system. Each data packet is preceded by a header which includes the following fields: (1) a destination address which the router circuits 18 and 30 use in coupling the proper output ports 14 and 26 to input ports 12 and 20, respectively, sending the data packet; (2) the length of the data packet in bytes; (3) a variable header which allows for various applications to be performed by the switching system; and (4) the body of the packet which is the data portion of the packet. Referring to Fig. 5, there is shown a format of a data packet which will be transmitted from a first node to a second or subsequent node. This format includes, in addition to all of the fields shown in Fig. 4, three additional fields to support a sliding-window protocol and error detection schemes. The additional fields are: (1) start of packet which is a sequence of bits indicating the start of a data packet; (2) sequence number is a plurality of bits which redundantly code a logical zero or a logical one in order to prevent duplicate packets from entering the multiprocessor system; and (3) a checksum field which facilitates the detection of errors in the transmitted data packet. Referring to Fig. 6, there is shown a state diagram of the data link protocol used in transmitting data packets within the multiprocessor system. Referring also to Figs. 1, 2 and 3, for purposes of illustration, let us assume the FIFO buffer associated with node A has a data packet that must be forwarded to the FIFO buffer associated with node B. The goal of the data link protocol is to successfully transfer the data packet. Moreover, in order to simplify the understanding of the state diagram shown in Fig. 6, the following is a list of symbols and acronyms used throughout the analysis of the protocol. Hardware: A Node A, the transmitting node. B Node B, the receiving node. IP n Input port n wherein n is defined as A or B. OP n Output port n wherein n is defined as A or B. D Data channel from OP A to IP B . C Control channel from IP B to OP A . R n The receiver in IP n . T n The transmitter in OP n . FIFO n FIFO buffer in IP n . FIFOC n FIFO controller in IP n . RP n The recovery processor at node n. Channel states: I The data channel idle pattern which is a particular sequence of bits indicating that the data channel is in the idle state. I G The control channel idle-go pattern indicating that IP B is able to accept a transmitted data packet. I W The control channel idle-wait pattern indicating that IP B is presently unable to accept transmission of a data packet. S The data packet labeled with sequence number S wherein S is encoded to represent a logical one or a logical zero. S′ The data packet with sequence number S′ or S wherein S is the logical complement of the current value of S. X Don't care, i.e. any permissible value is acceptable. A S Acknowledgment packet sent by the receiver indicating that the data packet with sequence number S was received correctly. A S Acknowledgment packet sent by the receiver indicating that the data packet with the sequence number S was received correctly. A X Either acknowledgement packet. Q RP-to-RP packet which may be transmitted on either D or C. G No valid data on a channel. G S Erroneous data on channel that is mistaken for a start-of-packet sequence. N A no-clock state of the data or control channels which is characterized by a complete loss of signal information on the given channel. Relevant data packet fields: L The contents of the length field of a data packet. SOP Start-of-packet sequence which distinguishes a data packet or acknowledgement packet from I or I G , respectively. The states in the state diagram are labeled by Sn where n is an integer and the transitions between states are labeled by integers only. Moreover, the sequence number variable S is used in the state diagram to avoid the need to enumerate conditions where S represents a logical zero and where S represents a logical one. Additionally, there are, as a result, cycles in the state diagram where the logical value of S must toggle. The transitions where S must be toggled are labeled with the letter T. Referring to Figs. 1, 2, 3 and 6, the packet switching system begins in state S1 which is a ready/waiting state. T A is transmitting I on D, receiving I G on C and is prepared to transmit a data packet from FIFO A as data packet S. T A indicates the intended transfer by asserting an Available signal to the router circuit 18. In a similar manner, R B is receiving I on D, transmitting I G on C and is ready to accept packet S from T A . Transmission of packet S begins with transition 1 from state S1 to state S2. The router circuit 18 connects FIFO A through switch 14 to T A and asserts a Connected signal to both the FIFOC A and T A indicating that a connection exists therebetween. T A then de-asserts the Available signal which informs the router circuit 18 that OP A is busy with packet transmission. T A also places a SOP sequence and the sequence number S on D and asserts a Send signal on the TransmitterStatus lines. The Send signal instructs the FIFOC A to begin forwarding bytes of the data packet, which is in a parallel format, through switch 14. T A then converts the bytes of the data packet to a serial format, encodes the data packet and transmits the data packet on D. During the forwarding of the data packet by FIFO A , T A continuously monitors whether an Underflow signal is asserted by FIFOC A . The assertion of the Underflow signal indicates that an error in the transmission of the data packet has occurred. When the Underflow signal is asserted by FIFOC A , the protocol takes transition 41 from state S2 to state S6. T A then stops the clock on D and takes transition 9 to an error state S7. R B responds by removing the clock from C. If the Underflow signal was not asserted, upon receiving SOP on D, R B decodes the first incoming byte of data, converts it to a parallel format and forwards the converted data to FIFOC B by asserting the StartOfPacket and Read strobe signals. Subsequent packet bytes are similarly decoded, converted to parallel format and strobed to FIFOC B . R B also extracts the sequence number S and the packet length L from the packet header. As the data packet is received, R B counts down from L, checksums the data packet and verifies that the sequence number S was the one expected. When L equals zero, the last byte of the data packet has been received from T A and transition 2 is taken. R B determines whether the checksum for the data packet is correct, strobes the Read line and forwards the last byte of data to FIFOC B . FIFOC B has then received the entire data packet without error. Simultaneously, when FIFOC A reaches the end of the data packet, it asserts the EndOfPacket signal. T A , in response to the EndOfPacket signal, transmits the last byte of data, starts a WaitingForAcknowledgement timer, and asserts a Wait signal on the TransmitterStatus lines. This places FIFOC A in a wait or suspended state. In transition 3 to state S3, R B encodes, serializes and transmits acknowledgement A S to T A . R B also toggles the value held in FIFOC B which represents the sequence number of the next-expected data packet. Transition 4 is taken from state S3 when FIFO B has room to accommodate a maximum-length data packet and R B has placed I G on C. This restriction exists because once data transmission from IP A through switch 14 to OP A to the adjacent node B has commenced, it can not be stopped and then restarted. Thus, the data link protocol requires the FIFOC B to refuse to accept a new data packet, which potentially is of maximum permissible length, until sufficient room exists in FIFO B . This refusal is represented by an idle-wait (I W ) pattern on the control channel. Additionally, transition 4 represents that T A has (1) received the acknowledgement before the WaitingForAcknowledgement timer expired, (2) verified that the acknowledgement received was for the data packet sent (positive acknowledgement) and then (3) asserted a Release signal on the TransmitterStatus lines. The asserting of the Release signal facilitates the relinquishing of the FIFOC A to do other work. Additionally, T A , in response to the appearance of I G on C, then toggles the sequence number stored in FIFOC A in preparation for sending the next data packet, S , and asserts the Available signal which informs the router circuit 18 that OP A is now available to transmit another data packet. If, as noted above FIFOC B refuses to accept a new data packet, FIFOC B asserts a FIFOFull signal, then transition 5 is taken from state S3 to state S4. The FIFOFull signal indicates that FIFO B has insufficient room therein to accommodate a maximum-length data packet. Subsequently, R B places I W on C. T A , having first seen the correct acknowledgement, A S , followed by I W , asserts Release on the TransmitterStatus lines. This assertion by T A frees FIFO A and starts WaitingForI G timer. Additionally, T A leaves the Available signal unasserted because it is not ready to send another data packet. Transition 6 is taken from state S4 to state S1 when FIFO B has room for the maximum-length data packet and FIFOC B de-asserts the FIFOFull signal. In response to the absence of the FIFOFull signal, R B places I G on C and toggles the sequence number stored in FIFOC B . If T A detects I G on C before the WaitingForI G timer expires, it will toggle the sequence number stored in FIFOC A , assert the Available signal which informs the router circuit 18 that it is free to send another data packet. As noted above, when R B receives the last byte of the data packet S, it checksums the packet in order to determine whether an error has occurred during the transmission thereof. If the checksum indicates that data packet S contained an error, transition 11 is taken from state S1 to state S8 wherein R B sends a negative acknowledgement, A S , to T A on C. Thus, R B is indicating that data packet S was the last packet it received correctly and which T A interprets to mean data packet S, the data packet just transmitted, must not have been received correctly by R B . R B then sends a Flush signal to FIFOC B which results in FIFOC B discarding any remnant of the data packet from FIFO B . Additionally, if FIFOC B has already started forwarding the data packet to an output port in node B, it will assert an Underflow signal which enables the output port to terminate the transmission by forcing the data channel it drives into a N state. Transition 12 from state S8 to an error state S7 implies (1) T A , upon receipt of a negative acknowledgement, always stops the clock on D irrespective of the state of C and (2) R B , upon transmitting a negative acknowledgement, always stops the clock on C irrespective of the state of D. Thus, in transition 12, R B stops the clock on C, signals RP B that an error has occurred and suspends operation. Similarly, T A stops the clock on D, informs RP A of the error, holds the TransmitterStatus lines in the Wait position and suspends operation. It should be noted that whenever the TransmitterStatus lines are in a Wait state, FIFOC A is held in a dormant state, with the data packet T A just attempted to transmit held in tact in FIFO A . The retention of the data packet by the FIFO A allows RP A to request FIFO A to resend the data packet until it is transmitted correctly. As stated above, during transition 2, T A started the WaitingForAcknowledgement timer. If because of some serious error at IP B , the timer expires, transition 13 is taken from state S1 to state S6 wherein T A stops the clock on D, signals RP A that an error has occurred and suspends operation. T A then holds its TransmitterStatus lines in the Wait state thus suspending operation of FIFOC A . R B , if it is able to do so, will then transit to the error state S7 as a result of transition 9. R B encodes and transmits to T A encoded acknowledgement information, I G and I W over C. As a result of the coding of I G and I W , a control channel reception error at OP A may cause T A to believe I W is appearing on C when I G is presently on C. If this condition is left unattended, this would lead to channel deadlock with T A waiting forever for I G which R B believes it is transmitting. To avoid this potential deadlock, the data link protocol requires T A to start the WaitingForI G timer whenever I W is detected. Thus, if the WaitingForI G timer expires prior to the detection of I G on C by T A , transition 46 will be taken from state S4 to state S6 wherein T A stops the clock on D, informs RP A of the error, holds the TransmitterStatus lines in the Wait state and suspends operation. R B will then take transition 9 to the error state S7. In state S2, T A is transmitting data packet S to R B . Before T A completes the transmission of the last byte of data of the packet S, an acknowledgement, A X , is received on C which is indicated by transition 7 to state S5. The presence of A X on C is a result of (1) the length field L erroneously indicated a packet length which is shorter than the actual length of S, hence R B responded with A X before the actual end of the packet S has been encountered or (2) the control channel C suffered a noise burst that looks like a valid acknowledgement. In either case, the protocol requires T A to take transition 8 from state S5 to state S6 whenever an acknowledgement is received prior to the completion of transmission of the data packet. T A stops the clock on D and takes transition 9 to the error state S7 It should be noted that although the transmission of data on D has stopped in state S6, T A must continue to accept data from FIFO A until the EndOfPacket signal is asserted. In essence, once a transmitter has asserted the Send signal, the data link protocol does not allow for an abort process between FIFO A and T A across the switch. R B , which detects the loss of clock information on D, takes transition 9 from state S6 to the error state S7. In transition 9, assuming the packet checksum has failed, R B signals Flush to FIFOC B , stops the clock on C, informs RP B of the error and suspends operation. FIFOC B , in response to the Flush signal, facilitates the discarding of the data packet from FIFO B . Additionally, if FIFOC B has already started forwarding the data packet to an output port in node B, it will assert the Underflow signal which enables the output port to terminate the transmission of the packet by forcing the data channel into the N state. Prior to the transmission of a data packet, T A transmits to R B the SOP, the sequence number and a header. During the transmission of the sequence number, the data channel D corrupts the packet sequence number in such a way that it appears to be its logical complement, S , transition 14 from state S1 to state S9. In state S9, R B determines that it has already received data packet S , discards the data packet as it is received without forwarding the packet to FIFOC B . R B then extracts the length field, L, from the packet only for the purposes of recognizing the end of the packet, transition 15, at which time it sends A S on C, transition 11 indicating that packet S was received correctly. T A interprets A S as a negative acknowledgement and takes transition 12 to the error state as described above. Additionally, while in state S9, a mid-transmission acknowledgement may be received on C, transition 21 to state S10. Thereafter, T A takes transition 22 from state S10 to S6 which is similar to transition 8 from state S5 to state S6. R B then takes the transition 9 from state S6 to the error state S7. In analyzing the state diagram shown in Fig. 6, the sequence of transitions 14, 15, 16, 17, 18, 19 and 20 may result in a loss of one or two packets. The likelihood of this particular sequence occurring is small in view of the fact that following two specific errors must occur sequentially: (1) the packet sequence number must be logically complemented in transition 14 and (2) acknowledgement A S must be logically complemented to A S in transition 16. Briefly, in state S10′, T A incorrectly believes the packet S has been properly received by R B and informs the FIFOC A to release the packet. Thus, the packet S is now lost. Toggling the logical meaning of S and S′ as transition 17 is taken, T A then transmits the next packet in transitions 18 and 19. R B , however, discards the packet as it is received because sequence number of the packet is the logical complement of the sequence number expected. R B subsequently sends A S to T A in transition 20 which falsely indicates that it has received the packet correctly. T A interprets A S to mean the packet just transmitted was received correctly. Hence, two packets are lost. As noted above, the probability of two sequential errors occurring is extremely small. It is possible for a FIFO buffer to run out of data to present to an output port while in the process of transmitting a packet as a result of a failed checksum or a mid-transmission acknowledgement as noted above. Assume that FIFOC A runs out of data to present to OP A because R A has detected a transmission error and has asserted the Flush signal. FIFOC A then asserts the Underflow signal. T A , in response to the Underflow signal, take transition 41 from state S2 or transition 42 from state S9 as stated above. In either case, T A will stop the clock on D, assert Release on the TransmitterStatus lines, inform RP A of the error and suspend operation. R B will take transition 9 to state S7 in a manner which is similar to that discussed above in response to a the mid-transmission acknowledgement. In order to recover, RP A must instruct T A to send a new packet with the same sequence number as the packet which underflowed. RP A will readily determine this fact when it detects that the Underflow signal is asserted in T A ′s state information. This procedure assures that OP A and IP B will not lose synchronism with respect to packet sequence numbers. Retransmission of the failed packet will proceed from the node where the packet was last buffered in its entirety. Referring to state S11, transition sequences 18, 23, 44 and 26, 28, 45 are analogous to transition sequences 1, 41, 9 and 14, 42, 9, respectively. Referring to Fig. 7, there is shown a state diagram which sets forth the transitions and states associated with error in sequence number, invalid data on the channels, false SOP and acknowledgement errors. An error in a sequence number is defined to be any corruption of the sequence number field other than logical complement which was discussed above as a wrong sequence number. The transition sequence begins in state S1 which is the main ready/waiting state. The sequences described hereafter could also occur while the protocol is in state S11 another ready/waiting state. However, in view of the similarity between state S1 and state S11, only references to state S1 will be made. In transition 66, all of the events described for transition 1 in the normal packet transmission occur up to the extraction of the sequence number S by R B . At this point, either R B is able to extract S from the packet but detects an error in the field or R B is totally unable to decipher what is detected on D. This condition is symbolized by G in the forward channel state in state S34. In either case, R B stops the clock on C, informs RP B of the error and suspends operation by taking transition 67 to state S32. Upon detecting loss of clock on C, T A stops the clock on D but must continue accepting data from FIFO A until the EndOfPacket signal is asserted. T A thereafter asserts Wait on the TransmitterStatus lines, signals RP A that an error has occurred and suspends operation by taking transition 62 to state S7. Referring again to state S1, while T A is waiting for an acknowledgement, R B determines that the packet was received without error and sends a positive acknowledgement to T A which is corrupted on C (transition 72). Thereafter, I A will either be able to extract the acknowledgement from C and detect an error in the packet or it will be completely unable to decipher what it saw on C. T A will immediately stop the clock on D, hold the TransmitterStatus lines in the Wait state, inform RP A of the error and suspend operation by taking transition 73 to state S17. R B then takes transition 32 to state S18 by stopping the clock on C, informing RP B of the error and suspending operation. It should be noted that following recovery, T A and R B will be out of synchronism with respect to S. However, the protocol will resolve this problem after one successful resending of the packet. Additionally, while T A is waiting for an acknowledgement in state S1, R B detects an error in the packet as a result of a checksum failure. R B then sends a negative acknowledgement to T A which is corrupted on C (transition 69). Similar action will occur in transition 70 as which occurred in transition 73. Transition 9 is then taken to the error state S7. In this case, synchronism between T A and R B is retained and the protocol will recover on the next transmission of the packet. While in state S1, R B is ready to receive a new packet, a noise burst strikes D such that what appears to be a valid SOP sequence and sequence number S are present on D. R B , will begin to receive what it believes to be a valid packet. The checksum should indicate an error in this packet upon which, in transition 61, R B sends a negative acknowledgement to T A , stops the clock on C, notifies RP B and suspends operation. T A then takes transition 62 to the error state S7. If at this time T A actually had been sending a real packet on D, it must continue accepting data from FIFOC A until the EndOfPacket signal is asserted. Recovery will proceed after the assertion of the EndOfPacket signal. In a similar manner while waiting for an acknowledgement in state S1, noise corrupts the idle pattern on C in such a way that it appears to T A that R B is sending an acknowledgement packet (transition 63). R B will extract the data packet from D. T A upon detecting an error in the acknowledgement or in time between the packet and what is expected to be I W or I G takes transition 64 to state S6 wherein T A stops the clock on D. R B responds in taking transition 9 to the error state S7 wherein the clock on C is removed. Referring to Fig. 8, shown therein is a state diagram which represents the protocol behavior when invalid data is detected on a channel or failure of an analog channel or port occurs. These events may occur at any time and as such are represented by the use of state variables w, x, y and z which are defined as: w The data packet the transmitter will send or is sending on the data channel wherein w represents S, S or Q. x The state or contents of the data channel wherein x represents I, Q, S, S , N, G or G S . y The packet the receiver expects to receive or is receiving on the data channel wherein y represents S, S or Q. z The state or contents of the control channel wherein z represents A S , A S , I G , I W , Q, N, G, or G S . If T A is unable to determine the binary value of an encoded symbol or value taken from C, it takes transition 74 to state S35 and stops the clock on D. T A thereafter informs RP A of an error and suspends operation. However, for example, if T A is in the process of transmitting a packet S to R B and an invalid data symbol is detected on C, T A must continue accepting data from FIFOC A until the EndOfPacket signal is asserted. Thereafter, T A informs RP A of the error, takes transition 75 to state S36 and suspends operation. The recovery process at nodes A and B will be determined by the states of OP A and IP A when the error was detected. Similarly, if R B is unable to determine the binary value of a data symbol detected on D, it takes transition 76 to state S37 and stops the clock on C. Certain actions may have to be performed before R B informs RP B of an error and suspends operation. Using the example immediately above, R B would have to first assert Flush, thereby ensuring that FIFOC B discards any traces of the packet received so far before signalling RP B and halting operation by taking transition 77 to state S36. Failure of any analog component along D, such as the physical channel itself, will result in complete loss of clock and data information on the channel. The sequence of transitions 74 and 75 represents a failure of the data channel and the subsequent transition to the error state. Similarly, failure of any analog component along C will result in complete loss of signal information on the channel. Sequence of transitions 76 and 77 represents this failure. Transition 78 from state S34 to error state S36 represents the events that take place when both channels D and C fail simultaneously. This type of failure may result if D and C were simultaneously disconnected or severed. Referring to Fig. 9a, there is shown a flow diagram of the steps of the protocol involved in transmitting a data packet from node A to node B. Referring also to Figs 1, 2 and 3, the protocol starts in a ready/waiting step 100 which is equivalent to state S1 shown in Fig. 6. Step 102 determines whether T A has asserted the Available signal to the router circuit 18 indicating that it is ready to transmit the data packet. Router circuit 18, in response to the assertion of the Available signal, facilitates the connection of T A through switch 14 to FIFO A in step 103. T A then de-asserts the Available signal, places the SOP sequence and the sequence number S on D and transmits the data packet S to R B in step 104. Additionally, T A starts the WaitingForAcknowledgment timer and asserts Wait on the TransmitterStatus lines. In step 105, a determination is made whether T A received an acknowledgement sequence from R B prior to the expiration of the timer. If the timer has expired in step 105, the protocol enters an error state represented by step 106. Alternately, T A in step 107 determines whether the Underflow signal is asserted by FIFOC A . In response to the assertion of the Underflow signal, the protocol moves into the error state step 106. In step 108, the protocol determines whether the transmission of the packet is complete. If an acknowledgement is received by T A prior to the completion of transmission of the packet, the protocol enters the error state by taking steps 109 and 110. Otherwise, R B determines in step 111 whether the data packet was received correctly. If there was an error in the transmission of the data packet, R B ,as set forth in steps 112, 113 and 106, sends a negative acknowledgement to T A , flushes FIFO B , asserts an Underflow signal if FIFO B is cut through, which facilitates the flushing of any downstream nodes and enters the error state, respectively. However, if the data packet was received correctly, R B , in step 114, sends a positive acknowledgement to T A over the control channel C. R B , in steps 115 and 116, places the I W sequence on C when FIFO B is full. T A , in response thereto, asserts the Release signal and starts an I G timer. If the timer expires prior to the appearance of I G on C, T A in step 117 facilitates node A to enter the error state. If the timer in step 117 did not expire and space becomes available in FIFO B , R B then places I G on C and returns to the ready/waiting step 100. Returning to step 115, if there was space in FIFO B for the entire data packet S, R B complements the sequence number and the protocol returns to the ready/waiting step 100. As stated above, when the protocol enters the error state, a port or channel failure has occurred between nodes A and B. Thereafter, the respective recovery processors 24 and 34 can autonomously determine whether recovery from the failure is possible. For purposes of illustration, let us assume that OP A has entered the error state, RP A must then examine the state information of the port prior to determining the recovery sequence. RP A may determine that transitions 1, 2, 11 and 12, as shown in Fig. 6, has occurred and may simply want T A to re-transmit the data packet to R B . Simultaneously, RP B attempts to recover the channel and may place node B in the ready/waiting state S1. As further stated above, the protocol includes the steps wherein the data packet is transmitted from a first node to a downstream and adjacent node where the packet is stored. The protocol also facilitates the transmission of a data packet from a first node through intermediate node or nodes to a final node without storing the packet in the intermediate node or nodes. This process is called virtual cut-through. Referring to Fig. 9b, there is shown a flow diagram of the virtual cut-through process. For purposes of illustration, let us assume that a data packet is to be transmitted from node A through B to node C. Node C is identical to nodes A and B as shown in Fig. 1. Referring to Figs. 9a and 9b, steps 100 through 107 are also used in the virtual cut-through process. When node B receives the header of the data packet S and determines that the packet is for node C, T B in step 150 asserts an Available signal to the router circuit 30 and FIFOC B then connects FIFO B to T B . Thereafter, T B transmits the packet to R C and starts a WaitingForAcknowledgment timer in steps 152 and 154. The packet S is transmitted to R C without being stored in FIFO B . If the timer expires prior to the receipt of an acknowledgement sequence from R C , T B facilitates the entering of the error state in steps 156 and 178. However, if an underflow signal is asserted by FIFOC B in step 160, FIFO B is flushed in step 162 and FIFO C is flushed in step 167. From step 167, a negative acknowledgement is sent to T B in step 165 and the error state in step 185 is entered. Returning to step 160, T B then determines in step 163 whether transmission of the data packet is complete and whether a premature acknowledgement was received in step 164. If a premature acknowledgement was received, the error state is entered in step 166, otherwise R C determines whether the data packet was received correctly in step 168. R C sends a positive acknowledgement to T B in step 170 if the packet was received correctly. The data packet is stored in FIFO C in step 172 and the sequence number is complemented in step 174. The protocol thereafter returns to the ready/waiting state. If in step 172, FIFO C was full, R C places I W on C, T B asserts the Release signal and starts the I G timer in step 176. If space becomes available prior to the timer expiring, R C complements the sequence number and returns to the ready/waiting state as set forth in steps 178, 180 and 182, respectively. Alternately, if the I G timer has expired prior to the appearance of I G on C, the protocol enters the error state. Thus, the data packet which was transmitted from node A through node B to its destination node C without storing the packet in the intermediate node B. This process can be used to transmit the data packet through any number of nodes to its destination node. It should be noted that if an Underflow signal is asserted by anyone of the intermediate nodes, the node asserting the signal and all downstream nodes flush their buffers and enter the error state. Re-transmission of the data packet will occur by the node where the data packet was last stored in its entirety.

Claims: A method for transmitting a data packet in an asynchronous digital network system having a plurality of nodes (A, B), characterized by the steps of: (a) transmitting a packet of data from a transmitter (16) of a first node (A) to a receiver (20) of a second node (B) over a first data channel (44), and initiating a time (WaitingForAcknowledgement timer) delay at said first node (A); (b) analyzing said data packet for errors at said second node (B); (c) transmitting from said second node (B) to a said first node (A) over a second data channel (control channel 46) an acknowledgement signal (A S ) when said packet is determined to have no errors, and a negative acknowledgement signal (A S ) when said packet is determined to have errors; i) storing each error free packet received at said second node in a FIFO buffer (38); ii) detecting whether said FIFO buffer (38) is full; iii) signalling said first node when said FIFO buffer (38) is full; iv) initiating a second time delay (WaitingFor I G timer) at said first node (A) when said FIFO buffer (38) is full; and, v) asserting a signal on said second channel (46) indicating to said first node (A) that said second node is available to receive traffic if said FIFO buffer (38) is not full before said second time delay expires; d) initiating a retransmission of said packet of data from said first node transmitter to said second node (A) if said acknowledgement signal (AS) has not been determined to be present during said first time delay; and, e) transmitting said packet from said second node (A) to a further node when said packet has been received by said second node (B). The method of claim 1 comprising the step of flushing the contents of said FIFO buffer (38) if an error is determined in a packet received by said second node (B).

Inventors: BLANER BARTHOLOMEW, BLANER, BARTHOLOMEW
Applicants: IBM, INTERNATIONAL BUSINESS MACHINES CORPORATION
Application year: 1987
Main classification: H04L12
IPC codes: H04L12, H04L29, H04L13, H04L1