Patent Publication Number: US-10324491-B2

Title: Reducing time of day latency variation in a multi-processor system

Description:
BACKGROUND 
     1. Technical Field 
     This disclosure generally relates to computer systems, and more specifically relates to multi-processor systems. 
     2. Background Art 
     Multi-processor systems include multiple processors that are interconnected so the processors can work together. Symmetric multiprocessing is one type of multi-processor system that includes multiple processors that share a common operating system and memory. The Power8 processor developed by IBM is an example of a processor that supports symmetric multiprocessing. 
     For multiple processors to work together, the processors must have a common time reference. Time of day (TOD) messages are periodically sent between processors to keep the time of day clocks for all processors synchronized. The time of day messages need to be reliable and have low variation in latency, which is sometimes referred to as TOD jitter. 
     The Power8 processor developed by IBM includes an interconnect known as a fabric that is used to interconnect processors. The fabric interconnecting Power8 processors includes electrical links that provide Error Correction Code (ECC) capabilities that allow correcting errors in the messages. The Power8 processor architecture provides links that have small latency variation, or small TOD jitter. 
     The next generation of the Power family of processors will have electrical links that can perform a cyclic redundancy check (CRC) replay if bit errors exist in the message. A CRC replay will cause latency variation, or jitter, in the TOD. Without a way to reduce TOD jitter in a multi-processor system, the TOD jitter will be excessive. 
     BRIEF SUMMARY 
     A time of day (TOD) synchronization mechanism in a first processor transmits a latency measure message simultaneously on two links to a second processor. In response, the receiver in the second processor detects latency differential between the two links, detects the delay in the second processor, and sends the latency differential and delay to the first processor on one of the two links. The first processor stores TOD delay values in the two links that account for the latency differential between the two links. When a TOD message needs to be sent, a link loads a counter with its stored TOD delay value, then decrements the counter until the TOD message is ready to be sent. The resulting counter value is the receiver delay value, which is transmitted to the receiver as data in the TOD message. Because the link delay values account for the latency differential between the two links, the TOD jitter between the two links is reduced. 
     The foregoing and other features and advantages will be apparent from the following more particular description, as illustrated in the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S) 
       The disclosure will be described in conjunction with the appended drawings, where like designations denote like elements, and: 
         FIG. 1  is a block diagram showing multiple processors interconnected in a multiprocessing system; 
         FIG. 2  is a block diagram showing transceivers on the processors in  FIG. 1  that allow the processors to communicate with each other; 
         FIG. 3  is a block diagram showing a sample format of a serial data packet that could be sent between processors in  FIG. 1 ; 
         FIG. 4  is a table showing examples of service packets that could be sent between processors in  FIG. 1  in the format shown in  FIG. 3 ; 
         FIG. 5  is a block diagram showing a 64/66 encoded message; 
         FIG. 6  is a block diagram of a data block for the 64/66 encoded message shown in  FIG. 5 ; 
         FIG. 7  is a block diagram of a control block for the 64/66 encoded message shown in  FIG. 5 ; 
         FIG. 8  is a table showing examples of 64/66 control blocks that could be sent between processors in  FIG. 1 ; 
         FIG. 9  is a flow diagram of a method for determining TOD delay for each of the two links shown in  FIG. 2 ; 
         FIG. 10  is a flow diagram of a method for sending a value for a number of cycles of the receiver should delay TOD packets to reduce TOD jitter; 
         FIG. 11  is a flow diagram of a method for electrical links between processors to process TOD packets; and 
         FIG. 12  is a flow diagram of a method for optical links between processors to process TOD packets. 
     
    
    
     DETAILED DESCRIPTION 
     A time of day (TOD) synchronization mechanism in a first processor transmits a latency measure message simultaneously on two links to a second processor. In response, the receiver in the second processor detects latency differential between the two links, detects the delay in the second processor, and sends the latency differential and delay to the first processor on one of the two links. The first processor stores TOD delay values in the two links that account for the latency differential between the two links. When a TOD message needs to be sent, a link loads a counter with its stored TOD delay value, then decrements the counter until the TOD message is ready to be sent. The resulting counter value is the receiver delay value, which is transmitted to the receiver as data in the TOD message. Because the link delay values account for the latency differential between the two links, the TOD jitter between the two links is reduced. 
     Referring to  FIG. 1 , a multi-processor system  100  includes multiple processors  110 A,  110 B, . . .  110 N interconnected with a communication fabric  130 . Each processor includes a time of day (TOD) synchronization mechanism. Thus, processor A  110 A includes a TOD synchronization mechanism  120 A; processor B  110 B includes a TOD synchronization mechanism  120 B, through processor N  110 N, which includes a TOD synchronization mechanism  120 N. The TOD synchronization mechanisms synchronize the time of day between processors. The communication fabric  130  can include multiple different communication channels. In the most preferred implementation, communication fabric  130  includes electrical links that communicate in one particular way, and optical links that communicate in a different way. The Power9 processor developed by IBM has electrical links and optical links in the communication fabric that interconnect processors. Note the optical links in the Power9 processor architecture do not actually process optical signals, but are electrical interfaces that process electrical signals. The terms “electrical links” and “optical links” are well-known in Power9 terminology, and are used herein to distinguish between two distinct electronic interfaces in the communication fabric  130  between processors that use different communication protocols without implying these interfaces have the same characteristics, properties or protocols as the Power9 processor architecture. 
     The communication fabric  130  preferably includes multiple bit lanes organized by protocol layer. The electrical links preferably define fifteen lanes, plus one spare lane. A packet on the electrical link is 30 bytes. The optical links preferably define ten bytes that correspond to ten lanes, plus two spare lanes. 
       FIG. 2  shows transceivers  200  that illustrate part of the communication fabric  130 . For this particular example, we assume Transceiver A  210 A is in processor A  110 A shown in  FIG. 1 , and Transceiver B  210 B is in processor B  110 B in  FIG. 1 .  FIG. 2  thus shows links that interconnect processors  110 A and  110 B in  FIG. 1  via the communication fabric  130 . Transceiver A  210 A includes a first link  220 A with a corresponding transmitter  222 A and receiver  224 A, and a second link  230 A with a corresponding transmitter  232 A and receiver  234 A. Transceiver B  210 B has a similar configuration, with a first link  220 B that includes a transmitter  222 B and a corresponding receiver  224 B, and a second link  230 B that includes a transmitter  232 B and a corresponding receiver  234 B. The transmitters and receivers are interconnected via the communication fabric  130  so transmitters are connected to receivers and receivers are connected to transmitters, as shown in  FIG. 2 . 
     Each transceiver  210 A and  210 B includes a corresponding TOD constant  212 A and  212 B. The TOD constants  212 A and  212 B are determined as explained below with reference to  FIG. 9 . Each link has a corresponding TOD delay that is determined by a latency measure interaction with the other transceiver, as described in more detail below with respect to  FIG. 9 . Thus, link  1   220 A in Transceiver A  210 A includes a TOD delay register  226 A; link  1   220 B in Transceiver B  210 B includes a TOD delay register  226 B; link  2   230 A in Transceiver A  210 A includes a TOD delay register  236 A; and link  2   230 B in Transceiver B  210 B includes a TOD delay  236 B register. The value stored in the TOD delay register for a link is used to derive a value for a receiver delay value transmitted in each TOD packet or TOD control block transmitted on that link. 
     In one specific implementation, the communication fabric  130  may include both electrical links and optical links. Transceivers  200  shown in  FIG. 2  are representative of both the electrical links and the optical links. 
     Referring to  FIG. 3 , a sample packet format for a serial packet is shown. The serial packet shown in  FIG. 3  could define packets, for example, that are used to communicate on the electrical links between processors. The sample packet format in  FIG. 3  includes an ACK bit at bit  0 , followed by a link bit at bit  1  that provides the logical link number, followed by a replay bit at bit  2  that indicates when a packet contains delayed data due to a replay, a nine bit sequence number at bits  3 - 11 , followed by 192 bits (24 bytes) for data at bits  12 - 203 , and 36 bits at bits  204 - 239  for a cyclic redundancy check (CRC). The sequence number in bits  3  through  11  can include coding that specifies types of service packets. In one particular implementation, when bits  3 - 11  are all ones, this indicates a service packet, and the first byte of data indicates the type of service packet. Referring to  FIG. 4 , two suitable types of service packets include a time of day (TOD) packet and a latency measure packet. Other service packets not shown in  FIG. 4  could also be defined, as needed. A packet is a TOD packet when the sequence number in bits  3 - 11  is all ones and the first byte of the data field has a value of 0x55. A packet is a latency measure packet when the sequence number in bits  3 - 11  is all ones and the first byte of the data field has a value of 0xB0. 
     Referring to  FIGS. 5-7 , a sample packet format for a serial block is shown. Such blocks could be used, for example, to communicate on the optical links between processors. The serial block shown in  FIGS. 5-7  are in 64/66 encoding, where the first two bits distinguish between data block and control blocks. Data blocks have a value of 01 in the first two bits, as shown in  FIG. 6 , followed by 64 bits (8 bytes) of data. Control blocks have a value of 10 in the first two bits, as shown in  FIG. 7 , followed by an eight bit field that specifies the type of control block, and 56 bits (7 bytes) that can be empty or can contain any suitable control information, data, or any suitable mixture of the two. Two types of control blocks that are used for TOD operations are shown in  FIG. 8  to include a TOD control block and a latency measure control block. A control block is a TOD control block when the value 0x78 is in the type field shown in  FIG. 7 . Similarly, a control block is a latency measure control block when the value 0xB4 is in the type field shown in  FIG. 7 . 
     Referring to  FIG. 9 , a method  900  shows how a processor may determine an TOD delay values for each link to keep TOD between processors in sync. This example assumes two processors P 1  and P 2  are interconnected. Method  900  is preferably performed by the TOD synchronization mechanisms (see  FIG. 1 ) in the respective processors P 1  and P 2 . P 1  sends a latency measure packet (for an electrical link) or control block (for an optical link) on both links (step  910 ). For the discussion of method  900  in  FIG. 9 , we use the term “latency measure message” as a general term that includes a latency measure packet if the communication takes place on electrical links, and that includes a latency measure control block if the communication takes place on optical links. Assuming Transceiver A  210 A in  FIG. 2  corresponds to P 1  in  FIG. 1  and Transceiver B  210 B in  FIG. 2  corresponds to P 2  in  FIG. 1 , Transceiver A  210 A sends the same latency measure packet or control block on both transmitters  222 A and  232 A in the two links  220 A and  230 A. The P 2  receiver detects any latency differential between the two links, and also keeps a count of the delay in P 2  for processing the latency measure packet or control block (step  920 ). P 2  then sends the latency differential and delay in P 2  to P 1  on the link that had the longer latency in sending the original message (step  930 ). Note the selection of the link that had the longer latency is a simple design choice, and the link that had the shorter latency could likewise be used. Again referring to  FIG. 2 , if the latency measure packet or control block was received first in the receiver  224 B in link  1   220 B, then was received in the second receiver  234 B in link  2   230 B, this means P 2  will send the latency differential and delay in P 2  via the transmitter  232 B in link  2   230 B. Note the latency differential and delay in P 2  can be included in the data portion of a latency measure packet as shown in  FIG. 3 , and in the 56 bits of control or data in the control block shown in  FIG. 7 . 
     When P 1  receives the message from P 2  with the latency differential between links and the delay in P 2 , P 1  determines the one-way latency from the total latency (step  940 ). For example, the one-way latency could be computed by taking the total latency, subtracting the delay in P 2 , and dividing by two. P 1  then reads a TOD constant (step  950 ). The TOD constant is preferably large enough to cover the worst case delay in sending the TOD message. In the most preferred implementation, the TOD constant is a hard-coded value. The TOD constant is then stored as the TOD delay for the link with the longer latency (step  960 ). The TOD constant is added to the latency differential between links and the resulting value is stored as the TOD delay for the link with the shorter latency (step  970 ). The TOD delay values thus have different values that account for the different latencies of the transmitters in the links, which results in the receiver receiving the same receiver delay value in TOD packets/control blocks regardless of which link is used to send the TOD packets/control blocks. 
     Method  1000  in  FIG. 10  shows how a link uses its corresponding stored TOD delay value. Method  1000  determines when a TOD packet or control block needs to be sent on a link (step  1010 ). Note that method  1000  applies to both links, so the terminology LinkX is used to denote that one of the two links has been selected. When no TOD packet or control block needs to be sent (step  1010 =NO), method  1000  loops back to step  1010  until a TOD packet or control block needs to be sent (step  1010 =YES). For this example, we assume Link  1  needs to send a TOD packet or control block (step  1010 =YES) Link  1  then loads a counter with the value stored in its corresponding TOD delay (step  1020 ). Thus, referring to  FIG. 2 , Link  1  reads the value from the TOD Delay register  226 A and loads a counter with that value. The counter is decremented until the TOD packet or control block is ready to be sent (step  1030 ). The value of the counter is then included in the TOD packet or control block sent to P 2  as the receiver delay value (step  1040 ). In one suitable implementation, the receiver delay value is a seven bit value. Note the difference in the values of the stored TOD delays is preferably the same as the latency differential detected in step  920  and sent to P 1  in step  930  in  FIG. 9 . Thus, if the differential between Link  1  and Link  2  is two cycles, the longer link will have one value stored as its TOD delay, and the shorter link will have the one value plus two stored as its TOD delay. The result is that the receiver should receive packets or control blocks that have a consistent receiver delay value regardless of which link was used to send the TOD packet or control block. By including the receiver delay value in all TOD packets/control blocks between P 1  and P 2 , the latency variation, or jitter, in the TOD is reduced. 
     Referring to  FIG. 11 , a method  1100  shows how to process TOD packets. Method  1100  could apply, for example, to the electrical links in the communication fabric  130 . P 1  sends a TOD packet to P 2  (step  1110 ). The data in the TOD packet includes the receiver delay value discussed above with reference to step  1040  in  FIG. 10 . As shown in  FIG. 3 , a TOD packet includes a CRC. When the CRC is correct (step  1120 =YES), the TOD packet is accepted and processed (step  1130 ). When the CRC is not correct (step  1120 =NO), error correction codes (ECC) are applied (step  1140 ) in an attempt to correct errors in the TOD packet. When the CRC is correct after applying the ECC (step  1150 =YES), the TOD packet is accepted and processed (step  1130 ). When the CRC is still incorrect after applying the ECC (step  1150 =NO), the TOD packet is rejected (step  1160 ). Method  1100  is then done. 
     Referring to  FIG. 12 , a method  1200  shows how to process TOD control blocks. Method  1200  could apply, for example, to the optical links in the communication fabric  130 . P 1  sends a TOD control block to P 2  on all lanes (step  1210 ). Assuming the optical links have ten lanes, this means the TOD control block is sent by P 1  to P 2  on all ten lanes. When P 2  sees the TOD control block on a majority of the lanes (step  1220 =YES), the TOD control block is accepted and processed (step  1230 ). When P 2  does not see the TOD control block on the majority of the lanes (step  1220 =NO), the TOD control block is rejected (step  1240 ). Method  1200  is then done. 
     A time of day (TOD) synchronization mechanism in a first processor transmits a latency measure message simultaneously on two links to a second processor. In response, the receiver in the second processor detects latency differential between the two links, detects the delay in the second processor, and sends the latency differential and delay to the first processor on one of the two links. The first processor stores TOD delay values in the two links that account for the latency differential between the two links. When a TOD message needs to be sent, a link loads a counter with its stored TOD delay value, then decrements the counter until the TOD message is ready to be sent. The resulting counter value is the receiver delay value, which is transmitted to the receiver as data in the TOD message. Because the link delay values account for the latency differential between the two links, the TOD jitter between the two links is reduced. 
     One skilled in the art will appreciate that many variations are possible within the scope of the claims. Thus, while the disclosure is particularly shown and described above, it will be understood by those skilled in the art that these and other changes in form and details may be made therein without departing from the spirit and scope of the claims.