Abstract:
A system includes at least two IEEE 1588 nodes. The nodes are connected by two paths: a low latency communication path and a high bandwidth path. The clocks within the nodes are synchronized to one another at a network interface. Data and timing are transmitted on the high bandwidth path while critical timing signals are passed on the low latency communication path thereby improving timing synchronization.

Description:
BACKGROUND 
       [0001]    To accommodate the IEEE 1588 standard for the Precision Time Protocol (PTP), it is convenient to monitor the Media Independent Interface (MII) for 10 and 100 BASE-T interfaces. The digital signals are interpreted to indicate a marker point, e.g. timestamp, for the departure and arrival of the Ethernet Frames, as they pass from the Media Access Layer (MAC) to the Physical Layer (PHY). 
         [0002]    For 1000 BASE-T interfaces, the equivalent interface is the Gigabit Media Independent Interface (GMII). The trend in silicon fabrication is to integrate the MAC and PHY functions into the same device. This has created a difficulty for 1588 implementers who want to monitor these signals in a 1000 BASE-T environment since the GMII interface is not exposed. 
         [0003]    Timing inaccuracies are introduced by the delays, delay asymmetries, and jitter from the following network related sources: PHYs, cables, and network devices, e.g. routers, switches, boundary clocks, and transparent clocks. 
       SUMMARY 
       [0004]    A system includes at least two IEEE 1588 nodes. The nodes are connected by two paths: a low latency communication path and a high bandwidth path, e.g. Ethernet. The clocks within the nodes are synchronized to one another at a network interface. Data and timing are transmitted on the high bandwidth path while critical timing signals are passed on the low latency communication path thereby improving timing synchronization. 
         [0005]    The high bandwidth path is selected from a group including Ethernet and PCI Express. The low-latency communication path is selected from a group includes a trigger wire, coaxial cable, LXI-Trigger Bus, RF signal, optical signal, optical signal in a fiber, trace on a back-plane, signal on a ground plane, and signal on a power supply bus. 
     
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]      FIG. 1  illustrates the basic IEEE-1588 operation with the master device issuing SYNC packets where timestamps TS 1  and TS 2  are collected (prior art). 
           [0007]      FIG. 2  illustrates an embodiment of the invention. 
           [0008]      FIG. 3  illustrates a LXI Class A Trigger Bus used as the out-of-band communication path. 
           [0009]      FIG. 4  illustrates PCI-Express architecture used as the out-of-band communication. 
           [0010]      FIG. 5  illustrates the multiple-lane nature of PCI-Express. 
       
    
    
     DETAILED DESCRIPTION 
       [0011]    The inventive concept to improve synchronization is for both ends of a communication path generate a timestamp for a common event. These timestamps that the master device collects must be transmitted to the slave device for additional processing. As an illustrative example, the concept will be described in the following figures using IEEE-1588 SYNC and DELAY_REQ packets. 
         [0012]      FIG. 1  illustrates the basic IEEE-1588 operation with the master device issuing SYNC packets where timestamps TS 1  and TS 2  are collected. The slave device issues DELAY_REQ packets where timestamps TS 3  and TS 4  are collected. The IEEE-1588 standard may be found at the http://ieee1588.nist.gov website. 
         [0013]      FIG. 2  illustrates an embodiment of the invention. Both the master and slave devices  10 A,  10 B are bidirectionally connected by the normal network path  24 , e.g. Ethernet, as well as the out-of-band communication path or low latency communication path  26 . 
         [0014]    Each device includes a processor with Media Access Control (MAC)  12 A,  12 B. The MAC  12 A,  12 B is bidirectionally communicates with the physical layer (PHY)  14 A,  14 B using a MII Bus  16 A,  16 B. The PHY  14 A,  14 B bidirectionally communicates with a connector, e.g. RJ45  18 A,  18 B. A transmit pattern detector  20 A,  20 B generates transmit detect (TX_DET) and transmit start-of-frame (TX_SOF) signals by monitoring the MII Bus  16 . A timestamp input circuit  22 A,  22 B is further included. 
         [0015]    In one embodiment, FPGA signals that exist in the current implementation are routed to the low-latency/low jitter communication path. This simplifies the design and allows the system to degrade back to standard IEEE-1588 communications when the low-latency/low-jitter communication path is unavailable. 
         [0016]    In operation, the “forward” path (shown as a solid line) carries the SYNC signal (the TX_SOF signal of the master device) from the master device to the slave device. The timestamp of input circuit of the slave device is used to capture the time-of-arrival of this trigger. This timestamp is an alternate to timestamp TS 2  (SYNC receive time). 
         [0017]    The “reverse” path (shown as a solid line) carries the DELAY_REQ signal (the TX_SOF signal of the slave device) from the slave device to the master device. Similarly, the timestamp of the input circuit of the master device is used to capture the time-of-arrival of this trigger. This timestamp is an alternative to timestamp TS 4  (DELAY_REQ receive time). 
         [0018]    As the same signal is used on the sending side as in standard IEEE-1588 communication, the timestamp TS 1  (SYNC send time) and timestamp TS 3  (DELAY_REQ send time) can be used without any additional processing. 
         [0019]    Thus, the PHY delay has been calibrated and the MII time-stamp has been adjusted accordingly. For TX packets, the time-stamp is always later to account for the PHY delay in the outbound direction. The timestamp reported in the FOLLOW_UP packet is only correct for slave devices that are time-stamping at their network interfaces. To further illustrate, when the PHY to the “out-of-band” connector delay is different from the FPGA to the “out-of-band” connector delay, the master device sends two FOLLOW_UP packets. One contains the PHY corrected time_stamp and the other contains the “out-of-band” corrected time-stamp. Alternatively, the two time-stamps may be in FOLLOW_UP message. 
         [0020]    To address “false positives”, the monitoring circuit of the device receiving the timestamp generates the TX_SOF (transmit start of frame) on any network traffic. Only later as the packet detector has processed the necessary header bytes does it generate the TX_DET (transmit detect) signal. The TX_SOF signal leads the TX_DET signal by numerous byte times. On 100-Base-T Ethernet Frame technology, this delay equals 36 μsec and corresponds to 45 bytes from the first byte of the packet (after the preamble) to the end of the IEEE-1588 version byte. 
         [0021]    When the solid lined “forward” and “reverse” paths are used, the receiving end is triggered on each network packet transmitted by the remote end. As the packet also follows the network path, a second timestamp is normally collected as the RX detector operates (not shown). Comparing these two timestamps, the receiver software can reject such “false positives”. 
         [0022]    When the dashed lined “forward” and “reverse” paths are used, there are no “false positives” as the signal is generated only upon a valid pattern match by the sender. However, the signal is “late” by the same delay, e.g. 36 μsec for 100-Base-T Ethernet Frame technology, discussed above. The receiver needs to be configured to subtract out this constant value from the timestamp. 
         [0023]    For SYNC packets, the slave device uses the timestamp resulting from the forward path rather than the timestamp from its RX detector. The master device requires no changes. The master device generates the same FOLLOW_UP packet as before (containing TS 1 ). 
         [0024]    For DELAY_REQ packets, the slave device behaves unaltered. When it sends the DELAY_REQ packet, its TX detector automatically generates the correct signal that traverses the “reverse” path back to the master device. The master device is configured to return the timestamp resulting, from the “reverse” path in the DELAY_RESP message. 
         [0025]    Suitable low-latency paths include a trigger wire, coaxial cable, LXI-Trigger Bus, RF signal, optical signal, optical signal in a fiber, trace on a back-plane, signal on a ground plane, and signal on a power supply bus. 
         [0026]      FIG. 3  illustrates a LXI Class A Trigger Bus used as the out-of-band communication path  26 . 
         [0027]    The LXI Trigger bus  26  has 8 low voltage differential signal (LVDS) pairs. The embodiment uses two of those pairs. One pair is used for SYNC “start-of-frame” signals in the “master-&gt;slave” direction. The other pair is used for reverse DELAY_REQ “start-of-frame” signals in the “slave-&gt;master” direction. 
         [0028]    To accommodate “false positives”, since the network packet arrives later over the Ethernet path, one approach is to associate the two timestamps. Thus, any timestamps resulting from non-IEEE-1588 event packets are discarded. 
         [0029]    In LXI Class A, the trigger bus  26  extends from device-to-device in a daisy chain fashion. To not run out of trigger lines in the reverse direction, the trigger line is time-shared. All slave devices connected to the same daisy chain are configured to send delay requests so that they are spaced out in time. During idle periods, they would tri-state that trigger line, e.g. wired or mode. 
         [0030]    As each slave device  10 B sends their DELAY_REQ, a pulse is automatically sent down the shared trigger line. The hardware on the slave side must: 1) automatically “un-tristate”, 2) send the pulse, 3) delay for a few μsec, and 4) re-tristate. The master device would see the edge and timestamp it. It is critical that the master device receive the DELAY_REQ packet over the network before the next timestamp comes down the trigger line. The DELAY_REQ signals from each slave must be sufficiently spaced apart in time such that the master device is not confused. 
         [0031]    Alternatively, the master device  10 A can participate in the time-share scheme. The scheduling of SYNC and DELAY_REQ messages must be spread out in time to avoid confusion. This is easily done as the Ethernet path may be used to achieve synchronization to better than 1 μsec so all devices have a coherent sense of time. 
         [0032]    The inventive concept can be used to synchronize over the network similar to normal IEEE 1588 PTP while using the “out of band” path to calibrate out the IEEE1588 network path errors. To illustrate, the time-stamps from the “out-of-band” path may be used to measure end-to-end asymmetries. The network path for IEEE1588 is used when the test system is in operation thus freeing up all the trigger lines in the LXI trigger bus. At start-up and during calibration, the trigger lines can be switched into a mode such that the IEEE1588 SW measures all the delays, jitter, and asymmetries from the network path. Once calibrated, the LXI trigger bus lines are “released” for use for traditional test system triggering. Other configurations that have trigger lines, e.g. VXI or PXI, may operate similarly. 
         [0033]      FIG. 4  illustrates PCI-Express architecture used as the out-of-band communication  26 . In PCI-Express, the interconnection scheme between the devices mimics a computer network. At its core is the addition of a switch integrated circuit (IC) into the center of the PC architecture. 
         [0034]    The “end point” devices and the host CPU must be synchronized with each other. For purposes of discussion, the “end point” devices are measurement instruments that need tight synchronization. The “switch” needs to be modified similar to an IEEE-1588 boundary clock. 
         [0035]    To allow for synchronization, the “end points” and the “switch” must recognize the “start-of-frame” signals. The “multiple lane” nature of PCI Express is shown in  FIG. 5 . 
         [0036]    A data packet, e.g. SYNC packet, gets divided into bytes that flow down different lanes. Inside the receiving device, these lines are re-assembled back into a packet. If timestamps are generated at that point, the delays and jitter introduced by the disassembly and the reassembly can be significant. 
         [0037]    In the present invention, the byte stream directly inside the PCI Express interface application specific integrated circuit (ASIC) is monitored. Data rates are extremely fast, e.g. 400 psec bit times, 2.5 Gbps per lane. A packet recognizer is needed to collect timestamps for critical IEEE-1588-like synchronization packets. The equivalent of SYNC and DELAY_REQ packets must flow in both directions to allow for synchronization and removal of any-offset bias. Similar to standard IEEE-1588, the symmetry assumption is needed unless the system can be calibrated and asymmetries measured. 
         [0038]    While the inventive concept has been described using IEEE-1588 SYNC and DELAY_REQ packets, improved timing synchronization may be achieved by the master device sending a periodic signal e.g. 1 pulse/second as the “event” that the slave device time-stamps. 
         [0039]    Alternatively, the “event” can come from a source, e.g. a remote television station, outside the system. Both the master and slave devices “see” the event and thereby timestamp the event. Appropriate messages are sent between the devices so that the slave can adjust its clock appropriately.