Abstract:
A node of a real-time scheduled packet network synchronizes an internal frequency reference to a master node by receiving first heartbeat packets that are transmitted by the master node at a predetermined frequency. The node calculates a link phase of a link over which the first heartbeat packets are received by subtracting a time at which the node transmitted a second heartbeat packet to another node from a time at which the node received a first heartbeat packet.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of U.S. Provisional Application No. 60/490,609 titled “Systems and Methods for the Synchronization of a Real-Time Scheduled Packet Network Using Relative Timing,” filed Jul. 28, 2003. 
    
    
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
     (Not applicable) 
     BACKGROUND OF THE INVENTION 
     The present invention relates generally to the implementation of a mechanism to synchronize a Real-Time Scheduled Packet Network. A Real-Time Scheduled Packet Network requires precise synchronization to ensure that packets transmitted at a precise time arrive at the next receiver at a predetermined time. 
     Many types of networks require synchronization; most require that the timing in each of the network elements is frequency-synchronized, but Real-Time Scheduled Packet Networks require both frequency synchronization and phase information. 
     BRIEF SUMMARY OF THE INVENTION 
     The present invention uses a novel approach to achieve phase alignment. Rather than requiring Global Positioning System (GPS) receivers in each network element, or by measuring propagation delay of each link, the present invention uses periodic synchronization packets (called Heartbeats) to synchronize transmitters and receivers in the network. These Heartbeats do not contain any timestamps. Furthermore, the timing sources in each network element do not need to be aligned in phase. Instead, the relative phase of each link is determined. This relative phase is called “Link Phase.” 
     The present invention allows any link to have asymmetric propagation delay, that is, the propagation delay from Point A to Point B can be significantly different from the propagation delay from Point B to Point A. Asymmetric propagation delay is common in certain types of transmission topologies, such as unidirectional path switched rings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       These and other features, advantages, aspects and embodiments of the present invention will become more apparent to those skilled in the art from the following detailed description of an embodiment of the present invention when taken with reference to the accompanying drawings, in which: 
         FIG. 1  is a block diagram showing an example of a heartbeat distribution tree having one Master Scheduled Switche/Router (SSR), in accordance with an illustrative embodiment of the disclosed system; 
         FIG. 2  is a block diagram showing an example of a heartbeat distribution tree having two Master SSRs, in accordance with an illustrative embodiment of the disclosed system; 
         FIG. 3  illustrates calculation of Link Phase for an illustrative embodiment of the present invention; 
         FIG. 4  illustrates operation of the disclosed invention in a network including three SSRs, all interconnected with direct connections; 
         FIG. 5  is a simplified flow chart showing operation of an illustrative embodiment of the present invention; and 
         FIG. 6  illustrates application of a calculated Link Phase to a packet schedule. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     U.S. Provisional Patent Application No. 60/490,609, titled “Systems and Methods for the Synchronization of a Real-Time Scheduled Packet Network Using Relative Timing,” filed Jul. 28, 2003, and U.S. patent application Ser. No. 09/746,744, titled “Scheduled Network Packet Switch,” filed Dec. 22, 2000, are hereby incorporated by reference herein. 
     The present invention provides techniques, in which the heartbeats of switches are not adjusted in phase. In accordance with the presently disclosed techniques, Link Phase provides a per-link adjustment in this regard. 
     The present disclosure has two main parts: Frequency Synchronization and Link Phase Calculation. 
     1. Frequency Synchronization 
     All Scheduled Switch/Routers (SSRs) and Scheduled End Points (SEPs) (collectively referred to herein as “nodes”) in a scheduled network are synchronized in frequency by the distribution of synchronization packets called Heartbeats. Frequency synchronization is used to overcome the variation of individual frequency references in each SSR and SEP. All individual frequency references are synchronized to the same frequency. Frequency synchronization is based on the frequency of arriving heartbeats, which is an inverse of the time period between heartbeats. 
     At least one SSR is selected to be a “Master” SSR. The SEPs and the other SSRs are synchronized, in frequency, to the Master SSR(s). A “Heartbeat tree” is created for each Master SSR. A Heartbeat tree is the set of paths that distributes Heartbeats from each Master SSR to all SEPs and the other SSRs. The Master SSR forms a root of a Heartbeat tree.  FIG. 1  shows an example of a scheduled network with a single Master SSR  100 , non-master SSRs  102 ,  104 ,  106  and  108 , SEPs  110 ,  112  and  114 , an optional external clock  130  and a heartbeat tree  120 . A single Master SSR can be synchronized to an external clock  130 , such as a GPS (Global Positioning System) or a BITS (Building Integrated Timing System), or it can simply run off its internal timing source. 
     The structure of the Heartbeat tree (i.e., its branches) are determined by the proximity (number of hops) of each SSR and each SEP to the Master SSR. The path to each SSR or SEP from the Master should comprise a minimum number of hops. On the tree, Heartbeats flow from the Master SSR to each SSR and SEP, synchronizing all the SSRs and SEPs to the Master SSR frequency reference, based on the frequency of arriving Heartbeats, i.e. based on the time period between Heartbeats. Although Heartbeats are sent on every scheduled port, only Heartbeats on the tree are used to synchronize SSRs and SEPs. Non-Master SSRs and SEPs synchronize the frequency of their internal frequency references to the incoming Heartbeats received on the designated port of the Heartbeat tree. The timing references of non-Master SSRs and SEPs are allowed to have any phase relative to the Master SSR, but once frequency-synchronized, this relative phase is locked to the Master SSR phase (i.e., the relative phase is constant). This frequency synchronization can be done using a variety of feedback mechanisms, such as digital phase-locked loops. Other mechanisms are well within the capability of those skilled in the art. 
     It is possible to designate more than one SSR as a Master for Heartbeat distribution. Multiple non-intersecting Heartbeat trees are determined, with a Master as the root of each tree. Since each Master, and its tree, operates independently, each Master should be synchronized to an external timing reference derived from a highly stable common source, such as a GPS or BITS. An example network diagram with two Master SSRs is shown in  FIG. 2 , including master SSRs  200  and  206 , non-master SSRs  202 ,  204  and  208 , SEPs  210 ,  212  and  214 , required external clocks  230  and  232 , and heartbeat trees  220  and  222 . 
     If any link of the Heartbeat tree fails, the SSR or SEP that was deriving Heartbeat timing from a port connected to that link transitions to a holdover mode. Downstream SSRs or SEPs continue to derive Heartbeat timing from their respective branches of the tree. The Heartbeat tree is modified as soon as possible to use other operational links, to ensure all nodes stay frequency-synchronized to a Master SSR. 
     Each SSR or SEP frequency reference has an arbitrary phase offset from the master SSR frequency reference, even though the frequencies of each reference are synchronized. There is no need to phase-align the timing references in each SSR or SEP. Instead, a phase calculation is determined on each link between SSRs or SEPs (in each direction). 
     2. Link Phase Calculation 
     The final synchronization step is to determine the phase of each scheduled link (the Link Phase). The determined Link Phase is the relative displacement of heartbeat packet arrival times at specific input ports. Once the Link Phase is known, the packet arrival times on any link from Point A to Point B can be calculated using the transmission time at Point A. 
     As previously noted, once Frequency Synchronization has been achieved, each SSR and each SEP independently generates heartbeat packets at the frequency determined during Frequency Synchronization. Accordingly, received heartbeat packets are not forwarded. Specifically, each SSR and each SEP transmits these Heartbeats on all scheduled ports in accordance with the frequency determined during Frequency Synchronization. A scheduled port is a port that: 1) supports scheduled packets and 2) has been determined to be connected to another port that supports scheduled packets. Heartbeats are transmitted on all ports of a node at the same time. 
     The Link Phase of a Link from Point A to Point B is calculated by the receiver (an SSR or SEP) at point B and is equal to the arrival time of Point A&#39;s Heartbeat minus the time of Point B&#39;s last generated Heartbeat.
 
LinkPhase AB =Heartbeat_Time AB −Heartbeat_Time B   (1)
         where Heartbeat_Time AB  is the arrival time of a Heartbeat packet at Point B (SSR-B or SEP-B) that was transmitted from Point A (SSR-B or SEP-A), and Heartbeat_Time B  is the transmit time of the Heartbeat at Point B that was transmitted at or before the arrival time of the Heartbeat packet from Point A.       

     Since the time between Heartbeats on all SSRs and SEPs is the same (because they are synchronized in frequency), the Link Phase of any link is in the range [0-T], where T is the time interval between Heartbeats. 
     The Link Phase in the opposite direction, from Point B to Point A, uses a similar formula but in general produces a different result than the Link Phase of the link in the direction from Point A to Point B:
 
LinkPhase BA =Heartbeat_Time BA −Heartbeat_Time A   (2)
 
       FIG. 3  illustrates an example of the Link Phase calculation between two SSRs.  FIG. 3  shows the times  300 ,  302 , and  304  at which a heartbeat packet is periodically transmitted by SSR-A, the times  310 ,  312  and  314  at which a heartbeat packet is periodically transmitted by SSR-B, arrival times  330 ,  332 , and  334  of heartbeat packets transmitted by SSR-A to SSR-B, arrival times  350  and  352  of heartbeat packets transmitted by SSR-B to SSR-A, the Link Phase  340  from SSR-A to SSR-B, and the Link Phase  360  from SSR-B to SSR-A. 
     Assume that SSR-A is the Master SSR. The phase offset of SSR-B (compared to the SSR-A Master) is arbitrary, and is shown as Φ B    320 , which can be measured in phase (0 to 360 degrees) or measured in terms of the Heartbeat period (0 to T, where T is the time interval between Heartbeats). The value of Φ B  does not need to be determined. 
     The Link Phase is calculated in the same manner by each receiving SSR, regardless of the phase relationship of SSR or SEP Heartbeat timing references at either end of the link. 
     It is not necessary to know the propagation delay (Delay) of any link in order to determine the Link Phase. Link Phase is, however, related to Delay. The formula relating Link Phase and Delay on a link from a Master SSR at point A to an SSR or SEP at point B is:
 
LinkPhase AB =(Delay AB −Φ B )mod T   (3)
 
LinkPhase BA =(Delay BA +Φ B )mod T   (4)
 
     Note: “mod” is the modulo function. The equation “y=x mod T ” yields a value such that y&lt;T, and (x−y) is divisible by T. The result of this modulo function is expressed as a value in the range [0-T] using the time reference of the SSR or SEP performing the calculation. For LinkPhase AB , the SSR-B receiver performs the Link Phase calculation. For LinkPhase BA , the SSR-A (Master SSR) receiver performs the Link Phase calculation. 
     Since the value of Φ B  (the phase of the timing reference at point B) is unknown and arbitrary, it is not possible to determine the one-way propagation delay of individual links using the present invention. However, the round-trip delay (Delay AB +Delay BA ) can be determined if it is known to be less than T, the time interval between Heartbeats. 
     Those skilled in the art can determine that Equation (3) and Equation (4) can be combined to yield:
 
(Delay AB,BA )mod T =(LinkPhase AB +LinkPhase BA )mod T   (5)
         where Delay AB,BA  (the round-trip delay between point A and point B)=Delay AB +Delay BA          

     To further illustrate the present invention, consider the case where there are three SSRs, all inter-connected with direct connections. This is illustrated by  FIG. 4 , which shows a configuration of three interconnected switches  400 ,  402  and  404 , where switch SSR-A  400  is connected to switch SSR-B  402  via links  406  and  412 , switch SSR-B is connected to switch SSR-C via links  408  and  410 , and switch SSR-C  404  is connected to switch SSR-A via links  414  and  416 . The links in  FIG. 4  may have unequal delays between the respective switches they connect. The Link Phase on each link is calculated by using equations equal or similar to Equations (1) and (2):
 
LinkPhase AB =Heartbeat_Time AB −Heartbeat_Time B   (6)
 
LinkPhase BA =Heartbeat_Time BA −Heartbeat_Time A   (7)
 
LinkPhase AC =Heartbeat_Time AC −Heartbeat_Time C   (8)
 
LinkPhase CA =Heartbeat_Time CA −Heartbeat_Time A   (9)
 
LinkPhase BC =Heartbeat_Time BC −Heartbeat_Time C   (10)
 
LinkPhase CB =Heartbeat_Time CB −Heartbeat_Time B   (11)
 
     The round-trip delay, if known to be less than T, can be calculated between each pair SSRs. Once again, assume that SSR-A is the master. SSR-B has a phase offset, Φ B . Likewise, SSR-C has an arbitrary clock offset, Φ C , so the resulting relationships among the Link Phases and the Delays are:
 
LinkPhase AB =(Delay AB −Φ B )mod T   (12)
 
LinkPhase BA =(Delay BA +Φ B )mod T   (13)
 
LinkPhase AC =(Delay AC −Φ C )mod T   (14)
 
LinkPhase CA =(Delay CA +Φ C )mod T   (15)
 
LinkPhase BC =(Delay BC +Φ B −Φ C )mod T   (16)
 
LinkPhase CB =(Delay CB −Φ B +Φ C )mod T   (17)
 
     Those skilled in the art can combine Equations (12) and (13), Equations (14) and (15) and Equation (16) and (17) to relate round-trip delay and Link Phase:
 
(Delay AB,BA )mod T =(LinkPhase AB +LinkPhase BA )mod T   (18)
 
(Delay AC,CA )mod T =(LinkPhase AC +LinkPhase CA )mod T   (19)
 
(Delay BC,CB )mod T =(LinkPhase BC +LinkPhase CB )mod T   (20)
 
       FIG. 5  is a simplified flow chart showing operation of an illustrative embodiment of the present invention. At  500 , at least one SSR is selected as a Master SSR. At  502 , if there exists more than one Master SSR, control passes to  504 , otherwise control passes to  506 . At  504 , all Master SSRs are synchronized to an external frequency timing source, such as a BITS or GPS. At  506 , the scheduled network topologies are determined. At  508 , a Heartbeat timing tree is created for each Master SSR. The root of each tree is one of the master SSRs. The number of hops from each SSR or SEP to its respective Master SSR is minimized. At  510 , the frequency reference of each SSR is synchronized to the incoming Heartbeats. As a result, all nodes are frequency-synchronized. At  512 , the Link Phase on each scheduled link is calculated. At  514 , the scheduled network is ready to schedule packet transmissions. 
     An illustrative example of a scheduled packet  600  passing from one SSR to another SSR via a third SSR is shown in  FIG. 6 , with reference to the switches  400 ,  402  and  404  and links  406  and  408  shown in  FIG. 4 . Once the SSRs are frequency-synchronized and all Link Phases are calculated, scheduled packets may commence. 
     In this example, SSR-A transmits a scheduled packet at relative time T A  (relative to SSR-A&#39;s heartbeat). This packet begins to arrive at SSR-B at relative time T B1  (relative to SSR-B&#39;s heartbeat), where
 
 T   B1 =( T   A +LinkPhase AB )mod T   (21)
 
     SSR-B begins to transmit this scheduled packet at relative time T B2  (relative to SSR-B&#39;s heartbeat).
 
 T   B2 =( T   B1   +X   B )mod T =( T   A   +X   B +LinkPhase AB )mod T   (22)
 
     where X B  is the transfer delay of a packet through SSR-B. 
     SSR-C begins to receive this scheduled packet at relative time T C  (relative to SSR-C&#39;s heartbeat), where:
 
 T   C =( T   B2 +LinkPhase BC )mod T   (23)
 
     It should be appreciated that modifications to and variations of the above-described methods and system may be made without departing from the inventive concepts disclosed herein. Accordingly, the invention is not to be viewed as limited except by the scope and spirit of the appended claims.