Patent Publication Number: US-11664968-B2

Title: Secondary phase compensation assist for PLL IO delay aligning sync signal to system clock signal

Description:
CROSS-REFERENCE TO RELATED APPLICATION(S) 
     This application is a continuation of U.S. patent application Ser. No. 16/836,713, filed Mar. 31, 2020, entitled “Secondary Phase Compensation Assist for PLL TO Delay Aligning Sync Signal to System Clock Signal,” naming Vivek Sarda as inventor, which application is incorporated herein by reference in its entirety. 
     This application relates to the application entitled “Secondary Phase Compensation Assist for PLL TO Delay”, naming Vivek Sarda as inventor, patent application Ser. No. 16/836,706, filed Mar. 31, 2020, which application is incorporated herein by reference in its entirety. 
    
    
     BACKGROUND 
     Field of the Invention 
     This invention relates to network timing and more particularly to reducing errors in network timing signals. 
     Description of the Related Art 
     Network communication boxes use timing protocols to ensure time of day (ToD) counters in the network are synchronized. SYNC signals are used to update time of day counters at the same time in the network. Any delay/offset and process, voltage, temperature (PVT) variation between the SYNC lines being supplied to the ToD counters in each line card in the network box results in an error that is classified as Continuous Time Error (CTE). The CTE budget for a network box is 5 ns for Class D type network boxes. Reducing sources of timing error in network boxes would give greater flexibility to designers to meet the CTE budget. 
     SUMMARY OF EMBODIMENTS OF THE INVENTION 
     Accordingly, in one embodiment a method includes generating a SYNC output signal that indicates when to update a time of day counter. The method further includes feeding back the SYNC output signal to an input terminal as a SYNC feedback signal and determining a clock edge of a first system clock signal closest in time to a transition of a SYNC input signal. A time difference is determined between the clock edge of the first system clock signal and a transition of the SYNC feedback signal and the timing of the SYNC output signal is adjusted based on the time difference. 
     In another embodiment an apparatus includes a first input terminal to receive an input SYNC signal and a second input terminal to receive a first system clock signal. A phase-locked loop is coupled to the first system clock signal and generates a second system clock signal. A divider circuit divides the second system clock signal and generates a SYNC output signal. A time of day counter is coupled to the SYNC output signal and is configured to update a count value responsive to the SYNC output signal. A second input terminal is coupled to receive the SYNC output signal as a feedback SYNC signal. Control logic is configured to determine a closest edge of the first system clock signal to a transition of the SYNC input signal corresponding to a transition of the SYNC feedback signal. The control logic is further configured to determine a time difference between the transition of the SYNC feedback signal and the closest edge of the first system clock signal, and to adjust a timing of the SYNC output signal based on the time difference. 
     In another embodiment a method includes receiving a SYNC input signal and receiving a first system clock signal. The method further includes generating a first time stamp based on the SYNC input signal, generating a second system clock signal in a phase-locked loop, and generating a SYNC output signal by dividing the second system clock signal in a divider circuit. The SYNC output signal is fed back as a SYNC feedback signal and a second time stamp is generated based on the SYNC feedback signal. The method determines a closest edge of the first system clock signal to the first time stamp and determines a time difference between the closest edge of the first system clock signal and the second time stamp. The timing of the SYNC output signal is adjusted based on the time difference using a coarse time adjustment by adjusting a divide ratio of the divider circuit and using a fine time adjustment in the phase-locked loop based on a residue of a remainder of the time difference not accounted for by the coarse time adjustment. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention may be better understood, and its numerous objects, features, and advantages made apparent to those skilled in the art by referencing the accompanying drawings. 
         FIG.  1    illustrates a typical communication network box with a slave line card (LC), a master timing card (TC), and multiple master line cards coupled through a backplane. 
         FIG.  2    illustrates an example of a time stamp exchange. 
         FIG.  3    illustrates additional details of a slave line card and a master timing card. 
         FIG.  4 A  illustrates another view of a portion of a network box. 
         FIG.  4 B  illustrates the timing relationships that can exist between the SYNC signal and the SYSCLK signal in the network box illustrated in  FIG.  4 A . 
         FIG.  4 C  illustrates a larger view of section E of  FIG.  4 B . 
         FIG.  5    illustrates an embodiment in which the SYNC output signal (SYNC_OUT) is looped back as an input. 
         FIG.  6    illustrates logic to time stamp and compare the time stamps. 
         FIG.  7    illustrates a SYNC control block that receives the time stamps, determines the IO delay, and controls both a coarse adjust and a fine adjust to remove the IO delay from the SYNC output signal (SYNC_OUT). 
         FIG.  8    illustrates a block diagram of an implementation of the coarse adjustment. 
         FIG.  9    illustrates an embodiment of a phase-locked loop (PLL) used for the fine adjustment. 
         FIG.  10    illustrates a flow diagram of the operation of the SYNC control logic. 
         FIG.  11    illustrates a timing diagram of possible relationships between the SYSCLK, SYNC signal, and SYNC_OUT signal. 
         FIG.  12 A  illustrates a block diagram showing the SYNC control block. 
         FIG.  12 B  illustrates a flow diagram of operation of control logic to realign the SYNC output signal (SYNC_OUT) to the input SYSCLK signal. 
     
    
    
     The use of the same reference symbols in different drawings indicates similar or identical items. 
     DETAILED DESCRIPTION 
       FIG.  1    shows a typical architecture of communication network box  100  with a slave line card (LC)  101 , a master timing card (TC)  103 , and multiple master line cards  105 . The data_out  109  from each line card is time stamped using time stamps from local Time of Day (ToD) counters  111 . One challenge is to keep the ToD on the slave line card in alignment with the network timestamps from the incoming data stream on data_in  116 . Another challenge is to maintain the ToD counters across different line cards in alignment over process, voltage, and temperature (PVT) variations so that all data_out  109  with their time stamps are aligned with each other and the incoming network time supplied on data_in  116 . 
     The master timing card  103  supplies a SYNC signal and system clock signal (SYSCLK) to the slave line card  101  generated using PLL  117  and dividers (not shown). The SYNC signal is also referred to as the FSYNC (frame sync) signal in certain contexts as the signal has different names (SYNC or F SYNC) at the system level or integrated circuit level inside the network box. The signal will be referred to as the SYNC signal herein for ease of reference. The master timing card  103  supplies the SYSCLK and SYNC signal to all of the master line cards  105  over backplane  119 . The SYNC signal is a global signal inside the network system box  100  that signifies the right moment/edge for the Time of Day (ToD) counters  111  to rollover. The SYNC signal has a frequency range of 1 kHz to pp2s (pulse per 2 seconds). In many network systems the SYNC signal is 1 pulse per second (1PPS). SYNC is an integer divided down and edge aligned version of the system clock signal SYSCLK. The SYNC output from the master timing card (TC) is the global SYNC used by all the line cards (LC) for their ToD rollover alignment. The various ToD counters  111  contain the same value and turnover at the same time based on the SYNC signal. Each of the line cards  101  and  105  generate the SYNC signal by dividing the SYSCLK generated by PLL  121  in a divider (not shown in  FIG.  1   ) to the desired frequency. 
     The exact position of the SYNC edge is derived using a precision time protocol (PTP) servo loop that uses the time information inside the incoming Synchronous Ethernet (SyncE) packet stream to the slave line card  101 .  FIG.  2    illustrates an example of a time stamp exchange  201  between an upstream PHY and the downstream pHY (e.g. PHY  123  in  FIG.  1   ). Each of the time stamps t 1 -t 4  represents the departure time (t 1 , t 3 ) or the receive time (t 2 , t 4 ). The timestamps exchange allows determination of one-way delay (OWD) and error offset between the upstream PHY and the downstream PHY shown at  203 . That time stamp exchange allows the slave line card to determine the correct time provided by the upstream PHY even with delays between the upstream PHY and the downstream PHY. Note that the high level description of the PTP servo loop is provided as background information to provide context in which various embodiments described herein can be utilized. 
     The slave line card and the master timing card also have a closed loop PTP servo system in accordance with the IEEE 1588 protocol that corrects the position of the SYNC signal over process, voltage, and temperature (PVT) and aligns the SYNC signals distributed by the master timing card  203  to the time stamps of the incoming packet stream to the slave line card. The servo loop ensures that the slave line card and the master timing card are synchronized. The slave line card  101  and the master timing card  103  exchange information in the closed loop system to adjust the CLK and SYNC pair on the master timing card such that the slave line card ToD is aligned with the network ToD of the chosen incoming data stream on data_in  116 . The PTP servo loop adjusts the timing of SYNC by adjusting PLL  117  so that the slave line card ToD is aligned in frequency and phase to the upstream ToD received by the slave line card on data_in  116 . The distributed SYSCLK is supplied as a reference clock to the PLL  121  within each of the line cards and the line card PLLs generate a local SYSCLK and SYNC signal that is phase and frequency aligned with the distributed SYSCLK and SYNC siganl. The master line cards  105  are duplicates (up to 64 copies) of the slave line card  101  but without the closed loop PTP servo loop. In other words, the distribution of the CLK/SYNC pair to the master line cards  105  is open loop (without the PTP closed loop adjustments). 
     Referring to  FIG.  3    additional aspects of the slave line card  301  and the master time card  303  are shown. In addition, to generating the SYSCLK  305 , the master time card generates a Synchronous Ethernet (SyncE) clock signal  307 . The SyncE clock signal is supplied to SyncE PLL  309  in the slave line cards so that the local SyncE clock signals are frequency and phase locked to the SyncE clock signal in the master timing card. Embodiments include a slave timing card  311  that functions as a backup timing card to the master timing card  303  by providing backup SYNC, SYSCLK, and SyncE signals. The FPGA  315  is part of the PTP loop and includes an MCU to implement PTP software. PM and SEC are primary and secondary data streams to select from for determining the network time. SEC is a backup of PRI. The PTP PLLs in the slave line card and the master timing card are used to adjust the SYSCLK (and SYNC) based on the PTP servo loop. The various time stamps required for the PTP servo loop are exchanged between the hosts on the slave line card and the master timing card. The hosts are microcontroller units (MCUs) or field programmable gate arrays (FPGAs) with some processing and communication abilities. The digitally controlled oscillator (DCO) in the master timing card  303  adjusts the phase of the SYNC and SYSCLK in accordance with the calculations of the PTP servo loop so that the SYNC at the ToD counter on the slave line card occurs at the desired time. 
     Any delay/offset and PVT variation between the SYNC lines to the ToD&#39;s in each line card and the slave timing card in the network box results in an error that is classified as Continuous Time Error (CTE). The CTE budget for a network box is 5 ns for Class D type network boxes. One source of error is that the SYNC signal and SYSCLK supplied by the local PLLs in the line cards still have to transit through the circuitry of the PLL to the ToD counters. A mismatch exists in SYNC signal delivery due to PVT differences between the line cards including the slave line card and the master line cards. That mismatch impacts the accuracy of the timestamps in every Master LC and impacts the Continuous Time Error (CTE) budget of 5 ns for a Class D network box. 
       FIGS.  4 A and  4 B  illustrate the timing relationships that can exist between the SYNC signal and the SYSCLK signal.  FIG.  4 A  shows a master timing card  401  supplying a line card  403  with the SYSCLK signal  405  and the SYNC signal  407  through backplane drivers, receivers and PCB traces  409 . At A, SYNC identifies the SYSCLK period that coincides with ToD rollover. In the example of  FIG.  4 B , that period is period 0 with period (−1) and period (+1) before and after period 0. In an embodiment SYNC can be adjusted to any SYSCLK period by aligning the rising edges of SYSCLK and SYNC. In the example of  FIG.  4 B  SYNC selects the SYSCLK period 0. The Δt boxes shown in  FIG.  4 A  represent the adjustments made to account for the delay in the SYNC and SYSCLK signals supplied from the master timing card  401 . Note that since SYSCLK may be frequency multiplied at the line card output as shown in  FIG.  4 A  (NX SYSCLK), setting this lower frequency SYSCLK period can be considered a coarse adjustment. Backplane drivers, receivers and PCB traces  404  cause a delay in the SYSCLK period 0 rising edge at C with respect to the alignment reference line  421 . In addition, there is a mismatch between SYNC and SYSCLK shown at  423 . At D, input delay adjustments in the line card realign the SYSCLK edges back to the original alignment reference line  421  but do not realign SYSCLK and SYNC. Even if the SYSCLK and SYNC edges remain misaligned as shown at D, the SYNC signal in the indicated range  425  will select the SYSCLK period 0. Input to output delay in the illustrated embodiment, causes misalignment of the SYSCLK rising edge with respect to the alignment reference line  421 . Referring now to E of  FIGS.  4 A and  4 B , if the misalignment is left without correction (open loop), the misalignment can range between ±0.5 ns around the alignment reference line as shown at  425 . Alternatively, a zero delay mode with respect to the SYSCLK can achieve ±100 ps as shown at  427 . As shown at  429 , the SYNC signal is realigned to the rising edge of SYSCLK period 0.  FIG.  4 C  shows a larger view of E of  FIG.  4 B . The SYNC signal can also be adjusted to any edge of N×SYSCLK (controlled at the line card). That is considered to be a fine adjustment. While open loop and zero delay options provide solutions, if the IO delay is measured and the adjustment is made to SYNC based on the IO delay, the error can be reduced to ±50 ps. 
     Accordingly, referring to  FIG.  5   , in order to measure the IO delay, the SYNC output signal (SYNC_OUT) is looped back to the slave line card input buffer  501  as the SYNC feedback signal (SYNC_FB)  503 . That allows the IO delay to be tracked and accounted for over PVT variations. The SYNC_FB signal is time stamped and that time stamp is compared to the current valid SYNC time stamp for the SYNC input signal  507  received on input buffer  509 . The measured IO delay can include delay caused by input buffers, the PLL  511 , divide logic  517 , and other clock tree buffers on the line card. While the slave line card is shown, the same approach is used for the master line cards, each of which feedback their SYNC signal to a master line card input buffer. 
     The SYNC signal  507  is received at buffer  509  and is used, along with SYSCLK  515 , to adjust the PLL  511  to ensure the SYNC_OUT and local SYSCLK  521  generated in the line card is phase and frequency aligned with the input SYNC and SYSCLK from the master timing card. The SYNC_OUT signal is generated by an integer divide in divider  517  of the local SYSCLK  521  generated by the PLL  511  that is aligned in phase and frequency to SYSCLK. 
       FIG.  6    illustrates logic to time stamp and compare the two time stamps. Receive buffer  501  receives the SYNC_FB signal  503  and supplies the SYNC_FB signal to time stamp logic  601 . Time stamp logic  603  receives the input SYNC signal  507  through buffer  509 . The time stamp logic functions as a time to digital converter and converts the transitions of the SYNC_FB signal and the input SYNC signal to digital values based on an available timing reference. Difference logic  605  receives the two time stamps and determines the difference between the time stamps of SYNC_FB  503  and the currently valid SYNC signal  507 . That difference  611  represents the input/output (TO) delay. The difference logic may be part of the SYNC control block  701  shown in  FIG.  7   . 
     Referring to  FIG.  7   , in an embodiment the SYNC control block  701  receives the time stamps, determines the IO delay, and controls both a coarse adjust  703  and a fine adjust  705  to remove the IO delay from the SYNC output signal to better align the SYNC output signal with the SYNC input signal. The functionality of the control block may be implemented as a state machine and other discrete control logic, in a programmed microcontroller, FPGA, or in a suitable combination of a programmed microcontroller, FPGA, and/or discrete control logic. Referring to  FIG.  8   , the coarse adjustment is implemented by adjusting the divider  517 . For example, if divider  517  divides by an integer N without IO correction, with IO correction divider  517  divides by N±M, where M is an integer corresponding to at least a portion of the IO delay as explained further herein. The fine adjustment is made by adjusting an offset to the phase and frequency detector (PFD)  901  on the feedback divider  905  of the PTP PLL  511  shown in  FIG.  9    in more detail. The combination of the coarse adjustment and the fine adjustment substantially eliminates the measured IO delay associated with the SYNC_out signal (or applies any desired offset to the SYNC_out signal). Still referring to  FIG.  9   , PLL  511  receives SYSCLK  515  as the reference clock signal. Once PLL  511  is locked to SYSCLK  515 , divider  517  generates the SYNC_OUT signal that is looped back as SYNC_FB. In an embodiment, the PTP PLL  511  has a loop bandwidth of between, e.g., 40 Hz and 100 Hz. 
     Referring back to  FIG.  4 A , conceptually, the input SYNC signal can be used to align the output SYNC signal by using the input SYNC signal  407  to reset the divider  415 . In that way, SYNC_OUT will then be aligned with the input SYNC signal. 
     Another approach utilizes the time stamps as discussed herein. Referring to  FIG.  10   , a flow diagram illustrates the operation of the SYNC control logic  701 . In  1001  the control logic determines the IO delay by comparing the time stamps for SYNC_FB  503  and SYNC  507 . The SYNC logic  701  calculates the quantized SYSCLK cycles of IO delay correction and applies that quantized value to the output divider  517  in the SYNC_OUT path. For example, in an embodiment the OSC  903  is a voltage controlled oscillator (VCO) providing a VCO output signal with a frequency of 10 GHz. With counter  517  being clocked by the VCO output signal, that results in the counter counting in 100 picosecond (ps) increments. Assume that the IO delay was measured to be 35.033 ns. The SYNC logic quantizes the IO delay in terms of a number of divider (or counter) cycles to add or subtract and applies the divider correction in  1003 . The SYNC control logic  701  then applies the residue left from the coarse IO delay correction to make the fine adjustment in  1005  using the PLL. For example, assume the divider is implemented as a counter that counts N SYSCLK cycles and then issues a pulse indicating the N count has been reached. The counter then resets and counts again, thus issuing a pulse every N cycles of SYSCLK. The 35 ns can be accounted for using the coarse correction by causing the counter to count 35 ns worth of fewer increments (in counter increments of 100 ps each) to zero out the IO delay (except for the residue). Note that the change in count is made for only one SYNC_OUT cycle. Adjusting the count value of the counter is one way of adjusting the divider value of the divider. While the example shown in  FIG.  9    utilizes a high speed counter clocked by the VCO, other embodiments utilize a counter clocked by a lower frequency clock signal. Thus, the SYNC control logic may quantize the IO error for one or more lower speed counters (or both low speed and high speed counters) before determining the residue. In the example being considered, the 0.033 ns (33 ps) cannot be accounted for by the coarse adjust. Referring again to  FIG.  9   , the PLL can be adjusted by adjusting the feedback divider  905 . In other embodiments in which the PFD  901  receives digital values, the fine adjustment can be made by adjusting the digital time stamp  907  of the feedback clock being supplied to the PFD  901  or even the digital time stamp  909  of the reference clock signal. The SYNC control logic applies the residue correction slowly, e.g., at less than ⅛ of the loop bandwidth of the PLL so as not to cause an undesirable large perturbation to the VCO output signal. The measured IO delay between SYNC_IN and SYNC_OUT is thereby reduced to zero (or other desired offset). Finally, in  1007 , the SYNC control logic recalculates the actual IO delay every S cycles of SYNC, where S is an integer, and makes the coarse and fine adjustments to zero out any detected IO delay. Note that embodiments compare the measured IO delay to a threshold amount of IO delay before making the coarse and fine adjustments so that the adjustments are not being continually made in response to random phase noise. 
     While the embodiment of  FIG.  10   , time aligns SYNC_OUT with SYNC, other embodiments align SYNC_OUT directly with SYSCLK. Referring to  FIG.  11   , the timing diagram illustrates how SYSCLK and SYNC may be misaligned. Ideally, the SYNC input signal is aligned at  1101  with SYSCLK. However, due to variations in delays associated with transmit and receive buffers and the backplane traces, the actual SYNC input signal received at the line card may be misaligned with respect the SYSCLK as shown at  1103 . The SYNC_FB IO delay with respect to the actual SYNC input signal is shown at  1105 . However, rather calculating the IO delay with respect to the time stamped actual SYNC input signal and realigning to the actual SYNC input signal, embodiments realign the SYNC output signal to SYSCLK. Thus, the correction is made to realign SYNC_OUT (SYNC_FB) to SYSCLK instead of the misaligned actual SYNC input signal by calculating the IO delay  1107  between SYNC_FB and SYSCLK rather than the delay  1105  between SYNC_FB and SYNC actual. The active edges (e.g., the rising edges) of SYSCLK may be time stamped by the time stamper  925  (see  FIG.  9   ) or another time stamper and SYNC and SYNC_FB time stamped as described in  FIG.  6   . 
     Referring to  FIG.  12 A , the functionality of the SYNC control block  1201  is modified from the SYNC control block  701  ( FIG.  7   ) to align the SYNC output signal to the input SYSCLK signal rather than the SYNC input signal. The modified control block receives the time stamps  1202  of SYSCLK, SYNC_FB and the SYNC input signal, determines the closest SYSCLK edge to the SYNC input signal time stamp, determines the IO delay to align the SYNC output signal to the closest input SYSCLK edge, and controls both a coarse adjust  1203  and a fine adjust  705  to remove the IO delay from the SYNC output signal to better align the SYNC output signal with the SYNC input signal. 
       FIG.  12 B  illustrates the control flow to realign to SYSCLK. The control functionality described in  FIGS.  12 A and  12 B  may be implemented in a state machine and/or other discrete control logic, in a programmed microcontroller or FPGA, or in a suitable combination of a programmed microcontroller, FPGA, and/or discrete control logic. The control logic in  1221  receives the time stamp for the SYNC input signal (for the valid SYNC signal) corresponding to the SYNC_FB signal, the time stamp for SYNC_FB, and time stamps of the SYSCLK edges. Then, the control logic determines in  1223  the closest SYSCLK edge to the SYNC input signal time stamp. In the example of  FIG.  11   , the closest edge is edge  1111 . The closest edge can be determined from SYSCLK time stamps and the SYNC input signal time stamp. A suitable number of SYSCLK time stamps are kept to ensure the closest SYSCLK edge is available. The closest SYSCLK edge is the smallest difference between the time stamps of the SYNC input signal and SYSCLK. Then, the IO delay is determined in  1225  as the difference between the SYSCLK time stamp (closest edge) and SYNC_FB (equivalent to SYNC_OUT). The IO correction is quantized in terms of the divider increments in  1225  and the quantized value applied in  1227  in a coarse correction by adjusting the divider supplying the SYNC_OUT signal. The residue after quantization is applied in  1229  using a fine correction as described earlier. In  1231 , the control logic waits for N cycles and then recalculates the IO delay to track changes in the IO delay due to temperature, voltage, or other environmental changes. 
     In at least one embodiment the IO delay realignment is programmable to be either to the SYNC input signal or to the SYSCLK and thus the SYNC control logic includes the functionality to determine the IO error with respect to both input signals. The programmability may be implemented over a programming interface (not shown) of the integrated circuit implementing the realignment. The description of the invention set forth herein is illustrative and is not intended to limit the scope of the invention as set forth in the following claims. Variations and modifications of the embodiments disclosed herein may be made based on the description set forth herein, without departing from the scope of the invention as set forth in the following claims.