Abstract:
Methods and systems for determining latency across a bus, such as a PCIe bus, coupling a field programmable gate array (FPGA) and a processor having different time incrementation rates. Both the FPGA and the processor count clock ticks independently, and using a calibration offset and the two incrementation rates, the processor converts the FPGA clock ticks into processor clock ticks in order to determine latency across the bus.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims priority to U.S. Provisional Patent Application Ser. No. 61/774,607, filed on Mar. 8, 2013 and entitled “Technique for sub-microsecond latency measurement across a PCIe bus,” the entire disclosure of which is incorporated herein by reference. 
     
    
     BACKGROUND 
       [0002]    The present invention relates to methods and systems for measuring the latency between two points connected by a bus with a reference clock synchronous to a processor clock, such as, for example, a PCI Express (“PCIe”) bus, and, more particularly, to methods and systems for accurately measuring latency between two points connected by a PCI Express bus in a combined field programmable gate array (“FPGA”) and conventional processor (“CPU”) processing system. 
         [0003]    The advent of FPGA based co-processors for conventional commodity servers has led to a number of issues regarding accurate and high precision processing time measurement. 
         [0004]    In systems with computation occurring in both an FPGA as well as a CPU, each with their own independent timing, measuring point to point latencies between both the FPGA and CPU segments is non-trivial due to, for example, clock drift and the difficulty in achieving accurate synchronization during calibration. 
         [0005]    As an example of latency concerns, in a system such as financial market data processing, the market data to be processed enters the system via an Ethernet or other connection directly to the FPGA. Once the FPGA has completed its processing, data is then passed to a CPU over a PCIe bus for further processing within or by a trading algorithm. 
         [0006]    The amount of time that expires during the FPGA processing is typically measured within the FPGA itself by instantiating a regularly clocked counter. Likewise, in the CPU, Time Stamp Counter (“TSC”) instructions allow checking of a similar regularly clocked counter for measuring times. 
         [0007]    However, in measuring time for a combined system, problems occur since the rate at which the FPGA and CPU counters increments can be different. Furthermore, the counters begin at different times. Even if the rate difference and start times can be accounted for in the system, independently running clocks quickly drift out of synchronization, among other concerns. This raises challenges when timing the complete processing from the input of Ethernet data into the FPGA, across the PCIe bus, into a trading algorithm running in the CPU. 
         [0008]    Accordingly, there is a continued need for systems and methods that accurately measure latency between two points connected by a PCIe bus in a combined FPGA and CPU processing system. 
       BRIEF SUMMARY 
       [0009]    Systems and methods for accurately measuring latency between two points connected by a PCIe bus in a combined FPGA and conventional processor processing system. According to a first aspect, a method for determining latency across a PCIe bus coupling a field programmable gate array (FPGA) comprising a first counter having first clock ticks with a first incrementation rate, and a processor comprising a second counter having second clock ticks with a second incrementation rate, wherein said first and second incrementation rates are not equal, the method comprising the steps of: (i) counting, by said first counter, a plurality of first clock ticks; (ii) counting, by said second counter, a plurality of second clock ticks; (iii) determining a calibration offset; and (iv) converting, using said first and second incrementation rates and the determined calibration offset, the plurality of first clock ticks to second clock ticks. 
         [0010]    According to a second aspect, a system for determining latency across a PCIe bus, the system comprising: (i) an FPGA having a first counter having first clock ticks with a first incrementation rate, wherein said first counter is programmed to count a plurality of first clock ticks; a processor having a second counter having second clock ticks with a second incrementation rate, wherein said first and second incrementation rates are not equal, the second counter programmed to count a plurality of second clock ticks; and a PCIe bus coupling said FPGA and said processor; wherein said processor is programmed to: (i) determine a calibration offset; and (ii) convert, using said first and second incrementation rates and the determined calibration offset, the plurality of first clock ticks to second clock ticks. 
         [0011]    According to a third aspect, a non-transitory storage medium comprising a computer program product stored thereon, the computer program product comprising instructions to: (i) count a plurality of first clock ticks by an FPGA counter having a first incrementation rate; (ii) count a plurality of second clock ticks by a processor counter having a second incrementation rate, wherein said first and second incrementation rates are not equal, and wherein said FPGA and said processor are coupled by a PCIe bus; (iii) determine a calibration offset; and (iv) convert, using said first and second incrementation rates and the determined calibration offset, the plurality of first clock ticks to second clock ticks. 
     
    
     
       BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S) 
         [0012]    The present invention will be more fully understood and appreciated by reading the following Detailed Description in conjunction with the accompanying drawings, in which: 
           [0013]      FIG. 1  is an FPGA/processor system containing a PCIe coupling according to an embodiment; and 
           [0014]      FIG. 2  is a method for latency determination according to an embodiment. 
       
    
    
     DETAILED DESCRIPTION 
       [0015]    According to an embodiment are methods and systems to allow latency measurement from any two points in a combined FPGA and CPU processing system using a PCI Express bus as interconnect. This measurement is performed by novel use of a PCI Express reference clock to eliminate drift between time references in the FPGA and CPU. According to another embodiment are methods and systems to achieve accurate calibration of the measurement. 
         [0016]    Provided herein are several embodiments which allow provision of point-to-point latency measurements in a combined FPGA/CPU computing system using a PCIe bus as interconnect. One application of a system for market data processing is used as a practical example, however the methods and systems described and envisioned herein not limited to this application, and can be used whenever precision latency measurement is required in an FPGA/CPU system across a PCIe bus. 
         [0017]    In the particular case of the market data application, market data from a financial exchange arrives, via Ethernet, at the FPGA for processing. Each message received from the exchange has its processing time tracked through the system. The aim of applying the measurement technique described is to find the latency between packet entry and any point in the processing, even if this in the CPU. The method described allows the entry time of each packet into the FPGA to be calculated in CPU TSC ticks. By taking the difference in the TSC tick count, point to point latency can be calculated, even across a PCIe bus. 
         [0018]    In one embodiment, a novel mechanism is used to eliminate drift between the FPGA and CPU clocks. A 100 MHz clock is present on the PCIe bus, for example, although many other types of clocks are possible. This clock is derived from the same source as the CPU TSC clock, and as such does not drift relative to it. The FPGA incorporates a counter which is incremented by a clock derived from this 100 MHz PCIe bus clock rather than one of its own internal clocks, as is typically implemented. The 100 MHz PCIe clock may be used directly to increment the counter on the FPGA, or it may pass through a further clock multiplier or divider. In the NovaSparks appliance, the value of this counter is applied to each packet as it enters the system. 
         [0019]    A simple formula allows the value of the drift-free FPGA reference counter to be converted into TSC units. When the FPGA/CPU system is initialized, the TSC and FPGA counters begin counting ticks independently. Since TSC and FPGA counters are derived from the same source there is no drift, FPGA ticks can be converted to TSC ticks as follows in Equation (1): 
         [0000]    
       
         
           
             
               
                 
                   
                     t 
                     TSC 
                   
                   = 
                   
                     
                       
                         t 
                         FPGA 
                       
                        
                       
                         ( 
                         
                           
                             f 
                             TSC 
                           
                           
                             f 
                             FPGA 
                           
                         
                         ) 
                       
                     
                     - 
                     δ 
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0020]    Where f TSC  and f FPGA  are the incrementation rates of the CPU and FPGA counters, and t TSC  and t FPGA  are the number of clock ticks recorded by the CPU and FPGA counters. Finally, δ is an offset between the CPU and FPGA tick counts measured during a calibration step, the calibration is described in further embodiments. f TSC  can be found by interrogating the host operating system for the current CPU clock rate. The value of f FPGA  is dependent on the FPGA implementation since a clock divider or multiplier may be applied to the 100 MHz PCIe clock within the FPGA, however it will be known by the FPGA circuit designer. In the case of the market data application, for example, the current value of the FPGA counter (t FPGA ) is stored at the time the input Ethernet packet enters the system in the header of the internal message(s) generated from this packet. Conversion of FPGA into TSC ticks is performed by the CPU using Equation (1), for example. 
         [0021]    Another embodiment allows the value of δ to be found during a calibration step. Naively, δ can be expressed as follows in Equation (2) below: 
         [0000]    
       
         
           
             
               
                 
                   δ 
                   = 
                   
                     
                       
                         u 
                         FPGA 
                       
                        
                       
                         ( 
                         
                           
                             f 
                             TSC 
                           
                           
                             f 
                             FPGA 
                           
                         
                         ) 
                       
                     
                     - 
                     
                       u 
                       TSC 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
         [0022]    Where u TSC  and u FPGA  are the values of the CPU and FPGA counters at the same instant of calibration, with other symbols as previously defined. 
         [0023]    In the practical example of the market data application, conversion of FPGA to TSC ticks is performed by the CPU, as is the calculation of δ. At the instant of calibration, the CPU writes the current value of TSC to a first PCIe register on the FPGA, where it may later be read as u TSC  by the CPU. This action triggers the current value of the FPGA counter to be latched into a second PCIe register, where it may later be read as u FPGA  by the CPU. The fact of writing TSC value in the first register removes the variability linked with CPU recording and reading this value in its associated memory. 
         [0024]    It should be noted that u FPGA  is not then recorded at the same instant as u TSC , there is a time difference equivalent to a PCIe register write between the readings. This can be accounted for as an additional term in the equation for δ, as shown in Equation (3) below: 
         [0000]    
       
         
           
             
               
                 
                   δ 
                   = 
                   
                     
                       
                         u 
                         FPGA 
                       
                        
                       
                         ( 
                         
                           
                             f 
                             TSC 
                           
                           
                             f 
                             FPGA 
                           
                         
                         ) 
                       
                     
                     - 
                     
                       
                         t 
                         write 
                       
                        
                       
                         f 
                         TSC 
                       
                     
                     - 
                     
                       u 
                       TSC 
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0025]    Where t write  is the time taken for a PCIe write operation. The value of t write  will depend on the PCIe implementation and CPU type. The value of t write  can be problematic to measure, halving the time taken for a CPU write/read roundtrip will not be accurate since the time taken for PCIe register read and write roundtrips are non-symmetric. 
         [0026]    A further embodiment allows the time for a PCIe write operation to be measured. First it is assumed that the time taken for a CPU to FPGA PCIe write is symmetric with an FPGA to CPU PCIe write. The state of the TSC is taken at the start of the procedure, and immediately a PCIe write is performed to the FPGA. This operation triggers the FPGA to perform a reply write to a known location in CPU memory. Meanwhile the CPU polls this location waiting for the FPGA write, as soon as it completed the state of the TSC is again recorded. The value of the t write  f TSC  term in the equation for δ can then be approximated as follows in Equation (4): 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       t 
                       write 
                     
                      
                     
                       f 
                       TSC 
                     
                   
                   ≈ 
                   
                     
                       
                         w 
                         TSC 
                       
                       - 
                       
                         v 
                         TSC 
                       
                     
                     2 
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
         [0027]    Where w TSC  is the value of the TSC counter at the end of the procedure and v TSC  is its value at the start. 
         [0028]    In a final embodiment, latency in transmitting the value of the FPGA counter for each piece of data flowing through the system can be reduced. This is achieved by transmitting only the least significant bits of the counter representation. The most significant bits of the counter may be ignored since a process running on the CPU tracks when the remaining truncated counter bits wrap to zero and reinserts the missing most significant bits on the CPU side. The disadvantage of this approach is that it limits the maximum time that can be measure by the technique. However, in the case of the market data application the upper bound on compute time is known so the length of the FPGA truncated counter representation can be correctly sized. 
         [0029]    Referring now to the drawings, wherein like reference numerals refer to like parts throughout, there is seen in  FIG. 1  a system  100  for latency measurement from any two points in a combined FPGA and CPU processing system using a PCI Express bus as interconnect, according to an embodiment. According to this embodiment, system  100  comprises a FPGA  102 , which can be, for example, a single FPGA, an FPGA matrix or array, and/or an FPGA node, among other embodiments of FPGA. 
         [0030]    System  100  may be, for example, directed to a market data application, among many other applications. In one example, a conventional system for processing market feed data relies on conventional server based components. Such systems can employ conventional CPU architectures and communicate to process a market data feed and communicate the processed data. According to an embodiment, FPGA  102  can include one or more connections to one or more conventional CPU processing systems  104 . In an example, information can be communicated to a CPU processing device to provide for analysis of the data received, data receive rates, data volume, and data delivery rates, among many other options. In some embodiments, downstream clients can interact with CPU processing device  104  to receive processed data, specific information on user defined symbols, and/or other user defined subsets of data delivered to the feed handler/matrix. 
         [0031]      FIG. 1  illustrates just one example of a conventional CPU system  104 , which includes a processor  108 , a non-transitory computer readable storage medium  106 , and a user interface  110 . The processor  108  can be any type of processor, multiprocessor or controller, including but not limited to commercially available microprocessors. 
         [0032]    The computer readable storage medium  106  may be, for example, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any combination of the foregoing. For example, computer readable storage medium  106  may contain or store a program for use by or in connection with the systems or methods described herein. Indeed, the instructions stored in the data storage may cause the processor  108  to perform one or more of the functions described herein. 
         [0033]    User interface  110  may be any user interface that allows the user or a third party to input and/or obtain information from system  100 , including but not limited to a monitor, display, mouse, keyboard, touchscreen, voice recognition sensor, and/or any of a wide variety of other components. 
         [0034]    Components FPGA  102  and CPU  104  are in communication via PCIe  112 , allowing the system  100  to send data between the components. PCIe  112  can include, for example, a 100 MHz clock is present, for example, although many other types of clocks are possible. The FPGA incorporates a counter which is incremented by a clock derived from this 100 MHz PCIe bus clock rather than one of its own internal clocks, as is typically implemented. The 100 MHz PCIe clock may be used directly to increment the counter on the FPGA, or it may pass through a further clock multiplier or divider. 
         [0035]    According to an embodiment, CPU  104  or another CPU in communication with one or more components of system  100  is programmed, adapted, and/or configured to calculate latency measurements in system  100 . For example, the CPU can include a computer readable storage medium  106  that contains or stores a program for use by or in connection with processor  108  to perform the following functions. 
         [0036]    The program allows the value of the drift-free FPGA reference counter to be converted into TSC units. For example, when the FPGA/CPU system is initialized, the TSC and FPGA counters begin counting ticks independently. Since TSC and FPGA counters are derived from the same source there is no drift, FPGA ticks can be converted to TSC ticks using Equation (1). 
         [0037]    Where f TSC  and f FPGA  are the incrementation rates of the CPU and FPGA counters, and t TSC  and t FPGA  are the number of clock ticks recorded by the CPU and FPGA counters. Finally, δ is an offset between the CPU and FPGA tick counts measured during a calibration step, the calibration is described in further embodiments. f TSC  can be found by interrogating the host operating system for the current CPU clock rate. The value of f FPGA  is dependent on the FPGA implementation since a clock divider or multiplier may be applied to the 100 MHz PCIe clock within the FPGA, however it will be known by the FPGA circuit designer. In the case of the market data application, for example, the current value of the FPGA counter (t FPGA ) is stored at the time the input Ethernet packet enters the system in the header of the internal message(s) generated from this packet. Conversion of FPGA into TSC ticks is performed by the CPU using Equation (1), for example. 
         [0038]    The program also allows the value of δ to be found during a calibration step. Naively, δ can be expressed in Equation (2), where u TSC  and u FPGA  are the values of the CPU and FPGA counters at the same instant of calibration, with other symbols as previously defined. 
         [0039]    In the practical example of the market data application, conversion of FPGA to TSC ticks is performed by the CPU, as is the calculation of δ. At the instant of calibration, the CPU writes the current value of TSC to a first PCIe register on the FPGA, where it may later be read as u TSC  by the CPU. This action triggers the current value of the FPGA counter to be latched into another PCIe register, where it may later be read as u FPGA  by the CPU. The fact of writing TSC value in the first register removes the variability linked with CPU recording and reading this value in its associated memory. 
         [0040]    It should be noted that u FPGA  is not then recorded at the same instant as u TSC , there is a time difference equivalent to a PCIe register write between the readings. This can be accounted for as an additional term in the equation for δ, as shown in Equation (3), where t write  is the time taken for a PCIe write operation. The value of t write  will depend on the PCIe implementation and CPU type. The value of t write  can be problematic to measure, halving the time taken for a CPU write/read roundtrip will not be accurate since the time taken for PCIe register read and write roundtrips are non-symmetric. 
         [0041]    The program can also allow the time for a PCIe write operation to be measured. First it is assumed that the time taken for a CPU to FPGA PCIe write is symmetric with an FPGA to CPU PCIe write. The state of the TSC is taken at the start of the procedure, and immediately a PCIe write is performed to the FPGA. This operation triggers the FPGA to perform a reply write to a known location in CPU memory. Meanwhile the CPU polls this location waiting for the FPGA write, as soon as it completed the state of the TSC is again recorded. The value of the t write  f TSC  term in the equation for δ can then be approximated as in Equation (4), where w TSC  is the value of the TSC counter at the end of the procedure and v TSC  is its value at the start. 
         [0042]    Shown in  FIG. 2  is a method  200  for measuring latency measurement between any two points in a combined FPGA and CPU processing system using a PCI Express bus as interconnect is disclosed. In step  202 , a combined FPGA and CPU processing system using a PCI Express bus as interconnect is provided. The FPGA and CPU can be any of the embodiments described herein or otherwise envisioned, and can include any of the components of the FPGA and/or CPU described in conjunction with  FIG. 1 . For example, the PCIe bus can include a clock such as a 100 MHz clock, although many other types of clocks are possible. The CPU can include a clock such as a Time Stamp Counter (TSC), although many other types of clocks are possible. The FPGA can measure expired time within the FPGA itself by instantiating a regularly clocked counter, although many other types of clocks are possible. In some embodiments, the TSC and FPGA counters begin counting ticks independently when the FPGA/CPU system is initialized. 
         [0043]    In step  204 , the time increments of the FPGA are converted to the CPU time increments using Equation (1), where f TSC  and f FPGA  are the incrementation rates of the CPU and FPGA counters, and t TSC  and t FPGA  are the number of clock ticks recorded by the CPU and FPGA counters, and δ is an offset between the CPU and FPGA tick counts measured during a calibration step as described herein. Since TSC and FPGA counters are derived from the same source there is no drift, so FPGA ticks can be converted to TSC ticks. 
         [0044]    According to an embodiment, f TSC  can be found by interrogating the host operating system for the current CPU clock rate. The value of f FPGA  is dependent on the FPGA implementation since a clock divider or multiplier may be applied to the 100 MHz PCIe clock within the FPGA, however it will be known by the FPGA circuit designer. In the case of the market data application, for example, the current value of the FPGA counter (t FPGA ) is stored at the time the input Ethernet packet enters the system in the header of the internal message(s) generated from this packet. 
         [0045]    At step  206 , the value of δ is determined during a calibration step using Equation (2), where u TSC  and u FPGA  are the values of the CPU and FPGA counters at the same instant of calibration, with other symbols as previously defined. 
         [0046]    In some embodiments, conversion of FPGA to TSC ticks is performed by the CPU, as is the calculation of δ. At the instant of calibration, the CPU writes the current value           TSC to a first PCIe register on the FPGA, where it may later be read as u TSC  by the CPU. This action triggers the current value of the FPGA counter to be latched into another PCIe register, where it may later be read as u FPGA  by the CPU. The fact of writing TSC value in the first register removes the variability linked with CPU recording and reading this value in its associated memory. If u FPGA  is not recorded at the same instant as u TSC , there is a time difference equivalent to a PCIe register write between the readings. This can be accounted for as an additional term in the equation for δ, as shown in Equation (3) where t write  is the time taken for a PCIe write operation. The value of t write  will depend on the PCIe implementation and CPU type. The value of t write  can be problematic to measure, halving the time taken for a CPU write/read roundtrip will not be accurate since the time taken for PCIe register read and write roundtrips are non-symmetric. 
         [0047]    At step  208 , the time for a PCIe write operation can be measured. First, it is assumed that the time taken for a CPU to FPGA PCIe write is symmetric with an FPGA to CPU PCIe write. The state of the TSC is taken at the start of the procedure, and immediately a PCIe write is performed to the FPGA. This operation triggers the FPGA to perform a reply write to a known location in CPU memory. Meanwhile the CPU polls this location waiting for the FPGA write, as soon as it completed the state of the TSC is again recorded. The value of the t write  f TSC  term in the equation for δ can then be approximated as in Equation (4), where w TSC  is the value of the TSC counter at the end of the procedure and v TSC  is its value at the start. 
         [0048]    According to one embodiment, latency in transmitting the value of the FPGA counter for each piece of data flowing through the system can be reduced. This is achieved by transmitting only the least significant bits of the counter representation. The most significant bits of the counter may be ignored since a process running on the CPU tracks when the remaining truncated counter bits wrap to zero and reinserts the missing most significant bits on the CPU side. The disadvantage of this approach is that it limits the maximum time that can be measure by the technique. However, in the case of the market data application the upper bound on compute time is known so the length of the FPGA truncated counter representation can be correctly sized. 
         [0049]    Although the present invention has been described in connection with a preferred embodiment, it should be understood that modifications, alterations, and additions can be made to the invention without departing from the scope of the invention as defined by the claims.