PRECISION TIMING ACROSS PCIe CEM NICS

Architectures and techniques are described that can facilitate precision timing synchronization with a network interface card (NIC) that is compliant with a peripheral component interconnect express (PCIe) card electromechanical (CEM) specification. In that regard, certain pins that are specified to provide JTAG functionality can be repurposed. Thereafter the particular JTAG pins that are repurposed for precision timing synchronization functions can be utilized to provide precision timing synchronization functions, USB functions, or both. Because the timing synchronization flows through the pins of the NIC, other cabling (e.g., coaxial cables) implemented for the express purpose of timing synchronization can be eliminated.

BACKGROUND

In networks today, and particularly telecommunication networks, precision timing refers to an accurate synchronization of clocks and timestamps across distributed systems in order to ensure precise coordination and timing of events. Precision timing can be crucial for maintaining the reliability, efficiency, and performance of network operations, particularly in scenarios where timing accuracy is critical, such as voice and video communication, financial transactions, industrial automation, scientific research, and so on. Precision timing relies on all devices and systems within a network to synchronize their clocks to a common time reference, typically provided by an external time source such as a global navigation satellite system (GNSS) like GPS (Global Positioning System) or Galileo. Clock synchronization ensures that all network components operate with consistent timing accuracy, minimizing timing errors and discrepancies.

DETAILED DESCRIPTION

Overview

As noted in the background section, precision timing can be crucial for many applications. In the context of open standards networks (e.g., open radio access networks (O-RAN) or the like), and related commodity-based hardware that provide a significant advantage for open standards networks, many standards do not account for precision timing functionality. For example, network interface cards (NICs) that are compliant with a peripheral component interconnect express (PCIe) card electromechanical (CEM) 6.x specification do not specify or account for precision timing functions such as the propagation of a discrete timing synchronization clock signal.

Thus, applications that leverage precision timing are sometimes implemented with purpose-built hardware that is generally proprietary in nature, which is essentially antithetical to open standards networks. Alternatively, widely used NICs such as PCIe CEM cards are designed to support precision timing via 50-ohm coaxial cable connectors.

To provide additional context, consider FIG. 1. FIG. 1 shows a pictorial image illustrating an example adaptor and/or network interface card (NIC) 100 that is compliant with a peripheral component interconnect express (PCIe) card electromechanical (CEM) specification in accordance with certain embodiments of this disclosure. As can be observed, to supplement the network functionality with precision timing elements, existing PCIe CEM NICs 100 comprise one or more coaxial cable connectors 102. Ordinarily, and in this example, two different coaxial cable connectors 102 are provided, labeled here as coaxial cable connector 102A and coaxial cable connector 102B. Coaxial cable connectors 102 are typically subminiature version A (SMA) type connectors.

Furthermore, PCIe CEM NICs 100 can comprise coaxial cable connector 104 that can be configured to couple to a global navigation satellite system (GNSS) antenna or the like, which can be a subminiature version B (SMB) type.

Propagating timing synchronization clocks within a common datacenter server presents challenges rarely observed in purpose-built telecommunication appliances. Timing synchronization clocks, normally frequency and a sync pulse, are commonly used in telecommunication applications such as 5G wireless and factory automation edge solutions to support high precision clock profiles such as G.8265.1, G.8275.1, and G.8275.2 as defined by the international telecommunications union (ITU) standardization body. These clocks synchronize both inter-system components as well as intra-system solutions. In the case of daisy chaining systems this may require up to four discrete clocks per NIC.

In the case of servers that have timing sync NICs included in their configuration, the NICs will require coaxial cables, custom cards, or both to support 5G and other timing sensitive applications. Today, NIC cards (e.g., PCIe CEM NIC 100), if supporting precision timing do so with coaxial cables that extend from the front of the bracket or internally. Such can present internal cabling challenges for systems developers. For example, provisioning coaxial cables can be problematic for both factories as well as field services because coaxial cables can be difficult to mount (e.g., screw into connectors 102, 104), difficult to bend, particularly for internal arrangements, and lack a presence indication which can cause debug challenges if the clocks are not fully seated.

Unfortunately, many NIC hardware standard bodies neglect time synchronization. While PCIe CEM-compliant NIC 100 comprises may pins 106 for data transfer, none of those pins 106 are dedicated to propagating discrete timing synchronization clocks. This causes equipment hardware architects to consider either customizing the pinouts of common connector standards or defining their own custom mezzanine cards, such as networks line cards used by purpose-built telecommunications appliances.

A better solution can be to route timing signals through the NIC card connectors, via pins 106. However, the PCIe CEM standard generally specifies other uses for most of the pins 106, meaning there is little to no pin availability to provide at a minimum four independent single-ended clocks to replicate what is being propagated with 50-ohm coaxial cabling. Moreover, the pin count requirements increases if differential clocking or separate synchronous Ethernet clocks are also required to be propagated between a system's motherboard and NIC.

However, recently the PCIe CEM 6.0 specification was updated to support the use of universal serial bus (USB) functionality. In that regard, PCIe CEM 6.0 specification transformed two existing pins defined by a joint test action group (JTAG) specification (e.g., defined by institute of electrical and electronics engineers (IEEE) 1149.1 specification) into USB 2.0 pins that can be used for higher speed system management. Prior to the introduction of USB pins, the system management (SM) bus was primarily used for system management communication. Given that the USB is significantly faster than SM bus, the USB interface is favored, resulting in growing disuse of JTAG pins. Such can provide an opportunity to extend the remaining (or all) JTAG pins to become timing synchronization signals.

In existing specifications, at least five total pins are allocated according to JTAG specification. Since two of these were specified for use with USB pins, three other JTAG are not used for the associated USB functionality in the PCIe CEM 6.0 specification.

Accordingly, the disclosed subject matter, in some embodiments, is directed to repurposing at least a portion of the pins allocated to JTAG (e.g., from among pins 106) for precision timing synchronization. By using JTAG pins to propagate precision timing across a NIC, the need for coaxial cables and associated connectors can be reduced or eliminated entirely. Such can be advantageous, as coaxial cables have been shown to be difficult to work with and are often highly disfavored by customers.

Certain techniques detailed herein can be codified into PCIe CEM standard (e.g., PCIe CEM 6.x) and can be leveraged to expand the functionality of PCIe CEM NICs to provide for precision timing synchronization across the NIC connector (as opposed to via coaxial cables) as well as other potential functions in addition to or as an alternative to the precision timing synchronization that is detailed herein. Such can extend the life of PCIe CEM specifications and can allow use cases not currently conceived to be achieved in a standardized way.

In that regard, this disclosure relies on the term “flexIO” in the context of a potential standardization technique and intended to represent a flexible input/output (flexIO) that can be utilized in connection with USB repurposing as well as timing synchronization repurposing. That is, JTAG pins can continue to be repurposed as flexIO pins in a standardized way, as has been done with USB repurposing. These flexIO pins can then be used in connection with precision timing synchronization, as detailed herein, or according to another function entirely.

Example Systems

With reference now to FIG. 2, a schematic block diagram is depicted illustrating an example device 200 that can repurpose certain JTAG pins to propagate precision timing synchronization signals across a PCIe connector instead of via a coaxial cable connector in accordance with certain embodiments of this disclosure. Precision timing synchronization can synchronize a board timing circuit 216 with a NIC timing circuit 240. In some embodiments, device 200 can be a server device such as a datacenter server or another suitable server that leverages precision timing synchronization.

Device 200 can comprise motherboard 202. Motherboard 202 can comprise baseboard management controller (BMC) 204. BMC 204 can be a specialized microcontroller or processor embedded on the motherboard of servers, high-end computers, and other devices. A primary function of BMC 204 can be to provide out-of-band remote management capabilities for the system, allowing administrators to monitor and manage servers remotely, regardless of the server's operating system or whether the server is turned on.

Furthermore, via, for example, SM bus 206, BMC 204 can communicate with NIC connector 208. NIC connector 208 can be configured to couple to, seat, or mount, a NIC, such as PCIe CEM compliant NIC 220. In some embodiments, NIC 220 can be PCIe CEM 6.x specification compliant.

NIC 220 can comprise multiple (e.g., gold finger) pins that can be defined by a PCIe CEM specification. For example, pins 106 indicated in connection with FIG. 1. A portion of pins 106 can be allocated as JTAG pins. As detailed herein, a portion of those JTAG pins can be repurposed as flexIO pins and/or can be otherwise reused to propagate precision timing synchronization signals via flexIO assignment. As noted, by supporting precision timing synchronization signals, applications associated with high precision clock profiles such as G.8265.1, G.8275.1, and G.8275.2 can be supported within a system. A representative example of the JTAG pins that can be repurposed can be found with reference to FIG. 3A.

While still referring to FIG. 2, but turning now as well to FIGS. 3A and 3B, block diagrams 300A and 300B are illustrated. Diagram 300A illustrates an example of specific PCIe CEM pins allocated as JTAG pins being repurposed according to a general flexIO allocation as well as a specific precision timing synchronization allocation of certain pins in accordance with certain embodiments of this disclosure. Diagram 300B illustrates an example of flexIO selection modes and leveraging a selection signal 250 in accordance with certain embodiments of this disclosure.

With specific reference to FIG. 3A, column 304 indicates the specific PCIe CEM pin (e.g., in this example in accordance with PCIe CEM 6.x). Column 306 indicates the JTAG function that was specified for the associated PCIe CEM pin 304. Column 308 indicates the specific flexIO pin assignment to which the associated PCIe CEM pin 304 can be assigned. Column 310 indicates the precision timing synchronization function or role of the associated PCIe CEM pin 304.

As can be seen, two JTAG pins 306 are allocated for USB functionality, namely the JTAG TCK pin (e.g., CEM_A5 pin) and the JTAG TDI pin (e.g., CEM_A6 pin). These pins can be defined as flexIO pins 308 as well, as indicated as USB_D− and USB_D+. In some embodiments, the disclosed techniques can provide timing synchronization without interfering with the USB functionality (e.g., without repurposing the JTAG TCK and TDI pins, which are allocated to USB functionality). Rather, JTAG TDO, TMS, and TRST #, which are not currently used by flexIO 308 to support USB functionality.

Hence, two JTAG pins can be allocated for USB functionality and the remaining three JTAG pins can be allocated for timing synchronization functionality, namely PCIe CEM pins A7, B8, and B9, which respectively correspond to JTAG TDO, TMS, and TRST #pins, can be used for timing synchronization functionality, comprising an embedded pulse per second (ePPS) out pin, an ePPS in pin, or a general purpose input/output (GPIO) pin. As illustrated by reference numeral 312, these three timing synchronization pins can be defined as flexIO pins 308 as well. In this example, the JTAG TDO pin is repurposed as the ePPS out pin, the JTAG TMS pin is repurposed as the ePPS in pin, and the TRST #pin is repurposed as the GPIO pin, but it is understood that any JTAG pin 306 can be repurposed for any specific timing synchronization role depending on implementation.

In this manner, PCIe CEM pins 304 can be transformed to flexIO pins 308 in order to support USB functionality, precision timing synchronization functionality, or both. In some cases other functionality can be supported by flexIO assignment. By convention a pin assigned to “out” represents transmitting from motherboard 202 to NIC 220 while “in” represents receiving by the motherboard 202 from NIC 220. In the context of precision timing synchronization function 310, ePPS pins and/or signals used to provide the timing synchronization functionality can relate to a merged single. As one example, the merged signal can be a 10 megahertz (MHz) low voltage complementary metal oxide semiconductor (LVCMOS) single-ended compatible square wave combined with a synchronization signal altering the duty cycle of the clock every one second, which is further detailed in connection with FIG. 5.

GPIO functionality and direction can also be indicated in record 223. The GPIO pins can be used to indicate physical loss-of-signal, squelch, alarm indicators, and so forth. It is understood that PCIe CEM pins 304, including CEM_A5 pin, CEM_A6 pin, CEM_A7 pin, CEM_B8 pin, and CEM_B9 pin, are intended to be representative and may include other additional or alternative PCIe CEM pins. Moreover, the actual set of PCIe CEM pins 304 are in this example assigned to specific flexIO pins 308, namely USB functions, and timing synchronization functions are merely examples. It is understood that both flexIO pins 308 and timing synchronization pins 310 can be assigned to different members of the set of PCIe CEM pins 304 than is illustrated in this example.

In some embodiments, NIC 220 can be produced without support for USB functionality depending on implementation. In that case, JTAG TCK and TDI pins can also be repurposed (e.g., via flexIO) for timing synchronization purposes. In that case, timing synchronization functions can use a total of five pins rather than three. Such could allow for different frequency modes (e.g., differential frequency mode, single-ended frequency mode) and/or different operating modes. With access to five pins, in some embodiments other singling modes apart from ePPS can be provided and the merging of the signals may not be utilized.

Still referring to FIGS. 2 and 3B, in order to establish the capabilities of an associated NIC 220, device 200 and/or motherboard 202 can leverage SM bus 206 that is routed to NIC 220. In that regard, motherboard 202 can query field replaceable unit (FRU) electrically erasable programmable read only memory (EEPROM) 222. For instance, the disclosed techniques can provide for a new or expanded record 223 that can be stored in FRU EEPROM 222, and that can, inter alia, specify operating modes and/or capabilities of NIC 220.

To select a given mode, the disclosed techniques can transform the selection signal 250 found on motherboard 202 into a two-pin FlexI/O selection bus represented as FlexIO_sel[1:0]. Such can be leveraged to drive a matrix of field effect transistor (FET) switches, buffers, or equivalent to create a flexible signal bus structure. In some embodiments, selection signal 250 can be an auxillary interface connector (AIC) signal such as AIC_TimeSync_EN signal that is similar to the AIC_USB_EN utilized for USB functionality.

Depending on how NIC 220 is provisioned in terms of supporting USB and precision timing synchronization, the FET switch matrix can replicate the associated format onto NIC 220, which can be an optional FET switch matrix. FlexIO data, network controller sideband interface (NCSI) data, and/or timing data can be provided from NIC connector 208 to the FET switch matrix (if present) via channel 210 or otherwise to application-specific integrated circuit (ASIC) 224. Channel 212 can provide for communication with board timing circuit 216 (e.g., on motherboard 202) and NIC timing circuit 240.

The FlexIO_sel[1:0] can operate the same or similar as the selection signal 250 during different power states in order to provide backward compatibility to PCIe CEM NICs that do not support FlexI/O.

FIG. 3B illustrates column 320 that indicates the flexIO_sel[1:0] values. Column 322 indicates a mode description that is associated with a given flexIO_sel[1:0] value of column 320. In this example, a value of “00” can indicate a dual active mode in which both USB and timing synchronization functionality are supported. A value of “01” can indicate a USB (only) active mode in which USB is supported, but potentially not timing synchronization functionality. A value of “10” can indicate that timing synchronization (only) active mode in which timing synchronization is supported by potentially not USB functionality. In this case, USB pins can be further repurposed as detailed above to provide further timing synchronization functionality. A value of “11” ‘can indicate a JTAG active mode (e.g., no support for USB or timing synchronization) which can be utilized for legacy JTAG support and/or for backward compatibility for legacy equipment.

In terms of feature support, motherboard 202 that is designed to support the additional functions can support both USB and timing synchronization modes as defined by the FlexI/O selection setting. The NIC 220 can have the option of supporting either USB, precision timing synchronization, or both. In the presence of legacy PCIe CEM cards as well as motherboards previously designed without FlexI/O, both the NIC 220 and associated motherboard 202 can relegate to supporting just JTAG and/or just USB.

FIG. 4 is intended to be examined in connection with FIG. 2. FIG. 4 illustrates a schematic block diagram 400 illustrating example type(s) of data that can be stored in record 223 in accordance with certain embodiments of this disclosure.

For example, record 223 can comprise operating mode data 402. Operating mode data 402 can relate to the types of operating modes that can be effectuated. For example, operating mode data 402 can represent any of the modes detailed in connection with FIG. 3B such as a dual active mode (e.g., selection signal 250 indicates “00”) in which both timing synchronization functions and USB functions are supported, timing synchronization active mode (e.g., selection signal 250 indicates “10”) in which timing synchronization functions are supported, USB active mode (e.g., selection signal 250 indicates “01”) in which USB functions are supported, or JTAG active mode (e.g., selection signal 250 indicates “11”) in which legacy JTAG functions are supported.

Record 223 can further comprise frequency mode data 404. Frequency mode data 404 can relate to the types of frequency operating modes that can be effectuated. For example, ePPS can be a default frequency mode data 404 indicator. In embodiments in which additional frequency modes can be supported such can relate to, for instance, a single-ended frequency mode, a differential frequency mode, a number of pulses per second, and/or another indicator relating to the pulses, and so on.

Record 223 can be polled to determine what modes are supported as well as instructed to set a particular mode. Such can apply to either or both of operating mode 402 or frequency mode data 404.

FIG. 5 illustrates an example diagram 500 of various aspects of an ePPS signal 506 including construction and transmission of the ePPS signal and receipt and deconstruction in accordance with certain embodiments of this disclosure. It is observed that ePPS has been widely adopted in custom timing solutions given the ability for an ePPS signal to reduce trace count, but to date has not been adopted in an open standard. Both ePPS in and ePPS out detailed in connection with timing synchronization pins 310 of FIG. 3A can combine a frequency clock signal 508 with a time synchronization pulse signal 510 in order to, e.g., reduce pin count. Such can be combined by any suitable equipment such as mixer 502.

In the present example, frequency clock signal 508 can be any suitable frequency, but a 10 MHz square wave signal is detailed here as representative. In this case, the square wave is asserted for approximately half of the duty cycle. Time synchronization pulse signal 510 is asserted for some time periodically. As a representative example, this time is one pulse per second, but it is understood that other periods can be used. One pulse per second is widely adopted and used in telecommunication applications as per the International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) G.703 specification, and therefore can be an advantageous selection in the present scenario regarding PCIe CEM use cases.

However, it is understood that in order to interoperate with a PCIe CEM environment for use with three pins, frequency clock signal 508 and time synchronization signal 510 can be combined and the associated ePPS signal 506 that results from the combination can conform to the 3.3 volt CEM electrical specification. For example, as illustrated, at every time synchronization pulse (e.g., every one second), the timing circuit of the transmitter (e.g., board timing circuit 216 or NIC timing circuit 240) modify the duty cycle of the frequency clock signal 508 from approximately 50% to a different duty cycle. This different duty cycle can correspond with the timing of time synchronization pulse signal 510. In the illustrated example, the duty cycle of frequency clock signal 508 is reduced from approximately 50% to approximately 25% for a single the corresponding wavelength.

The timing circuit of the receiver (e.g., board timing circuit 216 or NIC timing circuit 240) can employ demixer 504 to separate the frequency clock signal 508 and the time synchronization signal 510 into the two discrete constituent waveforms, namely the frequency clock signal 508 and time synchronization signal 510. ePPS signal 506 mixing and demixing can be supported by many timing component suppliers within, for example, associated phase-lock loop (PLL) product portfolios.

As a potential optional element, some systems that adopt the timing synchronization elements detailed herein can further make use of a GPIO pin (e.g., illustrated in connection with the JTAG TRST #pin in FIG. 3A). As noted, the GPIO pin's functionality and direction can be detailed by record 223, allowing for the use of physical loss-of-signal, squelch, alarm indications, and so forth. In several telecommunication use cases, such as enabling network port failover, providing this indication through software mechanisms may not be timely (e.g., fast enough) for a proper failover switching event. Accordingly, having a GPIO pin to indicate a loss-of-signal can allow the CEM adapter's hardware timing circuit indicate to the motherboard's timing circuit that the timing synchronization clock signals may not be relied upon and to switch over to a secondary synchronization clock source.

In accordance with the disclosed techniques, JTAG pins are specifically selected to add timing synchronization to PCIe CEM-compliant NICs such as NIC 220. It is appreciated that other pins (e.g., not allocated to JTAG) can also be used. For example, there are reserved/unused pins such as A19, A32, A33, A50, and B82. While these pins may be used in accordance with the disclosed techniques, it is noted that the above-mentioned pins can present certain signal-integrity or interoperability challenges. For instance, one issue that can present itself is that the introduction of a 10 MHz LVCMOS single-ended clock may then be adjacent to high-speed PCIe data lanes. These clock signals being adjacent to the data lanes can impact the associated signal integrity potentially causing compliance testing failure, operational issues, or both. Moreover, one or more of the above-mentioned reserved/unused pins can be better repurposed for other use cases.

With reference now to FIG. 6, a schematic block diagram is depicted illustrating an example system 600 that can facilitate precision timing synchronization across a NIC connector by repurposing JTAG pins in accordance with certain embodiments of this disclosure. In some embodiments, device 600 can be a server device such as server device 200, as detailed in connection with FIG. 2

Device 600 can comprise at least one processor 602 that, potentially along with time sync device 606, can be specifically configured to perform functions associated with precision timing synchronization. Device 600 can also comprise at least one memory 604 that stores executable instructions that, when executed by the at least one processor 602, can facilitate performance of operations. Processor(s) 602 can be a hardware processor having structural elements known to exist in connection with processing units or circuits, with various operations of processor 602 being represented by functional elements shown in the drawings herein that can require special-purpose instructions, for example, stored in memory 604 and/or time sync device 606. Along with these special-purpose instructions, processor 602 and/or time sync device 606 can be a special-purpose device. Further examples of the memory 604 and processor 602 can be found with reference to FIG. 11. It is to be appreciated that device 600 or computer 1102 can represent a server device or a client device of a network, data center, or telecommunications platform and computer 1102 can be used in connection with implementing one or more of the systems, devices, or components shown and described in connection with FIG. 6 and other figures disclosed herein.

Device 600 can comprise a NIC connector 607, which can be substantially similar to NIC connector 208 detailed in connection with FIG. 2. For example, NIC connector 607 can be configured to interface with a PCIe CEM such as NIC 630, which can be substantially similar to NIC 220 and can be compliant with PCIe CEM 6.0 specifications. As indicated, device 600 can further comprise board timing circuit 620, which can be substantially similar to board timing circuit 206 detailed in connection with FIG. 2.

As previously detailed in connection with NIC 220, NIC 630 can comprise a group of pins 612 that can interface with NIC connector 607. Group of pins 612 can be a portion of pins 106 described in connection with FIG. 1, namely the portion of pins 106 that are allocated to JTAG functions. Additionally, NIC 630 can comprise NIC timing circuit 622 which can be substantially similar to NIC timing circuit 240.

As illustrated at reference numeral 608, device 600 can perform precision timing synchronization procedure 610. Precision timing synchronization procedure 610 can, as indicated at reference numeral 614, utilize at least a portion 616 of group of pins 612 (e.g., pins allocated to JTAG functions) to facilitate a precision timing synchronization 618 between board timing circuit 620 can NIC timing circuit 622.

Turning now to FIG. 7, a schematic block diagram 700 is depicted illustrating additional aspects or elements of the system 600 that can facilitate precision timing synchronization across a NIC connector by repurposing RBT pins in accordance with certain embodiments of this disclosure.

For example, at reference numeral 702, device 600 can determine whether or not NIC 630 supports repurposing the portion 616 of the group of pins 612 that are configured for JTAG for use with precision timing synchronization procedure 610. Such can be accomplished in response to communication with EEPROM 704 of NIC 630. EEPROM 704 can be substantially similar to FRU EEPROM 222 detailed in connection with FIG. 2. For instance, if record 706 and/or record 223 does not exist, then it can be determined that NIC 630 does not support precision timing synchronization procedure 610 and/or flexIO functionality. Otherwise, record 706 can be examined to determine the configuration of NIC 630 in the context of precision timing synchronization procedure 610 or another suitable flexIO function.

At reference numeral 708, device 600 can set an operating mode 710 of NIC 630. For example, operating mode 710 can be set to timing sync active mode 710A in which the precision timing synchronization functions are asserted. Such can correspond to flexIO_sel[1:0] being set to “10” as was indicated in diagram 300B. As another example, operating mode 710 can be set to USB active mode 710B in which RBT functions are asserted. Such can correspond to flexIO_sel[1:0] being set to “01” was indicated in diagram 300B. As yet another example, operating mode 710 can be set to dual active mode 710C in which both the precision timing synchronization functions and the USB functions are supported. Such can correspond to flexIO_sel[1:0] being set to “00” was indicated in diagram 300B.

At reference numeral 712, device 600 can set a frequency operating mode 714 for NIC 630. Frequency operating mode 714 can be substantially similar to that described in connection with frequency mode data 404 of FIG. 4. As a default, particularly in the case of leveraging only three JTAG pins to allow concurrent use of USB, such can be set to an ePPS mode that represents a combination of the frequency clock signal 508 and time synchronization signal 510 detailed in connection with FIG. 5.

In the case of ePPS mode, device 600 can employ ePPS mixer/demixer circuitry 718 in order to combine frequency clock signal 508 and time synchronization signal 510 to form ePPS signal 506 or to separate ePPS signal 506 into the constituent frequency clock signal 508 and time synchronization signal 510 parts. Likewise, NIC 630 can comprise ePPS mixer/demixer circuitry 720 that can operate similar to ePPS mixer/demixer circuitry 718, but in the context of NIC 603 rather than device 600 and/or motherboard 202.

Example Methods

FIGS. 8 and 9 illustrate various methods in accordance with the disclosed subject matter. While, for purposes of simplicity of explanation, the methods are shown and described as a series of acts, it is to be understood and appreciated that the disclosed subject matter is not limited by the order of acts, as some acts may occur in different orders and/or concurrently with other acts from that shown and described herein. For example, those skilled in the art will understand and appreciate that a method could alternatively be represented as a series of interrelated states or events, such as in a state diagram. Moreover, not all illustrated acts may be required to implement a method in accordance with the disclosed subject matter. Additionally, it should be further appreciated that the methods disclosed hereinafter and throughout this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methods to computers.

Turning now to FIG. 8, exemplary method 800 is depicted. Method 800 can facilitate precision timing synchronization across a NIC connector by repurposing JTAG pins in accordance with certain embodiments of this disclosure. While method 800 describes a complete method, in some embodiments, method 800 can include one or more elements of method 900, reached via insert A, as discussed at FIG. 9.

At reference numeral 802, a device comprising at least one processor can determine that a NIC supports a precision timing synchronization operation mode. The determination that the NIC supports the precision timing synchronization operation mode can be made in response to interfacing the device to the NIC that is compliant with a PCIe CEM specification. For example, the NIC can be compliant with a PCI3 CEM 6.x specification.

At reference numeral 804, the device can instruct the NIC to utilize a group of pins specified by a joint test action group (JTAG) specification to perform a precision timing synchronization procedure. In other word, pins that are specified for JTAG functions can be repurposed to precision timing synchronization functions.

At reference numeral 806 the device can synchronize a card time synchronization circuit of the NIC with a board time synchronization circuit of a server motherboard according to the precision timing synchronization procedure. Because NIC pins are being used to effectuate the precision timing synchronization procedure, other cabling such as coaxial cabling that is traditionally used for precision timing synchronization can be unnecessary. Method 800 can terminate in some embodiments, or proceed to insert A in other embodiments, which is further detailed in connection with FIG. 9.

Turning now to FIG. 9, exemplary method 900 is depicted. Method 900 can provide for additional elements or functionality relating to facilitating precision timing synchronization across a NIC connector by repurposing JTAG pins in accordance with certain embodiments of this disclosure.

For example, at reference numeral 902, the device introduced in connection with FIG. 8 can examine a record stored in an electrically erasable programmable read-only memory (EEPROM) of the NIC in order to determine that the NIC supports the precision timing synchronization operation mode. In some embodiments, if the record does not exist, it can be determined that the NIC does not support the precision timing synchronization operation mode.

A reference numeral 904, the device can set an operating mode to a dual active mode that utilizes the group of pins to support USB functions and precision timing synchronization functions concurrently. In some embodiments, other operation modes can be set such as JTAG-only mode (e.g., in cases where the NIC does not support the precision timing synchronization mode), a precision timing synchronization-only mode, a USB-only operating mode, and so on.

A reference numeral 906, the device can utilize a mixer circuit that generates an embedded pulse per second (ePPS) signal. Hence, the device can generate the ePPS signal, via the mixer circuit, by combining a frequency waveform and a synchronization pulse waveform. In some embodiments, the device can utilize a demixer circuit to separate the ePPS signal into component waveforms.

Example Operating Environments

To provide further context for various example embodiments of the subject specification, FIGS. 10 and 11 illustrate, respectively, a block diagram of an example distributed file storage system 1000 that employs tiered cloud storage and block diagram of a computer 1102 operable to execute the disclosed storage architecture in accordance with example embodiments described herein.

Referring now to FIG. 10, there is illustrated an example local storage system including cloud tiering components and a cloud storage location in accordance with implementations of this disclosure. Client device 1002 can access local storage system 1090. Local storage system 1090 can be a node and cluster storage system such as an EMC Isilon Cluster that operates under OneFS operating system. Local storage system 1090 can also store the local cache 1092 for access by other components. It can be appreciated that the systems and methods described herein can run in tandem with other local storage systems as well.

As more fully described below with respect to redirect component 1010, redirect component 1010 can intercept operations directed to stub files. Cloud block management component 1020, garbage collection component 1030, and caching component 1040 may also be in communication with local storage system 1090 directly as depicted in FIG. 10 or through redirect component 1010. A client administrator component 1004 may use an interface to access the policy component 1050 and the account management component 1060 for operations as more fully described below with respect to these components. Data transformation component 1070 can operate to provide encryption and compression to files tiered to cloud storage. Cloud adapter component 1080 can be in communication with cloud storage 1 10951 and cloud storage N 1095N, where N is a positive integer. It can be appreciated that multiple cloud storage locations can be used for storage including multiple accounts within a single cloud storage location as more fully described in implementations of this disclosure. Further, a backup/restore component 1085 can be utilized to back up the files stored within the local storage system 1090.

Cloud block management component 1020 manages the mapping between stub files and cloud objects, the allocation of cloud objects for stubbing, and locating cloud objects for recall and/or reads and writes. It can be appreciated that as file content data is moved to cloud storage, metadata relating to the file, for example, the complete inode and extended attributes of the file, still are stored locally, as a stub. In one implementation, metadata relating to the file can also be stored in cloud storage for use, for example, in a disaster recovery scenario.

Mapping between a stub file and a set of cloud objects models the link between a local file (e.g., a file location, offset, range, etc.) and a set of cloud objects where individual cloud objects can be defined by at least an account, a container, and an object identifier. The mapping information (e.g., mapinfo) can be stored as an extended attribute directly in the file. It can be appreciated that in some operating system environments, the extended attribute field can have size limitations. For example, in one implementation, the extended attribute for a file is 8 kilobytes. In one implementation, when the mapping information grows larger than the extended attribute field provides, overflow mapping information can be stored in a separate system b-tree. For example, when a stub file is modified in different parts of the file, and the changes are written back in different times, the mapping associated with the file may grow. It can be appreciated that having to reference a set of non-sequential cloud objects that have individual mapping information rather than referencing a set of sequential cloud objects, can increase the size of the mapping information stored. In one implementation, the use of the overflow system b-tree can limit the use of the overflow to large stub files that are modified in different regions of the file.

File content can be mapped by the cloud block management component 1020 in chunks of data. A uniform chunk size can be selected where all files that are tiered to cloud storage can be broken down into chunks and stored as individual cloud objects per chunk. It can be appreciated that a large chunk size can reduce the number of objects used to represent a file in cloud storage; however, a large chunk size can decrease the performance of random writes.

The account management component 1060 manages the information for cloud storage accounts. Account information can be populated manually via a user interface provided to a user or administrator of the system. Each account can be associated with account details such as an account name, a cloud storage provider, a uniform resource locator (“URL”), an access key, a creation date, statistics associated with usage of the account, an account capacity, and an amount of available capacity. Statistics associated with usage of the account can be updated by the cloud block management component 1020 based on a list of mappings that the cloud block management component 1020 manages. For example, each stub can be associated with an account, and the cloud block management component 1020 can aggregate information from a set of stubs associated with the same account. Other example statistics that can be maintained include the number of recalls, the number of writes, the number of modifications, and the largest recall by read and write operations, etc. In one implementation, multiple accounts can exist for a single cloud service provider, each with unique account names and access codes.

The cloud adapter component 1080 manages the sending and receiving of data to and from the cloud service providers. The cloud adapter component 1080 can utilize a set of APIs. For example, each cloud service provider may have provider specific API to interact with the provider.

A policy component 1050 enables a set of policies that aid a user of the system to identify files eligible for being tiered to cloud storage. A policy can use criteria such as file name, file path, file size, file attributes including user generated file attributes, last modified time, last access time, last status change, and file ownership. It can be appreciated that other file attributes not given as examples can be used to establish tiering policies, including custom attributes specifically designed for such purpose. In one implementation, a policy can be established based on a file being greater than a file size threshold and the last access time being greater than a time threshold.

In one implementation, a policy can specify the following criteria: stubbing criteria, cloud account priorities, encryption options, compression options, caching and IO access pattern recognition, and retention settings. For example, user selected retention policies can be honored by garbage collection component 1030. In another example, caching policies such as those that direct the amount of data cached for a stub (e.g., full vs. partial cache), a cache expiration period (e.g., a time period where after expiration, data in the cache is no longer valid), a write back settle time (e.g., a time period of delay for further operations on a cache region to guarantee any previous writebacks to cloud storage have settled prior to modifying data in the local cache), a delayed invalidation period (e.g., a time period specifying a delay until a cached region is invalidated thus retaining data for backup or emergency retention), a garbage collection retention period, backup retention periods including short term and long term retention periods, etc.

A garbage collection component 1030 can be used to determine which files/objects/data constructs remaining in both local storage and cloud storage can be deleted. In one implementation, the resources to be managed for garbage collection include CMOs, cloud data objects (CDOs) (e.g., a cloud object containing the actual tiered content data), local cache data, and cache state information.

A caching component 1040 can be used to facilitate efficient caching of data to help reduce the bandwidth cost of repeated reads and writes to the same portion (e.g., chunk or sub-chunk) of a stubbed file, can increase the performance of the write operation, and can increase performance of read operations to portion of a stubbed file accessed repeatedly. As stated above with regards to the cloud block management component 1020, files that are tiered are split into chunks and in some implementations, sub chunks. Thus, a stub file or a secondary data structure can be maintained to store states of each chunk or sub-chunk of a stubbed file. States (e.g., stored in the stub as cacheinfo) can include a cached data state meaning that an exact copy of the data in cloud storage is stored in local cache storage, a non-cached state meaning that the data for a chunk or over a range of chunks and/or sub chunks is not cached and therefore the data has to be obtained from the cloud storage provider, a modified state or dirty state meaning that the data in the range has been modified, but the modified data has not yet been synched to cloud storage, a sync-in-progress state that indicates that the dirty data within the cache is in the process of being synced back to the cloud and a truncated state meaning that the data in the range has been explicitly truncated by a user. In one implementation, a fully cached state can be flagged in the stub associated with the file signifying that all data associated with the stub is present in local storage. This flag can occur outside the cache tracking tree in the stub file (e.g., stored in the stub file as cacheinfo), and can allow, in one example, reads to be directly served locally without looking to the cache tracking tree.

The caching component 1040 can be used to perform at least the following seven operations: cache initialization, cache destruction, removing cached data, adding existing file information to the cache, adding new file information to the cache, reading information from the cache, updating existing file information to the cache, and truncating the cache due to a file operation. It can be appreciated that besides the initialization and destruction of the cache, the remaining five operations can be represented by four basic file system operations: Fill, Write, Clear and Sync. For example, removing cached data is represented by clear, adding existing file information to the cache by fill, adding new information to the cache by write, reading information from the cache by read following a fill, updating existing file information to the cache by fill followed by a write, and truncating cache due to file operation by sync and then a partial clear.

In one implementation, the caching component 1040 can track any operations performed on the cache. For example, any operation touching the cache can be added to a queue prior to the corresponding operation being performed on the cache. For example, before a fill operation, an entry is placed on an invalidate queue as the file and/or regions of the file will be transitioning from an uncached state to cached state. In another example, before a write operation, an entry is placed on a synchronization list as the file and/or regions of the file will be transitioning from cached to cached-dirty. A flag can be associated with the file and/or regions of the file to show that the file has been placed in a queue and the flag can be cleared upon successfully completing the queue process.

In one implementation, a time stamp can be utilized for an operation along with a custom settle time depending on the operations. The settle time can instruct the system how long to wait before allowing a second operation on a file and/or file region. For example, if the file is written to cache and a write back entry is also received, by using settle times, the write back can be re-queued rather than processed if the operation is attempted to be performed prior to the expiration of the settle time.

In one implementation, a cache tracking file can be generated and associated with a stub file at the time the stub file is tiered to the cloud. The cache tracking file can track locks on the entire file and/or regions of the file and the cache state of regions of the file. In one implementation, the cache tracking file is stored in an Alternate Data Stream (“ADS”). It can be appreciated that ADS are based on the New Technology File System (“NTFS”) ADS. In one implementation, the cache tracking tree tracks file regions of the stub file, cached states associated with regions of the stub file, a set of cache flags, a version, a file size, a region size, a data offset, a last region, and a range map.

In one implementation, a cache fill operation can be processed by the following steps: (1) an exclusive lock on can be activated on the cache tracking tree; (2) it can be verified whether the regions to be filled are dirty; (3) the exclusive lock on the cache tracking tree can be downgraded to a shared lock; (4) a shared lock can be activated for the cache region; (5) data can be read from the cloud into the cache region; (6) update the cache state for the cache region to cached; and (7) locks can be released.

In one implementation, a cache read operation can be processed by the following steps: (1) a shared lock on the cache tracking tree can be activated; (2) a shared lock on the cache region for the read can be activated; (3) the cache tracking tree can be used to verify that the cache state for the cache region is not “not cached;” (4) data can be read from the cache region; (5) the shared lock on the cache region can be deactivated; (6) the shared lock on the cache tracking tree can be deactivated.

In one implementation, a cache write operation can be processed by the following steps: (1) an exclusive lock on can be activated on the cache tracking tree; (2) the file can be added to the synch queue; (3) if the file size of the write is greater than the current file size, the cache range for the file can be extended; (4) the exclusive lock on the cache tracking tree can be downgraded to a shared lock; (5) an exclusive lock can be activated on the cache region; (6) if the cache tracking tree marks the cache region as “not cached” the region can be filled; (7) the cache tracking tree can updated to mark the cache region as dirty; (8) the data can be written to the cache region; (9) the lock can be deactivated.

In one implementation, data can be cached at the time of a first read. For example, if the state associated with the data range called for in a read operation is non-cached, then this would be deemed a first read, and the data can be retrieved from the cloud storage provider and stored into local cache. In one implementation, a policy can be established for populating the cache with range of data based on how frequently the data range is read; thus, increasing the likelihood that a read request will be associated with a data range in a cached data state. It can be appreciated that limits on the size of the cache, and the amount of data in the cache can be limiting factors in the amount of data populated in the cache via policy.

A data transformation component 1070 can encrypt and/or compress data that is tiered to cloud storage. In relation to encryption, it can be appreciated that when data is stored in off-premises cloud storage and/or public cloud storage, users can request or require data encryption to ensure data is not disclosed to an illegitimate third party. In one implementation, data can be encrypted locally before storing/writing the data to cloud storage.

In one implementation, the backup/restore component 1085 can transfer a copy of the files within the local storage system 1090 to another cluster (e.g., target cluster). Further, the backup/restore component 1085 can manage synchronization between the local storage system 1090 and the other cluster, such that, the other cluster is timely updated with new and/or modified content within the local storage system 1090.

With reference again to FIG. 11, the example environment 1100 for implementing various example embodiments described herein includes a computer 1102, the computer 1102 including a processing unit 1104, a system memory 1106 and a system bus 1108. The system bus 1108 couples system components including, but not limited to, the system memory 1106 to the processing unit 1104. The processing unit 1104 can be any of various commercially available processors. Dual microprocessors and other multi-processor architectures can also be employed as the processing unit 1104.

The system bus 1108 can be any of several types of bus structure that can further interconnect to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. The system memory 1106 includes ROM 1110 and RAM 1112. A basic input/output system (BIOS) can be stored in a non-volatile memory such as ROM, erasable programmable read only memory (EPROM), EEPROM, which BIOS contains the basic routines that help to transfer information between elements within the computer 1102, such as during startup. The RAM 1112 can also include a high-speed RAM such as static RAM for caching data.

The computer 1102 further includes an internal hard disk drive (HDD) 1114 (e.g., EIDE, SATA), one or more external storage devices 1116 (e.g., a magnetic floppy disk drive (FDD) 1116, a memory stick or flash drive reader, a memory card reader, etc.) and an optical disk drive 1120 (e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.). While the internal HDD 1114 is illustrated as located within the computer 1102, the internal HDD 1114 can also be configured for external use in a suitable chassis (not shown). Additionally, while not shown in environment 1100, a solid state drive (SSD) could be used in addition to, or in place of, an HDD 1114. The HDD 1114, external storage device(s) 1116 and optical disk drive 1120 can be connected to the system bus 1108 by an HDD interface 1124, an external storage interface 1126 and an optical drive interface 1128, respectively. The interface 1124 for external drive implementations can include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are within contemplation of the embodiments described herein.

A number of program modules can be stored in the drives and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134 and program data 1136. All or portions of the operating system, applications, modules, and/or data can also be cached in the RAM 1112. The systems and methods described herein can be implemented utilizing various commercially available operating systems or combinations of operating systems.

Computer 1102 can optionally comprise emulation technologies. For example, a hypervisor (not shown) or other intermediary can emulate a hardware environment for operating system 1130, and the emulated hardware can optionally be different from the hardware illustrated in FIG. 11. In such an embodiment, operating system 1130 can comprise one virtual machine (VM) of multiple VMs hosted at computer 1102. Furthermore, operating system 1130 can provide runtime environments, such as the Java runtime environment or the .NET framework, for applications 1132. Runtime environments are consistent execution environments that allow applications 1132 to run on any operating system that includes the runtime environment. Similarly, operating system 1130 can support containers, and applications 1132 can be in the form of containers, which are lightweight, standalone, executable packages of software that include, e.g., code, runtime, system tools, system libraries and settings for an application.

A user can enter commands and information into the computer 1102 through one or more wired/wireless input devices, e.g., a keyboard 1138, a touch screen 1140, and a pointing device, such as a mouse 1142. Other input devices (not shown) can include a microphone, an infrared (IR) remote control, a radio frequency (RF) remote control, or other remote control, a joystick, a virtual reality controller and/or virtual reality headset, a game pad, a stylus pen, an image input device, e.g., camera(s), a gesture sensor input device, a vision movement sensor input device, an emotion or facial detection device, a biometric input device, e.g., fingerprint or iris scanner, or the like. These and other input devices are often connected to the processing unit 1104 through an input device interface 1144 that can be coupled to the system bus 1108, but can be connected by other interfaces, such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, a BLUETOOTH® interface, etc.

A monitor 1146 or other type of display device can be also connected to the system bus 1108 via an interface, such as a video adapter 1148. In addition to the monitor 1146, a computer typically includes other peripheral output devices (not shown), such as speakers, printers, etc.

The computer 1102 can operate in a networked environment using logical connections via wired and/or wireless communications to one or more remote computers, such as a remote computer(s) 1150. The remote computer(s) 1150 can be a workstation, a server computer, a router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other common network node, and typically includes many or all of the elements described relative to the computer 1102, although, for purposes of brevity, only a memory/storage device 1152 is illustrated. The logical connections depicted include wired/wireless connectivity to a local area network (LAN) 1154 and/or larger networks, e.g., a wide area network (WAN) 1156. Such LAN and WAN networking environments are commonplace in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can connect to a global communications network, e.g., the Internet.

When used in a LAN networking environment, the computer 1102 can be connected to the local network 1154 through a wired and/or wireless communication network interface or adapter 1158. The adapter 1158 can facilitate wired or wireless communication to the LAN 1154, which can also include a wireless access point (AP) disposed thereon for communicating with the adapter 1158 in a wireless mode.

When used in a WAN networking environment, the computer 1102 can include a modem 1160 or can be connected to a communications server on the WAN 1156 via other means for establishing communications over the WAN 1156, such as by way of the Internet. The modem 1160, which can be internal or external and a wired or wireless device, can be connected to the system bus 1108 via the input device interface 1144. In a networked environment, program modules depicted relative to the computer 1102 or portions thereof, can be stored in the remote memory/storage device 1152. It will be appreciated that the network connections shown are examples and other means of establishing a communications link between the computers can be used.

When used in either a LAN or WAN networking environment, the computer 1102 can access cloud storage systems or other network-based storage systems in addition to, or in place of, external storage devices 1116 as described above. Generally, a connection between the computer 1102 and a cloud storage system can be established over a LAN 1154 or WAN 1156 e.g., by the adapter 1158 or modem 1160, respectively. Upon connecting the computer 1102 to an associated cloud storage system, the external storage interface 1126 can, with the aid of the adapter 1158 and/or modem 1160, manage storage provided by the cloud storage system as it would other types of external storage. For instance, the external storage interface 1126 can be configured to provide access to cloud storage sources as if those sources were physically connected to the computer 1102.

Wi-Fi, or Wireless Fidelity, allows connection to the Internet from a couch at home, a bed in a hotel room, or a conference room at work, without wires. Wi-Fi is a wireless technology similar to that used in a cell phone that enables such devices, e.g., computers, to send and receive data indoors and out; anywhere within the range of a base station. Wi-Fi networks use radio technologies called IEEE 802.11 (a, b, g, n, etc.) to provide secure, reliable, fast wireless connectivity. A Wi-Fi network can be used to connect computers to each other, to the Internet, and to wired networks (which use IEEE 802.3 or Ethernet). Wi-Fi networks operate in the unlicensed 5 GHz radio band at a 54 Mbps (802.11a) data rate, and/or a 2.4 GHz radio band at an 11 Mbps (802.11b), a 54 Mbps (802.11g) data rate, or up to a 600 Mbps (802.11n) data rate for example, or with products that contain both bands (dual band), so the networks can provide real-world performance similar to the basic “10BaseT” wired Ethernet networks used in many offices.