Patent ID: 12259841

DETAILED DESCRIPTION

The technology provides an architecture for a multi-interface card environment, such as a server that includes multiple network interface cards (NICs) or peripheral component interconnect express (PCIe) cards. The architecture includes a passive optical splitter coupled between a leader clock and the multiple interface cards. The optical splitter can be used to distribute clock time from the leader clock to the interface cards. The architecture provides for distribution of timing in a scalable manner in multi-NIC environments for cloud deployments.

In single interface card environments, precision time protocol (PTP) is often used to provide clock information from the leader clock to the network interface card through a single physical link, such as a fiber or cable. When multiple network interface cards are present, requiring a separate cable for each individual network interface card can be costly. However, without dedicated circuitry, PCIe interconnects may cause an inaccurate transfer of clock information due to the jittery nature of PCIe connections.

According to PTP, at time T1 a leader clock sends a synchronization message to a network interface card that includes a follower clock. The synchronization message is received by the network interface card at time T2 due to a delay. The network interface card sends a delay request at time T3 back to the leader clock, which receives the delay request at time T4 and sends a delay response back to the network interface card. The delay request and response can be used to accurately calculate propagation time across the single physical link between the leader clock and the network interface card. The follower clock may calculate the delay as [(T2+T3)−(T1+T4)]/2, where the accurate delay measurement assumes identical delay in both directions.

The architecture in the present disclosure relies on an external passive optical splitter to transfer clock information via PTP to multiple network interface cards by relying on only one port of one of the network interface cards to calculate the delay. The passive optical splitter may be a 1:2 splitter or other type of splitter depending on the number of network interface cards to be synchronized. For example, a 1:2 splitter may be used to synchronize two network interface cards to the leader clock, a 1:4 splitter may be used to synchronize four network interface cards to the leader clock, etc. A 1:2 splitter passively splits light into a 50:50 ratio. Accordingly, light may be equally split between a first network interface card and a second network interface card. The first network interface card may be coupled to the leader clock through the optical splitter by a first path, whereas the second network interface card may be coupled to the leader clock through the optical splitter by a second path. When the path lengths of the first and second paths are similar, the second network interface card may estimate clock time by combining inputs from the leader clock and the delay information from the first network interface card.

In a process of combining inputs from the leader clock and the delay information from the first network interface card, the leader clock transmits a synchronization message containing the time at T1 which is then received by the first network interface card and the second network interface card at T2. Although the first network interface card and second network interface card may receive the synchronization message at slightly different times, this difference is negligible because the Ethernet receive paths have similar delays. At T3, the first network interface card sends a delay request to the leader clock. The delay request is received at the leader clock at time T4. There may be a non-negligible delay in the transmission due to the packet transiting the transmit chain of the first network interface card and the receive chain of the second network interface card. However, this delay does not affect the delay calculation. The leader clock transmits a delay response at T5 to both network interface cards, the delay response being received by each of the first and second network interface cards at time T6. The first and second network interface cards may receive the delay response at slightly different times, but the time when the second network interface card receives the delay response is not used for computing the estimated delay. The first network interface card may notify the second network interface card of the estimated delay.

The above process allows the second network interface card to receive the synchronization message and delay response from the leader clock, therefore obtaining an accurate clock time, without having to separately request a delay response. By combining the transmissions from the controller and the delay information from a first network interface card, a separate physical connection between the controller and the second network interface card is not needed.

FIGS.1A-1Cdepict block diagrams of an example passive clock synchronization architecture100. Referring toFIG.1A, the passive clock synchronization architecture may include leader clock102, first card104, second card106, and passive optical splitter108. leader clock102may include any type of electronic device that provides time synchronization signals to one or more follower clocks on a network. leader clock102may use any type of global time references and national time references. Global time references may be satellite-based and national time references may be radio-based. leader clock102may be a component of a router or a switch on a network. In other examples leader clock102may be a component within a server or other computing device.

follower clocks may refer to a hardware device that receives time information from a leader clock. follower clocks typically communicate on a clock network, which may either be a simple wired serial network, a wired pulse signal, or a more complex Ethernet network. First card104and second card106may be network interface cards (NICs) with controllers that may provide accurate time stamping of incoming and outgoing packets. The NICs may each include a physical hardware clock that functions as a smart clock and the smart clock may be communicable with the leader clock102. Passive optical splitter108may enable a signal on an optical fiber to be distributed among two or more fibers connected to network interface cards.

First card104may be locked to or directly communicable with leader clock102for PTP through network path112. First card104may not be locked to leader clock102but may receive sync messages from leader clock102via network path114.

Referring toFIG.1B, leader clock102may initiate the clock synchronization process by sending a ‘sync’ message to passive optical splitter108. The ‘sync’ message may then be sent to first card104and second card106through passive optical splitter108. Passive optical splitter108may be a 1:2 splitter or 1:N splitter depending on the number of network interface cards to be synchronized. If leader clock102sends the ‘sync’ message at T1, leader clock102records the timestamp at T1. First card104and second card106may receive the ‘sync’ message simultaneously at T2 and records the timestamps respectively.

Referring toFIG.1C, first card104may send a ‘delay_request’ message to passive optical splitter108. It is to be understood that the ‘delay_request’ message is not a query message asking leader clock102about the network delay, but rather, the ‘delay_request’ message may only be an additional message that is sent by one of the follower clocks to compute an accurate estimate of the network delay in the later steps. Once passive optical splitter108receives the ‘delay_request’ message, passive optical splitter108may send the ‘delay_request’ message to leader clock102and second card106may listen to the ‘delay_request’ message through passive optical splitter108. For example, leader clock102may receive the ‘delay_request’ message at T4 and records the timestamp at T4.

Referring toFIG.1D, leader clock102may send a ‘delay_response’ message to passive optical splitter108. The ‘delay_response’ message may be an additional message that leader clock102sends to follower clocks to compute a network delay. In the present example, passive optical splitter108may receive the ‘delay_response’ message first and distribute the message to both first card104and second card106. leader clock102may record the timestamp when the ‘delay_response’ message was sent. For example, leader clock102may record its timestamp at T5, and first card104and second card106may record the timestamps of the time when first card104and second card106received the ‘delay_response’ messages at T6. Since passive optical splitter108processes the received message and distribute the message signal at a 50:50 ratio, both first card104and second card106may receive the ‘delay_response’ message at the same time assuming there are negligible discrepancies between the cable length connecting first card104and passive optical splitter108and the cable length connecting second card106and passive optical splitter108.

FIG.2depicts a block diagram illustrating an example passive clock synchronization200. According to some examples, leader clock202may send messages208and210at time T1. First card204may receive message208at T2 and second card206may receive message210at T2. Messages208and210may include ‘sync’ messages and the message may be an initial message transmitted by leader clock202to synchronize clocks of first card204and second card206. First card204and second card206may interact with each other through a passive optical splitter as illustrated inFIG.1A-1C. At T3, first card204may transmit message212to leader clock202. Leader clock202may receive message212at T4. Message212may include messages equivalent to ‘delay_request’ messages described above in connection withFIGS.1A-1C. At T5, leader clock202may transmit message216to first card204and message218to second card206. First card204may receive message216at T6 and second card206may receive message218at T6. Synchronization220may be performed between first card204and second card206using the delay estimate from the first card204.

According to some examples, a delay measurement may be computed by averaging the time between T2 and T1 and the time between T3 and T4. The time between T2 and T1 may implicate a delay attributable to a cable or fiber length between the input and output ports of leader clock202and first card204. In other examples, the delay measurement may be computed by averaging the time between T3 and T4 and the time between T5 and T6. Second card206may only receive the above-computed delay measurement. That is, second card206may not need to directly engage with leader clock202to compute a delay measurement.

As the present disclosure utilizes a passive optical splitter that linearly connects to one or more network interface cards, each network interface card may receive the same delay measurement data from the first or representative network interface card that initially transmits a ‘delay_request’ message to leader clock202. If second card206interacts with leader clock202to compute second card206's own delay measurement without using a passive optical splitter, the result may be slightly different. A reason for such a discrepancy may be a difference in the length of the cable connected between first card204and leader clock202and the length of the cable connected between second card206and leader clock202.

The present disclosure may be advantageous because a delay measurement is computed once and applied to all other network interface cards connected to the passive optical splitter, thereby preserving the computing resources of the leader clock and reducing the risk of asynchronization among the clocks of multiple network interface cards.

FIG.3illustrates a computing device. leader clock102, first card104and second card105as illustrated inFIGS.1A-1Cmay be implemented on computing device300. Computing device300may include processor302, memory304, user input310, and user output312. Memory304may include instructions306and data308. Memory304may store information accessible by processor302including instructions306that can be executed by processor302. Processor302may also retrieve, manipulate or store data308. Processor302may include one or more central processing units (CPUs), graphic processing units (GPUs), field-programmable gate arrays (FPGAs), and/or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

Instructions306can include one or more instructions that when executed by processor302, cause one or more processors to perform actions defined by the instructions, such as sending ‘sync’, ‘delay_request’ or ‘delay_response’ messages. The instructions306can be stored in object code format for direct processing by the processor302, or in other formats including interpretable scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Instructions306can include instructions for implementing processes consistent with aspects of this disclosure.

Data308may include information relating to timestamps recorded when each ‘sync’, ‘delay_request’, or ‘delay_response’ message is transmitted or received. Data308can be retrieved, stored, or modified by processor302in accordance with instructions306. Data308can be stored in computer registers, in a relational or non-relational database as a table having a plurality of different fields and records, or as JSON, YAML, Protocol Buffers, or XML documents. Data308can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. Moreover, data308can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories, including other network locations, or information that is used by an appropriately configured device to calculate relevant data. Data308may also include relevant information to compute a delay measurement between a leader clock and a follower clock based on the recorded timestamps for each transaction of the above messages.

FIG.4depicts a flow diagram of an example method400for distributing clock information to multiple interface cards. According to block402, a synchronization message may be transmitted to a first network interface card through a passive optical splitter. A synchronization message may be transmitted by a computing device, such as a controller. The computing device may include a leader clock. The computing device may transmit the synchronization message simultaneously to the first network interface card and a second network interface card even though the delay measurement may only take into account the delay between the leader clock and the first network interface card. The computing device may transmit the synchronization message directly to a passive optical splitter and the passive optical splitter may distribute the synchronization message to the first network interface card and the second network interface card.

According to block404, the computing device may receive a delay request transmitted by the first network interface card through the passive optical splitter. The first network interface card may transmit a delay request message either directly to the computing device or through the passive optical splitter. The second network interface card may not need to transmit another delay request message.

According to block406, a delay response message may be transmitted to the first network interface card through the passive optical splitter. The computing device may transmit the delay response message to the passive optical splitter and the passive optical splitter may distribute the delay response message to both the first network interface card and the second network interface card.

According to block408, a second network interface card may listen to the delay request and the delay response through the passive optical splitter. The computing device may compute a delay measurement based on the timestamps recorded when the synchronization message was transmitted and received by the first network interface card and the delay request was transmitted by the first network interface card and the delay request was received by the computing device. The average may be computed using the time taken for the synchronization message and delay request message to travel between the computing device and the first network interface card.

According to block410, the second network interface card may synchronize the second network interface card to at least one of the computing device or the first network interface card based on the listening. According to some examples, the second network interface card may receive the above-computed delay measurement from the first network interface card through the passive optical splitter. In other examples, the second network interface card may receive the computed delay measurement from the computing device.

It is to be understood that the architecture of the present disclosure may include more than two components to be synchronized. For example, the concepts described herein may be used for synchronizing three, four, eight, or any other number of components. Moreover, the architecture of the present disclosure may be used for different environments. For example, the present disclosure may be used to synchronize the clocks between different components such as routers, switches, various microprocessors, and protocol stacks, etc. As such, the synchronized components may be any type of peripheral components, such as PCIe cards, NICs, etc. or other components.

The present disclosure may allow for consistent and reliable synchronization without separate physical connections between multiple network interface cards, and thus, a smaller number of ports are required for each device. Further, since each network interface card does not need to compute an estimated delay, the computing resources of each network interface card may be significantly preserved.

Conditional language used herein, such as, among others, “can,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain examples include, while other examples do not include, certain features, elements and/or states. Thus, such conditional language is not generally intended to imply that features, elements and/or states are in any way required for one or more aspects or that one or more aspects necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or states are included or are to be performed in any particular example. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list.

While the above detailed description has shown, described, and pointed out features as applied to various examples, it will be understood that various omissions, substitutions, and changes in the form and details of the machines or methods illustrated can be made without departing from the scope of the disclosure. As will be recognized, certain examples described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others.

Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing form the subject matter defined by the claims, the foregoing description should be taken by way of illustration rather than by way of limitation of the subject matter define by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as “such as”, “including” and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible examples. Further, the same reference numbers in different drawings can identify the same or similar elements.