Patent ID: 12199746

DETAILED DESCRIPTION

Generally disclosed herein are implementations for supporting precise time synchronization across a datacenter network. The time synchronization includes both a software based time synchronization mechanism and a hardware based time synchronization mechanism to provide more accurate time synchronization across various nodes in the datacenter network. The initial coarse time synchronization across the datacenter network can be performed by software while finer and more accurate time synchronization can be performed by hardware circuitry.

Each network interface controller (NIC) includes a hardware clock to be used within the NIC and its corresponding host as a global time. Each hardware clock can be set by writing a current time control status register (CSR) with a new value, writing a time adjust CSR with a desired step forward or backward, in cycles, or receiving a valid time sync message which requires a step forward or backward, in cycles. The time adjust or time sync clock adjustment can be performed by running the hardware clock slower or faster for as many cycles as needed by the step size. The hardware clock speed adjustment can be performed by adjusting a frequency that is incremented or decremented by itself right shifted by a configurable value. The configurable value can be set aggressively at startup and slowed down to smooth responses when a lock is acquired.

Each hardware clock includes a fine frequency adjustment, which can be implemented via a least significant bit for fractional clock counting. The clock frequency can be set faster or slower by changing the configurable value. The average of the previous number of time sync differences can be continually computed. This can enable NICs to use the fine frequency adjustment to attempt to match the frequency of the master NIC, minimizing differences seen with each time sync message.

Each hardware clock can maintain status bits that indicate a quality of the local current time, corresponding to several parameters to indicate an accuracy of each hardware clock. The time quality parameters can include a time locked parameter, a skip update parameter, and a software bits parameter. The time locked parameter can determine whether the time of the hardware clock is considered locked, which corresponds to being within a threshold range from a master clock. The skip update parameter can correspond to a number of skipped time sync messages reflected in the current time. The software bits parameter can be opaque to the hardware, such that software could use these bits to pass additional information between nodes without involving the hardware.

Current time is distributed from master hardware clocks to client hardware clocks (and from there to second-hop client hardware clocks, etc.) via the time sync messages. The time ordered set can include a subtype, time locked parameter, skip update parameter, software bits parameter, the time, and an error check.

Each NIC can be a master, client, or free running A master NIC can only transmit time sync messages. The master NIC can include software-based time synchronization to synchronize the clock using time adjust and fine frequency adjustments. A client NIC that can receive time sync messages from an upstream NIC (that can be a master or other client NIC closer to the master) or transmit time sync messages to a downstream NIC. The client NIC can include software-based time synchronization for initial setup of the rough time. A free running NIC is independent of master and client NICs. A free running NIC can include software-based time synchronization to synchronize the clock using time adjust and fine frequency adjustments.

Each NIC can transmit time sync messages on any number of output links. The time sync messages are transmitted at a configured interval that can be set by a time update timer register. On client hardware clocks, an incoming time sync message can cause an update to be sent downstream. The time sync message can indicate the actual time of transmission +/−delays. The delays are accounted for by incrementing the time field in the time sync message for each cycle that the time sync message waits to be transmitted.

Each NIC can receive time sync messages via a CSR that can control which port, if any, will honor the time sync messages. When an enabled port receives the time sync message, the time in the time sync message can be adjusted for any on-chip delays to get the time sync message to the hardware clock. Delays can include time to egress the far side NIC, the time across the channel, the time to ingress the local NIC, and flight time across the NIC to the hardware clock.

To consider variable length cables, channels, etc., each NIC includes a link latency measurement capability. The NIC can send a latency request message to a switch. The latency request can be generated on a chip reset or when a time lock is lost. The switch latches the latency request message and generates a latency response back to the NIC. Just as with time sync messages, the time field of the latency message is incremented every cycle the latency response waits to be transmitted. The NIC can receive the latency response and subtract the received time field from the current time.

Each NIC can be initialized with actual global time. The initial global time can be written to all NICs and then transmission and reception of time sync messages can be enabled. The time bound register can be set for an acceptable threshold so the different NICs can lock on to the master. Once all NICs have been locked, the time bound register can be made tighter.

For validity checks, to be accepted at a received NIC, the time sync message can be received on a port enabled for time sync reception, have a valid error check, indicate a time with lock threshold cycles of the local time, and have an incoming time that is locked. The hardware clock can internally maintain two CSR readable counts: a skipped updates count and a valid updates count. Any message that does not pass the validity checks involving time locks can lead to an increment of the skipped update count and a reset of the valid updates count. When the number of skipped updates increases beyond a time bound skipped messages threshold, the master time could have changed, leading to clearing the time lock parameter and interrupting the NIC. When a valid update is received, the hardware clock can move towards the received time and the valid updates count is incremented. The skipped update count can be reset to the incoming skipped update value. When the number of valid updates increases beyond a time bound locked messages threshold, the hardware clock can lock. The time lock parameter is updated based on internal skipped updates and valid update counts while the time itself is updated based on validity of incoming time sync messages.

FIG.1depicts a block diagram of an example network environment100with a network device110. The network environment100includes a network120of interconnected network nodes125. The network nodes125participate in the network120as data sources, data destinations or data sinks, and/or intermediary nodes such as switches, routers, and/or gateways propagating data from sources towards destinations through the network120. The network120includes the network device110with links127to various other participating network nodes125.

The network120can facilitate interactions between participant devices. Example networks include the Internet, a local network, a network fabric, or any other local area or wide area network. The network120can be composed of multiple connected sub-networks or autonomous networks. The network120can be a local-area network (LAN), such as a company intranet, a metropolitan area network (MAN), a wide area network (WAN), an inter-network such as the Internet, or a peer-to-peer network. Any type and/or form of data network and/or communication network can be used for the network120. The network can be public, private, or a combination of public and private. In general, the network120can be used to convey information between computing devices; for example, network nodes125, and the network device110.

The network device110can be a host device or server executing one or more virtual machines130on one or more processors of the network device110. The network device110can include a network interface driver140, a memory150, a network interface card160, a real operating system (OS)170, a hypervisor180, and the virtual machines130. The network device110can communicate with other network devices over the network120. The network device110and other network devices can create or participate in one or more virtual networks to facilitate communication between virtual machines130. The one or more virtual networks can overlay the physical network that makes up the network120. Data packets can be routed within one of the virtual networks overlaying the physical network using a system of data packet encapsulation. The network device110and the network nodes125can route the data packets according to virtual Internet protocol addresses (VIPs) and MAC addresses. In some examples, the network device110can host one or more virtual machines130that transmit data packets or receive data packets. In other examples, the network device110can be an intermediate node of a path between two virtual machines executing on two separate network devices. Accordingly, the network device110can act as a source, destination, or intermediate switch in one or more virtual networks.

The hypervisor180can manage operation of the virtual machines130, including spinning up and terminating the virtual machines130, managing allocation of memory150to the virtual machines130, and live migrating the virtual machines130to other network devices.

The memory150can store data and/or computer executable instructions related to the operation and use of the network interface driver140. The memory150can include a high bandwidth memory (HBM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a static random access memory (SRAM), and/or a flash memory, as examples. The memory150can store computer executable instructions of a transport protocol module175, such as a transmission control protocol (TCP) module or the TCP layer of a network stack, to be executed on a processor of the network device110. While shown in the network interface card160, the transport protocol module175can also be included in the real OS. Functionality occurring within a TCP layer of a network stack can be additionally or alternatively executed in another network protocol module within the transport layer, the network layer, or a combined transport/network layer of a network stack. For example, the functionality can be implemented in a user datagram protocol (UDP) module, reliable datagram protocol (RDP) module, reliable user datagram protocol (RUDP) module, or a datagram congestion control protocol (DCCP) module. A network layer, a transport layer, or a combined transport/network layer can generally be referred to as a packet layer of a network stack.

The network interface driver140can include a network interface driver software module running on the real OS170. As an example, the network interface driver140can be a collection of computer executable instructions stored in the memory150that when executed by a processor help facilitate network communications. As another example, the network interface driver140can be implemented as logic in a hardware processor or other integrated circuit, or as a combination of hardware and software logic. The network interface driver140can communicate with one of the software virtual machines130directly or via a guest OS of the virtual machine130and/or the hypervisor180.

The network interface driver140can be included within a layer of a network stack of the real OS170of the network device110. In an example, the network interface driver140can be included within a data link layer of a network stack and can communicate with a software module or application that is included in an application layer of the network stack. In another example, the network interface driver140can be included within a link layer of a network stack and can communicate with a TCP/IP module that is included in an internet/transport layer of the network stack. In other examples, the functionality occurring within the layer of the network stack can additionally or alternatively be configured to receive packets from another network or transport layer protocol module, such as a UDP module, RDP module, RUDP module, or a DCCP module. While not shown, the network interface driver140can be included as a portion of the network interface card160.

The network interface card160can be configured to transfer data packets over a network protocol, such as a layer 2 protocol. The network interface card160can include a packet forwarder162, a time synchronizer164, and a memory166.

The memory166can store data and/or computer executable instructions related to the operation and use of the network interface card160. For example, the memory166can store time sync messages as well as instructions related to validity checks. The memory166can include a SRAM, HBM, DRAM, SDRAM, and/or a flash memory, as examples.

The packet forwarder162can be implemented in an application-specific integrated circuit (ASIC) or as software executing on a microprocessor. The packet forwarder162can receive data packets for transmission from the virtual machines130and route them to one or more destination virtual machines executing on remote network devices. The packet forwarder162can also perform functions that allow the network interface card160to act as a destination for data packets received over the links127. For example, the packet forwarder162can receive a data packet over one of the lines127, decapsulate the data packet, and forward the data packet to one or more destination virtual machines130executing on the network device110. The packet forwarder162can transmit and receive data packets based on the connection entries stored in the cache166. Each transmission or reception of a data packet can correspond to deallocation or allocation of the memory150, respectively.

The time synchronizer164can be implemented in an application-specific integrated circuit (ASIC) or as a combination of hardware and software executing on a microprocessor. The time synchronizer164can support precise time synchronization via both a software based time synchronization mechanism and a hardware based time synchronization mechanism. Initial coarse time synchronization can be performed by software while subsequent finer time synchronization can be performed by hardware.

FIG.2depicts a block diagram of example time synchronization200of a network device202. The network device202can correspond to the network device110as depicted inFIG.1. The network device202includes a network interface controller (NIC)204that can include time sync software206, time sync hardware208, and a hardware clock210. The time sync software206, time sync hardware208, and hardware clock210can correspond to the time synchronizer164as depicted inFIG.1. The time synchronization200utilizing both time sync software206and time sync hardware208can allow for more accurate time synchronization, leading to more efficient database replicas and saving computing resources to maintain the database replicas.

The time sync software206can include one or more processors in communication with memory. The processors can be any logic circuitry that executes instructions fetched from the memory, including one or more central processing units (CPUs) and/or graphic processing units (GPUs). The time sync hardware208can be implemented as hardware logic circuitry, such as a combination of logic gates, registers, and/or flops, using field programmable gate arrays and/or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

The hardware clock210can be used within the NIC204and its corresponding network device202as a global time. The hardware clock210can count in units of core clock cycles, such as 900 MHz, 1.111 ns per cycle. The hardware clock210can correspond to a 62-bit clock that rolls over in about 180 years.

The NIC204can set the hardware clock210by writing a current time control status register (CRS) with a new value, writing a time adjust CSR with a desired step forward or backward in cycles, or receiving a valid time sync message requiring a step forward or backward in cycles. The NIC204can perform the time adjust or time sync clock adjustment by running the hardware clock210slower or faster for as many cycles as needed based on step size. For example, to step backwards 50 cycles, the hardware clock210can be incremented by 0.5 steps (instead of 1.0 steps) for 100 cycles, resulting in a loss of 50 cycles.

The NIC204can perform speed adjustment of the hardware clock210by adjusting a frequency via the time sync software206. To speed up, the frequency can be incremented by itself right shifted by a configurable value N. As an example, the configurable value N can range from 1 to 7. To slow down, the frequency can be decremented by itself right shifted by the configurable value N. This can result in the frequency being multiplied or divided by the configurable value N. At startup, the time sync software206can set the configurable value aggressively, such as setting N=1 to 2. Over time, the time sync software206can slow down the configurable value, such as setting N=2 to 7, to smooth responses when a lock is acquired.

The hardware clock210can include a fine frequency adjustment performed via the time sync hardware208. As an example, the fine frequency adjustment can have a resolution of 0.5 parts per billion. The fine frequency adjustment can be implemented via additional bits of least significant bits for fractional clock counting.FIG.3depicts a block diagram of an example configuration300for time adjustment. The configuration300includes a time register302, a time fraction adjustment register304, and a fine frequency adjustment register306. As examples, the time register can include 62 bits, the time fraction adjustment register can include 31 bits, and the fine frequency adjustment register306can include 32 bits. It should be noted the bits shown in each register are exemplary and not intended to be limiting.

The fine frequency adjustment register306can indicate how much to increment the time register302on each cycle. The fine frequency adjustment register306can default to incrementing the time register302by 1 bit each cycle, such as being indicated by setting the fine frequency adjustment register306as 0x80000000. The frequency of the hardware clock210can be set faster or slower by changing the configurable value N. For example, to set the hardware clock frequency faster, the fine frequency adjustment register306can be set to (0x80000000+((N<<<31)/1,000,000,000))).

The time fraction adjustment register304can be used for high resolution time adjustment. For example, the time fraction adjustment register304can adjust time at 0.5 parts per billion.

An average of previous time sync differences between the NIC204and upstream NICs can be continually computed by the NIC204. This can enable the NIC204to use the fine frequency adjustment via the time sync hardware208to attempt to match the frequency of a master NIC, minimizing differences seen with each time sync message. Averaging can essentially perform a low pass filter on the time sync differences. Because individual time sync messages can have varying differences due to, for instance, network effects, averaging can negate the impact of those network effects, which allows for achieving the master clock frequency sooner.

Referring back toFIG.2, the time sync hardware208can maintain status bits that indicate a quality of the local current time. The quality can correspond to several time quality parameters indicative of an accuracy of the hardware clock210. The time quality parameters can include a time locked parameter, a skip update parameter, and a software bits parameter.

The time locked parameter can determine whether the time of the hardware clock210is considered locked, which corresponds to being within a threshold range, such as +/−100 ns, from a master clock. The range for whether the time is considered locked can be configurable. At a master clock, the time locked parameter should always be locked, such as indicated by ‘1’ bit. At other hardware clocks, if the time is less than the lock threshold after a configurable number of time sync updates, such as 3, the time sync message can update with step counts less than the lock threshold. If the time is greater than the lock threshold after the configurable number of time sync updates, the time sync message can update with step counts greater than the lock threshold.

The skip update parameter can correspond to a number of skipped time sync messages reflected in the current time. At a master clock, the time sync messages should not be skipped, such as indicated by a ‘0’ bit. As other hardware clocks, after a timeout waiting for a time sync message or if an incoming time sync message is not accepted, such as based on validity checks, the skip update parameter is incremented, such as being incremented by a ‘1’ bit. The skip update parameter can be copied from incoming time sync messages. For example, a hardware clock two hops removed from a master clock may have missed an update, and an upstream hardware clock may have already had its skipped update incremented by 2, which would result in a local skipped update parameter equal to 3. This can also be sent in any outgoing time sync messages. The skip update parameter can saturate upon reaching its maximum value, such as a value of 7. Once the skip update parameter saturates, it is no longer incremented. The time sync hardware208can raise an interrupt to the NIC204once the skip update parameter saturates, so that the NIC204can take further action to identify why so many messages are being skipped.

The software bits parameter can be opaque to the time sync hardware208, such the NIC204could use these bits to pass additional information between network devices without involving the time sync hardware208or hardware clock210. At a master clock, the software bits parameter can be copied from a CSR into outgoing messages. At other hardware clocks, the software bits parameter can be copied from incoming messages and passed downstream. Any change in the software bits parameter on incoming messages to hardware clocks can cause an interrupt to be sent to the NIC204.

Current time is distributed from master hardware clocks to client hardware clocks (and from there to second-hop client hardware clocks, etc.) via time sync messages.FIG.4depicts an example time sync message400configured as a time ordered set. The message can include a type to indicate the message is a time sync message400, such as being represented by “10” inFIG.4. The time sync message400can include a number of bits that are always 0 for a time sync message, such as being represented by “0x00” inFIG.4. The time sync message400can also include a subtype402. The subtype402can correspond to the type of time sync message, such as a time update, latency request, or latency response.

The time sync message400can further include a time locked parameter404, a skip update parameter406, a software bits parameter408, the time410, and an error check412. The time sync message400can also include unused bits reserved for future use, such as being represented by “R” inFIG.4. The error check412can be a cyclic redundancy check (CRR) as an example. The time410included in the time sync messages400can correspond to least significant bits of the time. For example, the time bits can be the least significant 38-bits of the time, which gives a rollover period of about 5 minutes. Since the time sync message400can contain the least significant bits of the time, each NIC can compute the time sync by comparing its local time against the incoming time to determine the +0, +1, +2, etc.

Each NIC can be a master, client, or free running Referring back toFIG.2, the NIC204can correspond to a client NIC for a client device.FIG.2also depicts a master NIC212and other client NICs214,216for a master device218and other client devices220,222, respectively. While only two other client devices are shown for simplicity, any number of client devices can be included for time synchronization.

A master NIC, such as master NIC212, can only transmit time sync messages. The master NIC can include software based time synchronization224and hardware based time synchronization226to synchronize a master hardware clock228using time adjust and fine frequency adjustments. Software based time synchronization224can include a network time protocol or precision clock synchronization protocols. The hardware based time synchronization226can generate the timesync messages to send to downstream client NICs, such as NIC204.

A client NIC, such as the client NIC204,214, or216, can receive time sync messages from one or more upstream NICs and/or transmit time sync messages to one or more downstream NICs. Upstream NICs can be a master NIC or another client NIC closer in a time distribution tree to the master NIC, such as the client NIC204. Downstream NICs can be other client NICs further in a time distribution tree from the master NIC, such as the other client NICs214,216. Each client NIC can include software based time synchronization for initial setup of a rough time for a hardware clock and hardware based time synchronization for subsequent fine-tuning of the time for the hardware clock. As depicted inFIG.2, the other client NICs214,216respectively include time synchronization software230,232, time synchronization hardware234,236, and hardware clocks238,240.

A free running NIC (not shown) is independent of master and client NICs. In a free running NIC, the hardware-based time synchronization can be disabled, and only software-based time synchronization can be enabled. A free running NIC can include software-based time synchronization to synchronize the clock using time adjust and fine frequency adjustments. Free running allows the clock to count in units of time, such as ns, instead of cycles, by setting the fine frequency adjustment in the units of time, such as 1.25 ns for an 800 MHz clock.

Each NIC can transmit time sync message on any number of output links via one or more ports in one or more physical layers. For example, master NIC212can transmit time sync messages through physical layer242, the client NIC204can transmit time sync messages through physical layers244, and the other client NICs214,216can respectively transmit time sync messages through the physical layers246,248. The time sync messages are transmitted at a configured interval, such as about 1 us, 10, or 100 us. The configured interval can be set by a time update timer register in the time sync hardware. On client hardware clocks, an incoming time sync message can cause an update to be sent downstream even if the time update timer has not expired. This can ensure that even the farthest nodes in the time distribution tree do not drift much from the master clock. The time sync message can indicate the actual time of transmission +/−delays, which can be +/−1 ns, 2.5 ns, 5 ns, or 10 ns as examples. The delays are accounted for by incrementing the time field in the time sync message for each cycle that the time sync message waits to be transmitted, which includes flight time across the network device from the hardware clock and arbitration time to be transmitted.

Each NIC can receive time sync messages via a CSR that can control which port, if any, can accept the time sync messages. While each NIC can receive time sync messages from multiple upstream NICs, each NIC should accept and update its time based on only one upstream NIC. For example, if accepted, the client NIC204can receive time sync messages through the physical layer244and the other client NICs214,216can respectively receive time sync messages through the physical layers246,248. If none of its ports are programmed to accept time sync messages, an NIC can have only software based synchronization enabled, thus corresponding to a free running NIC. A single port in the NIC can be enabled to receive the time sync message to prevent the NIC receiving time sync messages from multiple upstream NICs, which could corrupt time computation. When an enabled port receives the time sync message, the time in the time sync message can be adjusted for any delays to get the time sync message to the hardware clock. Delays can include time to egress the transmitting NIC, time across a link in the network, time to ingress the receiving NIC, and flight time across the network device to the hardware clock.

Each port of an NIC can latch, e.g., extract and save, quality parameters of incoming time sync messages, regardless of whether the port is configured to accept the time sync message on that port. This can enable the NIC to check the quality of the clock received at each port, if any, to assist in deciding which port should be enabled for time sync message reception.

The NICs can include a link latency measurement capability to consider variable length cables, channels, etc. when transmitting and/or receiving time sync messages. Time synchronization software, such as the time sync software206in the NIC204, can send a latency request message to a switch250. The time sync software206can generate a latency request based on a reset of the network device202or when a time lock is lost. The switch250latches the latency request message and generates a latency response back to the time sync software206. Just as with time sync messages, the time field of the latency message is incremented every cycle the latency response waits to be transmitted. The time sync software206can receive the latency response and subtract the received time field from the current time. The link latency can indicate the roundtrip latency of physical channels, serializer/deserializer (SERDES), on-chip routing, etc., from one hardware clock to another. Because the clocks are hardware based, the link latency measurements can be fixed, allowing for more accurate synchronization than pure software based clocks.

The NICs can be initialized with actual global time. For example, the time sync software206can initialize the NIC204with the global time as a random value, since synchronization involves relative time. The initial time can be relatively inaccurate, for example +/−10 seconds, in the client NICs so long as thresholds in a time bound register are also large. The time bound register can be included as part of the time sync hardware, such as time sync hardware208. Once the initial global time is written to all NICs in the network, respective time sync software can enable transmission and reception of time sync messages. Each time bound register in respective NICs can be set for an acceptable threshold so the different NICs can lock on to the master NIC. Once all NICs have been time locked, the time bound register can be made tighter, such as about +/−1-100 ns.

The time between a master and any client should not be off by more than the configured threshold of the time bound register, such as initially +/−10 second but later tightened to +/−1-100 ns. Otherwise, the time sync software in an NIC can reset the time. During reset, the time sync software can unlock the time lock and can ensure time sync messages are not accepted or transmitted.

A time sync message can pass a validity check to be accepted at a received NIC, such as the NIC204. The validity check can be performed by the time sync hardware, such as time sync hardware208. The validity check can include determining the time sync message is received at a port enabled for time sync reception, the time sync message has a valid error check, the time sync message indicates a time within a lock threshold cycle of the local time, and/or have an incoming time that is locked.

The hardware clock, such as the hardware clock210, can internally maintain two CSR readable counts: a skipped updates count and a valid updates count. The skipped updates count can correspond to an amount, such as an integer amount, for determining whether the hardware clock should be reset. The valid updates count can correspond to an amount, such as an integer amount, for determining whether the hardware clock should be time locked. Any time sync message that does not pass the validity check involving time locks can lead to an increment of the skipped update count and a reset of the valid updates count. The skipped update count increment and valid updates count reset can be performed by the time sync hardware, such as time sync hardware208. When the number of skipped updates increases beyond a time bound skipped messages threshold, such as 7, the master time could have changed, leading to clearing the time lock parameter and interrupting the NIC. Clearing the time lock parameters and interrupting the NIC can also be performed by the time sync hardware, such as time sync hardware208.

When a valid update is received, the hardware clock can move towards the received time and the valid updates count is incremented. The skipped update count can be reset to the incoming skipped update value. The valid updates count increment and the skipped updates count reset can be performed by the time sync hardware, such as time sync hardware208. The incoming skipped updates are copied into the local skipped update to provide information on how much the current client clock could have drifted from the master clock. When the number of valid updates increases beyond a time bound locked messages threshold, such as 7, the hardware clock can lock. Locking can also be performed by the time sync hardware, such as time sync hardware208. The time lock parameter is updated based on internal skipped updates and valid update counts while the time itself is updated based on validity of incoming time sync messages.

FIG.5depicts a flow diagram of an example process500for synchronizing time between a master hardware clock and one or more client hardware clocks. The example process500can be performed on a system of one or more processors in one or more locations, such as the time synchronizer164of network interface card160as depicted inFIG.1.

As shown in block510, the time synchronizer164for a client hardware clock can receive a time sync message. The time sync message can be transmitted from time synchronizers for a master hardware clock or an upstream client hardware clock.

As shown in block520, the time synchronizer164can perform a validity check on the time sync message. For the validity check, the time synchronizer can determine the time sync message is received at a port of the network interface card160enabled for time sync message reception, the time sync message has a valid error check, the time sync message indicates a time within a lock threshold cycle of the client hardware clock, and/or have an incoming time that is locked.

As shown in block530, in response to the received time sync message passing the validity check, the time synchronizer164can update a time of the client hardware clock. Updating the time can include running the client hardware clock slower or faster for as many cycles as needed based on a step size. Updating the time can further include adjusting a frequency of the client hardware clock.

As shown in block540, in response to the received time sync message passing the validity check, the time synchronizer164can increment a valid update count. The time synchronizer164can also reset a skipped update count to an incoming skipped update value. The valid update counts and the skipped update counts can be maintained in the client hardware clock.

As shown in block550, in response to the valid update count incrementing above a threshold, the time synchronizer164can lock the time of the client hardware clock. Locking the client hardware clock can indicate the client hardware clock is within a desired threshold range from the master hardware clock.

FIG.6depicts an example computing system600, which can be configured as the network device110ofFIG.1. The computing system600can include one or more processors610in communication with memory620, at least one network interface controller650with network interface port660for connection to a network, input/output (I/O) components670, and other components680. The processors610can be in communication with the memory620, network interface controller650, I/O components670, and other components680via a bus640, as an example. The processors610can incorporate, or are connected to, cache memory630. In some instances, instructions are read from memory620into cache memory630and executed by the processors610from cache memory630.

The processors610can be any logic circuitry that executes instructions fetched from the memory620or cache630. The processors610can be microprocessor units or special purpose processors. The computing device600can be based on any processor, or set of processors, capable of operating as described herein. The processors610can be single core or multi-core processors. The processors610can be multiple distinct processors. The processors610can be implemented as circuitry on one or more chips.

The memory620can be any device suitable for storing computer readable data. The memory620can be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media and memory devices, semiconductor memory devices, e.g., EPROM, EEPROM, SDRAM, and flash memory devices, magnetic disks, magneto-optical disks, and optical discs, e.g., CD ROM, DVD-ROM, or Blu-Ray® discs. The computing system600can have any number of memory devices620.

The cache memory630can be a form of computer memory placed in close proximity to the processors610for fast access times. The cache memory630can be part of, or on the same chip as, the processors610. There can be multiple levels of cache630, e.g., level 2 and level 3 cache layers.

The network interface controller650can manage data exchanges via the network interface660. The network interface controller650can handle the physical and data link layers of the open systems interconnection (OSI) model for network communication. Some of the tasks of the network interface controller650can be handled by one or more of the processors610. The network interface controller650can be incorporated into the processors610, such as circuitry on the same chip.

The computing system600can have multiple network interfaces660controlled by a single controller650or multiple network interface controllers650. Each network interface660can be a connection point for a physical network link, e.g., a cat-5 Ethernet link. The network interface controller650can support wireless network connections and an interface port660can be a wireless, e.g., radio, receiver/transmitter. The network interface controller650can implement one or more network protocols such as Ethernet. The computing system600can exchange data with other computing systems via physical or wireless links through a network interface660. The network interface660can link directly to another device or to another device via an intermediary device, e.g., a network device such as a hub, a bridge, a switch, or a router, connecting the computing device600to a data network such as the Internet.

The computing system600can include, or provide interfaces for, one or more input or output (I/O) components670. Input devices can include keyboards, microphones, touch screens, sensors, and pointing devices, such as a mouse or trackball, as examples. Output devices can include video displays, speakers, and printers, as examples.

Other components680can include an I/O interface, external serial device ports, and any additional co-processors. For example, the computing system600may include an interface, e.g., a universal serial bus (USB) interface, for connecting input devices, output devices, or additional memory devices, e.g., portable flash drive or external media drive. The computing device600can include an additional co-processor, such as a math co-processor to assist the processors610with high precision or complex calculations.

Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as “such as,” “including” and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements.