Patent Description:
Computer and communication networks may use various schemes and protocols for synchronizing network nodes to a common time-base. One common example of such a protocol is the Precision Time Protocol (PTP) defined in IEEE standard <NUM>-<NUM>, and later versions thereof. PTP is used to synchronize clocks throughout a computer network and may achieve sub-microsecond accuracy.

<CIT> describes techniques for precise clock synchronization, e.g., in a network node. In some embodiments, an apparatus includes a real-time clock circuit, which is configured to output a local clock time, and which comprises a register, which is coupled to receive an offset value, and an adder, which is coupled to sum the local clock time with the offset value in the register so as to give an adjusted value of the local clock time, and a host processor configured to compute the offset value between the local clock time and a reference clock time, and to load the offset value into the register.

<CIT> describes a system for testing recovered clock quality and includes a test device for operating as a timing synchronization protocol master for communicating with a device under test functioning as a timing synchronization protocol slave or a timing synchronization protocol boundary clock to synchronize a clock of the device under test with a clock of the test device.

<CIT> describes a first network device comprising a first clock; and a second network device comprising a second clock, wherein the first network device and the second network device are configured to employ a frequency distribution scheme to attempt to set the second clock to operate at the same frequency as the first clock; the first network device is configured to generate and transmit a synchronous stream of timing packets to the second network device, wherein the timing packets are periodically transmitted based on the first clock; and the second network device is configured to receive the synchronous stream of timing packets and determine, based on comparing the synchronous stream of timing packets to the second clock, whether the second clock is out of sync with the first clock.

In order to illustrate the invention, aspects and embodiments which may or may not fall within the scope of the claims are described herein.

An embodiment of the present invention that is described herein provides a network device including one or more ports for connecting to a communication network, packet processing circuitry and clock circuitry. The packet processing circuitry is configured to communicate packets over the communication network via the ports. The clock circuitry includes a hardware clock configured to indicate a network time used for synchronizing network devices in the communication network, and a built-in accuracy test circuit configured to check an accuracy of the hardware clock. The built-in accuracy test circuit is configured to receive an external reference signal from outside the network device, and to sample the network time output from the hardware clock at a timing derived from the external reference signal.

In some embodiments, the hardware clock is configured to track the network time in accordance with a Precision Time Protocol (PTP).

In an example embodiment, the built-in accuracy test circuit is configured to transmit the sampled network time from the network device. Additionally or alternatively, the built-in accuracy test circuit may be configured to provide the sampled network time to a controller in the network device. In a disclosed embodiment, the external reference signal includes a pulse-per-second (PPS) signal. In some embodiments, the accuracy of the hardware clock is indicated by a deviation between the sampled network time and the external reference signal.

In a disclosed embodiment, the external reference signal includes a dedicated test signal, which differs from a pulse-per-second (PPS) signal and which comprises a predefined pattern, and the built-in accuracy test circuit is configured to identify the dedicated test signal and to sample the network time at the timing derived from the predefined pattern. In an embodiment, the external reference signal includes a <NUM> signal. In an embodiment, the hardware clock is configured to indicate the network time on a parallel output interface, and the built-in accuracy test circuit includes a set of Flip-Flops (FFs) that are configured to sample the parallel output interface at the timing derived from the external reference signal.

There is additionally provided, in accordance with an embodiment of the present invention, a method including, in a network device, communicating packets over a communication network. A network time, used for synchronizing network devices in the communication network, is indicated using a hardware clock in the network device. An accuracy of the hardware clock is checked using a built-in accuracy test circuit in the network device. Checking the accuracy of the hardware clock comprises receiving an external reference signal from outside the network device, and sampling the network time output from the hardware clock at a timing derived from the external reference signal.

In some embodiments, indicating the network time comprises tracking the network time in accordance with a Precision Time Protocol (PTP).

In an example embodiment, wherein checking the accuracy of the hardware clock further comprises transmitting the sampled network time from the network device. Additionally or alternatively, checking the accuracy of the hardware clock further comprises providing the sampled network time to a controller in the network device. In a disclosed embodiment, the external reference signal comprises a pulse-per-second (PPS) signal. In some embodiments, the accuracy of the hardware clock is indicated by a deviation between the sampled network time and the external reference signal.

In a disclosed embodiment, the external reference signal comprises a dedicated test signal, which differs from a pulse-per-second (PPS) signal and which comprises a predefined pattern, and wherein sampling the network time comprises identifying the dedicated test signal and sampling the network time at the timing derived from the predefined pattern. In an embodiment, the external reference signal comprises a <NUM> signal. In an embodiment, indicating the network time comprises outputting the network time on a parallel output interface, and wherein sampling the network time comprises sampling the parallel output interface at the timing derived from the external reference signal using a set of Flip-Flops (FFs).

Any feature of one aspect or embodiment may be applied to other aspects or embodiments, in any appropriate combination. In particular, any feature of a method aspect or embodiment may be applied to an apparatus aspect or embodiment, and vice versa.

Embodiments of the present invention that are described herein provide techniques for built-in accuracy testing of hardware clocks in network devices. The disclosed techniques are useful, for example, for testing the accuracy of PTP Hardware Clocks (PHCs) that are integrated into network adapters and network switches.

In some embodiments, a network device is configured to communicate packets over a communication network. Among other components, the network device comprises a hardware clock that is configured to track and indicate the network time used for synchronizing network devices in the communication network. The network device further comprises a built-in accuracy test circuit, which is configured to check the accuracy of the hardware clock.

Typically, the built-in accuracy test circuit tests the accuracy of the hardware clock relative to an external reference signal, e.g., a Pulse-Per-Second (PPS) signal, which is received from outside the network device. In an embodiment, the hardware clock outputs a digital word indicative of the current network time, and the built-in accuracy test circuit samples this digital word at a timing that is derived from the external reference signal. When using a PPS signal, for example, the built-in accuracy test circuit may sample ("take a snapshot of") the digital word produced by the hardware clock on the rising or falling edge of the PPS signal.

The sampled network time is thus indicative of the estimated network time, as tracked and indicated by the hardware clock at the time-of-arrival of the external signal in the network device. When the external reference signal is derived from some standard time-base, e.g., from a grandmaster clock of the network, the sampled network time is indicative of the accuracy of the hardware clock relative to this standard time-base.

In various embodiments, the built-in accuracy test circuit may act upon the sampled network time in various ways, e.g., send the sampled network time to various destinations for processing. The process of sampling and reporting may be performed, for example, periodically, in response to certain events, or on demand.

The disclosed techniques provide scalable and cost-effective means for testing the accuracy of hardware clocks in network devices. It may be possible in principle to sample the output of a hardware clock and test its accuracy using commercial test equipment. Such a measurement, however, is unsuitable for anything more than sporadic measurements on a small number of network devices. The disclosed techniques, in contrast, can be applied on an ongoing basis to a large number of network devices, e.g., to an entire data center, without requiring any external test equipment.

<FIG> is a block diagram that schematically illustrates a network device <NUM> employing built-in hardware-clock accuracy checking, in accordance with an embodiment of the present invention. Network device <NUM> may comprise, for example, a network adapter such as an Ethernet Network Interface Controller (NIC) or an Infiniband™ Host Channel Adapter (HCA), a network switch or router, a network-enabled Graphics Processing Unit (GPU), or any other suitable type of device capable of network communication.

Network device <NUM> comprises one or more network ports <NUM>, for receiving packets from a network (not shown) and for transmitting packets to the network. The network may comprise, for example, an Ethernet or Infiniband network, or any other suitable network type.

Network device <NUM> further comprises packet processing circuitry for communicating (transmitting and receiving) packets over the network via ports <NUM>. In the present example the packet processing circuitry comprises a data path <NUM>. Data path <NUM> receives packets from the network via ports <NUM>, processes the packets, and sends the packets to the network via the ports. This sort of data path, which is more typical of a switch or router, is depicted by way of example. In a network adapter, for example, the data path may receive packets from a host and send the packets to the network, and vice versa. Further alternatively, any other suitable packet processing circuitry, having any other suitable functionality, can be used.

Network device <NUM> further comprises a hardware clock, in the present example a PTP Hardware Clock (PHC) <NUM>. PHC <NUM> is configured to track the current network time, i.e., the common time-base used for synchronizing the various network devices in the network. To assist accurate tracking, PHC <NUM> may be adjusted ("disciplined") in various ways. In some embodiments, network device <NUM> comprises a PPS-IN input interface <NUM>, for receiving a PPS input signal used for disciplining PHC <NUM>. Additionally or alternatively, PHC <NUM> may receive adjustments from a local host over a suitable local interface (denoted "CLOCK ADJUSTMENTS" in the figure).

Typically, PHC <NUM> outputs a multi-bit digital word (denoted "CURRENT TIME" in the figure) that is indicative, at any given time, of the current network time as tracked by the PHC. In some embodiments, PHC <NUM> also produces a PPS output signal, which may be output from the network device via a PPS-OUT interface <NUM>.

The current time output ("CURRENT TIME") can be used for various purposes in network device <NUM>. For example, to support PTP, data path <NUM> may comprise ingress timestamping circuitry <NUM> and egress timestamping circuitry <NUM>. Ingress timestamping circuitry <NUM> is configured to time-stamp incoming PTP packets with the current time as they enter the network device, and egress timestamping circuitry <NUM> is configured to time-stamp outgoing PTP packets with the current time as they depart the network device.

As another example, data path <NUM>, or network device <NUM> as a whole, may perform various packet processing operations that depend on the current time. Techniques of this sort are described, for example, in <CIT>, entitled "Network Adapter with Time-Aware Packet-Processing Pipeline," in <CIT>, entitled "Packet Scheduling System with Desired Physical Transmission Time for Packets," in <CIT>, entitled "TDMA Networking using Commodity NIC/Switch," and in <CIT>, entitled "Packet Transmission Using Scheduled Prefetching.

Regardless of the specific usage of the network time in network device <NUM>, it is highly desirable to assess the accuracy of PHC <NUM> in tracking and indicating the network time. For this purpose, network device <NUM> comprises a built-in accuracy check (test) circuit <NUM>. The operation of circuit <NUM> will be described in detail further below. Briefly put, circuit <NUM> receives an external reference signal (denoted "EXT-REF") via PPS-IN interface <NUM>. The external reference signal may comprise the same PPS input signal used for disciplining PHC <NUM>, or a different signal, e.g., a dedicated signal used for accuracy testing. Circuit <NUM> samples the "CURRENT TIME" output of PHC <NUM> at a timing that is derived from the external reference signal. The sampled time (denoted "CURRENT TIME SAMPLED @EXT-REF" in the figure) is provided as output for analysis.

The configuration of network device <NUM> shown in <FIG> is an example configuration that is depicted purely for the sake of conceptual clarity. Any other suitable configuration can be used in alternative embodiments. The various elements of network device <NUM> may be implemented using suitable hardware, such as in one or more Application-Specific Integrated Circuits (ASIC) or Field-Programmable Gate Arrays (FPGA). The various elements of network device <NUM> may be implemented using hardware, using software, or using a combination of hardware and software elements.

<FIG> is a flow chart that schematically illustrates a method for built-in hardware-clock accuracy checking in network device <NUM>, in accordance with an embodiment of the present invention. The method begins with PHC <NUM> tracking the network time, at a tracking step <NUM>. At a reference input step <NUM>, built-in accuracy test circuit <NUM> receives an external reference signal ("EXT-REF") via PPS-IN interface <NUM>. At a sampling step <NUM>, circuit <NUM> samples the "CURRENT TIME" output of PHC <NUM> at a timing that is derived from the external reference signal.

In some embodiments, the external reference signal comprises a PPS signal (e.g., a PPS input signal used for disciplining PHC <NUM>, or another PPS signal). In other embodiments, the external reference signal comprises a dedicated test signal, which differs from a conventional PPS signal and is intended for accuracy checking of PHC <NUM>. For example, the external test signal may comprise a predefined pattern of rising and/or falling edges, e.g., three successive edges, whose timing is derived from the actual network time. Circuit <NUM> may monitor the PPS-IN input interface <NUM>, detect the dedicated test signal, and sample the PHC output at a timing defined by the predefined pattern of edges. The external reference signal may have any suitable frequency - In one embodiment the signal is a <NUM> signal.

In one non-limiting embodiment, PHC <NUM> has a parallel output interface, which outputs the current network time and is always valid for readout by clients. Test circuit <NUM> may comprise, for example, a set of Flip-Flops (FFs) that, when triggered by the external reference signal, sample the parallel output interface of PHC <NUM>. In this context, circuit <NUM> may be regarded as an additional client of the PHC. In alternative embodiments, any other suitable configuration can be used.

The accuracy of PHC <NUM> is assessed, at an accuracy estimation step <NUM>. In some embodiments, the accuracy of PHC <NUM> is estimated within network device <NUM>. For example, when network device <NUM> is a network switch, circuit <NUM> may send the sampled network time to a controller of the network switch, which runs software that estimates the PHC accuracy.

In other embodiments, circuit <NUM> may send the sampled network time to a destination that is external to network device <NUM>. Such a destination may comprise, for example, an analyzer or other suitable collector node. When network device <NUM> is a network switch, for example, circuit <NUM> may send the sampled network time over the network, e.g., in a communication packet, or output the sampled network time over a local output interface. When network device <NUM> is a network adapter, for example, circuit <NUM> may send the sampled network time to a local host, e.g., a compute node in which the network adapter is installed. Further additionally or alternatively, the sampled network time may be sent for analysis to any other suitable destination, or multiple destinations.

Assuming the external reference signal tracks the actual network time with high accuracy, the sampled time produced by circuit <NUM> is indicative of the accuracy of PHC <NUM> in tracking the network time.

In various embodiments, the sampled network time ("CURRENT TIME SAMPLED @EXT-REF") produced by circuit <NUM> can be used in various ways to estimate the deviation between the network time indicated by PHC <NUM> and the actual network time. In the present context, the term "deviation between the network time indicated by PHC <NUM> and the actual network time" may refer to a single, absolute difference between the network time indicated by PHC <NUM> and the actual network time, to some statistical measure of the difference between the network time indicated by PHC <NUM> and the actual network time, or to any other suitable form of deviation.

In some embodiments, the accuracy of PHC <NUM> is estimated based on a single measurement of circuit <NUM>, i.e., based on a single sampled network time. Multiple such measurements may be performed at different times. In other embodiments, the accuracy of PHC <NUM> is estimated based on multiple measurements, e.g., by averaging or applying any other suitable statistical calculation over multiple sampled network times.

For example, in some embodiments the sampling operation in circuit <NUM> has a constant delay relative to the external reference signal. In some embodiments the size of this constant delay is known. In these embodiments it is possible to deduce the absolute difference between the network time indicated by PHC <NUM> and the actual network time. In other embodiments the delay is constant but unknown. In these embodiments it is only possible to estimate statistical deviations (between the network time indicated by PHC <NUM> and the actual network time) over multiple measurements. For example, such measurements can be used for estimating the Root-Mean-Square (RMS) error or standard deviation of the network time indicated by the PHC, over some time interval. Further alternatively, the sampled network time ("CURRENT TIME SAMPLED @EXT-REF") can be used in any other suitable way to estimate any other suitable measure of the accuracy of PHC <NUM>.

The accuracy-testing techniques described herein can be used in various systems and applications, e.g., in testing of multiple hardware clocks of multiple network devices in a data center, in telco systems, automotive and industrial networks, robotic factories, and many others.

It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention is defined by the claims.

It will be understood that aspects and embodiments are described above purely by way of example, and that modifications of detail can be made within the scope of the claims.

Each apparatus, method, and feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination.

Claim 1:
A network device (<NUM>), comprising:
one or more ports (<NUM>) for connecting to a communication network;
packet processing circuitry configured to communicate packets over the communication network via the ports; and
clock circuitry, comprising:
a hardware clock (<NUM>) configured to indicate a network time used for synchronizing network devices in the communication network; and
a built-in accuracy test circuit (<NUM>) configured to
check an accuracy of the hardware clock, the network device characterized in
that the built-in accuracy test circuit is configured to receive an external reference signal from outside the network device, and to sample the network time output from the hardware clock at a timing derived from the external reference signal.