Patent Description:
Clock synchronization among network devices is used in many network applications, for example in applications in which a group of nodes in a network need to maintain a consistent, accurate, stable clock within the group. Another application of using a synchronized clock value is for measuring latency between devices. If the clocks are not synchronized the resulting latency measurement will be inaccurate. Synchronization requirements in many applications are extremely stringent. For example, ITU-T Recommendation G. <NUM> allows a maximum synchronization error of <NUM> ns among telecom boundary clocks.

Synchronous Ethernet (SyncE) is an International Telecommunication Union Telecommunication (ITU-T) Standardization Sector standard for computer networking that facilitates transfer of clock signals over the Ethernet physical layer. In particular, SyncE enables clock synchronization inside a network with respect to a master clock. Each network device, such as a switch, a network interface card (NIC), or a router, is required to recover the master clock from high-speed data received from a clock source and uses the recovered master clock for its own data transmission. In this manner, the master clock spreads throughout the network.

<CIT>, whose disclosure is incorporated herein by reference, describes a technique for precise synthesized clock synchronization between network devices. In this patent, a network device includes frequency generation circuitry configured to generate a clock signal, a phase-locked loop configured to generate a local clock based on the clock signal, and a plurality of receivers configured to receive respective data streams from respective remote clock sources. Each receiver of the plurality of receivers is configured to recover a remote clock from a respective data stream. A controller is configured to identify the remote clock recovered by one of the plurality of receivers as a master clock, find a clock differential between the identified remote clock and the local clock, and provide a control signal to the frequency generation circuitry responsively to the clock differential, which causes the frequency generation circuit to adjust the clock signal so as to iteratively reduce an absolute value of the clock differential. Another system for synchronizing clock signals, where a master clock is directly connected to local clock via a dedicated clock link, is disclosed by the document <CIT>.

In order to illustrate the invention, aspects and embodiments which may or may not fall within the scope of the claims are described herein.

Embodiments of the present invention that are described hereinbelow provide improved methods and devices for clock synchronization.

There is therefore provided, in accordance with an embodiment of the invention, a synchronized communication system, including a plurality of network communication devices, including one network communication device that is designated as a root device and one or more others of the network communication devices that are designated as slave devices. Each network communication device includes one or more ports, which are configured to transmit and receive respective communication signals over respective network links and communications circuitry, which is configured to process the communication signals received by the one or more ports so as to recover a respective remote clock from each of the communications signals. A synchronization circuit is integrated in the root device and is configured to provide a root clock signal to the communications circuitry in the root device. Clock links are coupled to convey the root clock signal from the root device to the slave devices so as to serve as a local clock signal for the communications circuitry in the slave devices. A host processor is coupled by control links to the network communication devices and is driven by software to select one of the ports of one of the network communication devices to serve as a master port, to find a clock differential between the root clock signal and the respective remote clock recovered from the master port, and to output, responsively to the clock differential, a control signal causing the synchronization circuit to adjust the root clock signal.

In the disclosed embodiments, the host processor is configured to select the master port from among the ports of both the root device and the slave devices.

In one embodiment, the clock links include cables interconnecting the network communication devices independently of the network links. In some embodiments, the clock links are configured to convey the root clock signal from the root device to a first one of the slave devices and from the first one of the slave devices in series to a second one of the slave devices. Additionally or alternatively, the clock links are configured to convey the root clock signal from the root device to at least first and second ones of the slave devices in parallel.

In the disclosed embodiments, the network communication devices include network interface controllers (NICs) and/or network switches. Additionally or alternatively, the system includes a peripheral component bus connecting the host processor to the network communication devices, wherein the control links are implemented on the peripheral component bus.

In some embodiments, the slave devices include respective synchronization circuits, which are configured to receive the root clock signal from the synchronization circuit of the root device and to generate the local clock signal responsively to the root clock signal. In a disclosed embodiment, the synchronization circuit includes a voltage-controlled oscillator (VCO) and is configured to adjust a voltage applied to the VCO responsively to the control signal.

There is also provided, in accordance with an embodiment of the invention, a method for synchronization, which includes connecting one or more ports in each of a plurality of network communication devices to transmit and receive respective communication signals over respective network links. The communication signals received by the one or more ports are processed so as to recover a respective remote clock from each of the communications signals. One of the network communication devices is designated as a root device and one or more others of the network communication devices as slave devices. A root clock signal is provided to communications circuitry in the root device from a synchronization circuit in the root device. The root clock signal is conveyed via clock links from the root device to the slave devices so as to serve as a local clock signal for the communications circuitry in the slave devices. A host processor is coupled via control links to the network communication devices. The host processor selects one of the ports of one of the network communication devices to serve as a master port. A clock differential is found between the root clock signal and the respective remote clock recovered from the master port. Responsively to the clock differential, a control signal is outputted, causing the synchronization circuit to adjust the root clock signal.

In the disclosed embodiments, the host processor may be configured to select the master port from among the ports of both the root device and the slave devices.

In one embodiment, the clock links comprise cables interconnecting the network communication devices independently of the network links. In some embodiments, conveying the root clock signal comprises transmitting the root clock signal from the root device to a first one of the slave devices and from the first one of the slave devices in series to a second one of the slave devices. Additionally or alternatively, conveying the root clock signal comprises transmitting the root clock signal from the root device to at least first and second ones of the slave devices in parallel.

In the disclosed embodiments, the network communication devices comprise network interface controllers (NICs) and/or network switches. Additionally or alternatively, the control links are implemented on a peripheral component bus.

In some embodiments, the slave devices comprise respective synchronization circuits, and the method comprises receiving the root clock signal from the synchronization circuit of the root device in the slave devices and generating the local clock signal in the respective synchronization circuits of the slave devices responsively to the root clock signal. In a disclosed embodiment, the synchronization circuit comprises a voltage-controlled oscillator (VCO), and providing the root clock signal comprises adjusting a voltage applied to the VCO responsively to the control signal.

In an embodiment, a synchronized communication system includes a plurality of network communication devices, one of which is designated as a root device and the others designated as slave devices. Each network communication device includes one or more ports and communications circuitry, which processes the communication signals received by the one or more ports so as to recover a respective remote clock from each of the signals. A synchronization circuit is integrated in the root device and provides a root clock signal, which is conveyed by clock links to the slave devices. A host processor selects one of the ports of one of the network communication devices to serve as a master port, finds a clock differential between the root clock signal and the respective remote clock recovered from the master port, and outputs, responsively to the clock differential, a control signal causing the synchronization circuit to adjust the root clock signal.

Any feature of one aspect or embodiment may be applied to other aspects or embodiments, in any appropriate combination. In particular, any feature of a method aspect or embodiment may be applied to an apparatus aspect or embodiment, and vice versa.

In SyncE-compliant switches, all of the switch ports are precisely locked to a master clock. The above-mentioned <CIT> describes efficient methods and circuits for identifying the port that is to serve as the source of the master clock and adjusting and distributing the master clock to the other ports.

To increase the number of synchronized ports in current SyncE networks, the master clock is transmitted over the Ethernet physical layer to other switches, meaning that the same path is used for the clock as for the network data. Each additional hop along this clock/data path adds noise and latency. For this reason, the SyncE scheme cannot readily be scaled to larger networks, containing large numbers of devices and Ethernet links.

Embodiments of the present invention that are described herein address these limitations by enabling multiple, separate network devices to share the same precise root clock, which is generated by a root device under the control of synchronization software running on a host processor. The root clock is distributed over dedicated clock links, such as cable links between the network devices, rather than relying on the data network for clock distribution as in conventional SyncE installations. The clock links may be arranged in any desired topology, including clock links in series and/or in parallel, in order to serve large numbers of network devices that are located in mutual proximity. Solutions of this sort are particularly well suited, for example, for sharing a precise root clock among multiple network interface controllers (NICs) serving the same host computer, or among multiple NICs or switches on different shelves of the same rack.

The disclosed embodiments are thus directed to a synchronized communication system comprising multiple network communication devices (such as NICs or switches, for example). Each network communication device comprises one or more ports, which transmit and receive communication signals over respective network links, along with communications circuitry, which processes the communication signals received by the ports of the device so as to recover a respective remote clock from each of the received signals. One of the network communication devices is designated as the root device, while the other network communication devices are designated as slave devices. A synchronization circuit integrated in the root devices provides a precise root clock signal to the communications circuitry in the root device. This root clock signal is conveyed over clock links from the root device to the slave devices so as to serve as a local clock signal for the communications circuitry in the slave devices. Thus, all of the network communication devices are precisely synchronized with one another by means of the root clock.

A host processor, running synchronization software, is coupled by control links to the network communication devices. To synchronize the root clock with a reference clock received from the network, the host processor selects one of the ports of one of the network communication devices to serve as the master port. (The host processor may make the selection autonomously, or in response to a network management command, for example. ) The master port can be selected from among the ports on any of the network communication devices and need not be on the root device. The host processor finds the clock differential between the root clock signal and the respective remote clock recovered from the master port. Based on this clock differential, the host processor outputs a control signal to the synchronization circuit in the root device, which causes the synchronization circuit to adjust the root clock signal so as to reduce the difference between the root clock signal and the remote clock recovered by the master port.

This process of clock adjustment typically continues iteratively, so that the clock differential gradually decreases, and the root clock is maintained within a small range around the reference clock. All the slave devices receive and use this precise reference clock as the source for their own local clocks. Methods and circuit elements that can be applied, mutatis mutandis, for the purpose of precisely adjusting the root clock in the synchronization circuit are described further in the above-mentioned <CIT> and in <CIT>, which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference.

<FIG> is a block diagram that schematically illustrates a synchronized communication system <NUM>, in accordance with an embodiment of the invention. System <NUM> in this example comprises a distribution unit (DU) <NUM>, which communicates over network communication links <NUM>, such as high-speed Ethernet links, with multiple radio units (RUs) <NUM> in a wireless telecommunications network, such as a cellular network. Such networks are subject to stringent synchronization requirements, and are thus a good example of the sort of environment in which embodiments of the present invention can be advantageously deployed. Alternatively, the principles of the present invention may be applied in other sorts of network communication systems in which precise clock control is needed, and particularly when scalability is important. Such alternative implementations and applications will be apparent to those skilled in the art after reading the present description and are considered to be within the scope of the present invention.

DU <NUM> comprises a host processor <NUM>, such as a server comprising a suitable central processing unit (CPU) and memory, which is connected by control links <NUM> to multiple network communication devices, in this case NICs <NUM>, <NUM>, <NUM>, <NUM>,. Each NIC typically comprises an integrated circuit chip or chip set on a respective circuit board. Control links <NUM> are implemented in this sort of unit on a peripheral component bus, such as a PCIe® bus, which connects the host processor to the NICs. Alternatively or additionally, the control link may comprise network links, such as Ethernet links. Each NIC <NUM>, <NUM>, <NUM>, <NUM> comprises two ports <NUM> (labeled PORT <NUM> and PORT <NUM>), which transmit and receive respective communication signals over respective network links <NUM> under the control of communications circuitry <NUM>. Alternatively, each NIC or other network communication device in DU <NUM> may comprise a single port or three or more ports.

Communications circuitry <NUM> carries out network interface and control functions that are known in the art, including Ethernet physical layer (PHY) and medium access control (MAC) functions in the present example. The PHY functions include processing the communication signals received by ports <NUM> from network links <NUM> so as to recover a respective remote clock from each of the received communications signals. A synchronization circuit <NUM> in each NIC <NUM> provides a local clock signal, which is applied by communications circuitry <NUM> in modulating communications signals that are transmitted through the corresponding ports <NUM> to network links <NUM>. The interface and control functions of communications circuitry <NUM> are typically implemented in digital logic circuits, which may be hardwired or programmable. Alternatively or additionally, at least some of the functions of the communications circuitry may be performed by an embedded microprocessor or microcontroller, under the control of suitable software or firmware.

One of the network communication devices - NIC <NUM> in the present example - is designated as the root device for purposes of clock synchronization. The other NICs <NUM>, <NUM>, <NUM> are designated as slave devices. Synchronization circuit <NUM> in the root device generates a root clock signal, under the control of synchronization control software <NUM>, which runs on host processor <NUM>. (Details of the circuits and process used in generating the root clock are shown in the figures that follow and are described hereinbelow with reference thereto. ) Clock links <NUM> convey the root clock signal from the root device to the slave devices so as to serve as the local clock signal for communications circuitry <NUM> in the slave devices.

Typically, synchronization circuits <NUM> in the slave devices, such as in NICs <NUM>, <NUM>, <NUM>,. , receive the root clock signal from the synchronization circuit of NIC <NUM> (the root device) and generate respective local clock signals at the same frequency as the root clock signal. Since the slave devices rely on the root clock signal in this manner, synchronization circuits <NUM> in the slave devices may be simple in their construction and capabilities, for example comprising a phase-locked loop (PLL) with appropriate input and output connections. Alternatively, for greater versatility and robustness in the configuration and operation of system <NUM>, the slave devices may comprise more complex synchronization circuits, similar to those in the root device, as described below.

Clock links <NUM> may comprise, for example, dedicated cables or printed circuit traces, with lengths and transmission characteristics chosen to ensure that synchronization circuits <NUM> in the slave devices are able to synchronize their respective local clocks precisely with the root clock signal received over links <NUM>. In the pictured example, clock links <NUM> connect NIC <NUM> to NICs <NUM>, <NUM> and <NUM> in series. Alternatively or additionally, the clock links may be arranged so that at least some of the slave devices are connected to the root device in parallel (for example in a fan-out configuration, in which each of NICs <NUM>, <NUM>, <NUM> is connected directly by a respective clock link to synchronization circuit <NUM> of NIC <NUM>).

Host processor <NUM> is driven by synchronization control software <NUM> to interact with synchronization circuits <NUM> and control the synchronization of NICs <NUM>, <NUM>, <NUM>, <NUM>,. Software <NUM> may be downloaded to host processor <NUM> in electronic form, for example over a network. Alternatively or additionally, this software may be stored on tangible, non-transitory computer-readable media, such as optical, magnetic, or electronic memory media.

As described further hereinbelow, software <NUM> causes host processor <NUM> to select one of ports <NUM> of one of the NICs to serve as the master port, for example PORT <NUM> on NIC <NUM>. (As noted earlier, the master port may be on any of the NICs, including both the root and slave devices, and software <NUM> may even cause host processor <NUM> to change the master port from time to time. ) Communications circuitry <NUM> in NIC <NUM> recovers a remote clock from the signal received by the master port over the respective network link <NUM> and reports the remote clock frequency to host processor <NUM>. The host processor calculates a clock differential between the frequency of the root clock signal and this remote clock frequency. Alternatively, communications circuitry <NUM> in NIC <NUM> may calculate and report the clock differential to the host processor. In either case, based on this clock differential, host processor <NUM> outputs a control signal to synchronization circuit <NUM> on NIC <NUM> (the root device), instructing the synchronization circuit to adjust the root clock signal so as to reduce the clock differential. This process continues iteratively during the operation of system <NUM>.

The root clock signal generated by synchronization circuit <NUM> in NIC <NUM> is thus locked precisely, with low jitter and low wander, to the remote clock recovered from the communication signals that are received at the master port. When the synchronization is implemented with sufficient precision, for example as described below, the frequency of the root clock will converge to within a few parts per billion (PPB) of the received signal clock. Consequently, NICs <NUM>, <NUM>, <NUM>, <NUM>,. , are able to lock their transmitted symbol rates precisely to the received symbol rate to within a few PPB, as well, in accordance with SyncE requirements.

<FIG> is a block diagram that schematically shows details of communications circuitry <NUM> and synchronization circuit <NUM> in NIC <NUM>, in accordance with an embodiment of the invention. The elements of communications circuitry <NUM> that are shown in <FIG> are those that are used in clock recovery and are thus common to all of NICs <NUM>, <NUM>, <NUM>, <NUM>,. Because NIC <NUM> is the root device in the present example, synchronization circuit <NUM> is configured for precise clock adjustment and control. As noted earlier, slave devices, such as NICs <NUM>, <NUM>, <NUM>, may have similar synchronization circuits or alternatively, their synchronization circuits may be simpler and may even be limited to PLLs <NUM> (shown as part of communications circuitry <NUM> in <FIG>).

Communications circuitry <NUM> comprises two receivers <NUM>, <NUM>, which receive incoming communications signals via PORT <NUM> and PORT <NUM> from the respective network links <NUM>. Receivers <NUM> and <NUM> demodulate and buffer the data carried by the signals in respective buffers <NUM>. A clock and data recovery (CDR) circuit <NUM> running in each receiver <NUM>, <NUM> recovers a remote clock from the received signal, for example based on transitions in the signal level. The clock recovery may be implemented based on any suitable process that is known in the art, such as a delay-locked loop or digital oversampling of the incoming signal. Receivers <NUM>, <NUM> forward the data received in buffers via control links <NUM> to the memory of host processor <NUM> for further processing and forwarding.

Synchronization circuit <NUM> generates a clock signal (in this case the root clock signal), which it conveys to PLLs <NUM>, as well as conveying this clock signal to other NICs <NUM>, <NUM>, <NUM> via clock links <NUM>. Based on this clock signal, PLLs <NUM> generate a local clock, for example in the GHz range. CDR circuit <NUM> of each receiver <NUM>, <NUM> outputs the recovered frequency value and/or a clock differential, which is the difference between the remote clock frequency recovered from the respective network link <NUM> and the local clock frequency generated by PLL <NUM>. Communications circuitry <NUM> conveys the recovered frequencies or clock differentials over control links <NUM> to synchronization control software <NUM> running on host processor <NUM>. Similar clock frequencies or differentials are computed by the receivers in the other NICs and are similarly conveyed to the host processor.

Synchronization control software <NUM> chooses one of ports <NUM> on one of the NICs as the master port and generates a control signal based on the clock differential of the corresponding receiver. The choice of the master port can be controlled by an external network management function. Alternatively or additionally, software <NUM> itself may choose the master port based on suitable criteria and may change the choice of master port, if appropriate, during the operation of the system. For example, the master port may be selected in response to messages indicating the clock quality or based on knowledge of the quality of the crystal oscillators driving the clock signals.

Synchronization circuit <NUM> in the present embodiment comprises a wander cleaner <NUM>, which adjusts the frequency of the root clock signal in response to the control signal generated by synchronization control software <NUM> running on host processor <NUM>. Wander cleaner <NUM> may advantageously comprise a voltage-controlled oscillator (VCO) with a voltage controller that adjusts the voltage applied to the VCO in response to the control signal from software <NUM>. The voltage adjustment may be positive or negative, depending on the sign of the clock differential computed at the master port, and may be applied in very fine steps, for example on the order of <NUM> part per billion (PPB) of the clock frequency, or even less. In one embodiment, wander cleaner <NUM> comprises an Ultra-Low Jitter Network Synchronizer Clock LMK05318, available from Texas Instruments Inc. (Dallas, Texas), which uses VCO technology to generate a precise, stable output clock. This component has inputs for two different clocks, including a crystal oscillator (XO) <NUM> as its main frequency source and a temperature-controlled crystal oscillator (TCXO) <NUM> as a reference frequency source. It is capable of locking the root clock signal to the reference frequency received from the master port to within a few PPB.

Alternatively, synchronization circuit <NUM> may comprise other types of frequency generation circuits, as are known in the art. For example, synchronization circuit <NUM> may comprise an oscillator with clock switching circuitry and a frequency mixer and PLL. As another example, synchronization circuit <NUM> may comprise an analog or digital frequency synthesizer, based on a VCO or other frequency synthesis component. Further details of these sorts of frequency generation circuits are presented in the above-mentioned <CIT> and in <CIT>.

Reference is now made to <FIG> and <FIG>, which schematically illustrate a method for clock synchronization in system, <NUM>, in accordance with an embodiment of the invention. <FIG> is a block diagram showing details of synchronization control software <NUM> and its interaction with NICs <NUM>, <NUM> and <NUM> in the clock synchronization process, while <FIG> is a flow chart showing steps in the method. In this example, NIC <NUM> is the root device and is connected by parallel clock links <NUM> to NICs <NUM> and <NUM>. PORT <NUM> of NIC <NUM> has been selected as the master port.

As shown in <FIG>, synchronization control software <NUM> assigns all the NICs <NUM>, <NUM>, <NUM> that share the same root clock to a synchronization group <NUM>. Software <NUM> running on host processor <NUM> may assign all of the NICs (or other network devices) that are connected to the host processor to be in the same synchronization group, or it may alternatively manage multiple groups, each with its own root clock.

Within each such group <NUM>, NICs <NUM>, <NUM>, <NUM> register their respective ports <NUM> as synchronization providers <NUM>, at a registration step <NUM> (<FIG>). NIC <NUM> is registered as the source of the root clock for the group. Software <NUM> selects the master synchronization provider, i.e., the master port (PORT <NUM> of NIC <NUM>), at a master selection step <NUM>. NIC <NUM> measures the clock frequency of the received communication signal on PORT <NUM>, as explained above, and communicates the frequency (or the frequency differential relative to the root clock) to software <NUM>.

Based on the frequency differential, software <NUM> computes a frequency adjustment to be applied to the root clock, at a frequency computation step <NUM>. The adjustment may be positive or negative, as noted earlier, and it may be applied gradually, in iterative steps, as described in <CIT>. Software <NUM> drives host processor <NUM> to generate and convey a frequency control signal to NIC <NUM>, at a clock synchronization step <NUM>. This frequency control signal may take the form, for example, of a clock synchronization command <NUM>, which is transmitted via control link <NUM> to NIC <NUM>. Synchronization circuit <NUM> in NIC <NUM> adjusts the root clock accordingly. The root clock is transmitted continuously by physical transfer over clock links <NUM> to NICs <NUM> and <NUM>, in a clock transfer step <NUM>.

Although the embodiments described above relate, for the sake of concreteness and clarity, to a particular type of network and system configuration, the principles of the present invention may similarly be implemented in other hardware and software environments, in order to precisely synchronize communications not only of NICs, but also switches and other network devices. All such alternative applications and implementations are considered to be within the scope of the present invention. It will thus be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.

It will be understood that aspects and embodiments are described above purely by way of example, and that modifications of detail can be made within the scope of the claims.

Each apparatus, method, and feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any appropriate combination.

Claim 1:
A synchronized communication system, comprising:
a plurality of network communication devices (<NUM>, <NUM>, <NUM>, <NUM>), including one network communication device that is designated as a root device (<NUM>) and one or more others of the network communication devices that are designated as slave devices (<NUM>, <NUM>, <NUM>), each network communication device comprising:
one or more ports (<NUM>), which are configured to transmit and receive respective communication signals over respective network links; and
communications circuitry (<NUM>), which is configured to process the communication signals received by the one or more ports so as to recover a respective remote clock from each of the communications signals;
a synchronization circuit (<NUM>), which is integrated in the root device and is configured to provide a root clock signal to the communications circuitry in the root device;
clock links coupled to convey the root clock signal from the root device to the slave devices so as to serve as a local clock signal for the communications circuitry in the slave devices; and
a host processor (<NUM>), which is coupled by control links to the network communication devices and is driven by software (<NUM>) to select one of the ports of one of the network communication devices to serve as a master port, to find a clock differential between the root clock signal and the respective remote clock recovered from the master port, and to output, responsively to the clock differential, a control signal causing the synchronization circuit to adjust the root clock signal.