Synchronizing Scheduler Interrupts Across Multiple Computing Nodes

A method, system and program code for synchronizing scheduler interrupts across multiple nodes of a cluster. Network timers and local scheduling timers are clocked off a system source clock. A processor in each computing node repeatedly reads a network time of day counter. The start of scheduler interrupts is synchronized when the time of day counter is at an integer multiple of a synchronizing integer number of network timer ticks. The processor sends an interprocessor scheduler interrupt to other processors in the node to synchronize scheduling timers in the computing node and throughout the cluster.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

A computer cluster is a set of computers running their own operating systems, connected to a high speed network, and coupled so that the computers can act in coordination with one another. Computers in clusters often share data storage, e.g., using a clustered file system. A computer in the cluster can be any single system image created, without limitation, by a microprocessor, a personal computer processor, a supercomputer, a high performance computer system, or any such computer or computer system running an operating system.

The OS (operating system) software running on the computers of a cluster may include, for example, a Windows® operating system by Microsoft Corporation of Redmond, Wash., or a Linux operating system. The BIOS for such a computer may be provided as firmware by a hardware manufacturer, such as Intel Corporation of Santa Clara, Calif.

A generalized computer cluster architecture that may make use of embodiments of the present invention is shown inFIG. 1. The hardware computing nodes on the computer cluster100ofFIG. 1are housed within one or more chassis110,112,114,116. A chassis is an electronic chassis that is configured to house, power, and provide high-speed data communications among a plurality of computing nodes. In specific embodiments, the computing nodes are stackable, modular electronic circuit boards called blades120,122,124. The modular design of a chassis permits the blades to be connected to power and data lines with a minimum of cabling and vertical space. Each blade includes enough computing hardware to act as a standalone computing server. Indeed, there may be a number of processors190,192,194running in any given blade.

Each chassis typically has a chassis management controller130(also referred to as a “chassis controller” or “CMC”) for managing system functions in the chassis110and a number of blades120,122,124for providing computing resources. Each chassis controller130manages the resources for just the blades in its chassis. The chassis controller130is physically and electrically coupled to the blades120-124inside the chassis110by means of a local management bus138. The hardware in other chassis112-116may be similarly configured.

The chassis controllers communicate with each other over a local area network150. The local area network150may be a high-speed LAN, for example, running an Ethernet communication protocol, or other data bus. By contrast, the blades in a chassis communicate with each other using a computing data connection137. To that end, the computing data connection137illustratively has a high-bandwidth, low-latency system interconnect, such as Infiniband.

A system source clock200is contained in a network switch203on the local area network150. The source clock200provides a master clock for the entire cluster. The source clock200may be an external clock in a top level network switch, or it may be designated on one of the chassis or computing nodes on the network150. Any number of clock signals can be derived from the master clock. By varying the ratio of number of master clock ticks to the tick rate of the desired clock signal, a plurality of clock signals can be propagated at differing frequencies and yet based on the same master. Of significance to the embodiments of the present invention, the processors190,192,194in the computing nodes each have a local scheduling timer210,212,214run with a signal clocked off of the system source clock200. The local scheduling timer would exhibit a local scheduling timer tick every certain number of system source clock ticks. The network clock signal would exhibit a network timer tick every certain number of system source clock ticks, which may usually differ from the frequency of local scheduling timer ticks.

Chassis110is shown with greater detail inFIG. 1. In this figure, parts not relevant to the immediate description have been omitted. The chassis controller130is shown with its connections to the local area network150. The chassis controller130may be provided with a chassis data store134for storing chassis management data. In some embodiments, the chassis data store134is volatile random access memory (“RAM”). In other embodiments, the chassis data store134is non-volatile storage such as a hard disk drive (“HDD”) or a solid state drive (“SSD”).

Whereas clusters may be formed with computing nodes connected directly to the local area network150, in the embodiment shown the computing nodes are all housed in chassis. To service such an architecture, chassis controller130includes a network switch330. In specific embodiments of the invention, the local area network150supports the Ethernet protocol and the network switch330is an Ethernet switch. The network switch includes one or more physical layer transceivers (PHY)332that connect to the local area network150. The PHY transmits and receives network packets and also provides a clock signal recovered from the receive path from local area network150. The recovered clock signal from the PHY332is distributed to all of the computing nodes in the chassis110. Preferably, the clock signal from the PHY is fed through a clean-up phase locked loop to provide a clean clock signal for use as the root of all clock signals in the computing nodes. By basing the clock signals in the processors190,192,194and the clock signals on the LAN150on the source clock200, frequency synchronization is achieved.

The network switch330is also responsible for distributing time of day to all of the computing nodes in the chassis110. In some embodiments, the network switch relies upon a transparent clock functionality.

One unique example of an alternate system and method for deriving a synchronous local clock signal from a system source clock is disclosed in copending application docket number 3549/123, U.S. Ser. No. 13/798,604, filed Mar. 13, 2013, entitled “Global Synchronous Clock” and assigned to the same assignee as the present application. The full disclosure of this copending application is hereby incorporated by reference herein. In this alternative method for recovering the synchronous clock signal locally, a physical layer transceiver (“PHY”) is provided in each of the computing nodes120,122,124.

FIG. 1shows relevant portions of a specific implementation of blade120. The other blades may be similarly constructed. The blade120includes a blade management controller160(also called a “blade controller” or “BMC”) that executes system management functions at a blade level, in a manner analogous to the functions performed by the chassis controller at the chassis level. In addition, the blade controller160may have its own RAM162to carry out its management functions. The chassis controller130communicates with the blade controller of each blade using the local management bus138.

The blade120also includes one or more processor chips170,172that are connected to RAM182,184. Blade120may be alternately configured so that multiple processor chips may access a common set of RAM on a single bus, as is known in the art. The processor chips170,172are connected to other items, such as a data bus that communicates with I/O devices188, a data bus that communicates with non-volatile storage186, and other buses commonly found in standalone computing systems. (For clarity,FIG. 1shows only the connections from processor170to these other devices.) The processor chips170,172may be, for example, Intel® Core™ processor chips manufactured by Intel Corporation. The I/O bus may be, for example, a PCI or PCI Express (“PCIe”) bus. The storage bus may be, for example, a SATA, SCSI, or Fibre Channel bus. It will be appreciated that other bus standards, processor types, and processor manufacturers may be used in accordance with illustrative embodiments of the present invention.

It should also be appreciated that processor chips170,172may include any number of processors190,192,194such as central processing units (“CPUs”) or cores, as is known in the art. Each processor190,192,194is associated with a local scheduling timer210,212,214. In specific embodiments, the local scheduling timers may be LAPIC timers. The scheduling timer issues an interrupt at every scheduling timer tick to trigger operating system housekeeping tasks. The processors in the cluster may also be programmed to recognize major scheduling frames every certain number of scheduling timer ticks. The number of scheduling timer ticks in a frame is made known throughout the cluster. At the beginning of each frame of processing, larger housekeeping tasks may take place in the processors thoughout the cluster. Since the processors on a cluster are only loosely coupled, the scheduling timer ticks issued in each local scheduling timer are not synchronized without performing a synchronization. A method for synchronizing the scheduling timer ticks will be described below with regard toFIG. 2.

In order to enhance the accuracy of time synchronization of scheduling timer ticks, in accordance with embodiments of the invention, time synchronization is provided for the clock signal used on the LAN150. In specific embodiments, precision time protocol is used to synchronize clocks throughout the local area network150. One particular protocol is known as IEEE 1588. Thus, the network controller230selected for use in each computing node, according to such an embodiment, supports the IEEE 1588 protocol. This is a protocol that uses time stamps to identify latencies in the network and then compensates for any such latencies. Thus, a time of day can be provided to all computing nodes so that they all have simultaneous and synchronous time of day. Each network controller230includes a time of day counter234and a clock timer232. The network controllers230that support IEEE 1588 include time stamping capability. The source clock200periodically sends IEEE 1588 SYNC packets on the LAN150. Network switches along the way are transparent clocks which account for the variable propagation delay of the IEEE 1588 packets through the switch. Preferably, this is a one-step grand master source clock that places the current time in the IEEE 1588 SYNC packet as it leaves the switch and one-step transparent clocks that adjust the IEEE 1588 SYNC packet time field as it flows through the switch. The network controllers230in the computing nodes act as ordinary slave clocks as they receive the IEEE 1588 SYNC packets and use them to synchronize their local time of day counters234. Thus, the IEEE 1588 protocol synchronizes the time of day counters in all of the computing nodes in the cluster.

In accordance with aspects of the present invention, the network timing is used by a processor in each computing node to synchronize the timing of the local scheduling timers. The synchronization method shall now be described with reference toFIG. 2. As explained with regard to the cluster architecture, clock signals in the processors of the computing nodes are derived from a system source clock200. Thus, the local processor timers, and hence the local scheduling timers, are clocked402off the system source clock. This provides frequency synchronization across computing nodes. Network timers are also clocked404off the system source clock. In a large cluster, time delays across the local area network150may result in time variations across multiple nodes on the cluster. To overcome such issues, it is desirable to also time synchronize the network clock signal across the cluster. In accordance with a specific embodiment, time synchronization may be achieved by implementation of IEEE 1588 on the local area network150. The synchronization of network timing in frequency and time is reflected in the time of day counters234in all of the computing nodes.

Program code is written into a non-transitory computer-readable medium, such as processor memory, for synchronizing the scheduling timer ticks. The code is performed upon starting up a cluster or joining a cluster. In specific embodiments, the code is made part of the Linux kernel. At least one processor in each computing node120performs the code. The processor is instructed to read406the network time of day counter234in the computing node. The processor determines408whether the time of day indicates an integer multiple of a synchronizing integer number of network timer ticks have elapsed since a predetermined start time. Typically the start time will be time zero, but another agreed start time may be used instead. The synchronizing integer corresponds to a number n of network timer ticks in an interval between simultaneous occurrences of a network timer tick and a local scheduling timer tick when the desired synchronization has been achieved. If inFIG. 3, signal A is the desired network timing signal and signal B is the local scheduling timer signal, then the number 6 could be used as the synchronizing integer, since every 6 network timer ticks, the network timer tick coincides with the scheduling timer tick. This was a simple example where all of the ticks in signal B occur simultaneously with a tick in signal A. Mathematically, one could determine a synchronizing integer by dividing the size of a scheduling tick interval by the size of a network tick interval and multiplying by an integer which results in an integer. For example, when there are 6 network ticks per scheduling tick 6/1 times 1 equals 6. If there are 5 network ticks per every 17 scheduling ticks, then each scheduling tick interval is 5/17 the size of a network tick. Multiplying by the integer 17 produces the integer 5 which can be used as the synchronizing integer.

It is further desirable to synchronize scheduling frames across the multiple nodes of the cluster. To achieve this objective, the synchronizing integer n used to compare with the time of day counter should also equal a number of network timer ticks between desired simultaneous occurrences of a network timer tick and a local scheduling timer tick at the beginning of a scheduling frame. Typically, a preset multiple of the number of local scheduling timer ticks in each scheduling frame will satisfy this requirement. A synchronizing integer that satisfies this requirement will necessarily satisfy the first requirement for synchronizing scheduler interrupts. To better understand the synchronizing integer used in the time of day counter comparison408, we refer toFIG. 3. Assume that the clock signal A is the local scheduling timer clock signal and the clock signal B is the network clock. Note that both are frequency synchronized as they are both derived from the same source clock. The synchronizing integer is predetermined to set all of the scheduling timers in a processing node to synchronously produce scheduling ticks and for the processors in the node to start scheduling frames on the same clock tick used in all the nodes on the cluster. For illustration purposes, we assume that the cluster has set the length of a scheduling frame at13local clock ticks. If a tick on the local scheduling timer clock signal A and a tick on the clock signal B both coincide with the start of a scheduling frame, the next time ticks on both signals will coincide with the start of a scheduling frame will be 13 ticks later on the network clock. Indeed, every thirteen ticks on the network clock there will be a new scheduling frame. Since all nodes on the cluster have access to read a time of day counter, frame scheduling may begin for any node when the time of day counter is at an integer multiple of 13 network timer ticks have elapsed since a predetermined start time. In this example, each scheduling interval is ⅙ the size of a network timer interval multiplied by 6 equals 1. So a synchronizing integer of 1 or indeed any multiple of 1 can be used as the synchronizing integer in this example for scheduler interrupt synchronization. To achieve frame synchronization, the integer needs to be a preset multiple of 13. Thus the synchronizing integer may be 13, 26, 39, . . .

By using IEEE 1588 to time synchronize the time of day counters even better synchronization of scheduling frames across the cluster is achieved. The program only needs to be run once upon joining the cluster to synchronize the scheduling ticks and frames for a computing node. If desired the program code can be run diagnostically to make sure there has been no divergence from the synchronized schedule for initiating scheduling frames. If a problem is detected, the program can be run to resynchronize the scheduling frames.

Now assume the network has a faster clock than the local scheduling timer. In this example, clock signal A is the network clock and the clock signal B is the local scheduling timer clock signal. When this example was discussed above, the synchronizing integer was determined to be a multiple of 6 because every 6 network timer ticks the network timer tick coincides with a local scheduling timer tick. For an illustration of frame synchronization, we assume that the cluster has set the length of a scheduling frame at 13 local clock ticks in signal B. If a tick on the local scheduling timer clock signal B and a tick on the network clock signal A both coincide with the start of a scheduling frame, the next time ticks on both signals will coincide with the start of a scheduling frame will be 78 ticks later on the network clock A. Indeed, every 78 ticks on the network clock there will be a start of a new scheduling frame. The synchronizing integer must be a multiple of 6 to achieve scheduling tick synchronization and a multiple of 13 to achieve frame synchronization. 6 times 13 equals 78. So the synchronizing integer in this example can be 78, 156, 234 . . . to satisfy the requirements for scheduler interrupt synchronization and frame synchronization. Since all nodes on the cluster have access to read a time of day counter, frame scheduling may begin for any node when the time of day counter is an integer multiple of 78.

If the time of day counter does not indicate an integer multiple of the synchronizing integer has elapsed since the predetermined start time, the processor repeats the read of the time of day counter. A start time of time zero is shown inFIG. 2. The counter is repeatedly read until an integer multiple of the synchronizing integer has elapsed. Upon detecting a time of day corresponding to an integer multiple of the synchronizing integer beyond the start time, the processor sends410an interprocessor scheduler interrupt to the other processors in the computing node. This will synchronize the scheduling timers and, if frame synchronization is being used, will indicate to the processors the beginning of a frame. Upon receiving the interprocessor scheduler interrupt each of the local scheduling timers is set to properly time its scheduling ticks and hence the scheduler interrupts associated with each of the scheduling ticks. A scheduling timer sends a scheduler interrupt to its associated processor at each scheduling tick. The processors will thereafter count from the start of a frame the local scheduler interrupts per frame. When the count indicates a next frame starts, the processor will begin the major housekeeping tasks in unison with all the other processors on the cluster.

By synchronizing the local scheduling timers from the network time of day counter, the scheduling timer ticks and the scheduling frames in all processors on the computing node can be set to start synchronously. By time synchronizing the network timers in the cluster, the scheduling timer ticks and frames for all processors across the multiple nodes on the cluster are synchronized to begin at the same time. By synchronizing the scheduling frames across the cluster, the cluster more efficiently attends to performance critical computing tasks.

In an alternative embodiment, the disclosed apparatus and methods (e.g., see the various flow charts described above) may be implemented as a computer program product for use with a computer system. Such implementation may include a series of computer instructions fixed either on a tangible medium, such as a computer readable medium (e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to a computer system, via a modem or other interface device, such as a communications adapter connected to a network over a medium.

The medium may be either a tangible medium (e.g., optical or analog communications lines) or a medium implemented with wireless techniques (e.g., WIFI, microwave, infrared or other transmission techniques). The series of computer instructions can embody all or part of the functionality previously described herein with respect to the system. The process ofFIG. 2is merely exemplary and it is understood that various alternatives, mathematical equivalents, or derivations thereof fall within the scope of the present invention.