Abstract:
A method and apparatus for controlling processor clock rates of a synchronous multi-processor system in response to an environmental condition of a processor. In one embodiment, a processor-reported an environmental condition is stored in a register and all processors are interrupted simultaneously. Upon interrupt, each processor reads the contents of the register and responds by adjusting its local clock rate synchronously with the other processors. In another embodiment, the processor&#39;s environmental status is polled by software control. Upon notification of an environmental condition, the software control notifies each processor to adjust its local clock rate synchronously with the other processors.

Description:
FIELD OF THE INVENTION 
     The present invention relates generally to controlling system clocks and specifically to controlling system clock rate in response to an environmental condition. 
     BACKGROUND OF THE INVENTION 
     Through increasingly larger die sizes and circuit miniaturization techniques, advances in integrated circuit (IC) fabrication have lead to the development of individual ICs housing millions of transistors. At the same time, these densely populated ICs are performing greater and greater numbers of operations per second. As a result, today&#39;s microprocessors operate at higher power levels producing heat that must be managed to prevent failures. External cooling devices can be used to dissipate heat. These devices include both passive devices, such as heat sinks, and active devices such as fans and thermoelectric coolers. More recent techniques include the incorporation of thermal sensors within ICs that monitor on-die temperature and provide control signals to an active cooling system. Other solutions include an ability to control IC temperature without the use of external cooling mechanisms. One solution incorporates a clock-throttling mechanism used to slow the processing speed of the IC in response to an over-temperature condition reported by the IC&#39;s thermal sensor. The reduction in processing speed reduces power consumption thereby reducing the amount of heat to be dissipated. 
     Fault-tolerant computing systems that use hardware redundancy may be constructed with multiple modules, such as central processing units (CPUs), operating in a synchronous, lock-step relationship (performing the same instructions on the same clock cycle). It is desirable for these systems to employ current-technology, enabling commodity ICs to reap the benefits of reduced size, increased performance and reduced cost. Being subject to the thermal conditions described above, it is desirable to utilize ICs with a thermal sensing capability in combination with a clock- throttling technique to manage the thermal load while preserving system reliability. However, allowing each processor within a fault-tolerant system to reduce its own clock rate as its temperature rose above a predetermined threshold would remove the ability of the processors to operate in lock step. This is because thermal gradients caused by ambient conditions would result in the on-die temperature of individual processors increasing above a common threshold at different times. This loss of synchronization would then result in system failures. The present invention avoids this problem. 
     SUMMARY OF THE INVENTION 
     The present invention relates to methods and apparatus for controlling central processing unit (CPU) power consumption in response to a reported environmental condition by varying the clock rate of each processor of a synchronous multi-processor system. The object of this invention is to maintain synchronization of all processors before, during, and after any clock-rate variation response to reported environmental conditions. 
     In accordance with one embodiment of the invention, a computer system employs two or more identical CPUs, each containing a microprocessor executing the same instructions at substantially the same time, according to the processor clock rate. Each microprocessor includes a thermal sensor that continuously monitors the on-die temperature and compares it to a pre-stored threshold value. In this embodiment, one feature of the invention is that when the measured on-die processor temperature crosses the threshold value, the microprocessor writes the result to a common, external interrupt register. 
     Another feature of the invention is the simultaneous interrupt notification to all CPUs of the multi-processor system of a reported over-temperature condition on any CPU. Upon interrupt notification, each microprocessor halts all applications and enters a service-handling routine where the contents of the interrupt register are read to determine the cause of the interrupt. Yet another feature of the invention is the simultaneous reduction of each processor clock rate to a lower rate in response to the reported over-temperature interrupt. In one embodiment, the microprocessor controls a local phase-locked loop (PLL) to reduce its local clock rate. When the microprocessor detects an over-temperature condition, it actuates a reduction of the local clock rate to a lower rate. Operation at the lower clock rate will reduce microprocessor power consumption resulting in an eventual reduction of the microprocessor&#39;s, on-die temperature. 
     In another embodiment, one feature of the invention is a software-controlled polling of each CPU to report any over-temperature conditions. In this embodiment, each microprocessor is in communication with a register having a bit, or multiple bits, that effects control of the processor clock rate. Other embodiments are envisioned where the register effecting control of the processor clock rate may be contained within the microprocessor. Another feature of the invention is to simultaneously set the clock rate reduction bit, or multiple bits, within each register. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The invention is pointed out with particularity in the appended claims. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. Like reference characters in the respective drawing figures indicate corresponding parts. The advantages of the invention described above, as well as further advantages of the invention, may be better understood by reference to the description taken in conjunction in the accompanying drawings, in which: 
     FIG. 1 is a system block diagram of an embodiment of a fault-tolerant computer system constructed in accordance with the present invention. 
     FIG. 2 is a more detailed block diagram of an embodiment of a fault-tolerant computer system shown FIG.  1 . 
     FIG. 3 is a block diagram of an embodiment of a fault-tolerant computer system constructed in accordance with the present invention. 
     FIG. 4 is a system block diagram of an embodiment of a fault-tolerant computer system constructed in accordance with the present invention. 
     FIG. 5 is a graph the processor clock rate versus temperature. 
     FIG. 6 is a flowchart depicting an interrupt-driven embodiment of the invention. 
     FIG. 7 is a flowchart chart depicting a software-polling embodiment of the invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     One embodiment of the present invention, shown in FIG. 1, is a fault-tolerant, synchronous, multi-processor system that alters the clock rate of each processor of the multi-processor system in response to a measurement of an environmental condition in at least one processor of the multiprocessor system. In one embodiment, the environmental condition represents the on-die temperature. In other embodiments, the environmental condition represents Time of Day (TOD), electrical current, humidity, and barometric pressure. 
     An embodiment of a fault-tolerant computer includes a number of redundant Central Processing Units (CPU),  100   a ,  100   b  (generally  100 ), that are operating in a synchronous, lock step relationship, that is, performing the same operations at the same time. Each of the redundant CPUs  100  is in electrical communication with a bus  900  that is in electrical communication with a common external System Module (SM)  302 . 
     Each redundant CPU  100  includes a processor  110 . In one embodiment, each processor  110  is of a variety that includes an environmental sensor, such as a thermal sensor  120  that provides an output signal ENV_STATUS  220  in response to the measured environmental condition exceeding a predetermined threshold value in its respective processor. In one embodiment, the processor  110  is an Intel Pentium III® XEON™ processor and the environmental condition is its on-die temperature. Other embodiments are possible in which environmental condition-measuring circuitry external to the processor  110  determines that the environmental condition of the processor  110  exceeds a predefined threshold. 
     The ENV_STATUS  220  signal from each CPU  100 , or environmental condition-measuring circuitry, is provided as an input to the SM  302  notifying the SM  302  that the environmental condition of a processor  110  has exceeded a predefined threshold. In one embodiment, the SM  302  includes an interrupt register  320 . The interrupt register  320  stores the results ENV_STATUS  220 . In response to an environmental condition exceeding a predefined threshold for any processor  110  being reported by ENV_STATUS  220 , the SM  302  generates an interrupt  230  to all processors  110  simultaneously. In response to the interrupt, each of the processors  110  reads the contents of the interrupt register  320  to determine the cause of the interrupt. 
     Referring to FIG. 2, in one embodiment, each processor  110  contains a thermal sensor  120  that monitors the on-die temperature and a register  410  that holds a programmable threshold temperature. The thermal sensor  120  compares the measured on-die temperature with the threshold value stored in register  410  of processor  110  and provides an output signal ENV_STATUS  220  when the temperature crosses the threshold value. Results of this comparison for each processor  110  are similarly conveyed to the SM  302  and stored in interrupt register  320 . In one embodiment, ENV_STATUS  220  indicates that the processor temperature measured by the thermal sensor  120  is equal to or greater than the threshold value stored in register  410 . In response to such an over-temperature indication by ENV_STATUS  220 , a value is written into a “too hot” bit  460 , or series of bits, within interrupt register  320 . In one embodiment, a single bit is used to indicate the presence or absence of an over-temperature condition. Whereas, in other embodiments, a series of bits are used indicating the presence or absence of an over-temperature condition, and locating the temperature condition within a predetermined range of temperatures. In response to the same over-temperature indication by ENV_STATUS  220 , the SM  302  sends an interrupt  230  signal substantially simultaneously to all processors  110 . Upon their servicing of an interrupt, the processors  110  read, through the electrical communications bus  900 , the information stored in “too hot” bit(s)  460  of interrupt register  320 . 
     Referring to FIG. 3, in another embodiment, the same CPUs  100  and their processors  110  containing the thermal sensors  120  are in communication with a common external System Management Module (SMM)  300 . In this embodiment, the SMM executes a controlled process whereby it repeatedly polls each of the processors  100  to report thermal condition status in ENV_STATUS  220 . In response to an over-temperature reported on any processor  110 , the SMM  300  communicates with all processors  110  to set a clock-throttle control bit substantially simultaneously within a register in communication with each processor  110 . Having the clock-throttle bit set, the processors  110 , operating in lock-step, respond by uniformly reducing their clock value by a set amount to a lower rate. 
     Yet in more detail, in one embodiment shown in FIG. 2, each processor  110  contains a processor unit  430  that executes instructions and a phase-locked loop (PLL)  440  that is used to generate local clock signals from an external system clock  500 . An interrupt  230  signal causes each processor  110  to perform an interrupt handler routine designated for that interrupt whereby the contents of the SM  302  interrupt register  320 , specifically the “too hot” bit  460 , are read by the handler routine. In another embodiment, the SMM  300  polls each processor  110  for its thermal status and sets a clock-throttle bit within a processor  110  register in response to an over-temperature condition. When the contents of the “too hot” bit  460  indicate the existence of a temperature having exceeded a predefined threshold, or alternatively, if the clock throttle bit has been set in a processor  110  register, each processor unit  430  modifies its local clock rate a predefined amount by altering an input control signal (CTRL)  470  to the PLL  440 . The PLL  440  is synchronized to an external system clock  500  and generates the processor local clock (LOCAL CLK)  480 . 
     Referring to FIG. 4, in a preferred embodiment a Fault Tolerant Computer (FTC) system comprises multiple redundant CPUs  100  operating in a synchronized lock-step manner. Each of the CPUs  100  includes multiple independent processors  110 . Each processor  110  of a particular CPU  100  operates in combination with the other processors  110  of the same CPU, not necessarily performing the same instructions on the same clock cycles, comprising a symmetric multi-processing CPU  100 . On the redundant CPUs  100 , each processor  110  operates in lock step with the respective processors on the other CPUs  100 . That is, processor  110   a  of CPU  100   a  is operating in lock step with processors  10   a  of redundant CPU(s)  100   b.  Through a bus  900 , each processor is in communication with a common external Input/Output (I/O) module  305 . In this embodiment, the bus  900  is a proprietary bus that functions as a PCI bridge to interconnect PCI busses resident on each CPU  100  and the I/O module  305 ; however, other communication busses may be used. In one embodiment shown in FIG. 4, the I/O module comprises an Application Specific Integrated Circuit (ASIC)  301 , that further comprises an interrupt register  320 . The temperature sensor  120  of each processor  110  communicates the results of an over-temperature condition in ENV_STATUS  220  to the ASIC  301  interrupt register  320 . The ASIC  301 , having been notified of an over-temperature condition on any of the processors  110 , generates an interrupt  230  substantially simultaneously to all processors  110 . In response to the interrupt  230 , the processors  110  halt all then current processes and, maintaining lock-step operation, enter an interrupt service handling routine where the contents of the interrupt register  320  are read to determine the cause of the interrupt. 
     In one embodiment, all processors  110  of all CPUs  100  respond by reducing their clock rate in a lock-step manner to a reported over-temperature condition from any processor  110 . In another embodiment, only respective processors  110  of each redundant CPU  100  respond to an over-temperature condition reported by a corresponding processor. That is, processors  110   a  of each of CPU  100   a  and CPU  100   b  would respond by reducing their operating clock rates in response to an over-temperature condition reported from processor  110   a  of any CPU  100 , while processor  110   b  would continue to operate at the normal clock rate. 
     Referring to FIG. 5, in one embodiment comprising a single temperature threshold, the processor clock rate is shown as a function of on-die temperature. In this embodiment processor  110  runs at its normal clock rate (CR 0 )  610  for all temperatures below the programmed threshold temperature (T 0 )  600 . As the on-die temperature rises above T 0    600 , the processor clock rate is reduced by a set amount, delta clock rate (Δ 0 )  620 , to a new lower clock rate of CR 1    612 . The clock rate remains at this lower rate until the measured on-die temperature falls below T 0    600 . When the temperature falls below T 0    600 , the processor clock rate increases by the same set amount, Δ 0    620 , returning to the normal processor clock rate, CR 0    610 . Also referring to FIG. 5, in another embodiment having multiple threshold temperatures, the relationship between processor clock rate and on-die temperature is shown with multiple threshold temperatures: T 1    601  through T N    603 ; multiple clock rates: CR 2    612  through CR N    613 ; and multiple delta clock rate values: Δ 1    621  through Δ N    622 . 
     An embodiment of the processor clock rate modification process in a system of processors having internal temperature monitors is depicted in FIG.  6 . After system boot and once the synchronized operation of all primary and redundant CPUs has been attained, the thermal sensor  120  of each processor  110  measures the on-die temperature (step  10 ). The thermal sensor  120  of each processor  110  compares the measured temperature to the stored threshold value (step  20 ). The thermal sensor  120  of each processor  110  continues to measure the on-die temperature and compare the measured value to the stored threshold value until the measured value is equal to or greater than the threshold value. Once the measured on-die temperature of any processor  110  has reached or surpassed the threshold value, a value is reported by ENV_STATUS  220  and written and stored within the “too hot” bit(s)  460  of the interrupt register  320  (step  30 ). After the “too hot” bit(s)  460  has been set, the SM  302  sends an interrupt substantially simultaneously to all processors  110  (step  40 ). In response to the interrupt, each processor  110 , which has been operating in lock step with the other processors, halts processing of all applications and enters an interrupt handling service routine, during which time, the “too hot” bit(s)  460  is read from the interrupt register  320  substantially simultaneously by each processor  110  (step  50 ). Upon reading the “too hot” bit(s)  460 , the processor unit  430  of each processor  110 , operating in lock-step with respective processors  110  of redundant CPUs  100 , proceeds to reduce its clock rate by a set amount maintaining lock-step relationship by executing the same clock-throttle command on the same clock cycle (step  60 ). 
     While operating at the reduced clock rate, the thermal sensor  120  of each processor  110  continues to measure the on-die temperature (step  70 ) and compare it to the threshold value (step  80 ). Once the measured temperature of all processors  110  have reduced to values below the threshold value, the results are reported by ENV_STATUS  220  and the “too hot” bit(s)  460  of the interrupt register  320  are reset (step  90 ). After the “too hot” bit(s)  460  has been reset, the SM  302  sends an interrupt substantially simultaneously to all processors  110  (step  100 ). In response to the interrupt, each processor  110 , which has been operating in lock step with the other processors, halts processing of all applications and enters an interrupt handling service routine, during which time, the “too hot” bit(s)  460  is read from the interrupt register  320  substantially simultaneously by each processor  110  (step  110 ). Upon reading the “too hot” bit(s)  460 , the processor unit  430  of each processor  110  operating in lock step with other processors  110 , proceeds to increase its clock rate to resume normal clock rate operation (step  120 ). In one embodiment, the processor  110  clock resumes its normal rate in a single step. Whereas, in other embodiments, the processor  110  clock gradually resumes its normal rate gradually, in multiple steps. 
     In another embodiment, a processor  110  clock-throttle process is depicted in FIG.  7 . After system boot and once synchronized operation of all primary and redundant CPUs  100  has been attained, a common external System Management Module (SMM)  300  polls every processor  110  for the output value of its thermal sensor  120  (step  10 ). SMM  300  reads the value of ENV_STATUS  220  reported from each processor  110  in response to each poll (step  20 ) to determine if the measured temperature of any processor  110  is equal to or greater than the threshold value (step  30 ). The SMM  300  continues to poll each processor  110  until a reported ENV_STATUS  220  indicates that the measured temperature has met or exceeded the threshold value. If ENV_STATUS  220  of any processor  110  indicates an over-temperature condition, the SMM  300  substantially simultaneously sets a corresponding register bit on all processors  110  (step  40 ). Setting the appropriate processor register bit of each processor  110  in this manner, prompts each processor  110  operating in lock-step relation to other processors,  110  to modify its local processor clock rate to a slower rate, maintaining lock-step relationship by executing the same clock-throttle command on the same clock cycle. 
     While operating at the reduced clock rate, the SMM  300  continues to poll each processor  110  for the output value of its thermal sensor  120  (step  50 ). The SMM  300  reads the values of ENV_STATUS  220  reported from each processor  110  in response to each poll (step  60 ). The SMM  300  examines the resulting ENV_STATUS  220  values to determine if the measured temperatures of all processors  110  are below the threshold value (step  70 ). Once the measured temperature of all processors  110  have reduced to values below the threshold value, the clock-throttle bit of each processor  110  is reset, prompting each processor to substantially simultaneously increase its clock rate to resume normal clock rate operation (step  80 ), while maintaining lock-step relationship. 
     EXAMPLE 
     The following example is one way of using the invention to control the clock rate in response to an environmental condition within a multi-processor Fault-Tolerant Computer (FTC) system where redundant, multi-processor CPUs  110  are operating in lock step. In this example, an embodiment of the invention is used to control power consumption in relationship to the Time of Day (TOD). Power consumption of a FTC system is reduced by throttling the processor clock rate during certain time periods, such as during hours of peak utility usage, and not reducing power consumption during other time periods. 
     In this example, the processor  110  includes a register that holds a regularly updated value representing a time-reference, such as the TOD, or “wall-clock” time. Another register is used to store a threshold time(s). The value of the TOD register is regularly compared with the stored threshold value. One or more bits are used for storing the time values in the processor  110  registers depending on the required resolution of the time threshold. For example, a single binary bit would be sufficient to discriminate between a.m. and p.m.; whereas additional bits would allow further resolution of hours, minutes, seconds, etc. When results of the comparison indicate that the processor  110  TOD value has exceeded the stored threshold value, the processor clock rate, LOCAL_CLK  480 , is slowed by a predetermined amount. 
     In this example, the clock rate of LOCAL_CLK  480  is controlled by a PLL  440 , shown in FIG. 2, comprising a Voltage Controlled Oscillator (VCO), a divider, and a phase detector. The output of the VCO represents the output of PLL  440 , LOCAL_CLK  480 . Within the PLL  440 , the VCO output is divided by a number, resulting in a lower-frequency, time-varying signal. Within the PLL  440 , a phase detector compares the divided signal with the system clock reference and generates a voltage proportional to any phase-offset. Within the PLL  440 , the phase-offset signal is input into the VCO to adjust and stabilize the frequency of LOCAL_CLK  480 . Thus, the clock rate of LOCAL_CLK  480  is determined by the number used in the divider of PLL  440 . 
     In this example, results of the comparison of the contents of the TOD register to the contents of the time threshold register, provide a numeric value that is input into the divider of the PLL. In one embodiment of the invention, a value of “2” is written into the divider register of the PLL  440 , when the threshold is exceeded. This causes the VCO output signal to be divided by “2” and results in a reduction of LOCAL_CLK  480  clock rate by one half. Other embodiments store multiple threshold values, where multiple clock rates are required at different times. This is accomplished by generating different PLL  440  divider numbers depending on which time threshold value(s) have been exceeded and using these numbers to control the clock in a similar manner. The LOCAL_CLK  480  clock rate could be divided by 2if the TOD has exceeded a first threshold, and divided by 4 if the TOD has exceeded a second threshold, etc. 
     Having shown the preferred embodiments, one skilled in the art will realize that many variations are possible within the scope and spirit of the claimed invention. It is therefor the intention to limit the invention only by the scope of the claims.