Abstract:
Some embodiments of the invention accurately account for power dissipation in memory systems that include individual memory modules by keeping track of the number of read requests, the number of write requests, and the number of activate requests that are applied to the individual memory modules during selected time periods. If the sum of these totals exceeds a threshold level, the embodiments throttle the memory system, either by throttling the entire memory system based in response to the most active memory module, or by throttling individual memory modules as needed. Other embodiments of the invention may assign the same or different weights to activate requests, read requests, and write requests. Other embodiments are described and claimed.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     Digital processors, such as microprocessors, use a memory subsystem to store data and processor instructions. Some processors communicate directly with memory, and others use a dedicated controller chip, often part of a “chipset,” to communicate with memory.  
         [0002]     Conventional memory subsystems are often implemented using memory modules. Referring to  FIG. 1 , which illustrates an example conventional memory subsystem, a microprocessor  20  communicates with a memory controller/hub (MCH)  30  that couples the microprocessor  20  to various peripherals. One of these peripherals is system memory, shown as dual in-line memory modules (DIMMs)  40 ,  42 , and  44  inserted in card slots  50 ,  52 , and  54 . In this example, each of the DIMMs  40 ,  42 ,  44  includes a number of memory devices  35 , which may be DRAM memory devices. When connected, the DIMMs are addressed from MCH  30  whenever MCH  30  asserts appropriate signals on an Address/Control Bus  60 . Data transfers between MCH  30  and one of DIMMs  40 ,  42 , and  44  occur on a Data Bus  70 .  
         [0003]     Thermal throttling refers generally to methods used to reduce the workload experienced by processor-based electronic system components in response to overheating. For example, some processors are equipped with a pin that signals when the processor die temperature has exceeded a threshold level. When the threshold is exceeded, the processor is “throttled” or operated at a slower speed for a period of time in order to reduce the amount of heat generated by the processor.  
         [0004]     Memory modules are another type of component that may be found in processor-based electronic systems that may be thermally throttled.  
         [0005]     For example,  FIG. 2  is a flowchart illustrating a conventional method  200  of thermal throttling that may be applied to the memory subsystem of  FIG. 1 . In process  210 , the MCH  30  counts the number of read requests R that are directed at any of the DRAMs  35  on the DIMMs  40 ,  42 ,  44  during a first time period Δt 1 . The first time period Δt 1  may be referred to as a global sample window (GSW). In process  220 , the number of read requests that occur during the GSW is compared to a first preset read threshold n 1 . If r is less than or equal to n 1 , process  210  is repeated. If r is greater than n 1 , a thermal throttling mode is entered at process  230  for a second time period Δt 2 , where Δt 2  is greater than or equal to the first time period Δt 1 . The second time period Δt 2  may be referred to as the Read Throttle Period (RTP).  
         [0006]     At process  240 , the number of read requests occurring during a third time period Δt 3  is tracked by the MCH  30 . The third time period At 3  may be referred to as the Read Monitor Period (RMP). The length of the second time period Δt 2  (RTP) is n times the length of the third time period Δt 3  (RMP). In process  250 , the number of read requests R is compared to a second preset read threshold n 2 . If R is greater than n 2 , process  260  prevents additional read requests from being issued to the memory interface for the rest of the time period Δt 3  (RMP). Regardless of the outcome of process  250 , in process  270  the number of elapsed third time periods Δt 3  (RMPs) is checked for equality with the second time period Δt 2  (RTP). If the RTP has not expired, a return to process  240  occurs and the number of reads is checked for another RMP. If the RTP has expired, then the throttling mode also expires and a return to process  210  occurs.  
         [0007]     In the above example, write requests that are directed at DRAMs  35  on the DIMMs  40 ,  42 ,  44  are handled in an identical manner, but using a separate mechanism. Thus, the thermal throttling mode could be triggered either by the number of read requests or the number of write requests exceeding a threshold level.  
         [0008]     In the example described above, all reads and writes are treated identically, and no distinction is made based upon which individual DIMM  40 ,  42 ,  44  contains the DRAM  35  that is the target of the memory transaction. This approach works well for desktop systems because it successfully accounts for the total dissipated power (TDP) during read and write cycles for the entire memory subsystem. However, contemporary server systems can dissipate more heat compared to desktop systems and the primary thermal concern is the thermal density for individual memory modules. Also, compared to desktop traffic, server traffic is generally more random and spread across various memory modules as compared to desktop traffic.  
         [0009]     Thus, if one assumes that that reads are well-distributed across all the memory modules in a server system (a fairly safe assumption), the result will be a threshold that is set too high. In such situations, the memory system might become vulnerable to damage by a power virus, which is a virus designed to concentrate memory accesses on one DIMM or even on one DRAM. Power viruses such as these have the potential to destroy the particular memory module that is attacked. Even in the absence of a power virus a “hot spot” can occur under some reasonable workloads, or when memory modules of different size are used.  
         [0010]     Conversely, if one assumes that all reads or writes will be targeted to one memory module, the threshold will be set too low and the attainable performance of the system will be constrained due to overly frequent and unnecessary throttling of the memory interface.  
         [0011]     Furthermore, while the power dissipated by read and write requests is accounted for by the above example, it fails to recognize the power dissipated during activates on the DRAM interface. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0012]      FIG. 1  is a block diagram illustrating an example conventional memory subsystem.  
         [0013]      FIG. 2  is a flowchart illustrating a conventional method of thermal throttling that may be applied to the memory subsystem of  FIG. 1 .  
         [0014]      FIG. 3  is a block diagram illustrating an example memory subsystem utilizing memory modules that may be used in conjunction with some embodiments of the invention.  
         [0015]      FIGS. 4A and 4B  are diagrams that illustrate in further detail the memory modules of  FIG. 3 .  
         [0016]      FIG. 5  is a flowchart illustrating a method of thermal throttling according to some embodiments of the invention. 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0017]      FIG. 3  is a block diagram illustrating some components of an example memory subsystem  100  utilizing DIMMs that may be used in conjunction with embodiments of the invention. It should be emphasized that embodiments of the invention are not limited only to memory subsystems that are implemented with DIMMs. For example, embodiments of the invention work equally well with memory subsystems that utilize single inline memory modules, or SIMMs. Thus, the generic term “memory module” is intended to include DIMMs, SIMMs, and other memory modules that include a plurality of memory devices. The number of memory modules in the memory subsystem may be more or less than the number illustrated in  FIG. 3 .  
         [0018]     Referring to  FIG. 3 , the memory subsystem  100  includes a host  110 , four memory modules M 1 , M 2 , M 3 , and M 4 , four memory channels  112 ,  122 ,  132 , and  142 , and a low-speed system management bus (SMBus)  160 . The host  110  includes four counters  111 , each counter corresponding to one of the memory modules M 1 -M 4 . Host  110  may also include one or more microprocessors, signal processors, memory controllers, graphics processors, etc. The processors and memory controllers may also be separate from the host, and a memory controller may be included as part of the processor. Typically, a memory controller coordinates access to system RAM memory, and the memory controller is the component of host  110  connected directly to the host memory channel  112 , which is connected to the first memory module M 1 .  
         [0019]     Memory module M 3  is typical of modules M 1 -M 4 . A memory module buffer (MMB)  146  connects module M 3  to a host-side (upstream) memory channel  132  and to a downstream memory channel  142 . A number of memory devices, for example, Dynamic Random Access Memory chips (DRAMs)  144 , communicate with memory module buffer  146  through a memory device bus (not shown in  FIG. 2 ) to provide addressable read/write memory for subsystem  100 . Other example memory subsystems compatible with embodiments of the invention need not have memory module buffers.  
         [0020]      FIGS. 4A and 4B  are schematic diagrams that further illustrate the memory modules M 1 -M 4  of  FIG. 3 . A set of card edge connectors  148  provide electrical connections for upstream and downstream memory channels, reference voltages, clock signals, SMBus  160 , etc. In this instance, MMB  146  is centrally located on one side of module M 3 , flanked on each side by four DRAM devices  144 , with ten more DRAM devices  144  occupying the opposite side of module M 3 .  
         [0021]     Each memory channel  112 ,  122 ,  132 , and  142  in  FIG. 3  is a point-to-point connection between two devices, either two MMBs  146  or the host  110  and an MMB  146 . The direct connection allows the memory channels to run preferably at relatively high data rates.  
         [0022]     Although the memory subsystem  100  of  FIG. 3  illustrates only memory modules M 1 -M 4  and a host  110 , there may be a repeater (not shown) located between any two of the components illustrated in  FIG. 3 . For example, a repeater may be placed between the host  110  and the module M 1  or between the module M 1  and the module M 2 .  
         [0023]     Each of the memory channels  112 ,  122 ,  132 , and  142  is composed of two uni-directional buses for data traffic in both directions. That is, commands and data can travel in the direction away from the host  110  and status and data can travel towards the host  110 . For convenience, the movement of command and data through the memory channels in a direction away from the host  110  will henceforth be referred to as “southbound.” Likewise, movement of status and data through the memory channels in the direction toward the host  110  will be referred to “northbound.” It should be apparent that these terms have nothing to do with the actual geographic orientation of the memory channels.  
         [0024]     The actual signal paths that make up the memory channels are implemented using high-speed serial differential signals. The number of differential signals in the southbound direction may be different than the number of signals in the northbound direction.  
         [0025]     In normal mode of operation, host  110  accesses the memory space of module M 3  by sending commands and data, addressed to module M 3 , southbound on host memory channel  112 . The MMB  146  of module M 1  receives the commands/data and resends it, without modification, on memory channel  122  to the MMB  146  of memory module M 2 . The MMB  146  of module M 2  next receives the command and resends it on memory channel  132  to MMB  146  of memory module M 3 . On module M 3 , MMB  146  detects that the command is directed to it, decodes it, and transmits DRAM commands and signaling to the DRAMs (e.g.,  144 ) controlled by that buffer. When a response is expected (such as when a read is requested), MMB  146  receives the data from the DRAMs, encodes/formats the data, and sends it backwards (northbound) along the memory channels  132 ,  122 , and  112 , repeated without modification by the MMBs  146  on modules M 2  and M 1 , to host  110 .  
         [0026]      FIG. 3  also illustrates a control bus (SMBus)  160  routed to the host  110  and to each of the modules M 1 , M 2 , M 3 , and M 4 . Although proprietary or other standard buses or signaling may be used for other memory module subsystems, an SMBus is illustrated in  FIG. 3 . A SMBus is a particular type of control bus that conforms to the  System Management Bus  ( SMBus )  Specification,  SBS Implementers Forum, Version 2.0, Aug. 3, 2000. SMBus  160  provides a reliable low-speed (10 to 100 kbps) serial channel that is typically used in a computer system to control peripherals such as a battery management system, fans, laptop display settings, memory module recognition and configuration, etc.  
         [0027]      FIG. 5  is a flowchart illustrating a method  500  of thermal throttling according to some embodiments of the invention.  
         [0028]     As was explained above, there is a counter  111  corresponding to each of the memory modules M 1 -M 4  in the memory subsystem  100  illustrated in  FIG. 3 . Process  510  of  FIG. 5  generally refers to module Mn so that the method  500  is applicable to any number n of memory modules. Over a first time period Δt 1 , the value Tn in each counter corresponding to the memory module Mn is incremented for every read request (Rn), activate command (An), and write request (Wn) that is directed at the particular memory module Mn. The first time period Δt 1  may be referred to as a global sample window (GSW).  
         [0029]     In process  520 , every value Tn corresponding to each of the memory modules Mn is compared with a first threshold value n 1 . If Tn is not greater than n 1 , then the counter is reset and process  510  is repeated for another first time period Δt 1 . If Tn is greater than n 1 , then the corresponding memory module Mn is placed in throttle mode (process  530 ) for a second time period Δt 2 , where Δt 2  is greater than or equal to the first time period Δt 1 . The second time period Δt 2  may be referred to as the Read Throttle Period (RTP).  
         [0030]     At process  540 , for every module Mn that is in the thermal throttling mode, the total number Tn of read requests (Rn), activate commands (An), and write requests (Wn) occurring is again tracked by the corresponding counter, but this time for a third time period Δt 3 . The length of the second time period Δt 2  is n times the length of the third time period Δt 3 . The third time period Δt 3  may be referred to as the Read Monitor Period (RMP).  
         [0031]     In process  550 , the total number of reads/activates/writes Tn is compared to a second threshold value n 2 . If Tn is greater than n 2 , process  560  prevents further read requests, activate commands, or write requests to be issued to the corresponding memory module Mn for the rest of the time period Δt 3  before moving on to process  570 . If Tn is not greater than n 2 , process  570  will take place immediately after process  550 .  
         [0032]     In process  570 , the number of elapsed third time periods Δt 3  is checked for equality with the second time period Δt 2 . If the second time period Δt 2  has not expired, the value of Tn is reset and a return to process  540  occurs, where Tn is again tracked for another third time period Δt 3 . In process  570 , if the second time period Δt 2  has expired, then the throttling mode for the corresponding memory module Mn also expires and a return to process  510  occurs.  
         [0033]     Thus, according to the embodiments of the invention described above, a separate counter exists for each memory module, and the counter is incremented for every read, write, or activate cycle that is targeted at the module. Thermal throttling methods can then be applied to individual memory modules that exceed a threshold level. Alternatively, thermal throttling techniques may be applied to all modules once at least one of the memory modules exceeds the threshold level. In this alternative arrangement, the memory module having a corresponding counter with the highest count will effectively determine when the entire memory subsystem enters the thermal throttling mode. In either implementation, the threshold levels may be set between the two extremes so as to protect against a power virus while also ensuring that the performance impact is minimized.  
         [0034]     According to some other embodiments of the invention, a programmable instruction weighting may be applied to differentiate between Read/Write commands and Activate commands. Read/write commands targeted at a particular memory module may increment the corresponding counter by a fixed amount but an activate command may increment the counter by an amount specified by a programmable field. For example, a read command or write command targeted at a particular memory module may increment the corresponding counter by 2. However, the amount that the counter is incremented by an activate command may be specified by a programmable two-bit field. When the two-bit field is ‘00’, the activate command will also increment the counter by 2, so that the ratio between activate commands and read/write commands is 2:2. When the two-bit field is ‘11’, the ratio becomes 5:2. By assigning a greater weight to activate commands, the embodiments may effectively account for the increased power dissipation associated with those commands.  
         [0035]     According to alternative embodiments of the invention, the weight assigned to each of the three commands may be separately controlled. In the example above, read and write commands were given the same weight. However, according to these alternative embodiments each of the read commands, write commands, and activate commands would increment the corresponding counter by a different amount.  
         [0036]     It is also possible that during a given cycle, a read or write could occur to one memory module along with an activate to a different memory module. In this situation, two different counters, each corresponding to one of the modules, would be incremented.  
         [0037]     Assuming that a chipset using an embodiment of the invention has two modules on the memory channel, that a memory module dissipates about 20 W of power, and that a server can cool 8 Watts per module under peak theoretical bandwidth conditions, the chipset will be able to operate at approximately 80% of its peak theoretical bandwidth while a conventional chipset that does not utilize module by module thermal throttling will be limited to about 40% of peak.  
         [0038]     Embodiments of the invention can achieve the above advantages because they reduce the probability of entering throttle mode. The embodiments also improve the accuracy of monitoring the heat dissipation by taking into account the activate cycles. Compared to the conventional example of thermal throttling described in  FIG. 2 , embodiments of the invention also simplify the associated logic because one set of counters and associated logic for writes are eliminated.  
         [0039]     In high performance servers, both heat dissipation and high performance must be addressed simultaneously. Embodiments of the invention directly address these issues by throttling the memory device interface only when really required, thus allowing it to operate at a higher “sustained” bandwidth than its predecessors. In other words, the throttling mechanism is not allowed to interfere with or limit processor/system performance.  
         [0040]     Instead of treating all reads/writes as if they were targeting a single memory module, embodiments of the invention treat accesses to different memory modules separately, thereby reducing the frequency that the memory device interface is throttled. Thus, the system may deliver substantially higher sustained bandwidth in a server environment compared to the conventional thermal throttling techniques.  
         [0041]     The preceding embodiments are exemplary. Those of skill in the art will recognize that the concepts taught herein can be tailored to a particular application in many other advantageous ways. In particular, those skilled in the art will recognize that the illustrated embodiments are but one of many alternative implementations that will become apparent upon reading this disclosure.  
         [0042]     Although the specification may refer to “an”, “one”, “another”, or “some” embodiment(s) in several locations, this does not necessarily mean that each such reference is to the same embodiment(s), or that the feature only applies to a single embodiment.  
         [0043]     Many of the specific features shown herein are design choices. The number and type of memory modules, the number and type of memory chips on a memory module, control bus protocols, etc., are all merely presented as examples. For instance, memory modules are not required to have memory module buffers as was illustrated in the example above. Likewise, functionality shown embodied in a single integrated circuit or functional block may be implemented using multiple cooperating circuits or blocks, or vice versa. Such minor modifications are encompassed within the embodiments of the invention, and are intended to fall within the scope of the appended claims.