Abstract:
One embodiment of the present invention sets forth a hybrid memory module that combines memory devices of different types while presenting a single technology interface. The hybrid memory module includes a number of super-stacks and a first interface configured to transmit data between the super-stacks and a memory controller. Each super-stack includes a number of sub-stacks, a super-controller configured to control the sub-stacks, and a second interface configured to transmit data between the sub-stacks and the first interface. Combining memory devices of different types allows utilizing the favorable properties of each type of the memory devices, while hiding their unfavorable properties from the memory controller.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 12/057,306, filed Mar. 27, 2008, now U.S. Pat. No. 8,397,013, which is a continuation-in-part of U.S. patent application Ser. No. 11/611,374, filed Dec. 15, 2006, now U.S. Pat. No. 8,055,833, which also claims benefit of U.S. Provisional Patent Application Ser. No. 60/849,631, filed Oct. 5, 2006, all of which are herein incorporated by reference. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     Embodiments of the present invention generally relate to memory modules and, more specifically, to hybrid or mixed-technology memory modules. 
     2. Description of the Related Art 
     Memory systems in computers are typically built from memory modules. An example of a common type of memory module is a Dual Inline Memory Module or DIMM. The most common type of memory module is a DIMM that contains just one type of memory technology known as DRAM. 
     There are several new forms of semiconductor memory that are currently being developed, some as potential replacements for one-transistor-cell DRAM. Examples are Magnetoresistive Random Access Memory (MRAM), Phase-change memory (also known as PCM, PRAM, PCRAM, Ovonic Unified Memory and Chalcogenide RAM or C-RAM), Ferroelectric RAM (FRAM), Resistor RAM (RRAM), and Molecular Wire Crossbar Memory (MWCM). Most of these new memory technologies have system properties (most notably the read and write bandwidth and read and write latency) that are different from DRAM. In addition there are several flash memory technologies, such as NAND and NOR flash, that are attractive (because of their high density and low cost) for use as system memory, but again have system properties that are different from DRAM. Therefore, an ideal memory module would incorporate different types of memory while having a single memory interface with a host computer. 
     As the foregoing illustrates, what is needed in the art is a mixed-technology memory module architecture that combines different memory technologies while presenting a single, preferably standard, technology interface. 
     SUMMARY OF THE INVENTION 
     One embodiment of the present invention sets forth a hybrid memory module comprising a plurality of super-stacks and a first interface configured to transmit data between the plurality of super-stacks and a memory controller. Each super-stack includes a plurality of sub-stacks, a super-controller configured to control the plurality of sub-stacks, and a second interface configured to transmit data between the plurality of sub-stacks and the first interface. 
     One advantage of the disclosed hybrid memory module is that combining memory devices of different types allows utilizing the favorable properties of each type of the memory devices, while hiding their unfavorable properties from the memory controller. More specifically, combining slow memory devices and fast memory devices using the architecture of the hybrid memory module allows the long latency of the slow memory devices to be hidden, so that the hybrid memory module appears to the memory controller as a standard fast memory device characterized by having low latency. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments. 
         FIG. 1A  shows an abstract and conceptual model of a mixed-technology memory module, according to one embodiment. 
         FIG. 1B  is an exploded hierarchical view of a logical model of a HybridDIMM, according to one embodiment. 
         FIG. 2  shows a HybridDIMM Super-Stack with multiple Sub-stacks, according to one embodiment. 
         FIG. 3  shows a Sub-Stack showing a Sub-Controller, according to one embodiment. 
         FIG. 4  shows the Sub-Controller, according to one embodiment. 
         FIG. 5  depicts a physical implementation of a 1-high Super Stack, according to one embodiment. 
         FIG. 6A  depicts a physical implementation of 2-high Super-Stacks, according to one embodiment. 
         FIG. 6B  depicts a physical implementation of a 4-high Super-Stack, according to one embodiment. 
         FIG. 7  shows a method of retrieving data from a HybridDIMM, according to one embodiment. 
         FIG. 8A  shows a method of managing SRAM pages on a HybridDIMM, according to one embodiment. 
         FIG. 8B  shows a method of freeing SRAM pages on a HybridDIMM, according to one embodiment. 
         FIG. 9  shows a method of copying a flash page to an SRAM page on a HybridDIMM, according to one embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     Embodiments of the invention disclosed herein describe how to build standard DIM Ms that contain more than one type of memory technology. An example of a mixed-technology DIMM is a module that combines flash memory with SRAM. 
       FIG. 1A  shows an abstract and conceptual model of a mixed-technology memory module, according to one embodiment. 
     The mixed-technology memory module  100  shown in  FIG. 1A  has both slow memory and fast memory, with the combination architected so as to appear to a host computer as fast memory using a standard interface. The specific embodiment of the mixed-technology memory module  100 , which will also be referred to as a HybridDIMM  100 , shows both slow, non-volatile memory portion  104  (e.g. flash memory), and a latency-hiding buffer using fast memory  106  (e.g. using SRAM, DRAM, or embedded DRAM volatile memory), together with a controller  108 . As shown in  FIG. 1A , the combination of the fast and slow memory is presented to a host computer over a host interface  110  (also referred to herein as a DIMM interface  110 ) as a JEDEC-compatible standard DIMM. In one embodiment, the host interface  110  may communicate data between the mixed-technology memory module  100  and a memory controller within a host computer. The host interface  110  may be a standard DDR3 interface, for example. The DDR3 interface provides approximately 8 gigabyte/s read/write bandwidth per DIMM and a 15 nanosecond read latency when a standard DIMM uses standard DDR3 SDRAM. The host interface  110  may present any other JEDEC-compatible interface, or even, the host interface may present to the host system via a custom interface, and/or using a custom protocol. 
     The DDR3 host interface is defined by JEDEC as having 240 pins including data, command, control and clocking pins (as well as power and ground pins). There are two forms of the standard JEDEC DDR3 host interface using compatible 240-pin sockets: one set of pin definitions for registered DIMMs (R-DIMMs) and one set for unbuffered DIMMs (U-DIMMs). There are currently no unused or reserved pins in this JEDEC DDR3 standard. This is a typical situation in high-speed JEDEC standard DDR interfaces and other memory interfaces—that is normally all pins are used for very specific functions with few or no spare pins and very little flexibility in the use of pins. Therefore, it is advantageous and preferable to create a HybridDIMM that does not require any extra pins or signals on the host interface and uses the pins in a standard fashion. 
     In  FIG. 1A , an interface  105  to the slow memory  104  may provide read bandwidth of 2-8 gigabyte/s with currently available flash memory chips depending on the exact number and arrangement of the memory chips on the HybridDIMM. Other configurations of the interface  105  are possible and envisioned by virtue of scaling the width and/or the signaling speed of the interface  105 . However, in general, the slow memory  104 , such as non-volatile memory (e.g. standard NAND flash memory), provides a read latency that is much longer than the read latency of the fast memory  106 , such as DDR3 SDRAM, e.g. 25 microseconds for current flash chips versus 15 nanoseconds for DDR3 SDRAM. 
     The combination of the fast memory  106  and the controller  108 , shown as an element  107  in  FIG. 1A , allows the “bad” properties of the slow memory  104  (e.g. long latency) to be hidden from the memory controller and the host computer. When the memory controller performs an access to the mixed-technology memory module  100 , the memory controller sees the “good” (e.g. low latency) properties of the fast memory  106 . The fast memory  106  thus acts as a latency-hiding component to buffer the slow memory  104  and enable the HybridDIMM  100  to appear as if it were a standard memory module built using only the fast memory  106  operating on a standard fast memory bus. 
       FIG. 1B  is an exploded hierarchical view of a logical model of the HybridDIMM  100 , according to one embodiment. While  FIG. 1A  depicts an abstract and conceptual model of the HybridDIMM  100 ,  FIG. 1B  is a specific embodiment of the HybridDIMM  100 .  FIG. 1B  replaces the simple view of a single block of slow memory (the slow memory  104  in  FIG. 1A ) with a number of sub-assemblies or Sub-Stacks  122  that contain the slow memory (flash memory components  124 ).  FIG. 1B  also replaces the simple view of a single block of fast memory (the fast memory  106  in  FIG. 1A ) by SRAM  144  in a number of Sub-Controllers  126 . Further, the simple view of a single controller (the controller  108  in  FIG. 1A ) is replaced now in  FIG. 1B  by the combination of a Super-Controller  116  and a number of Sub-Controllers  126 . Of course, the particular HybridDIMM architecture shown in  FIG. 1B  is just one of many possible implementations of the more general architecture shown in  FIG. 1A . 
     In the embodiment shown in  FIG. 1B , the slow memory portion in the Sub-Stack  122  may use NAND flash, but, in alternative embodiments, could also use NOR flash, or any other relatively slow (relative to DRAM) memory. Also, in the embodiment shown in  FIG. 1B , the fast memory in the Sub-Controller  126  comprises an SRAM  144 , but could be comprised of DRAM, or embedded DRAM, or any other relatively fast (relative to flash) memory etc. Of course it is typical that memory made by use of differing technologies will exhibit different bandwidths and latencies. Accordingly, as a function of the overall architecture of the HybridDIMM  100 , and in particular as a function of the Super-Controller  116 , the differing access properties (including latency and bandwidth) inherent in the use of different memories are managed by logic. In other words, even though there may exist the situation where a one memory word is retrieved from (for example) SRAM, and another memory value retrieved from (for example) flash memory, the memory controller of the host computer (not shown) connected to the interface  110  is still presented with signaling and protocol as defined for just one of the aforementioned memories. For example, in the case that the memory controller requests a read of two memory words near a page boundary, 8 bits of data may be read from a memory value retrieved from (for example) SRAM  144 , and 8 bits of data may be read from a memory value retrieved from (for example) the flash memory component  124 . 
     Stated differently, any implementation of the HybridDIMM  100 , may use at least two different memory technologies combined on the same memory module, and, as such, may use the lower latency fast memory as a buffer in order to mask the higher latency slow memory. Of course the foregoing combination is described as occurring on a single memory module, however the combination of a faster memory and a slower memory may be presented on the same bus, regardless of how the two types of memory are situated in the physical implementation. 
     The abstract model described above uses two types of memory on a single DIMM. Examples of such combinations include using any of DRAM, SRAM, flash, or any volatile or nonvolatile memory in any combination, but such combinations not limited to permutations involving only two memory types. For example, it is also possible to use SRAM, DRAM and flash memory circuits together in combination on a single mixed-technology memory module. In various embodiments, the HybridDIMM  100  may use on-chip SRAM together with DRAM to form the small but fast memory combined together with slow but large flash memory circuits in combination on a mixed-technology memory module to emulate a large and fast standard memory module. 
     Continuing into the hierarchy of the HybridDIMM  100 ,  FIG. 1B  shows multiple Super-Stack components  102   1 - 102   n  (also referred to herein as Super-Stacks  102 ). Each Super-Stack  102  has an interface  112  that is shown in  FIG. 1B  as an 8-bit wide interface compatible with DDR3 SDRAMs with x8 organization, providing 8 bits to the DIMM interface  110 . For example nine 8-bit wide Super-Stacks  102  may provide the 72 data bits of a DDR3 R-DIMM with ECC. Each Super-Stack  102  in turn comprises a Super-Controller  116  and at least one Sub-Stack  114 . Additional Sub-Stacks  113   1 - 113   n  (also referred to herein as Sub-Stacks  113 ) may be optionally disposed within any one or more of the Super-Stack components  102   1 - 102   n . 
     The Sub-Stack  122  in  FIG. 1B , intended to illustrate components of any of the Sub-Stack  114  or the additional Sub-Stacks  113 , is comprised of a Sub-Controller  126  and at least one slow memory component, for example a plurality of flash memory components  124   1 - 124   n  (also referred to herein as flash memory components  124 ). Further continuing into the hierarchy of the HybridDIMM  100 , the Sub-Controller  126  may include fast memory, such as the SRAM  144 , queuing logic  154 , interface logic  156  and one or more flash controller(s)  146  which may provide functions such as interface logic  148 , mapping logic  150 , and error-detection and error-correction logic  152 . 
     In preferred embodiments, the HybridDIMM  100  contains nine or eighteen Super-Stacks  102 , depending for example, if the HybridDIMM  100  is populated on one side (using nine Super-Stacks  102 ) of the HybridDIMM  100  or on both sides (using eighteen Super-Stacks  102 ). However, depending on the width of the host interface  110  and the organization of the Super-Stacks  102  (and, thus, the width of the interface  112 ), any number of Super-Stacks  102  may be used. As mentioned earlier, the Super-Controllers  116  are in electrical communication with the memory controller of the host computer through the host interface  110 , which is a JEDEC DDR3-compliant interface. 
     The number and arrangement of Super-Stacks  102 , Super-Controllers  116 , and Sub-Controllers  126  depends largely on the number of flash memory components  124 . The number of flash memory components  124  depends largely on the bandwidth and the capacity required of the HybridDIMM  100 . Thus, in order to increase capacity, a larger number and/or larger capacity flash memory components  124  may be used. In order to increase bandwidth the flash memory components  124  may be time-interleaved or time-multiplexed, which is one of the functions of the Sub-Controller  126 . If only a small-capacity and low-bandwidth HybridDIMM  100  is required, then it is possible to reduce the number of Sub-Controllers  126  to one and merge that function together with the Super-Controller  116  in a single chip, possibly even merged together with the non-volatile memory. Such a small, low-bandwidth HybridDIMM  100  may be useful in laptop or desktop computers for example, or in embedded systems. If a large-capacity and high-bandwidth HybridDIMM  100  is required, then a number of flash memory components  124  may be connected to one or more of the Sub-Controller  126  and the Sub-Controllers  126  connected to the Super-Controller  116 . In order to describe the most general form of HybrdDIMM  100 , the descriptions below will focus on the HybridDIMM  100  with separate Super-Controller  116  and multiple Sub-Controllers  126 . 
       FIGS. 2 through 4  illustrate various implementations of the Super-Stack  102 , the Sub-Stack  122 , and the Sub-Controller  126 . 
       FIG. 2  shows a HybridDIMM Super-Stack  200  with multiple Sub-Stacks, according to one embodiment. The HybridDIMM Super-Stack  200  shown in  FIG. 2  comprises at least one Sub-Stack  204  including the slow memory and at least one Super-Controller  206 . The HybridDIMM Super-Stack  200  shown in  FIG. 2  may also comprise optional Sub-Stacks  202   1 - 202   n  including the slow memory. Interfaces  210  between the Sub-Stack  204  (and/or the Sub-Stacks  202   1 - 202   n ) and the Super-Controller  206  may be an industry-standard flash-memory interface (e.g. NAND, NOR, etc.) and/or they may be a flash memory interface designed for flash-memory subsystems (e.g. OneNAND, ONFI, etc.). The embodiment shown includes the Super-Controller  206  that communicates over the interface  112  (as shown in  FIG. 1B ) to the memory controller of the host computer, using a standard memory interface (such as DDR3). 
     The Super-Controller  206  in  FIG. 2  operates to provide error-detection and management of the interfaces  210  and  112 , as well as management of the Sub-Stack  204 ,  202   1 - 202   n  (also referred to herein as Sub-Stack components  204 ,  202   1 - 202   n ). The Super-Stack interface  112  appears as if Super-Stack  200  was a standard memory component. In a preferred embodiment, the interface  112  conforms to JEDEC x8 DDR3 standard, however in other embodiments, it could be x4 or x16 DDR3, or could be DDR, DDR2, GDDR, GDDR5 etc. In still other embodiments, the interface  112  could include a serial memory interface such as an FBDIMM interface. 
     The interfaces  210  in  FIG. 2 , between the Super-Controller  206  and one or more Sub-Stacks  204 ,  202   1 - 202   n , may be variously configured. Note first that in other embodiments the Super-Controller  206  may optionally connect directly to one or more flash memory components  124  illustrated in  FIG. 1B  (not shown in  FIG. 2 ). In some embodiments that use an optional direct interface to the flash memory components  124 , the protocol of interface  210  is one of several standard flash protocols (NAND, NOR, OneNAND, ONFI, etc). Additionally, and strictly as an option, in the case that the interface  210  communicates with Sub-Stacks  204 ,  202   1 - 202   n , the interface protocol may still be a standard flash protocol, or any other protocol as may be convenient. 
     With an understanding of the interfaces  210  and  112  of the Super-Stack  200 , it follows to disclose some of the various functions of the Super-Stack  200 . 
     The first internal function of the Super-Controller  206  is performed by a signaling translation unit  212  that translates signals (data, clock, command, and control) from a standard (e.g. DDR3) high-speed parallel (or serial in the case of a protocol such as FB-DIMM) memory channel protocol to one or more typically lower speed and possibly different bus-width protocols. The signaling translation unit  212  may thus also convert between bus widths ( FIG. 2  shows a conversion from an m-bit bus to an n-bit bus). The signaling translation unit  212  converts the command, address, control, clock, and data signals from a standard memory bus to corresponding signals on the sub-stack or flash interface(s). The Super-Controller  206  may provide some or all (or none) of the logical functions of a standard DRAM interface to the extent it is “pretending” to be a DRAM on the memory bus. Thus in preferred embodiments, the Super-Controller  206  performs all the required IO characteristics, voltage levels, training, initialization, mode register responses and so on—as described by JEDEC standards. So, for example if the memory interface at  112  is a standard x8 DDR3 SDRAM interface then the Super-Controller memory interface as defined by the signaling translation unit  212  behaves as described by the JEDEC DDR3 DRAM standard. 
     A second internal function of the Super-Controller  206  is performed by protocol logic  216  that converts from one protocol (such as DDR3, corresponding to a fast memory protocol) to another (such as ONFI, corresponding to a slow memory protocol). 
     A third internal function of the Super-Controller  206  is performed by MUX/Interleave logic  214  that provides a MUX/DEMUX and/or memory interleave from a single memory interface to one or more Sub-Stacks  204 ,  202   1 - 202   n , or alternatively (not shown in  FIG. 2 ) directly to one or more flash memory components  124 . The MUX/Interleave logic  214  is necessary to match the speed of the slow memory  104  (flash) to the fast memory  106  (DRAM). 
       FIG. 3  shows a Sub-Stack  302  including a Sub-Controller  306 , according to one embodiment. As shown in  FIG. 3 , the Sub-Stack  302  includes the Sub-Controller  306  and a collection of NAND flash memory components  308 ,  304   1 - 304   n . The interface  210  between the Sub-Stack  302  and the Super-Controller, such as the Super-Controller  206  or  116 , has already been described in the context of  FIG. 2 . Interfaces  310  between the Sub-Controller  306  and the flash memory components  308 ,  304   1 - 304   n  are standard flash interfaces. The interfaces  310  are defined by the flash memory components  308 ,  304   1 - 304   n  that are used to build the Sub-Stack  302 . 
     The flash memory components  308 ,  304   1 - 304   n  are organized into an array or stacked vertically in a package using wire-bonded connections (alternatively through-silicon vias or some other connection technique or technology may be used). The Sub-Stack  302  shown as an example in  FIG. 3  has 8 active flash memory components  304   1 - 304   n  plus a spare flash memory component  308 , resulting in an array or stack of 9 flash memory components  308 ,  304   1 - 304   n . The spare flash memory component  308  is included to increase the yield of the Sub-Stack  302  during assembly. The capacity of the flash memory in the Sub-Stack  302  in aggregate (exclusive of any spare capacity) is any arbitrary size (e.g. 8 gigabit, 16 gigabit, 32 gigabit, etc), and prophetic configurations are envisioned to be arbitrarily larger, bounded only by the practical limits of the availability of the flash memory components  308 ,  304   1 - 304   n . Thus, for example, the total flash capacity on a HybridDIMM with 9 Super-Stacks (eight data and one for ECC) with four Sub-Stacks each containing eight 8-gigabit flash chips would be 32 gigabytes. Of course any known or derivative technology for flash may be used, including SLC, MLC, etc. 
       FIG. 4  shows the Sub-Controller  306 , according to one embodiment. The Sub-Controller  306  contains (physically or virtually) as many flash controllers  406   1 - 406   n  as there are flash memory components  308 ,  304   1 - 304   n  in the Sub-Stack  302 , the fast memory  404 , plus (optionally) additional components to provide interfacing features and advanced functions. The optional components include Command Queuing logic  414  and High-Speed Interface logic  416 . The interface  210  shown in  FIG. 4  between the Sub-Controller and Super-Controller has already been described in the context of both  FIG. 2  and  FIG. 3 . The interface  310  between the flash controllers and the flash chips was described in the context of  FIG. 3 . 
     It should be noted that each flash controller  406  in  FIG. 4  may be a single block implementing one or more flash controllers, or it may be a collection of flash controllers, one each dedicated to controlling a corresponding flash memory device. 
     The High-Speed Interface logic  416  is configured to convert from a high-speed interface capable of handling the aggregate traffic from all of the flash memory components  308 ,  304   1 - 304   n  in the Sub-Stack  302  to a lower speed interface used by the flash controllers and each individual flash memory component  308 ,  304   1 - 304   n . 
     The Command Queuing logic  414  is configured to queue, order, interleave and MUX the data from both the fast memory  404  and array of slow flash memory components  308 ,  304   1 - 304   n . 
     Each flash controller  406  contains an Interface unit  408 , a Mapping unit  418 , as well as ECC (or error correction) unit  412 . The Interface unit  408  handles the I/O to the flash components in the Sub-Stack  302 , using the correct command, control and data signals with the correct voltage and protocol. The ECC unit  412  corrects for errors that may occur in the flash memory in addition to other well-known housekeeping functions typically associated with flash memory (such as bad-block management, wear leveling, and so on). It should be noted that one or more of these housekeeping functions associated with the use of various kinds of slow memory such as flash may be performed on the host computer instead of being integrated in the flash controller. The functionality of the Mapping unit  418  will be described in much more detail shortly and is the key to being able to access, address and handle the slow flash memory and help make it appear to the outside world as fast memory operating on a fast memory bus. 
       FIG. 5  depicts a cross-sectional view of one possible physical implementation of a 1-high Super-Stack  502 , according to one embodiment. In this embodiment, the Super-Stack  502  is organized as two vertical stacks of chips. A first vertical stack comprising a Super-Controller  506  and a Sub-Controller  508  situated on one end of a multi-chip package (MCP) substrate, and a second vertical Sub-Stack  504  comprises a plurality of flash memory components. The stacks in  FIG. 5  show connections between flash memory components made using wire bonds. This is a typical and well-known assembly technique for stacked chips. Other techniques such as through-silicon vias or other chip-stacking techniques may be used. In addition there is no requirement to stack the Super-Controller  506  and Sub-Controller  508  separately from the flash memory components. 
       FIG. 6A  depicts a physical implementation of 2-high Super-Stack  602 , according to one embodiment. This implementation is called “2-high” because it essentially takes the 1-high Super-Stack shown in  FIG. 5  and duplicates it. In  FIG. 6A , element  604  comprise the flash chips, element  608  is a Sub-Controller, and element  610  is a Super-Controller. 
       FIG. 6B  depicts a physical implementation of 4-high Super-Stack  652 , according to one embodiment. In  FIG. 6B , element  654  comprise the flash chips, element  658  is a Sub-Controller, and element  610  is a Super-Controller. 
     Having described the high-level view and functions of the HybridDIMM  100  as well as the details of one particular example implementation we can return to  FIG. 1A  in order to explain the operation of the HybridDIMM  100 . One skilled in the art will recognize that the slow memory  104  (discussed above in embodiments using non-volatile memory) can be implemented using any type of memory—including SRAM or DRAM or any other type of volatile or nonvolatile memory. In such as case the fast memory  106  acting as a latency-hiding buffer may emulate a DRAM, in particular a DDR3 SDRAM, and thus present over the host interface  110  according to any one (or more) standards, such as a JEDEC-compliant (or JEDEC-compatible) DDR3 SDRAM interface. 
     Now that the concept of emulation as implemented in embodiments of a HybridDIMM has been disclosed, we may now turn to a collection of constituent features, including advanced paging and advanced caching techniques. These techniques are the key to allowing the HybridDIMM  100  to appear to be a standard DIMM or to emulate a standard DIMM. These techniques use the existing memory management software and hardware of the host computer to enable two important things: first, to allow the computer to address a very large HybridDIMM  100 , and, second, to allow the computer to read and write to the slow memory  104  indirectly as if the access were to the fast memory  106 . Although the use and programming of the host computer memory management system described here employs one particular technique, the method is general in that any programming and use of the host computer that results in the same behavior is possible. Indeed because the programming of a host computer system is very flexible, one of the most powerful elements of the ideas described here is that it affords a wide range of implementations in both hardware and software. Such flexibility is both useful in itself and allows implementation on a wide range of hardware (different CPUs for example) and a wide range of operating systems (Microsoft Windows, Linux, Solaris, etc.). 
     In particular, embodiments of this invention include a host-based paging system whereby a paging system allows access to the mixed-technology memory module  100 , a paging system is modified to allow access to the mixed-technology memory module  100  with different latencies, and modifications to a paging system that permits access to a larger memory space than the paging system would normally allow. 
     Again considering the fast memory  106 , embodiments of this invention include a caching system whereby the Hybrid DIMM  100  alters the caching and memory access process. 
     For example, in one embodiment of the HybridDIMM  100  the well-known Translation Lookaside Buffer (TLB) and/or Page Table functions can be modified to accommodate a mixed-technology DIMM. In this case an Operating System (OS) of the host computer treats main memory on a module as if it were comprised of two types of memory or two classes of memory (and in general more than one type or class of memory). In our HybridDIMM implementation example, the first memory type corresponds to fast memory or standard DRAM and the second memory type corresponds to slow memory or flash. By including references in the TLB (the references may be variables, pointers or other forms of table entries) to both types of memory different methods (or routines) may be taken according to the reference type. If the TLB reference type shows that the memory access is to fast memory, this indicates that the required data is held in the fast memory (SRAM, DRAM, embedded DRAM, etc.) of the HybridDIMM (the fast memory appears to the host as if it were DRAM). In this case a read command is immediately sent to the HybridDIMM and the data is read from SRAM (as if it were normal DRAM). If the TLB shows that the memory access is to slow memory, this indicates that the required data is held in the slow memory (flash etc.) of the HybridDIMM. In this case a copy command is immediately sent to the HybridDIMM and the data is copied from flash (slow memory) to SRAM (fast memory). The translation between host address and HybridDIMM address is performed by the combination of the normal operation of the host memory management and the mapper logic function on the HybridDIMM using well-known and existing techniques. The host then waits for the copy to complete and issues a read command to the HybridDIMM and the copied data is read from SRAM (again now as if it were normal DRAM). 
     Having explained the general approach, various embodiments of such techniques, methods (or routines) are presented in further detail below. In order to offer consistency in usage of terms, definitions are provided here, as follows: 
     va—virtual address that caused the page fault 
     sp—SRAM page selected in Step 1 
     pa—a physical address 
     Page Table and Mapper requirements: 
     PageTable[va]==pa 
     Mapper[pa]==sp 
     Hence: Mapper[PageTable[va]]=sp 
     How do we select a physical address “pa”? 
     Must not already map to an active SRAM location 
     Must map to the BigDIMM that contains the “sp” 
     The caches must not contain stale data with “pa” physical tags 
     No processor in the coherence domain must contain a stale TLB entry for “va” 
       FIGS. 7 through 9  illustrate interactions between the OS of the host computer and the mixed-technology memory module  100  from the perspective of the OS. Although the method steps of  FIGS. 7-9  are described with respect to the memory management portion of the computer OS, any elements or combination of elements within the OS and/or computer configured to perform the method steps, in any order, falls within the scope of the present invention. 
       FIG. 7  shows a method  700  for returning data resident on the HybridDIMM to the memory controller. As an option, the present method  700  may be implemented in the context of the architecture and functionality of  FIG. 1  through  FIG. 6 . Of course, however, the method  700  or any operation therein may be carried out in any desired environment. 
     The method  700  as described herein may be entered as a result of a request from the memory controller for some data resident on a HybridDIMM. The operation underlying decision  702  may find the data is “Present” on the HybridDIMM (it is standard and well-known that an OS uses the terms “Present” and “Not Present” in its page tables). The term “Present” means that the data is being held in the fast memory on a HybridDIMM. To the OS it is as if the data is being held in standard DRAM memory, though the actual fast memory on the HybridDIMM may be SRAM, DRAM, embedded DRAM, etc. as we have already described. In the example here we shall use fast memory and SRAM interchangeably and we shall use slow memory and flash memory interchangeably. If the data is present then the BigDIMM returns the requested data as in a normal read operation (operation  712 ) to satisfy the request from the memory controller. Alternatively, if the requested data is “Not Present” in fast memory, the OS must then retrieve the data from slow memory. Of course retrieval from slow memory may include various housekeeping and management (as already has been described for flash memory, for example). More specifically, in the case that the requested data is not present in fast memory, the OS allocates a free page of fast memory (operation  704 ) to serve as a repository, and possibly a latency-hiding buffer for the page containing the requested data. Once the OS allocates a page of fast memory, the OS then copies at least one page of memory from slow memory to fast memory (operation  706 ). The OS records the success of the operation  706  in the page table (see operation  708 ). The OS then records the range of addresses now present in fast memory in the mapper (see operation  710 ). Now that the initially requested data is present in fast memory, the OS restarts the initially memory access operation from the point of decision  702 . 
     To make the operations required even more clear the following pseudo-code describes the steps to be taken in an alternative but equivalent fashion: 
     A. If Data is “Present” (e.g. present in memory type DRAM) in the HybridDIMM: 
     
         
         
           
             The HybridDIMM SRAM behaves the same as standard DRAM
 
B. Data “Not Present” (e.g. present in memory type Flash)—there is a HybridDIMM Page Fault:
 
           
         
       
    
     1. Get free SRAM page 
     2. Copy flash page to SRAM page 
     3. Update Page Table and/or TLB 
     4. Update Mapper 
     5. Restart Read/Write (Load/Store) 
     We will describe the steps taken in method or code branch B above in more detail presently. First, we must describe the solution to a problem that arises in addressing or accessing the large HybridDIMM. In order to access the large memory space that is made possible by using a HybridDIMM (which may be as much as several terabytes), the host OS may also modify the use of well-known page-table structures. Thus for example, a 256 terabyte virtual address space (a typical limit for current CPUs because of address-length limitations) may be mapped to pages of a HybridDIMM using the combination of an OS page table and a mapper on the HybridDIMM. The OS page table may map the HybridDIMM pages in groups of 8. Thus entries in the OS page table correspond to HybrdDIMM pages (or frames) 0-7, 8-15, 16-23 etc. Each entry in the OS page table points to a 32 kilobyte page (or frame), that is either in SRAM or in flash on the HybridDIMM. The mapping to the HybridDIMM space is then performed through a 32 GB aperture (a typical limit for current memory controllers that may only address 32 GB per DIMM). In this case a 128-megabyte SRAM on the HybridDIMM contains 4096 pages that are each 32 kilobyte in size. A 2-terabyte flash memory (using 8-, 16-, or 32-gigabit flash memory chips) on the HybridDIMM also contains pages that are 32 kilobyte (made up from 8 flash chips with 4 kilobyte per flash chip). 
     The technique of using an aperture, mapper, and table in combination is well-known and similar to, for example, Accelerated Graphics Port (AGP) graphics applications using an AGP Aperture and a Graphics Address Relocation Table (GART). 
     Now the first four steps of method or code branch B above will be described in more detail, first using pseudo-code and then using a flow diagram and accompanying descriptions: 
     
       
         
               
             
               
               
               
             
           
               
                   
               
               
                 Step 1 - Get a free SRAM page 
               
               
                   
               
             
             
               
                   
               
             
          
           
               
                   
                 Get free SRAM page( ) 
                   
               
               
                   
                  if SRAM page free list is empty( ) then 
                   
               
               
                   
                   Free an SRAM page; 
                   
               
               
                   
                  Pop top element from SRAM page free list 
                   
               
               
                   
                 Free an SRAM page: 
                   
               
               
                   
                  sp = next SRAM page to free; // depending on chosen replacement  
                   
               
               
                   
                 policy 
                   
               
               
                   
                  if sp is dirty then 
                   
               
               
                   
                   foreach cache line CL in sp do // ensure SRAM contains last  
                   
               
               
                   
                 written data; 
                   
               
               
                   
                               // could instead also set caches to  
                   
               
               
                   
                 write-through  
                   
               
               
                   
                    CLFlush(CL);      // &lt;10μps per 32KB 
                   
               
               
                   
                   fp = Get free flash page; // wear leveling, etc. is perfomed here 
                   
               
               
                   
                   Send SRAM2flashCpy(sp, fp) command to DIMM; 
                   
               
               
                   
                   Wait until copy completes; 
                   
               
               
                   
                  else 
                   
               
               
                   
                   fp = flash address that sp maps to; 
                   
               
               
                   
                  Page Table[virtual address(sp)] = “not present”, fp; 
                   
               
               
                   
                 // In MP environment must handle multiple TLBs using additional  
                   
               
               
                   
                 code here 
                   
               
               
                   
                 Mapper[sp] = “unmapped” 
                   
               
               
                   
                 Push sp on SRAM page free list 
               
               
                   
               
             
          
         
       
     
     Step 2—Copy flash page to SRAM Copy flash page to SRAM page: 
     Send flash2SRAMCpy(sp, fp) command to DIMM; 
     Wait until copy completes; 
     Step 3—Update Page Table 
     Update Page Table: 
     // Use a bit-vector and rotate through the vector—cycling from 0 GB up to the 32 GB aperture and then roll around to OGB, re-using physical addresses 
     pa=next unused physical page; 
     if (pa==0) then 
     WBINVD; // we have rolled around so flush and invalidate the entire cache 
     PageTable[va]=pa; 
     Now we shall describe the key elements of these steps in the pseudo-code above using flow diagrams and accompanying descriptions.  FIG. 8A  shows a method  800  for the OS to obtain a free page of fast memory (“Get free SRAM page” in the above pseudo-code). Remember we are using fast memory and SRAM interchangeably for this particular example implementation. As an option, the present method  800  may be implemented in the context of the architecture and functionality of  FIG. 1  through  FIG. 7 . Of course, however, the method  800  or any operation therein may be carried out in any desired environment. 
     The operation  704  from  FIG. 7  indicates an operation for the OS to get a page of fast memory. Although many embodiments are possible and conceived, one such operation is disclosed here, namely the method  800 . That is, the method  800  is entered at entry point  802  whenever a new page of fast memory is needed. The decision  804  checks for a ready and available page from the page free stack. If there is such an available page, the OS pops that page from the page free stack and returns it in operation  810 . Alternatively, if the free stack is empty then the decision  804  will proceed to operation  806 . Operation  806  serves to acquire a free fast memory page, whether acquired from a pool or reused resources or whether from a newly allocated page. Once acquired then, the OS pushes the pointer to that page onto the page free stack and the processing proceeds to operation  810 , returning the free fast memory page as is the intended result to the method  800 . 
       FIG. 8B  shows a method  850  for the OS to free a page of fast memory (“Free an SRAM page” in the above pseudo-code). As an option, the present method  850  may be implemented in the context of the architecture and functionality of  FIG. 1  through  FIG. 8A . Of course, however, the method  850  or any operation therein may be carried out in any desired environment. 
     The operation  806  from  FIG. 8A  indicates an operation for the OS to free a page of fast memory. Although many embodiments are possible and conceived, one embodiment of such an operation is disclosed here, namely the method  850 . That is, the method  850  is operable to free a page of fast memory, while maintaining the fidelity of any data that may had previously been written to the page. 
     As shown, the system is entered when a page of fast memory is required. In general, a free fast memory page could be a page that had previously been allocated, used and subsequently freed, or may be a page that has been allocated and is in use at the moment that the method  850  is executed. The decision  856  operates on a pointer pointing to the next fast memory page to free (from operation  854 ) to determine if the page is immediately ready to be freed (and re-used) or if the page is in use and contains data that must be retained in slow memory (a “dirty” page). In the latter case, a sequence of operations may be performed in the order shown such that data integrity is maintained. That is, for each cache line CL (operation  858 ), the OS flushes the cache line (operation  860 ), the OS assigns a working pointer FP to point to a free slow memory page (see operation  862 ), the OS writes the ‘Dirty’ fast memory page to slow memory (operation  864 ), and the loop continues once the operation  864  completes. 
     In the alternative (see decision  856 ), if the page is immediately ready to be freed (and re-used), then the OS assigns the working pointer FP to point to a slow memory address that SP maps to (operation  868 ). Of course since the corresponding page will now be reused for cache storage of new data, the page table must be updated accordingly to reflect that the previously cached address range is (or will soon be) no longer available in cache (operation  870 ). Similarly, the OS records the status indicating that address range is (or will soon be) not mapped (see operation  872 ). Now, the page of fast memory is free, the data previously cached in that page (if any) has been written to slow memory, and the mapping status has been marked; thus the method  850  pushes the pointer to the page of fast memory onto the page free stack. 
       FIG. 9  shows a method  900  copying a page of slow memory to a page of fast memory. As an option, the present method  900  may be implemented in the context of the architecture and functionality of  FIG. 1  through  FIG. 8B . Of course, however, the method  900  or any operation therein may be carried out in any desired environment. 
     The operation  706  from  FIG. 7  indicates an operation to copy page of slow memory to a page of fast memory. In the embodiment shown, the OS is operable to not only perform the actually copy, but also to perform bookkeeping and synchronization. In particular, after the actual copy is performed (operation  904 ) the OS sends the fact that this copy has been performed to the HybridDIMM (operation  906 ) and the method  900  waits (operation  908 ) until completion of operation  906  is signaled. 
     These methods and steps are described in detail only to illustrate one possible approach to constructing a host OS and memory subsystem that uses mixed-technology memory modules. 
     While the foregoing is directed to embodiments of the present invention, other and further embodiments of the present invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.