Patent Description:
Computer systems such as network servers, personal computers, PDAs, mobile phones, video games, scientific instrumentation, industrial robotics, medical electronics, and so on, rely heavily on the capacity and throughput of their system or main memories and the speed of accessing them for optimal performance. Currently, dynamic random-access memory (DRAM) is commonly used as system memory. DRAM is a type of random-access memory that stores each bit of data in a separate capacitor in an integrated circuit. The capacitor can be either charged or discharged so that these two states are taken to represent the two values of a bit, conventionally called <NUM> and <NUM>. Since capacitors leak charge, the information eventually fades unless the capacitor charge is refreshed periodically. Because of this refresh requirement, it is a dynamic memory as opposed to SRAM and other static memory.

The structural simplicity of DRAM allows DRAM chips to reach very high densities, because billions of transistor and capacitor pairs can fit on a single memory chip. On the other hand, DRAM is volatile memory - it loses its data quickly when power is removed. Compared to Flash memory, which is a type of electronic non-volatile computer storage medium that can be electrically erased and reprogrammed, DRAM is also much more expensive. For example, high density DRAM can cost as much as <NUM> times more than highperformance Flash devices. Furthermore, Flash chips can have much higher density than DRAM chips, allowing a same-sized memory module to pack much more to reach a much larger memory capacity.

There are two main types of Flash memory, the NAND type and the NOR type, which are named after the NAND and NOR logic gates. The NOR type allows a single machine word (byte) to be written or read independently. NAND type Flash memory may be written and read in blocks (or pages), which are generally much smaller than the entire device. NAND Flash also has reduced erase and write times, and requires less chip area per cell, thus allowing greater storage density and lower cost per bit than NOR Flash. Moreover, NAND Flash also has up to ten times the endurance of NOR Flash. Thus, NAND Flash has been more widely used than NOR Flash.

Besides its advantages, Flash memory also has certain limitations, which pose many challenges to make it useful as main memory. One limitation of Flash memory, especially NAND Flash, is that it can only be erased a "block" at a time. Erasing a block generally sets all bits in the block to <NUM>. Starting with a freshly erased block, any location within that block can be programmed a byte or a word at a time in a random access fashion. However, once a bit has been set to <NUM>, only by erasing the entire block can it be changed back to <NUM>. In other words, Flash memory does not offer arbitrary random-access rewrite or erase operations.

Another limitation is that Flash memory has a finite number of program-erase cycles (typically written as P/E cycles). Most commercially available Flash products are guaranteed to withstand around a certain number of cycles (e.g., <NUM>,<NUM> P/E cycles) before the wear begins to deteriorate the integrity of the storage. Some chip firmware or file system drivers perform the so-called wear leveling technique by counting the writes and dynamically remapping blocks in order to spread write operations between sectors. For portable consumer devices, these wear-out management techniques typically extend the life of the Flash memory beyond the life of the device itself, and some data loss may be acceptable in these applications. For high reliability data storage, however, it is not advisable to use Flash memory that would have to go through a large number of programming cycles. <CIT> discloses a memory module that is coupleable to a memory controller hub (MCH) of a host system includes a non-volatile memory subsystem, a data manager coupled to the non-volatile memory subsystem, a volatile memory subsystem coupled to the data manager and operable to exchange data with the non-volatile memory subsystem by way of the data manager, and a controller operable to receive read/write commands from the MCH and to direct transfer of data between any two or more of the MCH, the volatile memory subsystem, and the non-volatile memory subsystem based on the commands. <CIT> discloses a plurality of specialized controllers each one adapted to control a particular type of data transfer operation, control the flow of data between a system bus and a local bus on a computer adapter card. The datasheet "<NPL> discloses a technical specification of the MT29F2G08AACWP, MT29F4G08BACWP, MT29F8G08FACWP NAND Flash Memory devices by Micron Technology Inc.

As shown in <FIG>, a computer or server system (computer system) <NUM> according to certain embodiments includes a central processing unit (CPU) or processor, a memory controller (MC), and one or more memory modules coupled to a system bus. The one or more memory modules provide a system memory, and may further provide storage. In certain embodiments, the MC may be integrated into the CPU. In further embodiments, the computer system may also include a direct data management controller (DMA) also coupled to the system bus. The CPU with or without the MC and/or the DMA, or the computer system <NUM> in part or in while, is sometimes referred to hereafter as the "System" or "system.

In certain embodiments, the computer system <NUM> may further include a network card and one or more I/O devices such as keyboard, monitor, touchscreen, microphone, speaker, etc. The network card may or may not be integrated into the CPU and provides network interface functions (wired or wireless) between the computer system <NUM> and local and/or wide area networks. The computer system <NUM> may further include a PCI bus, which is coupled to the system bus via a north bridge and one or more storage devices, such as a hard drive, a CD/DVD drive, and a USB drive, via a south bridge.

<FIG> is a block diagram of a Hypervault™ dual-in-line memory module (HVDIMM) <NUM>, which can be used to provide the system memory and/or the storage of the computer/server system according to certain embodiments. As shown in <FIG>, the HVDIMM <NUM> includes a volatile memory subsystem (HV-DRAM) <NUM>, a non-volatile memory subsystem (HV-Flash <NUM>) <NUM>, and a module control subsystem (HV Controller) <NUM>, mounted on a module board <NUM>, which may include one or more printed circuit boards. The HVDIMM <NUM> may also include a data buffer (HV-DB), and may also include a network interface controller (HV-NIC). In certain embodiment, the HV-DB <NUM> includes DRAM memory, such as terabyte DRAM memory (TBM). The HV-DRAM <NUM> and the HV Controller <NUM> are coupled to each other, to the system, and to the HV-Flash <NUM> via data signal lines (as represented by the dashed double-arrow lines) and control/address (C/A) signals lines (as represented by the solid double or single-arrow lines). As shown in <FIG>, data signal lines <NUM>, <NUM>, and <NUM>, together with the HV Controller <NUM> and the HV-DB <NUM>, form a dedicated data path between the HV-DRAM <NUM> and the HV-Flash <NUM>, allowing data to be transferred between the volatile and non-volatile subsystems without going through the memory channel or the CPU. In certain embodiment, the dedicated data path is a highbandwith data path.

As is also shown in <FIG>, the HVDIMM <NUM> further includes switches, FET-A (e.g., Field-effect transistor or FET switches). In certain embodiments, there are two sets of switches in the HVDIMM <NUM>, data switches and command/address switches. These switches do not need to be fast switches but they should support relatively short input to output delay time. In certain embodiments, the signal propagation time for both sets of switches should be a small fraction of a data period (e.g., <NUM>-<NUM> ps), so that the delay time can be hidden from the system.

As shown in <FIG>, data from the system is directly connected to HV-DRAM <NUM> data input/output (I/O) (not shown) while the system command/address signals are connected to the HV-DRAM <NUM> via the FET switches, such that the HV-DRAM <NUM> either can receive command/address from the system during, for example, normal operations when the system accesses the memory spaces in the HVDIMM <NUM>, or from the HV Controller <NUM> during, for example, backup/restore operations when the HVDIMM <NUM> backs up the content in the HV-DRAM after a power failure or restore the content back into the DRAM after power is resumed. The FET switches can be controlled by the HV Controller <NUM>.

In certain embodiments, the HV-Flash <NUM> includes MLC NAND Flash, which are partitioned to support fast access as well as enhance the error correction capability for virtual duplication. In certain embodiments, the HV-Flash <NUM> includes, for example, 256GB/512GB of main Flash and 32GB of scratch Flash. The main Flash can serve as a large storage with direct data bus on the HVDIMM <NUM> to the DRAM. The scatch Flash facilitates a mechanism to prolong the life time of the Flash memory cells in the HV-Flash <NUM>, as described below.

<FIG> is a block diagram of the HV-DRAM <NUM> subsystem according to certain embodiments. As shown in <FIG>, the HV-DRAM <NUM> subsystem includes DRAM devices <NUM>, a registered control device (RCD) or control register <NUM> to interface with the MC or the HV Controller <NUM>, and load-reduction data buffers (LRDB) <NUM>. In certain embodiment, the RCD <NUM> can be a standard register, which is a register in compliance with an industry standard, such as the Joint Electron Device Engineering Council Double Data Rate <NUM> Load Reduction Dual In-Line Memory Module (JEDEC DDR4 LRDIMM) standard, so that the HV-DRAM <NUM> can be compatible with a standard system memory interface. In certain embodiments, the data transmission circuits described in commonly owned <CIT>, can be used as the LRDB <NUM>. Although <FIG> shows the LRDB <NUM> as one unit, in practice, the LRDB <NUM> can include multiple units distributed across the module board <NUM> and coupled to respective groups of memory devices, as described in <CIT>.

In certain embodiments, the HV-DRAM <NUM> provides main memory functions for the HVDIMM <NUM> when the HVDIMM <NUM> is used to provide system memory. In cerain embodiments, the HV-DRAM <NUM> acts as buffer memory for the HV-Flash <NUM> when the HVDIMM <NUM> is used to provide storage. In certain embodiments, cache-line-wide reads from the DRAM is mapped to the Flash. There are, however, differences between a standard DRAM module (e.g. JEDEC DDR4 LRDIMM) and the HV-DRAM <NUM>. In certain embodiments, the HV-DRAM <NUM> may include data switches (FET-D), in addition to the command/address switches (FET-A). The data switch FET-D is coupled between DRAM data buffers (LR-DB) and the DRAM, while the command/address switch FET-A is coupled between the memory channel C/A bus and the RCD <NUM>. The FET switches, FET-A and FET-D, can be controlled by the HV Controller <NUM> to transition the HVDIMM <NUM> between different operation modes. In certain embodiments, either or both of these switches, FET-A and FET-D, are not required, and the different modes of operation can be accomplished by tristating the relevant I/Os in the DRAM, the HV Controller <NUM>, and/or the LRDB <NUM>.

<FIG> is a block diagram of the HV Controller <NUM>, which can include an applicationspecific integrated circuit (ASIC) device or a programmable field gate array (FPGA) device. As shown in <FIG>, the HV Controller <NUM> includes control logic <NUM>, a data buffer <NUM>, local memory <NUM> and registers <NUM>. The HV Controller <NUM> further includes a DRAM interface <NUM>, a Flash interface <NUM>, a system management Bus interface <NUM>, and a network interface <NUM>. In certain embodiments, the HV Controller <NUM> controls data transfers between the HV-DRAM <NUM> and HV-Flash <NUM>. It keeps an address management table in the local memory on-chip memory space, operates FET switches, and generates proper commands to the HV-Flash <NUM> and HV-DRAM <NUM> for data transfers therebetween.

In certain embodiments, the HV Controller <NUM> also keeps an HVDIMM <NUM> level bad block table in the local memory and manages the scratch Flash, as discussed below. In certain embodiments, the HV Controller <NUM> is configured to execute error detection/correction routines to insure the integrity of data stored in the Flash, to perform Flash life extension operations by averaging out the HV-Flash <NUM> block usage and/or using the scratch Flash to minimize/reduce program/erase operations in the main Flash.

In certain embodiments, the control logic <NUM> includes logic circuits and may further include one or more processing units to pre-process data being transferred from the Flash to the DRAM, so as to save DRAM memory space and reduce data traffic in the memory channel by off-loading some of the computing work traditionally done by the CPU, as discussed further below.

In certain embodiments, the HVDIMM <NUM> can be operated to back up data in the DRAM in response to power failure events. The HV Controller <NUM> provides correct timings for HV-DRAM <NUM> to be operated in an DLL-off mode when data in the DRAM is being transferred to the Flash. The HV Controller <NUM> also provides proper operational procedure for the back-up and restore processes. The FET switches, FET-A, can be configured to isolate the RCD <NUM> and the HV Controller <NUM> from the C/A bus and to allow the RCD <NUM> to receive C/A signals from the HV Controller <NUM> during the back-up and restore processes.

In certain embodiments, the HVDIMM <NUM> is configured to allow the system to access it via a system management (SM) bus using, for exampe, the I<NUM>C protocol or any other system control bus interface. For example, the system can also use the SM bus to configure the HV Controller <NUM> by setting certain registers in the HV Controller <NUM>. The HV Controller <NUM> can also use the SM bus to notify the system when certain operation is completed or when an error is encountered, either using a preconfigured interrupt signal, or by updating a predefined status register in the system bus interface of the HV Controller <NUM>, or in the DMA.

In certain embodiments, the HV Controller <NUM> also manages network interfaces between the HVDIMM <NUM> and any local or wide-area networks in conjunction with HV-NIC so as to facilitate direct data transfers between the HVDIMM <NUM> and other storage devices in the local or wide-area networks. In certain embodiments, the HV Controller <NUM> includes a network interface and/or is coupled to a network interface card (HV-NIC), which can take the data from the HV-DRAM <NUM> and/or HV-Flash <NUM>, and constructs network packets with proper source and destination addresses. In general, the source address is preconfigured by the system. In certain embodiments, the HV-NIC or network interface and some or all of the other components of the HV Controller <NUM> can be embedded into a same ASIC or FPGA.

In addition to the hardware components shown in <FIG>, the computer system <NUM> also includes software/firmware components. In certain embodiments, the software/firmware components can be roughly represented as a stack of software/firmware layers <NUM> over a hardware layer <NUM>. As shown in <FIG>, the stack of software/firmware layers include an applications layer <NUM> sitting on an operating system layer <NUM>. The applications <NUM> are software programs that perform specific tasks. The operating system <NUM> manages the hardware and software resources of the computer system <NUM> and acts as an intermediary between the application programs <NUM> and the hardware components of the computer system <NUM>.

The operating system <NUM> includes a kernel <NUM>, which are computer programs that manages input/output requests from other software programs (or processes), and which translates the requests into data processing instructions for the CPU and/or other hardware components of the computer system <NUM>. The kernel can include an interrupt handler that handles all requests or completed I/O operations that compete for the kernel's services, a scheduler that determines which programs share the kernel's processing time in what order, and a supervisor that actually gives use of the computer to each process when it is scheduled. The kernel may also include a manager of the operating system's address spaces in memory or storage. The kernel's services are requested by other parts of the operating system or by applications through a specified set of program interfaces sometimes referred to as system calls.

Between the kernel and the hardware layer is the basic input/output system (BIOS) layer <NUM>, which in certain embodiments is firmware stored in some sort of permanent memory (e.g., programmable read-only memory (PROM), or electrically programmable read-only memory (EPROM)) and includes program codes for initializing and testing the system hardware components, and to load the operating system from a mass memory device when the computer system <NUM> is boot up. The BIOS may additionally provides an abstraction layer for the hardware components so as to provide a consistent way for application programs and operating systems to interact with the hardware components such as the system memory and input/output devices.

In certain embodiments, the software stack further includes an HV driver <NUM> in, for example, the kernel. The HV driver <NUM> is a software program for controlling system access to the HVDIMM <NUM>. Thus, the HVDIMM <NUM> can operate like a standard DIMM (e.g., DDR4 LRDIMM) without requiring any changes to the BIOS. The HV driver <NUM> has access to a memory space <NUM> in the CPU or one or more other memory devices in the computer/server system, which is used to store lookup tables or other configuration information, and which the HV driver <NUM> can consult with and/or update as needed. In certain embodiments, the driver intercepts system calls to access the HV-Flash <NUM> and/or HV-DB <NUM> and directs the memory controller to send control, address and data signals in response to the system calls and in compliance with the memory interface standard the system is using (e.g., the JEDEC DDR4 LRDIMM Standard). In certain embodiments, the FET switches, FET-A, are configured to allow both the RCD <NUM> and the HV Controller <NUM> to receive the C/A signals from the MC during normal system operation, and the HV Controller <NUM> is configured to monitor the C/A signals from the memory controller and to recognize and act upon C/A signals fomulated in response to system calls to access the HV-Flash <NUM> and/or the HV-DB <NUM>.

For example, as shown in <FIG>, which illustrates different data paths supported by the HVDIMM <NUM>. The dash/dotted lines represent the normal system read/write data path <NUM>, by which the system performs read/write operations with the HV-DRAM <NUM> via the stadard memory interface. The dotted lines represent a page-out (or swap-out) data path <NUM>, by which the HVDIMM <NUM> transfers data from the HV-DRAM <NUM> to the HV-DB <NUM> and/or the HV Flash under the control of the HV Controller <NUM> and in response to system request to move the data from the main memory to the storage so as to, for example, make space in the main memory for data to be swapped in. The dashed lines <NUM> represent a page-in (or swap-in) data path, by whith the HVDIMM <NUM> transfers data from the HV-DB <NUM> or the HV-Flash <NUM> to the main memory under the control of the HV Controller <NUM> and in response to system request to move the data from the storage to the main memory for random access by one or more software programs.

In certain embodiments, normal system access to the main memory is conducted between the system and the HV-DRAM <NUM>, without much involvement from the HV driver <NUM> or the HV Controller <NUM>. In certain embodiments, the memory interfaces in the computer system <NUM> are designed to be slave interfaces without per command handshake provision. So, the system does not have knowledge about whether any on-DIMM (intra-module) activities are occupying the DRAM I/Os. For example, if the DRAM I/Os are being occupied for transferring data between main memory and storage, and the system (memory controller) initiates a data read or write command with the expectation that the DRAMs would be ready to execute the system command, the DRAMs would fail to execute the system read/write operation and the system would experience a 'memory failure', which leads to a system failure.

In certain embodiments, for operations involving the HV-Flash <NUM> or HV-DB <NUM>, such as a swap-in or swap-out operation, the HV driver <NUM> and the HV Controller <NUM> work together to move data in or out of the main memory without causing conflict with normal system access to the main memory. In certain embodiments, the HV-driver sends a memory access request to the memory controller when it needs to transfer data between DRAM (main memory) and Flash (storage) and provides the DRAM and the Flash addresses with this request. The memory controller may interleave the HV-driver requests with normal system memory access requests.

In certain embodiments, after receiving a page-in command to transfer data from the HV-Flash <NUM> to the HV-DRAM <NUM>, the HV-controller monitors the memory read/write commands from the memory controller. If the memory address for a write command matches the target DRAM address in the page-in command, the HV-controller replace the write data from the system with the data from the Flash. On the other hand, after receiving a page-out command to transfer data from the HV-DRAM <NUM> to the HV-Flash <NUM>, the HV-controller monitors the memory read/write command from the memory controller. If the memory address for a read command matches the source DRAM address in the page-out command, the HV-controller snoops the DRAM read data, and transfer the DRAM read data to the Flash.

For example, as shown in <FIG>, when a page-in request is issued, the HV Driver would intercept the page-in request and formulate a page-in command and memory-mapped I/O (MMIO) signal values according to the page-in request, which may include a source address range in the storage and a destination address in the main memory. The HV driver <NUM> instructs the memory controller to issue the page-in command to the HVDIMM <NUM>. The page-in command uses the same set of C/A signals as a standard write command but with one or more designated C/A signals (e.g., chip select signal(s)) asserted or deasserted to indicate that this is not a normal write command for the DRAM devices <NUM> to respond to. The memory controller would schedule and send the page-in command as if it is a standard write command. The RCD <NUM> in the HV-DRAM <NUM> is configured to recognize this page-in command and would not forward the C/A signals to the DRAM devices <NUM>. The HV Controller <NUM> on the other hand has been monitoring the C/A signals and would act upon the page-in command by controlling the data transfer between HV-DRAM <NUM> and the HV-Flash <NUM> or HV-DB <NUM>.

In certain embodiments, the HV Controller <NUM> in response to the page-in command may set the FET switches, FET-D, to direct the data signals associated with the page-in command to the HV Controller <NUM>. These data signals represent the MMIO values fomulated by the HV driver <NUM> and include further information/instructions related to the swap-in request, such as what addresses to take data from in the HV-Flash <NUM> and what addresses in the HV-DRAM <NUM> to place the data. In certain embodiments, after receiving the MMIO signals, the HV Controller <NUM> may check whether the requested page-in data has already been loaded into the HV-DB <NUM>, and if not, the HV Controller <NUM> would initiate and control data transfer from the HV-Flash <NUM> to the HV-DB <NUM> by reading the page-in data from the HV-Flash <NUM> and writing the page-in data to the HV-DB <NUM> using the data buffer <NUM> in the HV Controller <NUM> as temparary storage for the page-in data between the read and write operations. In certain embodiments, after all page-in data are transferred to the HV-DB <NUM>, the HV Controller <NUM> may reload some of the page-in data into the data buffer <NUM> in the HV Controller <NUM>, reset the FET-D switches to allow the HV-DRAM <NUM> to perform normal memory operations with the memory controller, and wait for the command from the memory controller to write the data into the HV-DRAM <NUM>.

In certain embodiments, the HV driver <NUM> is configured to wait for a certain amount of time to allow the page-in data to be transferred from the HV-Flash <NUM> to the HV-DB <NUM>. The HV driver <NUM> may determine the amount of time based on how much data is being paged-in. Afterwards, the HV driver <NUM> would instruct the memory controller to schedule and send a dummy write command to the HVDIMM <NUM>. In certain embodiments, the dummy write command is like a normal write command except that it is followed with dummy data or no data from the memory controller. The HV Controller <NUM> would recognize the dummy write command since it is directed at the same addresses the page-in data should be placed. In response, the HV Controller <NUM> would set the FET-D switches and would provide the page-in data to the DRAM devices <NUM>. The dummy data from the memory controller is thus ignored or discarded. In certain embodiments, the HV Controller <NUM> outputs the page-in data to the DRAM devices <NUM> a certain time period after receiving the dummy write command so that the data appears at the DRAM I/Os in accordance with the CAS latency parameters of the D RAM devices <NUM>. After page-in data associated with the dummy write command has been written into DRAM, the HV Controller <NUM> would reset the FET-D switches to allow the DRAM to perform normal system memory operations.

In certain embodiments, the HV Controller <NUM> is confgured to monitor the memory commands from the memory controller and schedule on-DIMM (intra-module) data transfers accordingly to avoid data access conflicts. In certain embodiments, the HV Controller <NUM> would work around system memory accesses when placing the page-in data at the DRAM I/Os, so as to avoid system failure caused by such data access conflicts. For example, as illustrated in <FIG>, as data A thruogh G are being paged in from the HV-DB <NUM> (TBM) to the main memory (MM), the system may also be issuing memory access commands to write data M, M+<NUM>, M+<NUM>, M+<NUM> into the main memory (MM). The memory controller may schedule the memory commands from the system and the dummy write commands from the HV driver <NUM> as follows:.

Before the system issues the CAS TBM B command, the HV Controller <NUM> (referred to in the figure as "FPGA") may have issued CAS TBM A', CAS TBM B', and CAS TBM C' commands to the TBM to output data A, data B, and data C to the HV CONTROLLER. The HV Controller may preload data A and data B from the TBM (as shown by the data blocks A and B in the "FPGA pg-in input from TBM") and place it in the data buffer <NUM> in the HV Controller. Afterwards, data C is output from the TBM in response to CAS TBM C' from the HV Controller.

The HV Controller continues to issue CAS TBM D' to the TBM when the HV Controller observed the CAS sys M command from the system. In response, the HV Controller issues a DES/NOP command to the TBM to pause the data transfer between the TBM and the MM. Thus, FPGA page-in (Pg-in) output is paused, as shown by the gap between data B and data C in the FPGA Pg-in output, and system data M (Sys) is received by at the MM input. Afterwards, the HV Controller continues to issue CAS TBM E' to the TBM when it observed CAS sys M+<NUM> and later CAS sys M+<NUM> from the system. In response, the HV Controller issues two consecutive DES/NOP commands to pause the TBM from outputting data to the HV Controller. As a result, no data is output between data E and data F from the TBM, and no data between data C and data D is driven from the HV Controller to the MM, leaving a gap in the HV Controller output to the MM to allow the MM to take system data M+<NUM> and M+<NUM> (Sys).

In certain embodiments, the HV Controller <NUM> is further configured to perform shadowed data transfer operations between the HV-DRAM <NUM> and the HV-Flash <NUM>. For example, when a system command targes a DRAM address that has been preprogrammed as an address that requires data to be transferred from the HV-Flash <NUM>, the HV Controller <NUM> would perform such a transfer to enable proper system access to this preprogrammed address.

The page-out operations can be performed similarly but in opposite direction and in a different order, as exemplified in <FIG>. Thus, the page-out/page-in process can be orders of magnitude faster than using PCIe SSD or conventional memory channel storage because the page-in and page-out data can be transferred between the main memory and the storage on the HVDIMM <NUM>, without going through the memory channel or the CPU. In addition, system performance is further improved because the data transfer between HV-Flash <NUM> and HV-DRAM <NUM> also frees up the main memory channel and the CPU. In certain embodiments, data in the HV-Flash <NUM> is stored in DRAM format, so there is no need to convert the data format as data is being moved between the HV-Flash <NUM> and HV-DRAM <NUM>, which is conventionally performed by the CPU.

In certain embodiments, as shown in <FIG>, multiple HVDIMMs <NUM>, e.g., HVDIMM <NUM>, HVDIMM <NUM>, etc., can be used together to provide the system memory and/or the storage coupled to the CPU/MC via the system bus, which includes a system control/address bus and a system data bus. Since the operating system sees the Flash space of all HVDIMM <NUM> as a unified HVDIMM <NUM> storage, and the system may not know which physical Flash devices are located on which physical HVDIMM <NUM>. As a result, the HV driver <NUM> could issue a page-out command with the DRAM address on one HVDIMM <NUM> and the Flash address on another HVDIMM <NUM>.

To address this issue, the HV driver <NUM> in certain embodiments builds a memory association table, as shown in <FIG>. Since the operating system views the storage provided all of the HVDIMM <NUM> as one storage disk (say, the K-disk), the driver can partition the K-disk into a plurality of sub-disks, K1, K2,. Kn, each associated with the a respective HVDIMM <NUM>. For example, as shown in <FIG>, the memory association table has <NUM> entries per CPU, which in certain embodiments is the number of DIMM socket per CPU. Each entry correspond to a respective HVDIMM <NUM> and includes the sub-disk number, the HVDIMM <NUM> ID, the minimum and maximum address bounds for the HV-DRAM <NUM> on the DIMM, and the minimum and maximum address bounds for the HV-Flash <NUM> on the DIMM. Thus, by consulting the memory association table, the HV driver <NUM> would try to swap data within the address bounds of the HV-DRAM <NUM> and the HV-Flash <NUM> on the same HVDIMM <NUM>.

In general, for cost/performance/power reasons, the memories in a computer/server system are arranged in layers such that faster and smaller memories are located within (or close) to a memory cores (e.g., first layer cache), and density and access time increase as memory is physically and electronically further away from the core. There are layers of cache memories in a CPU/MC package, and the memory module(s) that are connected to the MC via a dedicated memory channel in the system bus is regarded as the main memory, which provides dynamic random data access by the CPU. The storage devices are further away from the CPU and are usually very large memories in the system, such as hard disc devices (HDD), solid-state storage devices (SSD), etc., but they do not provide dynamic random access capabilities.

The memories in the computer system <NUM> are somewhat similarly structured, as shown in <FIG> except that the storage provided by the HVDIMM <NUM> are not far away from the CPU and data from the storage can be moved into the HV-DRAM <NUM> for random access without going through a south bridge, or a PCI bus, or even the memory channel. Furthermore, the HVDIMM <NUM> provides the HV-DB <NUM>, which can act as a cache memory for the HV-DRAM <NUM> by storing data which the HV-DRAM <NUM> does not have space to hold and which can be quickly moved into the DRAM when needed in response to a dummy write command from the memory controller.

In certain embodiments, the HVDIMM <NUM> is configured to provide a very large, configurable, expandable, dynamic random access system memory to a computer system. The HVDIMM <NUM> incorporates novel memory cache layer techniques, i.e., the Memory Window techniques, where the HV-DRAM <NUM> holds contiguous and complete sections of HV-Flash <NUM> for dynamic access by the computer system. Thus, the HV-Flash <NUM> works as a data vault to the HV-DRAM <NUM>, such that the computer system can open up a Memory Window (MW) in the HV-Flash <NUM> and bring needed data stored in the HV-Flash <NUM> to the HV-DRAM <NUM> via Memory Window for dynamic random access by the System.

In certain embodiments, the HVDIMM <NUM> can perform two types of operations concurrently: standard memory operation (SMO), as discussed above, and Memory Window operation (MWO). During SMO, the HVDIMM <NUM> provides a standard main memory space via a standard protocol (e.g., the JEDEC DDR4 LRDIMM protocol). During MWO, as shown in <FIG>, a specifc memory area (MW), such as a Flash segment or block, in the HV-Flash <NUM> is opened up to support high speed dynamic random access by the computer system. Requested data in the specific memory area is moved from the Flash to the DRAM. If the system requests to open up more MWs than the DRAM space is allowed, the system has the option to have the HVDIMM <NUM> overwrite the least recently used (LRU) DRAM area, or overwrite a specific DRAM location. The data from a DRAM area is moved back from the HV-DRAM <NUM> to the HV-Flash <NUM> either when there is no more open pages for a specific duration (by default), or when the system specifically requests to save the data. The MWO can be a background operation that is controlled by the HVDIMM <NUM> controller (HV Controller <NUM>).

Thus, to the computer system, the HV-Flash <NUM> can also be viewed as a very highspeed access storage because data does not need to be moved from a separate storage unit to the main memory, and because data from any specific memory area in the Flash can be accessible via the memory channel upon request from the system to open up a MW. In certain embodiments, the system can make a single request to open a MW with a certain size.

In one embodiment, the HVDIMM <NUM> is a multi-rank (e.g., <NUM>-rank) DIMM, and the HV Controller <NUM> controls each rank independently, so that the system can access one rank while the HV Controller <NUM> performs an MWO. In general, however, it is preferred that the MWO be executed on both ranks in unison for better memory management. Regardless of whether the system executes MWO per rank or on both ranks in unison, the HV Controller <NUM> can set its internal (per rank) register bits to indicate completion of an MWO. As stated above, the system can also configure the HV Controller <NUM> to generate an interrupt signal when the MWO is completed instead of or in addition to setting the register bits.

In certain embodiments, the HV Controller <NUM> also controls the boot-up process for the HVDIMM <NUM>. There are two types of boot; Fresh boot (booting after a clean shut down) and Reboot (booting after a power failure). Unlike the Fresh boot case (where there is no valid data in HVDIMM <NUM>), Reboot requires the HV Controller <NUM> to populate the HV-DRAM <NUM> with the same data that was in HV-DRAM <NUM> at the time of power-loss.

The HVDIMM <NUM> can have two very different operation frequencies, the HV mode frequency and the LRDIMM mode frequency. The HV mode frequency is used to transfer data between HV-DRAM <NUM> and HV-Flash <NUM> while the LRDIMM mode frequency is used to transfer data between HVDIMM <NUM> and the system. In certain embodiments, the HV-DRAM <NUM> has two operational modes, a standard operation mode and a MW mode. During the standard operation mode, the HV-DRAM <NUM> fully supports standard memory operations (e.g., the JEDEC DDR4 LRDIMM operations) including the initialization and training protocols. When the HV-DRAM <NUM> is in the MW mode, the HV-DRAM <NUM> operates with its DLL turned off since the MW mode of operation frequency (HV mode frequency) is much slower (e.g., an order of magnitude slower) than the frequency range of the DLL, which covers the standard operation mode frequency. The HV Controller <NUM> uses the HV mode frequency for MW operations, during which the DRAM and RCD <NUM> is put into the JEDEC DLL-off operational state.

In certain embodiments, commencement of a MWO is initiated by a request from the System. <FIG> is a block diagram of certain components in the CPU with an integrated MC according to certain embodiments. As the memory channel between the MC and the HVDIMM <NUM> may not allow sufficient number of address bits to address the entire memory space in the HV-Flash <NUM>, the System may keep a look-up table about which areas of the HV-Flash <NUM> have been copied in the HV-DRAM <NUM> for random access. As shown in <FIG>, when the System needs to access a certain memory area in the HV-Flash <NUM>, the System would check the lookup table to determine whether data stored in the memory area has been copied to a certain area in the HV-DRAM <NUM>, i.e., a whether a MW is opened in the HV-Flash <NUM> to allow the System access to the data. If the answer is yes, the MC would proceed to perform memory operations to access the memory area in the DRAM. If the answer is no, the System would send a request to open the MW in the HVDIMM <NUM>. The request would include identification of the memory area in the HV-Flash <NUM> to be accessed, such as a starting address of the memory area and a size of the memory area, and a destination address in the DRAM, to which data from the memory area is to be transferred. If the System needs more than one MWs, more than one requests can be sent one after another. In certain embodiments, the request is sent directly to the HV Controller <NUM> in the HVDIMM <NUM> via the I<NUM>C bus or any other system management/control bus. In certain other embodiments, the request is sent to and processed by the DMA controller so that the System can continue to perform other memory operations via the memory channel while the MW is being opened.

In certain embodiment, the DMA controller is used to control certain aspects of the MWO processes. <FIG> is a block diagram of an exemplary DMA controller. In certain embodiments, the (DMA) controller can be used in conjunction with the CPU/MC to initiate and monitor MWO in the HVDIMM <NUM>. Without DMA, the CPU/MC can be occupied for part of or the entire duration of a MWO, and is thus unavailable to perform other tasks. With the DMA, the CPU can simply initiate a data transfer request to the DMA and then performs other operations while the transfer is in process. When the data transfer is done, the CPU is notified by an interrupt from the DMA controller. Thus, the DMA can offload extensive memory operations from the CPU. Many hardware systems use DMA, including disk drive controllers, graphics cards, network cards and sound cards. DMA is also used for intra-chip data transfer in multi-core processors. Computers that have DMA channels can transfer data to and from devices with much less CPU overhead than computers without DMA channels. Similarly, a processing element inside a multi-core processor can transfer data to and from its local memory without occupying its processor time, allowing computation and data transfer to proceed in parallel.

In certain embodiments, as shown in <FIG>, the DMA receives a request from the CPU/MC to open a MW (MW) in the HVDIMM <NUM>. The DMA can buffer the request and forward the same or reformulated request to the HV Controller <NUM> via the I<NUM>C bus. The DMA can actively monitor the associated MWO in the HVDIMM <NUM> and inform the CPU/MC via an interrupt when the MW is opened.

Characteristics associated with Flash memories such as limited endurance and slow writes may require the HVDIMM <NUM> to obtain support from an operating system (OS) running on the CPU. The OS may also need the knowledge of the movement of pages between the DRAM and Flash so as to know when to hide the weak characteristics of the Flash. At boot up, the OS needs to allocate memory pages in the DRAM. After write to a certain page, the OS may also need to know an estimated time when a next write to the page can be performed. The OS may also need to set page-table entries for the DRAM pages and Flash pages. Some or all of these tasks can be offloaded to the DMA, which include status registers, internal memories and control logic <NUM> to keep track of these activities.

For example, the DMA can store information regarding how long the HV Controller <NUM> may need to transfer a certain amount of data from the HV-Flash <NUM> to the HV-DRAM <NUM>. Thus, the DMA does not need to wait to receive a notification from the HV Controller <NUM> before telling the CPU/MC that the memory window has been opened for dynamic random access. Alternatively or addtionally, the DMA can break a request to open a Memory Window into multiple requests each for a smaller chunk of data of a predetermined size (e.g., 4KB), as the time required to complete each of such data transfers is more predictable.

Thus, the HVDIMM <NUM> allows the system to open up a Memory Window in a HV-Flash <NUM> block for dynamic operation. The system sends the starting address and the data size to HV, and the HV Controller <NUM> opens up the block of memory containing the data and transfers the amount of requested data into the HV-DRAM <NUM>. In certain embodiments, the minimum transfer size per HV-Flash <NUM> block is 32KB, and the maximum size is the HV-Flash <NUM> block size (e.g., 4MB). Therefore, if the transfer size per HV-Flash <NUM> block is 32KB, for 8GB DRAM, the system can open up to <NUM> HV-Flash <NUM> blocks simultaneously.

In certain embodiments, the HV Controller <NUM> is configured to provide HV-Flash <NUM> address management and keeps track of physical HV-Flash <NUM> addresses in relation to virtual-physical addresses known to the system. This can be done by creating and updating an address mapping table, which maps the system (virtual-physical) address to the HV-Flash <NUM> physical address for tracking the address of any particular Flash block, and to the offset address of each opened (and copied to HV-DRAM <NUM>) memory location within each block. HV Controller <NUM> uses the offset addresses to correctly place data from HV-DRAM <NUM> back into proper locations within a particular block of Flash memory. Table <NUM> lists description and sizes for a few address types according to certain embodiments.

<FIG> illustrates certain processes carried out by the HV Controller <NUM> to open a memory window in the HV-Flash <NUM> for dynamic random address by the system according to certain embodiments. In certain embodiments, the HV Controller <NUM> may notify the DMA or CPU/MC that a Memory Window has been opened after data from the memory window has been successfully transferred to the DRAM. In other embodiments, the HV Controller <NUM> may predict the time when the transfer would be completed based on the request for the Memory Window and historical information, and send the notification to the DMA or CPU/MC before the completion of the transfer so that the data transfer will be completed when the CPU/MC receives the interrup from the DMA or the notification directly from the HV Controller <NUM> and gets around to start the memory operation with the memory window.

<FIG> illustrates certain processes carried out by the HV Controller <NUM> to close a memory window according to certain embodiments. The system may request to close a HV-Flash <NUM> area via the I<NUM>C interface or some other system control bus interface. The request may include a starting HV-DRAM <NUM> address and the size of the Memory Window. In certain embodiments, if the system needs to close one Memory Window but leaves other Memory Windows within a HV-Flash <NUM> block open, the HV Controller <NUM> would not close (update) the HV-Flash <NUM> block until all Memory Windows within the block are closed.

In case of a catastrophic system failure due to, for example, power loss, the HV Controller <NUM> may also assume the responsibility of moving data from HV-DRAM <NUM> to HV-Flash <NUM> and closes the open blocks. If none of updated HV-DRAM <NUM> data has been stored into HV-Flash <NUM>, the maximum size of data that HV Controller <NUM> may need to move can be as large as the DRAM size, e.g., 8GB.

In certain embodiments, to make room for a Memory Window in the HV-DRAM <NUM>, certain DRAM pages may need to be moved to Flash. The HV Controller <NUM> would execute one or more pre-erased Flash pages and copy the one or more DRAM pages into the Flash. The copying can be completed without slowing or stalling the CPU. Small copies of the DRAM pages can also be staged or held in the DRAM or in a cache-buffer/scratch-pad in the HVDIMM <NUM> for the OS to the control.

In certain embodiments, as shown in <FIG>, the HV Controller <NUM> includes a built-in processor <NUM> and associated hardware (e.g., a dual ARM cortex A9 core configuration integrated as a part of an FPGA SOC), and can act as a co-processor to provide on-DIMM (intra-module) data handling, such as searching, sorting, screening, categorizing, structureing, formatting, etc. Thus, certain tasks traditionally performed by the system CPU can be offloaded to the co-processor so that the overall system performance can be significantly improved. Examples of such tasks include, but are not limited to, in-memory compression/decompression (e.g., source of data in one segment of DRAM and processed data in another segment of DRAM), in-memory encryption/decription; security authentication, etc. Since the co-processor has direct access to the DRAM main memory or the Flash storage without system CPU's involvement, the co-processor can compress, encrypt, and authenticate data in on-DIMM memories without system-wide overhead.

In certain embodiment, software or firmware packages with Application Programming Interfaces (API) exposed to the system software are provided to support on-DIMM computing/processing. The software or firmware packages are run by the coprocessor and may include, for example, software development kits (SDK), such as data compression SDK, data encription SDK, data authentication SDK, etc. The firmware packages can be stored on the HVDIMM <NUM> or they can be downloaded from the system. The software or firmware packages can be configured to support different features and the configurations (e.g., license bits, bits indicating enabled functions) can be stored in a onetime programmable devce (OTP) on the HVDIMM <NUM>. For example, the HV Controller <NUM> can compress or decompress a certain block or section of data stored in the main memory or the storage on the HVDIMM <NUM> using a type of compression algorithm specified by corresponding configration bits stored in the OTP.

As a further example, when the CPU is running a search process involving a large amount of data stored in the on-DIMM storage, the HV Controller <NUM> can pre-screen the data to reduct the data size to be handled by the CPU as the data is being transferred from the on-DIMM storage to the main memory. The HV Controller <NUM> can also sort the data to reduce the data categorization and collectiont time. The HV Controller <NUM> can also search the data to support fast querying of meta data information. In a further example, the data from the storage can be presearched so that only entries that are considered relevant to the search criteria are required to go through ECC and be loaded into main memory.

In further embodiments, the HV Controller <NUM> uses the HV-NIC (which can be, for example, an Ethernet interface controller) to provide direct data transfer between a network and on-DIMM memory, and data extraction/correction using the on-DIMM coprocessor, so that data can be loaded directly from the network, to the storage, the main memory, or both simultaneouly, or vice versa, without going through the system CPU. Thus, the HVDIMM <NUM> supports efficient data sharing in a cluster environment.

For example, multiple HVDIMM <NUM> can be used in a Hardoop processing framework, which is an open-source software framework for storage and large scale processing of data sets on clusters of CPUs each representing a DataNode in a distribued computing environment. Each DataNode can include a number of HVDIMM <NUM>, which together can contain, for example, <NUM> GB of main memory and <NUM>-<NUM> TB of memory channel storage. Very fast memory channel storage through put rate (e.g., <NUM> GB per each <NUM> CPU server) can be achieved because of parallel data transfer betweent the Flash and the DRAM on multiple HVDIMM <NUM>. Furthermore, the storage on the HVDIMM <NUM> can be accessed with very low latency (comparable to the latency for accessing the DRAMs) because the storage is accessed through the memory channel. Since data is moved between HV-DRAM <NUM> and HV-Flash <NUM> without having to go through a storage channel or PCI interface, very large blocks of data (e.g., up to <NUM> GB) can be accessed in read dominated operations.

In certain embodiments, as shown in <FIG>, the HVDIMM <NUM> provides a high bandwidth dedicated data path <NUM> between the main memory and the storage on the HVDIMM <NUM> to support on-DIMM data processing, and fast 'page swap' and 'demand page' operations. In further embodiment, the on-DIMM data buffer (HV-DB <NUM>) can be a very large data buffer such as terabit memory (TBM) to serve as temporary storage for the on-DIMM processing. In certain embodiments, as shown in <FIG>. the HVDIMM <NUM> also provides another data path <NUM> to allow data transfer between the system and the HV-Flash <NUM> via the HV Controller <NUM> and the HV-DB <NUM> without going through the HV-DRAM <NUM>.

In further embodiments, As shown in <FIG>, in certain embodiments, the Flash storage on the HVDIMM <NUM> includes a number of (e.g., <NUM>) standard embedded multi-media card (eMMC) packages each having an embedded multi-media interface, a Flash controller and Flash memory. The HV Controller <NUM> also includes built-in redundant array of independent disks (e.g., RAID <NUM>) circuit <NUM> that provides dynamic hardware-based errorcorrection, full data recovery and data reconstruction, resulting in increased Flash life time. The RAID <NUM> feature also minimized requirement for data duplication.

The eMMCs generally support error correction in hardware. Issues can arise, however, when one of the eMMCs cannot correct certain errors in data from the Flash memory in its package during a read operation. In certain server systems, storage networks are built with redundancies (e.g., RAID) to enable further correction of errors at the storage system level. Before eMMCs were used to form the Flash storage, such redundancies were helpful as the Flash storage would output uncorrected data with error indication. An eMMC, however, does not produce output data if it cannot correct the errors in the data. Thus, the missing bit from an eMMC can cause system failure and/or unrecoverable data error.

To address this problem, the HVDIMM <NUM> according to certain embodiments includes at least one parity eMMC <NUM> (e.g., the 9th eMMC in <FIG>) that is used to store parity bits associated with the data stored in the rest of the eMMCs ("data eMMCs," e.g., the 1st to the 8th eMMCs in <FIG>). The HVDIMM <NUM> further includes an error correction (e.g., RAID <NUM>) circuit in, for example, the HV Controller <NUM>. In certain embodiments, the RAID <NUM> circuit is included in the data paths between the storage and the system, which may also includes the TBM and/or the main memory, as shown in <FIG>.

In certain embodiment, as shown in <FIG>, when write data is to be written to the storage, the RAID <NUM> circuit receives each set (e.g., <NUM> bytes) of write data from, for example, the TBM, and generates a paraty byte for each set of data bytes. The RAID <NUM> circuit then outputs the data bytes together with its associated parity byte for storing in respective eMMC circuits. During a read operation, as shown in <FIG>, the HV Controller <NUM> outputs control and address signals to instruct the eMMCs to output read data. The RAID <NUM> circuit would receive sets of data bytes from the data eMMCs and parity bytes from the parity eMMC. For each set of data bytes received in parallel, the RAID <NUM> circuit would determine if the set of data bytes is missing a byte from one of the data eMMCs. If no data byte is missing, the RAID <NUM> circuit would move the set of data bytes along its intended path. If a data byte is missing from one of the data eMMCs, the RAID <NUM> circuit would reconstruct the missing data byte from the set of data bytes and the parity byte received in parallel with the set of data bytes and generate a reconstructed set of data bytes, which are placed in the TBM for forwarding to the main memory or the system, or stored in a buffer memory in the HV Controller <NUM> for further processing by the on-DIMM processor.

The HV-DB <NUM> is used to temporarily store data so as to make data transfers in the HV-DB <NUM> faster and more efficient. Since normally data may be transferred in and out of Flash memory at a slower speed than data is transferred to and from the system, the HV-DB <NUM> is used to buffer data to/from the Flash memory so the system does not have to slow down and wait for data to be written to or read from the storage subsystem. When the system writes data to the storage subsystem, the data is buffered into the HV-DB <NUM> at DRAM data I/O speed, which is much faster than Flash data I/O speed. The buffered data is written into the Flash memory on, for example, first in, first out basis. The same is true for the read direction. Thus, while reading from the storage subsystem, the CPU can engage in other processes with the main memory until the HV-DB <NUM> buffer has buffered a predetermined amount of data for transferring to the main memory or the system at the DRAM speed. On the other hand, when data is transferred from the main memory to the storage, the data is read from the DRAM according to a set of control/address (C/A) signals from the system or the HV Controller <NUM> and written into the HV-DB <NUM> according to another set of C/A signals from the HV Controller <NUM>. While the DRAM can be engaged with the system on other tasks, the HV Controller <NUM> can transfer the data from the HV-DB <NUM> to the storage by reading the data from the HV-DB <NUM> and writing the data to the storage.

The components in the HVDIMM <NUM>, e.g., the HV Controller <NUM>, the main memory subsystem (or volatile memory subsystem), the HV-DB <NUM>, the storage subsystem (or non-volatile memory subsystem), can be mounted on a same printed circuit board or disposed in close proximity to each other to allow fast and smooth data transfer therebetween.

NAND Flash can be prone to low-reliability issues due to random errors generated by physical effects in the geometry of the NAND gates. Thus, in certain embodiments, to improve data integrity, the HV Controller <NUM> is configured to carry out a set of error detection/correction routines to detect and correct errors in the data stored in the HV-Flash <NUM>. For example, every time when data is transferred from the HV-DRAM <NUM> to the HV-Flash <NUM>, the HV Controller <NUM> would perform error correction coding on the data. In certain embodiments, as shown in <FIG>, the data bits are grouped and each group of data bits are arranged in a three-dimensional matrix. Cyclic Redundance Check (CRC) codes can be computed using predetermine algorithm along each of the X, Y and Z axis for the three dimensions of the matrix. The CRC codes are then stored together with the data into the HV-Flash <NUM>. In certain embodiments, the HV Controller <NUM> includes on-chip memory spaces (e.g., <NUM>-<NUM> of SRAM) and/or shift registers to store a copy of the data for the CRC calculation while the data is being transferred from the DRAM to the Flash.

When the data is transferred from HV-Flash <NUM> to HV-DRAM <NUM>, the HV Controller <NUM> would have each group of data arranged again into the same matrix format, and CRC codes are computed again using the same predetermined algorithm along each of the axis as shown in <FIG>. The newly computed CRC codes are compared with the corresponding CRC codes received with the data. If there were no error in the data, the newly computed CRC codes would match the corresponding CRC codes received with the data. If the received data contain one or more errors as shown in <FIG> and <FIG>, there would be mismatch between the newly computed CRC codes and the received CRC codes. With the data arranged in the matrix format, such mismatch in the CRC codes can be used to identify the location of an erroneous data bit, and correction can be made by simply flipping the erroneous data bit at the identified location.

<FIG> illustrates an exemplary situation where one of the bits in a X-Y plane of the data matrix (e.g., bit D210) is erroneous. This can be detected with CRC check in just the X and Y directions to pinpoint the bit location with the error. As shown in <FIG>, bit D210 is part of a row of bits D010, D110, D210,. and D710 along the X direction, and also part of a column of bits D200, D210, D220,. , D290 along the Y direction. So, if both the newly calculated CRC code for the row of bits (CRC810, CRC910, and CRCA10) and the newly calculated CRC code for the column of bits (CRC2A0, CRC2B0, and CRC2C0) do not match the corresponding CRC codes received with the data, while all other newly calculated CRC codes in the same X-Y plane match the corresponding CRC codes received with the data, the location of the error would be at the intersection of the row of bits and the column of bits, i.e., D210, as shown in <FIG>.

<FIG> illustrates an exemplary situation where two of the data bits in a same X-Y plane in the data matrix (e.g., bit D210 and bit D600) are erroneous. To properly pinpoint the locations of the bit errors, CRC check needs to be conducted in X, Y and Z directions. As shown in <FIG>, CRC check conducted in just the X and Y directions in this situation would indicate four possible locations of bit error (e.g., D200, D210, D600 and D610). Additional CRC check in the Z direction is thus use to pinpoint the exact locations of bit error (e.g., D210 and D600).

In certain embodiments, the HV Controller <NUM> performs CRC checks on copies of the data which are being transferred from the HV-Flash <NUM> to the HV-DRAM <NUM>. Therefore, by the time HV Controller <NUM> detects a bit error, the erroneous data bit may have already been written into the DRAM. To correct the bit error, the HV Controller <NUM> can perform a read-modify-write operation to read a segment of data containing the erroneous bit from the DRAM, modify the erroneous bit, and then write the data segment back into the DRAM.

As stated above, Flash memory has a finite number of program-erase cycles, and frequent erase/write operations can cause Flash memory cells to wear out, causing reliability issues. To prolong the life of HV-Flash <NUM>, the HVDIMM <NUM> includes a scratch Flash in additional to the main Flash (as shown in <FIG>) and the HV Controller <NUM> is configured to execute an error detection/correction process when data is moved back and forth between the HV-DRAM <NUM> and HV-Flash <NUM>. By using the error detection/correction process, which is discussed below, the HVDIMM <NUM> can support random updates to Flash without reducing the Flash life by avoiding program/erase (P/E) operations to the main Flash as much as possible.

In certain embodiments, the scratch Flash is employed to hold updated (modified) data when a Flash block is filled. The modified data in the scratch Flash can be incorporated into the Flash block when the system is ready to close the block. For example, if a particular task/application requires <NUM> updates to each of <NUM> Flash pages in Flash block, <NUM>,<NUM> updates would be needed. If there is <NUM> initially unfilled page areas in a block, this task/application requires <NUM> P/E operations, which amounts to <NUM>. 2GB of data being rewritten. However, if the modified data is stored in the scratch Flash, then the block only needs <NUM> P/E operation. As for the scratch Flash area, only <NUM>,<NUM> pages will be written, which amounts to only 80MB of data being rewritten.

In certain embodiments, the scratch Flash, unlike storage or main Flash, does not have to follow the block concept, although it follows standard Flash operations. Thus, each page update is written into the next open (unwritten) page space. A written page is marked as 'stale' when either the page is updated again and the updated page is written into a new location, or the page is copied into the storage Flash by the system closing a block in the storage Flash. When all pages in a physical block in the scratch Flash are marked as 'stale', the block is erased and then marked as open (or available). In certain embodiments, for a block that contains mostly 'stale' pages, the pages that are not marked 'stale' are copied into a new location so that the block can be erased.

<FIG> illustrate a conventional SSD or Flash operation where, according to industry standard replacement algorithm, when a system requests to open a segment in a Flash block (Block K), the system must select a block (Block N) in the main memory (DRAM) to be replaced by the Flash block. A new block in the Flash is opened to accommodate the data in Block N before Block N is replaced by the data in Block K. Another block in the Flash where block N was originally taken from is thereafter erased.

<FIG> illustrate a Memory Window operation in the HVDIMM <NUM> according to certain embodiments. Here, in response to a system request to open a new segment in the main Flash, and a block is selected in the HV-DRAM <NUM>, but only the pages in the DRAM block that are required for the selected segment (e.g., X pages for segment Y) are actually subject to replacement. The X pages are written to the scratch Flash before data from segment Y is written into the space held by the X pages in the DRAM. Thus, data is moved between HV-DRAM <NUM> and HV-Flash <NUM> page by page and no new blocks in the Non-volatile memory are required to be written or erased when the new segment is opened for dynamic random access.

At some point, a current scratch Flash block may be filled up as data is moved from the Flash to the DRAM page by page. For example, as shown in <FIG>, if a memory window operation requires M+<NUM> pages to be replaced in DRAM and the current scratch block j only has M pages left unfilled, the first M pages of the M+<NUM> pages can be written into Block j, while the M+<NUM>st page is written into Block j+<NUM> in the scratch Flash.

The HV Controller <NUM> is configured to keep track of data movements among the main Flash, the DRAM and the scratch Flash. When the system requests to close a block in the DRAM, a new block in the Flash is opened to accommodate the data in the to-be-closed block in DRAM. Since some of the data in the to-be-close block may have been put in the scratch Flash, the HV Controller <NUM> is further configured to merge data in the to-be-closed block from the DRAM with the data taken from the to-be-closed block and stored in the scratch Flash, as illustrated in <FIG>. The merged data is stored in the new block in the Flash and an old block in the Flash where the to-be-erased block in the DRAM was taken from is thus erased.

In addition to using the scratch Flash to prolong the life of the main Flash, the HV Controller <NUM> is also configured to perform wear leveling by equalizing average usage time of each block in the HV-Flash <NUM>. In certain embodiments, such wear leveling can be done by a round robin method. The HV Controller <NUM> uses its address mapping management capabilities and relatively large memory space to keep track of the associated address changes.

<FIG> illustrates a simplified example of a HVDIMM <NUM> round-robin wear leveling technique, according to certain embodiments. As shown in <FIG>, both Event Progress Cases <NUM> and <NUM> have Blocks <NUM> and <NUM> opened for Memory Window operations. In Event Progress Case <NUM>, File A is closed first while in Event Progress Case <NUM>, File C is closed first. Thus, for Case <NUM>, File A is written into the first empty (erased) block (B#<NUM>). For Case <NUM>, File C is written into block B#<NUM>.

Instead of or in addition to being used as main memory, the HVDIMM <NUM> can also be used as a storage device to facilitate direct data transfers within an intranet network.

Recent developments in cloud networking and computing require efficient ways to transfer and store data. Since the cloud environment supports many different types of applications that share computational power as well as database, any particular server or storage bottleneck can impact the overall performance of the cloud network.

There are two types of data transfers, intranet and internet. An intranet provides a closed network within an organization, which shares computing resources and information, while internet networks are between intranets or between organizations.

Internet (between organizations) data transfers are generally less concerned about data transfer latency. On the other hand, intranet data transfers require prompt responses and is less tolerant of data transfer latency. This is especially true when a task is farmed out to multiple servers for parallel computation using shared operating system, program, and database. In such cases, data coherency is required among these servers for correctly executing the task. Therefore, any data transfer latency in one server can slow down the task execution.

There are also two types of data transfers in an intranet: data transfers within each individual server and data transfers amongst various servers. The data transfers amongst various servers use internet protocol technology to share information, operational systems, or computing resources. The data transfers within a server is generally handled by the CPU, and occur amongst memory devices and network connections via the CPU.

Currently, transferring data between intranet servers requires a transmitting server CPU to gather the data from either a storage or from the main memory, packetize the data, and put it onto the network. The receiving server CPU needs to extract the data and to store it in a storage or the main memory. Transferring data within a server requires the CPU to read data from one memory coupled to the CPU and write the data into another memory device also coupled to the CPU.

For example, when a process running on the CPU attempts to allocate more memory than the system has available, the OS would swap memory pages to and from the storage. If a page is selected for replacement and "Page Out" is referenced again, it has to be paged in (read in from storage). This would involve waiting for I/O completion and the total responding time is the sum of: <MAT> where <MAT> and <MAT>.

In both cases, data transfer latency can be reduced if the data does not need to go through the CPU. In other words, if direct data transfer occurs from a memory device to the network, then the data transfer latency amongst servers will be minimized. Data transfer latency within a server can also be reduced if the data is transfered directly between memory devices without going through the CPU.

In the intranet network, if a process running on the CPU attempts to allocate more memory than the system has available, the OS would swap memory pages to and from the IO Storage. If the page is selected for replacement and "Page Out" is referenced again, it has to be paged in. This would involve waiting for I/O completion, but the total responding time is now the sum of: <MAT> where <MAT> and <MAT> Thus, the total responding time is significantly shortened.

In certain embodiments, the HV Controller <NUM> is configured to facilitate data transfers between different HVDIMMs <NUM> by providing a network interface ("Share™ Direct DIMM Interface") via the HV-NIC. For example, as shown in <FIG>, an HVDIMM <NUM> (on the left) can be coupled directly to another HVDIMM <NUM> (on the right) or any other storage devices via their respective NIC devices, and the HV Controller <NUM> in either HVDIMM <NUM> is configured to transfer data between the DRAM on one HVDIMM <NUM> and the Flash in the other HVDIMM <NUM>, between the DRAM on one HVDIMM <NUM> and the DRAM on the other HVDIMM <NUM>, between the Flash on one DIMM and the Flash on the other DIMM, and also between the HVDIMM <NUM> and any other storage devices, using similar techniques as discussed above.

Claim 1:
A memory module (<NUM>) coupled to a memory controller via a memory channel of a computer system (<NUM>), the memory channel including a data bus and a control/address, C/A, bus, the memory module (<NUM>) including a volatile memory subsystem (<NUM>), and a non-volatile memory (<NUM>) subsystem, the memory module (<NUM>) configured to:
receive a first command from the memory controller via the C/A bus;
in response to the first command being a command to transfer first data from the non-volatile memory subsystem (<NUM>) to the volatile memory subsystem (<NUM>), receive via the data bus first information associated with the first command, the first information including at least one of a source address in the non-volatile memory subsystem (<NUM>) and a destination address in the volatile memory subsystem (<NUM>), and read the first data from the non-volatile memory subsystem (<NUM>);
characterised in that the memory module (<NUM>) is further configured to:
receive a first dummy write memory command from the memory controller via the C/A bus, wherein a dummy write memory command is like a normal write command except that it is followed with dummy data or no data from the memory controller;
in response to the first dummy write memory command being associated with the first command, provide a first portion of the first data to the volatile memory subsystem (<NUM>);
receive the first portion of the first data at the volatile memory subsystem (<NUM>) in response to the first dummy write memory command;
receive a normal write memory command on the C/A bus for writing third data from the memory controller into the volatile memory subsystem (<NUM>);
receive a second dummy write memory command after receiving the normal write memory command;
in response to the second dummy write memory command being associated with the first command, provide a second portion of the first data to the volatile memory subsystem (<NUM>) after the volatile memory subsystem (<NUM>) receives the third data in response to the normal write memory command; and
receive the second portion of the first data at the volatile memory subsystem (<NUM>) in response to the second dummy write memory command.