Patent Abstract:
A computer-implemented method for transferring data from a computer system programmed to perform the method includes receiving in a memory buffer in a first memory module hosted by the computer system, a request for data stored in RAM of the first memory module from a host controller of the computer system, retrieving with the memory buffer, the data from the RAM, in response to the request, formatting with the memory buffer, the data from the RAM into formatted data in response to a defined software transport protocol, and initiating with the memory buffer, transfer of the formatted data to a storage destination external to the first memory module via an auxiliary interface of the memory buffer, bypassing the host controller of the computer system.

Full Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    The present invention is a non-provisional of Application No. 61/619,736 filed Aug. 21, 2012 (Attorney Docket No. 929RO-000710US) and claims priority to and is a continuation-in-part of application Ser. No. 13/359,877 filed Jan. 27, 2012 (Attorney Docket No. 408-28-050) that claims priority to Application No. 61/473,889 filed Apr. 11, 2011. These applications are hereinby incorporated by reference, for all purposes. 
     
    
     BACKGROUND 
       [0002]    The present invention relates generally to memory systems, and more particularly to computer systems which include memories systems with auxiliary interfaces. 
         [0003]    Traditional computer systems, such as servers, workstations, desktops and laptops, all use pluggable memory which can be inserted into slots on the computer&#39;s motherboard as needed. The most common form of pluggable memory is the Dual In-line Memory Module (DIMM). Historically, DIMMs contain multiple RAM chips—typically DRAM—each of which has a data bus width of 4 or 8 bits. Typically, eight or nine 8-bit DRAM chips (or twice as many 4-bit DRAM chips) are arranged in parallel to provide each DIMM with a total data bus width of 64 or 72 bits; the data bus, typically referred to as the ‘DQ’ bus, is connected to a host controller. Each arrangement of 64 or 72 data bits using DRAM chips in parallel is termed a ‘rank’. 
         [0004]    A command/address (CA) bus also runs between the host controller and each DIMM; the CA and DQ busses together form a ‘system’ bus. With a basic unbuffered DIMM, the CA bus is connected to every DRAM on the DIMM. As a result, there is a high electrical load on the CA bus, given by the product of the number of DRAMs times the number of ranks. For the DQ bus, the number of electrical loads is equal to the number of ranks. 
         [0005]    A buffering device is employed to reduce loading in a ‘load reduction’ DIMM (LR-DIMM), an example of which is illustrated in  FIG. 1 . An LR-DIMM  10  containing multiple DRAM chips  12  uses a logic device  14  to buffer the DQ and CA signals between the DRAMs and a host controller  16 . Logic device  14  may be, for example, a single device such as the iMB (isolating Memory Buffer) from Inphi Corporation. Memory systems of this sort are described, for example, in co-pending U.S. patent application Ser. Nos. 12/267,355 and 12/563,308, which are incorporated herein by reference, for all purposes. 
       SUMMARY 
       [0006]    The present invention is directed to memory systems which include a memory buffer having one or more auxiliary interfaces. In various embodiments, the auxiliary interfaces enable memory systems to send and receive data to and from other memory systems having such auxiliary interfaces and using a defined data protocol. Additionally, various embodiments enable separate computer systems having such memory systems to send and receive data to and from other memory systems having such auxiliary interfaces. 
         [0007]    Embodiments of the present invention include a memory buffer having an interface arranged to buffer data and/or command bytes being written to or read from a plurality of RAM chips (typically DRAM) residing on a DIMM by a host controller (of a computer system). The memory buffer includes at least one additional interface arranged to buffer data and/or command bytes between the host controller (of the computer system) or RAM chips and one or more external devices (e.g. DIMM, remote computer system) coupled to the at least one additional (e.g. auxiliary) interface. 
         [0008]    In one embodiment, the memory buffer may include a SATA interface and be arranged to convey data between the host controller or RAM chips and FLASH memory devices coupled to the SATA interface. Other types of interfaces may include, for example, an Ethernet interface, an optical interface, a radio interface, or the like. In various embodiments, the memory buffer may reside on the DIMM along with the RAM chips with which it interfaces, or be separate from the DIMM (e.g. on a motherboard, a daughter card, or the like). 
         [0009]    In various embodiments, the memory buffer is provided with a predefined protocol to which data is transported out on the auxiliary interface and/or received into the auxiliary interface. In some embodiments, the protocol includes framing protocols, interframe signals, start of frame delimiters, error detecting protocols, CRC protocols, and the like. In other embodiments, many other types of protocols including protocol framing, error correction, or the like may also be used to format data outbound from an auxiliary interface or deformat data inbound from the auxiliary interface. 
         [0010]    According to one aspect of the invention, a computer-implemented method for transferring data from a computer system programmed to perform the method is described. One method includes receiving in a memory buffer in a first memory module hosted by the computer system, a request for data stored in RAM of the first memory module from a host controller of the computer system, and retrieving with the memory buffer, the data from the RAM, in response to the request. A process includes formatting with the memory buffer, the data from the RAM into formatted data in response to a defined software transport protocol, and initiating with the memory buffer, transfer of the formatted data to a storage destination external to the first memory module via an auxiliary interface of the memory buffer, bypassing the host controller of the computer system. 
         [0011]    According to another aspect of the invention, a computer system configured to transferring data to a remote computer system is described. One apparatus includes a host controller configured to output instructions, and a processor coupled to the host controller, wherein the processor is configured to provide the instructions to the host controller. A device includes a memory module coupled to the host controller, wherein the memory module including RAM configured to store and provide data, a memory buffer coupled to the RAM and to the host controller, wherein the memory buffer is configured to receive the instructions from the host controller, wherein the memory buffer is configured to retrieve the data from in the RAM in response to the instructions, and wherein the memory buffer is configured to format the data from the RAM into formatted data in response to a defined software transport protocol and to the instructions, and an auxiliary interface coupled to the memory buffer and to the remote computer system, wherein the auxiliary interface is configured to output the formatted data to the remote computer system, bypassing the host controller of the computer system. 
         [0012]    These and other features, aspects, and advantages of the present invention will become better understood with reference to the following description and claims. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    In order to more fully understand the present invention, reference is made to the accompanying drawings. Understanding that these drawings are not to be considered limitations in the scope of the invention, the presently described embodiments and the presently understood best mode of the invention are described with additional detail through use of the accompanying drawings in which: 
           [0014]      FIG. 1  is a block diagram of a known memory system which includes a DIMM containing a memory buffer; 
           [0015]      FIG. 2   a  is a block diagram of a memory system which includes a memory buffer in accordance with the present invention; 
           [0016]      FIG. 2   b  is a block diagram of a memory system which includes a memory buffer in accordance with the present invention, in which the memory buffer resides on a DIMM. 
           [0017]      FIG. 3  is another possible embodiment of a memory system containing a memory buffer in accordance with the present invention; 
           [0018]      FIG. 4  is an embodiment of a memory system which includes two memory buffers in accordance with the present invention; 
           [0019]      FIG. 5  is another possible embodiment of a memory system which includes memory buffers in accordance with the present invention; and 
           [0020]      FIG. 6  is an embodiment of a memory system which includes a plurality of memory buffers in accordance with the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0021]    The present memory system includes one or more memory buffers, each of which is arranged to buffer data and/or command instructions (e.g. bytes) being written to or read from random access memory (RAM chips) residing on a Dual In-line Memory Module (DIMM) by a host controller of a host computer system. In various embodiments, the RAM chips are referred to hereinafter as DRAM, though the invention is not limited to any specific memory chip type or technology. 
         [0022]    In various embodiments, the memory buffer includes one or more additional hardware interfaces which are arranged to provide access to data stored in the DRAM chips, bypassing the host controller. In one example, the memory buffer might include a serial ATA (SATA) interface, or the like such that in addition to serving as a buffer between a host controller and the DRAM, the memory buffer also serves as a buffer between the host controller and/or DRAM and external devices capable of interfacing with a SATA interface. 
         [0023]    One possible embodiment of this concept is shown in  FIG. 2A . In this embodiment, a memory buffer  20  interfaces with a host controller  22  (on a host computer system) via a host interface  23 . In various embodiments, memory buffer  20  is coupled to DRAM residing on one or more DIMMs  24  via a DIMM interface  25  and operates as a buffer for command and data lines. Additionally, as illustrated, memory buffer  20  also includes an auxiliary interface  26  (e.g. SATA interface  26 ), In operation, memory buffer  20  also operates as a buffer between host controller  22  and/or any devices coupled to the auxiliary interface  26  (e.g. the SATA interface). For example, FLASH memory devices  28  might be connected to SATA interface  26  via a controller  30 , such that data from host controller  22  or DIMM  24  can be written to the FLASH memory via SATA interface  26 . 
         [0024]    In other embodiments, the external devices may be connected directly to a memory buffer  20  via an auxiliary interface  26 , or be coupled to them via intervening controller devices such as controller  30 , as needed. 
         [0025]    Various embodiments include memory systems having memory buffers  20  that control individual memory chips  24  and that interface between the individual memory chips  24  and a host controller  22 . Various embodiments of the present invention may be adapted to standard memory interface specifications, such as DDR3, load-reduced DIMM (LRDIMM), registered or unregistered DIMM (RDIMM, UDIMM), non-volatile DIMM (NV-DIMM), or any DDR interface standard specifications. 
         [0026]    In one specific example, a non volatile DIMM includes a memory buffer for reading and writing write RAM data to a FLASH memory. The FLASH memory maintains the data in the event of a power loss. In various embodiments, the NV-DIMM includes: a SATA interface (internal or external to the memory buffer); one or more a NV-DIMMs (per the arrangement shown in  FIG. 2A ); then, data stored in the DRAMs can be efficiently maintained in the FLASH memory if power is lost. 
         [0027]    A SATA interface is merely one example of an interface that could be added to a memory buffer as described above. Other possible interfaces include an Ethernet interface, optical interface, and/or a radio interface, as well as interfaces capable of interfacing with additional known or future memory element types including, but not limited to, FLASH and DRAM. 
         [0028]    In various embodiments, memory buffer  20  may be an integrated circuit that is separate and distinct from the DIMM ICs  24 , as shown in  FIG. 2   a . In such embodiments, the DIMMs may be standard DIMMs—i.e., with no modifications. 
         [0029]    In other embodiments, the memory buffer  33  may reside on the DIMM  32  containing the DRAM  34 . This is illustrated in  FIG. 2B  where DIMM  32  includes a plurality of DRAM chips  34 , and a memory buffer  33 . In various embodiments, memory buffer  33  may include a DRAM interface  35  arranged to buffer data and/or command data (e.g. bytes) being written to or read from the DRAM  34  by a host controller  36  via a host interface  37 . In some embodiments, memory buffer  33  includes at least one additional interface  38  arranged to buffer data and/or command bytes between host controller  36  and/or DRAM  34 . In some embodiments, one or more external devices are coupled to DIMM  32  via interface  38 , bypassing the data channel of the host controller  36 . These external devices may include other DIMMS, network storage, FLASH memory, or the like. 
         [0030]    In some embodiments, the form factor of DIMM  32  deviates from a standard form to accommodate the memory buffer, and/or additional interfaces  38 . Because of the above additional functionality, DIMM  32  may require additional I/O connectivity between host interface  37  and host controller  36 . In various embodiments, additional I/O pins are added to DIMM&#39;s  32  main connector, and in other embodiments, DIMMs  32  include one or more additional connectors in addition to DIMM&#39;s  32  main connector. 
         [0031]    In some embodiments, the approach described herein is applicable to virtually any memory system that employs a memory buffer which serves as an interface between the individual memory chips on a DIMM and a host controller (as long as the system is capable of providing full control of the memory chips to the memory buffer). 
         [0032]    In contrast to the above, previous memory systems that employed a memory buffer merely used the memory buffer as a ‘slave’ device. As a slave device, the memory buffer simply received commands from the host controller and then passed them to the DRAM interface. They did not control the DIMM interface of its own accord. In contrast, with embodiments of the present invention, the memory buffer typically has full control of the DRAM chips so that it can independently send and receive data between the DRAM and the additional interfaces (as well as performing general housekeeping and power management). 
         [0033]    In various embodiments, read and write control can be given to memory buffer  33  in several ways or under several conditions. For example, the change of control can occur when power to the memory system  32  fails, is indeterminate, and/or is restored. As another example, full control of the DRAM  34  might also be given to the memory buffer  33  via direct or indirect commands from the host controller  36 . In one example, a direct command may take the form of one or more special commands or signals sent by the host computer via host controller  36  to the memory buffer  33 . In another example, an indirect command could may occur within a window of time when, when the host controller  36  is not going to access the DRAM  34  for a predetermined amount of time. Within this window, memory buffer  32  is thereby provided with control. 
         [0034]    Multiple interfaces  38  of the same type, or of different types, can be incorporated into a memory buffer  33  as needed. An exemplary embodiment illustrating the addition of two such interfaces is shown in  FIG. 3 . In this example, a memory buffer  40  provides a buffer for the command and data lines between a host controller  42  (via a host interface  43 ) and one or more DIMMs  46  (via a DIMM interface  47 ). As illustrated in  FIG. 3 , memory buffer  40  also includes two external interfaces  48 ,  50  (e.g. SATA) and operates as a buffer between host controller  42  and/or DIMMs  46  and external devices  52 ,  54 . In the embodiment shown, the external devices  52  and  54  are FLASH memory devices and interface with SATA interfaces  48 ,  50  via one or more controllers  56 ,  58 . 
         [0035]    In various embodiments, the configuration illustrated in  FIG. 3  may be implemented with a memory buffer  40  that is separate from DIMMs  46 . In other embodiments, memory buffer  40  may be implemented on the DIMM containing DRAM. 
         [0036]    In some embodiments of the present invention, a memory system may include a memory buffer having an interface which enables two or more host controllers to be ‘cross-connected. Such embodiments enable memory and/or data sharing between hosts, bypassing the host controller bus for data transfer. An exemplary embodiment is shown in  FIG. 4 . In the present example, memory buffers  60  and  62  interface with host controllers  64  and  66  via respective host interfaces  68 ,  70  (of respective host computer systems). As shown, each memory buffer  60 ,  62  also includes a DIMM interface  72 ,  74  that operates as a buffer for the command and data lines connected to DIMMs  76 ,  78 . Additionally, memory buffer  60  includes a cross-connect interface  80 , which is arranged to buffer data and/or command bytes between host controller  64  or DIMMs  76  and external devices. Similarly, memory buffer  62  includes a cross-connect interface  82 , which is arranged to buffer data and/or command bytes between host controller  66  or DIMMs  78  and external devices. When cross-connect interfaces  80  and  82  are coupled together as shown, memory buffers  60  and  62  can operate as buffers between each other, as well as between host controllers  64  and  66 , enabling data and/or command bytes to be exchanged between the DIMMs and/or host controllers via the cross-connect interfaces. 
         [0037]    In various embodiments of the present invention, memory buffer  60  may format data from DIMMS  76  or from host  64  into a transport protocol, prior to output on cross-connect interface  80 ; and memory buffer  62  may unformat data received from cross-connect interface  82  from the transport protocol, prior to controller  66 . The formatting may also be performed by memory buffer  62  and unformatting by memory buffer  60 . 
         [0038]    In various embodiments, the transport protocol may include the Ethernet protocol based upon frames of data. For example, in one embodiment, memory from DIMMs  76  or host controller  64  may be separated into a number of Ethernet frames. In one example, each frame may include, one or more of the following types of data, a preamble, start of frame (SOF) delimiters, a source and/or destination address, a frame length, the data, padding, a checksum, and the like. Other examples may also include other types of error correction mechanisms, such as Cyclic redundancy check (CRC); end of field indicators; framing protocols; interframe signals; and the like. In light of the present patent disclosure, one of ordinary skill in the art will recognize that many other types of data transfer mechanisms and formats may be used with embodiments of the present invention. 
         [0039]    In various embodiments, a typical memory buffer will be able to both format and unformat data packets sent and received from another typical memory buffer. In other embodiments, one type of memory buffer may only format and transmit formatted data packets, and another type of memory buffer may only receive and unformat formatted data packets. 
         [0040]    As with the other embodiments described herein, this sort of configuration can be implemented with memory buffers  60  and  62  being separate from DIMMs  76  and  78  as illustrated in  FIG. 4 . Alternatively, memory buffers  60  and  62  could be implemented onboard the DIMMs containing DRAM (e.g. DIMMs  76  and  78 , respectively). 
         [0041]    Another possible embodiment is shown in  FIG. 5 , in which two or more memory buffers  90  and  92  enable two or more DIMMs  106  and  108  and/or two or more host controllers  94  and  96  to be ‘cross-connected’ via a switch. As illustrated in this example, memory buffers  90  and  92  interface with host computers (host controllers  94 ,  96 ) via respective host interfaces  98 ,  100 . 
         [0042]    In various embodiments, each memory buffer  90  and  92  also includes a DIMM interface  102 ,  104  which operates as a buffer for the command and data lines connected to one or more DIMMs  106 ,  108 . Each memory buffer  90  and  92  also includes an ingress and an egress cross-connect interface ( 110  and  112  for memory buffer  90 ,  114  and  116  for memory buffer  92 ), which are connected to a switch  118 . In various embodiments, the switch  118  can be arranged to enable unidirectional communication (in either direction), or bidirectional communication, between the respective cross-connect interfaces. Switch  118  thereby enables data and/or command bytes to be exchanged between the DIMMs  106 ,  108  and/or host controllers  94 ,  96  via the cross-connect interfaces  110 ,  112  and  114 ,  116 . 
         [0043]    In some embodiments of the present invention, switch  118  may include a series of data packet switches and/or multiplexers. In such examples, as described above, memory buffer  90  and/or  92  may format the data to be transferred as a series of formatted data packets, and unformat the series of data packets to recover the data. In one embodiment, switch  118  may be embodied as a packet switcher that receives each packet of formatted data, determines the destination address, and then routes each packet to the respective destination address. 
         [0044]    In additional embodiments, switch  118  may include additional capability. For example, in some embodiments, switch  118  may include a memory that buffers or stores formatted packets. Such configurations enable switch  118  to provide flow control and quality of service functionality. For example, switch  118  may buffer formatted packets that are destined for memory buffer  92 , if memory buffer  92  is busy storing data from host controller  96  to DIMMs  108 . In operation, when memory buffer  92  is available, memory buffer  92  indicates to switch  118  that it can receive data, and in response, switch  118  can output formatted packets that have been buffered within switch  118 . 
         [0045]    In another example, switch  118  may buffer formatted packets that are destined for a third memory buffer while delivering formatted packets that are destined for a fourth memory buffer. In such cases, switch  118  may have quality of service (QoS) requirements that prioritize delivery of formatted data to the third memory buffer, accordingly delivery of formatted to the second memory buffer may be delayed, or the like. In light of the present patent disclosure, one of ordinary skill in the art will now recognize that many other types of QoS and flow control functionality may be implemented by embodiments of switch  118 . 
         [0046]    In another example, switch  118  may include the capability to analyze and/or reformat data packets. In some embodiments, switch  118  may analyze one or more portions of the packetized data, such as framing protocols, interframe signals, start of frame (SoF) delimiters, end of frame (EoF) delimiters, data length, or the like. In some embodiments, if such portions are incorrect, switch  118  may attempt to correct the packet and outputs a reformatted data packet. In other embodiments, switch  118  may also analyze one or more error detecting protocols, CRC protocols, or the like, to determine the validity of the data packet. In light of the present patent disclosure, one of ordinary skill in the art will now recognize that many other types of packet reformatting functionality may be implemented by embodiments of switch  118 . 
         [0047]    Another possible embodiment is shown in  FIG. 6 . In various embodiments, a common host (CPU  120 ) is connected to multiple memory buffer chips  122  via memory channels  124  (e.g. DDR3, DRAM memory interface); and each memory buffer chip  112  is in turn coupled to one or more DIMMs (2 DIMMs in the example shown in  FIG. 6 ). In this example, each memory buffer  112  may be configured as described herein to include one or more additional or auxiliary interfaces. For example, each memory buffer  122  includes a separate ingress interface and an egress interface (illustrated in  FIG. 5 ) or a cross connection interface (illustrated in  FIG. 4 ). In some embodiments, these interfaces may be directly physically cross-connected to other interfaces, as desired or required, and in other embodiments, these interfaces may be cross-connected to other interfaces via one or more switches (as illustrated in  FIG. 5 ). 
         [0048]    In some embodiments, as described above, memory buffers  122  may include a direct memory access (DMA) controller capable of writing data to or reading data from the DIMMs. Additionally, memory buffers  112  may control and convey data between DIMMs, CPU  10 , and one or more auxiliary interfaces, as described above. In one example, if a memory buffer  122  includes a SATA interface, the DMA controller may be arranged to exchange data between the DIMMs and the SATA interface. 
         [0049]    In other embodiments of the present invention, memory buffers  122  may include a self-test engine capable of reading data from the DIMM and comparing it with one or more data patterns. One such memory buffer  122  may include, for example, a MemBIST (Memory Built-In Self Test) engine incorporated into an iMB memory buffer available from the present assignee of the present application, Inphi Corp. One function of this self-test engine may be to test the integrity of a DIMM&#39;s DRAM by reading data from it and comparing it against an expected pattern stored in the memory buffer. 
         [0050]    In such an engine, an ‘error’ occurs if the data read does not match the expected pattern, and as a result, the offending DRAM address is logged and the test continues. In various embodiments, by changing a polarity of the error signal generated by the self-test engine, it is possible to change the operation performed by the self-test engine from ‘test’ to ‘search’. Using this ‘search’ capability, when a pattern read from DRAM matches a pattern stored or inputted into the memory buffer  122 , the DRAM address of the “found” data is logged and the search continues. In other embodiments, this search capability could be extended so that there are multiple possible match patterns, as well as “don&#39;t care” bits, which would enable searches to be executed for multiple patterns in parallel with wild cards. 
         [0051]    In various embodiments, a memory buffer as described herein could be implemented as a single integrated circuit (IC), or with a multiple chip chipset with various functions spread among several ICs. For example, a memory system based on the DDR4 standard employs DIMMs which include nine separate data buffer chips arranged close to the connector contacts and provides an interface between the connector and the DRAMs. The standard also provides for a central control element which functions as the register section of the DIMM and includes an extra interface to control the data buffers. For this type of chipset implementation, implementing an auxiliary port as described herein requires a new path from the data buffers to the central controller. 
         [0052]    The embodiments shown in the figures and described above are merely exemplary. The present system encompasses any memory system which employs a memory buffer that serves as an interface between the individual memory chips on a DIMM and a host, and which includes at least one additional, auxiliary interface which enables the memory buffer to serve as an interface between the host and/or memory chips and additional external devices. 
         [0053]    In other embodiments, a system may include more than one host computer (each with host controller) wherein each host computer includes a memory buffer having a RAM interface and an auxiliary interface, as described herein. The auxiliary interfaces of the memory buffer of one host computer may be directly coupled to an auxiliary interface of the memory buffer of another host computer, or may be coupled via one or more switches. As described herein, such configurations enable the transfer of data from one RAM to another RAM bypassing data paths of the host controllers. 
         [0054]    The embodiments of the invention described herein are exemplary and numerous modifications, variations and rearrangements can be readily envisioned to achieve substantially equivalent results, all of which are intended to be embraced within the spirit and scope of the invention as defined in the appended claims.

Technology Classification (CPC): 6