Patent Publication Number: US-9411519-B2

Title: Implementing enhanced performance flash memory devices

Description:
This application is a continuation application of Ser. No. 14/571,861 filed Dec. 16, 2014. 
    
    
     FIELD OF THE INVENTION 
     The present invention relates generally to the data processing field, and more particularly, relates to a method and apparatus for implementing enhanced performance in a flash memory system. 
     DESCRIPTION OF THE RELATED ART 
     In flash memory devices, data are written in pages. Memory can only be erased in larger units called blocks typically made up of multiple pages. If the data in some of the pages of the block are no longer needed, such as stale pages, only the pages with good data in that block are read and re-written into another empty block, such as a previously erased empty block. Then the free pages left by not moving the stale data are available for new data. This process is called garbage collection (GC). 
     All solid state drives (SSDs) include some level of garbage collection, while the SSDs typically differ in when and how fast the garbage collection (GC) process is performed. Current methods for GC use the mainline bus to perform erase operations that are essential to free up blocks to accommodate subsequent writes from host. This GC process however locks up the mainline bus bandwidth for erase operations and therefore overall performance reduces significantly. 
     There are numerous arrangements and industry practices to balance between erase and write operations to optimize the performance; however, in such known methods, erase operations are still essential operations of the GC process that requires I/O bus bandwidth. 
     Scrub operations are vital to ensure data integrity, to catch and correct errors preemptively, before mainline reads hit the errors. However, the scrub operation also locks up the read bandwidth by the flash controller. 
     A need exists for an effective method and a flash memory system having the ability for implementing enhanced performance including garbage collection (GC) and scrub on flash memory chips in the flash memory system. 
     SUMMARY OF THE INVENTION 
     Principal aspects of the present invention are to provide a method and apparatus for implementing enhanced performance in a flash memory system in a computer system. Other important aspects of the present invention are to provide such method, and apparatus substantially without negative effects and that overcome many of the disadvantages of prior art arrangements. 
     In brief, a method, and apparatus for implementing enhanced performance in a flash memory system in a computer system. A flash memory chip includes a function engine performing garbage collection and scrub operations using an internal bus for data movement, preserving I/O bandwidth. The system includes an on-flash chip memory buffer buffering garbage collection and scrub requests. Garbage collection and scrub operations are interleaved with mainline reads and writes. 
     In accordance with features of the invention, a switching infrastructure multiplexer in the flash memory chip selects between the internal bus and the mainline bus. 
     In accordance with features of the invention, the flash controller communicates a list of blocks to be scrubbed and the function engine performs the scrub taking control of the internal bus, resulting in a significant decrease in I/O bandwidth consumption. 
     In accordance with features of the invention, mainline write operations from controller are performed normally when subsequent write operation have enough free blocks to accommodate those write operations. All blocks are connected to the mainline bus by default. 
     In accordance with features of the invention, host writes having dependency on the creation of free blocks before the write operations can happen require a garbage collection process. Controller firmware can exploit a queue portion or a full write queue to identify a set of blocks to be garbage collected. Controller firmware can communicate the list of such blocks to the flash memory device and initiate a background GC process at the flash memory device level. Controller can drive mainline writes with an order of write operations and erase operations for garbage collection in the queue being rearranged or reordered to ensure enough blocks are freed up in the flash memory device, as a background process in parallel to current writes, to accommodate upcoming writes to ensure maximum performance. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention together with the above and other objects and advantages may best be understood from the following detailed description of the preferred embodiments of the invention illustrated in the drawings, wherein: 
         FIG. 1  is a block diagram of an example computer system embodying the present invention; 
         FIG. 2  is a block diagram illustrating an example flash memory system for implementing enhanced performance in the flash memory system in the computer system of  FIG. 1  in accordance with preferred embodiments; 
         FIGS. 3A, 3B, and 3C  illustrate example erase operations and write operations for garbage collection within a flash memory system of  FIG. 2  for implementing enhanced performance in accordance with the preferred embodiments; 
         FIGS. 4A, 4B, and 4C  illustrate example operations of background erase operations in parallel to write operations within the flash memory system of  FIG. 2  for implementing enhanced performance in accordance with the preferred embodiments; 
         FIGS. 5A, 5B, and 5C  illustrate example operations of background erase operations in parallel to write operations within the flash memory system of  FIG. 2  for implementing enhanced performance in accordance with the preferred embodiments; 
         FIGS. 6A, 6B, and 6C  illustrate example operations of background erase operations in parallel to write operations within the flash memory system of  FIG. 2  for implementing enhanced performance in accordance with the preferred embodiments; 
         FIG. 7A  is a block diagram illustrating example erase operation process and write operation process in the example flash memory system of  FIG. 2  in accordance with preferred embodiments; 
         FIGS. 7B and 7C  are flow charts respectively illustrating example steps of the erase operation process and write operation process of  FIG. 7A  in the example flash memory system of  FIG. 2  in accordance with preferred embodiments; 
         FIG. 8  is a block diagram illustrating a computer program product in accordance with the preferred embodiment. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     In the following detailed description of embodiments of the invention, reference is made to the accompanying drawings, which illustrate example embodiments by which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the invention. 
     The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. 
     In accordance with features of the invention, a method and apparatus are provided for implementing enhanced performance in a flash memory system. Managing garbage collection and scrub operations are provided at flash memory device level. A controller communicates blocks for garbage collection and pages for scrub to a function engine within the flash device. 
     Having reference now to the drawings, in  FIG. 1 , there is shown a computer system embodying the present invention generally designated by the reference character  100  for implementing enhanced performance in a flash memory system in accordance with the preferred embodiment. Computer system  100  includes one or more processors  102  or general-purpose programmable central processing units (CPUs)  102 , #1-N. As shown, computer system  100  includes multiple processors  102  typical of a relatively large system; however, system  100  can include a single CPU  102 . Computer system  100  includes a cache memory  104  connected to each processor  102 . 
     Computer system  100  includes a memory system  106  including a memory controller  108  and a main memory  110  connected by a bus  112 . Bus  112  is one or more busses that send address/command information to main memory  110  and send and receive data from the memory  110 . Main memory  110  is a random-access semiconductor memory for storing data, including programs. Main memory  110  is comprised of, for example, a dynamic random access memory (DRAM), a synchronous direct random access memory (SDRAM), a current double data rate (DDRx) SDRAM, non-volatile memory, optical storage, and other storage devices. 
     I/O bus interface  114 , and buses  116 ,  118  provide communication paths among the various system components. Bus  116  is a processor/memory bus, often referred to as front-side bus, providing a data communication path for transferring data among CPUs  102  and caches  104 , memory controller  108  and I/O bus interface unit  114 . I/O bus interface  114  is further coupled to system I/O bus  118  for transferring data to and from various I/O units. 
     As shown, computer system  100  includes a storage interface  120  coupled to storage devices, such as, a direct access storage device (DASD)  122 , and a CD-ROM  124 . Computer system  100  includes a terminal interface  126  coupled to a plurality of terminals  128 , #1-M, a network interface  130  coupled to a network  132 , such as the Internet, local area or other networks, and a I/O device interface  134  coupled to I/O devices, such as a first printer/fax  136 A, and a second printer  136 B. 
     I/O bus interface  114  communicates with multiple I/O interface units  120 ,  126 ,  130 ,  134 , which are also known as I/O processors (IOPs) or I/O adapters (IOAs), through system I/O bus  116 . System I/O bus  116  is, for example, an industry standard PCI bus, or other appropriate bus technology. 
     Computer system  100  is shown in simplified form sufficient for understanding the present invention. The illustrated computer system  100  is not intended to imply architectural or functional limitations. Although main memory  110  of main memory system  106  is represented conceptually in  FIG. 1  as a single entity, it will be understood that in fact the main memory is more complex. In particular, main memory system  106  comprises multiple modules and components. The present invention can be used with various hardware implementations and systems and various other internal hardware devices. 
     An example flash memory system is illustrated and described with respect to  FIG. 2  for implementing enhanced performance in accordance with embodiments of the invention. 
     Referring now to  FIG. 2  there is shown an example flash memory system generally designated by the reference character  200  for implementing enhanced performance in the computer system  100  in accordance with the preferred embodiment. Flash memory system  200  includes a flash controller  202  and a flash memory device  204  connected by a mainline bus  206 . Flash memory device  204  includes an internal bus (I-bus)  208  for implementing enhanced performance in accordance with embodiments of the invention. 
     In accordance with features of the invention, a host controller or the flash controller  202  tracks blocks for garbage collection and communicates the list of blocks to be garbage collected to the flash memory device  204 . The flash controller  202  communicates a list of pages/blocks to be scrubbed the flash memory device  204 . 
     In accordance with features of the invention, flash memory device  204  includes a function engine  210  for performing device level garbage collection and scrub processes. The garbage collection and scrub function engine  210  is connected by the internal bus (I-bus)  208  to a switching infrastructure multiplexer  212  within the flash memory chip  204 . The switching infrastructure multiplexer  212  is coupled to a plurality of memory blocks #1-N,  214  selecting between the internal bus  208  and the mainline bus  206 . The garbage collection and scrub function engine  210  includes a multiplexer control  216  and a command decoder  218  that control the switching infrastructure multiplexer  212  to select between the internal bus  200  and the mainline bus  206 . The flash memory device  204  includes a buffer memory  220  to buffer all garbage collection erase and scrub requests. The flash memory device  204  performs these garbage collection erase and scrub operations internally, with interleaved mainline traffic in accordance with embodiments of the invention. 
     In accordance with features of the invention, the function engine performs garbage collection and scrub operations internally by taking control of internal bus (I-bus)  208 . Mainline writes are performed as usual; for example, all blocks are connected to mainline bus  206  by default. 
     Referring to  FIGS. 3A, 3B, and 3C , there are shown example erase operations and write operations for garbage collection within a flash memory system  200  of  FIG. 2 . In  FIG. 3A , there are shown read, write and erase garbage collection operations generally designated by the reference character  300 . As shown, a read operation is approximately 50 microseconds for a 16 KB page, a write operation is approximately 1.6 milliseconds for a 16 KB page, and an erase operation is about 5 milliseconds per block. 
     In  FIG. 3B , there are shown write and erase garbage collection operations generally designated by the reference character  310  with 9 writes, followed by 4 erase, and 5 writes. 
     In  FIG. 3C , there are shown write and erase garbage collection operations generally designated by the reference character  320  with each write followed by an erase. 
     Referring to  FIGS. 4A, 4B, and 4C  there are shown example operations of background erase operations in parallel to write operations within the flash memory system  200  for implementing enhanced performance in accordance with the preferred embodiments. 
     In  FIG. 4A , there are shown read, write, erase garbage collection operations and garbage collection commands generally designated by the reference character  400 . As shown, a read operation is approximately 50 microseconds for a 16 KB page, a write operation is approximately 1.6 milliseconds for a 16 KB page, an erase operation is about 5 milliseconds per block, and the garbage collection command of approximately 10 microseconds for providing a background erase operation. The background erase operations shown in brackets are performed in parallel to write operations within the flash memory device  204 . 
     In  FIG. 4B , there are shown write and erase garbage collection operations generally designated by the reference character  410  with 9 writes, followed by 4 background erase operations in parallel to write operations shown in brackets and 5 writes. 
     In  FIG. 4C , there are shown write and erase garbage collection operations generally designated by the reference character  420  with each write followed by a background erase operation shown in brackets. 
     Referring to  FIGS. 5A, 5B, and 5C , there are shown example operations of background erase operations in parallel to write operations within the flash memory system  200  for implementing enhanced performance with bandwidth preservation in accordance with the preferred embodiments. 
     In  FIG. 5A , there are shown read, write, erase garbage collection operations and garbage collection commands generally designated by the reference character  500 . As shown, a read operation is approximately 50 microseconds for a 16 KB page, a write operation is approximately 1.6 milliseconds for a 16 KB page, an erase operation is about 5 milliseconds per block, and the garbage collection command of approximately 10 microseconds providing a background erase operation. Also the background erase operations shown in brackets are performed in parallel to write operations within the flash memory device  204 . 
     In  FIG. 5B , there are shown the write and erase garbage collection operations  310  with 9 writes, followed by 4 erase, and 5 writes of  FIG. 3B , together with write and erase garbage collection operations generally designated by the reference character  510  with 9 writes, followed by 4 background erase operations shown in brackets and 5 writes illustrating gained bandwidth indicated by an arrow labeled GAINED BW 1   
     In  FIG. 5C , there are shown write and erase garbage collection operations  320  with a write followed by an erase of  FIG. 3C , together with write and erase garbage collection operations generally designated by the reference character  420  with a write followed by background erase operation shown in brackets illustrating gained bandwidth indicated by an arrow labeled GAINED BW 2 . 
     Referring to  FIGS. 6A, 6B, and 6C , there are shown example operations of background erase operations in parallel to write operations within the flash memory system  200  for implementing enhanced performance with bandwidth preservation in accordance with the preferred embodiments. 
     In  FIG. 6A , there are shown read, write, erase garbage collection operations and garbage collection commands generally designated by the reference character  600 . Also as shown, a read operation is approximately 50 microseconds for a 16 KB page, a write operation is approximately 1.6 milliseconds for a 16 KB page, an erase operation is about 5 milliseconds per block, and the garbage collection command of approximately 10 microseconds to communicate a list of blocks to be garbage collected providing a background erase operation. 
     In  FIG. 6B , there are shown write and erase garbage collection operations generally designated by the reference character  610  with 9 writes (9×1.6 ms or 14.4 ms), followed by 4 background garbage collection (GC) commands (4×5 ms or 20 ms with garbage collection in device  204  performed in background) and 5 writes (5×1.6 ms or 8 ms). 
     In  FIG. 6C , there are shown write and erase garbage collection operations generally designated by the reference character  620  for eight continuous cycles with a write (1×1.6 ms)×8 cycles or 12.8 ms), followed by background erase operation (1×5 ms)×8 cycles or 40 ms with garbage collection in device  204  performed in background). 
     Referring to  FIGS. 7A, 7B, and 7C , there are shown garbage collection operations in accordance with the preferred embodiments. In  FIG. 7A  there is shown an example, garbage collection embodiment generally designated by the reference character  700  of the flash memory system  200 . In  FIG. 7A , example erase and write operations are respectively indicated by encircled labels A, B, C, D, E, F and encircled labels A, B, C, G, H in accordance with preferred embodiments.  FIGS. 7B and 7C  respectively illustrate the erase operation and the write operation of  FIG. 7A  in the example flash memory system  200  in accordance with preferred embodiments. 
     As indicated at A in  FIG. 7A  and as indicated in a block  720  in  FIG. 7B , erase operation begins with the host sending a list of blocks to be erased, for example, through control frames. As indicated at B, C, D in  FIG. 7A  and as indicated in a block  722  in  FIG. 7B , the command decoder  218  in the garbage collection and scrub function engine  210  processes the control frames to direct the multiplexer control  216 , and multiplexer  212  to select the internal bus  208  for the intended block  214  to be erased. As indicated at E in  FIG. 7A  and as indicated in a block  724  in  FIG. 7B , the garbage collection and scrub function engine  210  performs the erase operation on the selected block. The bus is connected back to the mainline bus after the erase operation is completed, as indicated in a block  726  and at G in  FIG. 7A . 
     Referring to  FIG. 7C , the write operation begins as indicated at A in  FIG. 7A , and as indicated in a block  730  with the host sending a list of one or more pages to be written. As indicated at B, C, G in  FIG. 7A  and as indicated in a block  732  in  FIG. 7C , the command decoder  218  in the garbage collection and scrub function engine  210  processes the control frames to direct the multiplexer control  216 , and multiplexer  212  to select the mainline bus  206  for the intended block  214  to be written. As indicated at H in  FIG. 7A , and as indicated in a block  734 , the garbage collection and scrub function engine  210  performs the write operation on the selected pages. 
     Referring now to  FIG. 8 , an article of manufacture or a computer program product  800  of the invention is illustrated. The computer program product  800  is tangibly embodied on a non-transitory computer readable storage medium that includes a recording medium  802 , such as, a floppy disk, a high capacity read only memory in the form of an optically read compact disk or CD-ROM, a tape, or another similar computer program product. Recording medium  802  stores program means  804 ,  806 ,  808 , and  810  on the medium  802  for carrying out the methods for implementing enhanced performance in the flash memory system  200  of  FIG. 2 . 
     A sequence of program instructions or a logical assembly of one or more interrelated modules defined by the recorded program means  804 ,  806 ,  808 , and  810 , direct the flash memory system  200  for implementing enhanced performance of the preferred embodiment. 
     While the present invention has been described with reference to the details of the embodiments of the invention shown in the drawing, these details are not intended to limit the scope of the invention as claimed in the appended claims.