Abstract:
Embodiments of this disclosure relate to improving solid-state non-volatile memory management. Embodiments improve the management of solid-state non-volatile memory by providing an execution manager responsible for controlling the timing of providing a request to a memory unit for execution. In embodiments, the execution manager traverses a list of received requests for memory access and dispatches commands for execution. In embodiments, if a request is directed to memory units which have reached a threshold for outstanding requests, the request may be skipped so that other requests can be dispatched for memory units which have not yet reached the threshold.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a divisional application of U.S. patent application Ser. No. 13/166,985, filed Jun. 23, 2011, entitled “System and Method for Multistage Processing in Memory Storage Subsystem, the specification of which is incorporated herein by reference. U.S. patent application Ser. No. 13/166,985 claims priority from provisional U.S. Patent Application Ser. No. 61/385,132, filed on Sep. 21, 2010, the specification of which is incorporated herein by reference. 
    
    
     BACKGROUND OF THE INVENTION 
     Systems for managing access to solid-state memory often perform complicated tasks such as maintaining drive coherency and mapping logical to physical addresses. In addition to receiving and processing memory access requests from host systems, these management systems often also process memory access requests related to internal maintenance operations such as garbage collection and wear leveling. The management systems may also be configured to respond to changing conditions in the physical memory media, including handling return status of memory commands executed in the media. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Systems and methods which embody the various features of the invention will now be described with reference to the following drawings, in which: 
         FIG. 1  illustrates an overview of an embodiment of a media access control system. 
         FIG. 2  illustrates a more detailed overview of an embodiment of a media access control system. 
         FIG. 3  is a flowchart showing the processing of a media access request object according to an embodiment. 
         FIG. 4  illustrates a structure of an embodiment for a media access request object. 
         FIG. 5  illustrates a detailed overview of an embodiment for a media access request manager. 
         FIG. 6  illustrates a detailed overview of an embodiment for a media access request executor. 
         FIGS. 7A-7D  illustrate the handling media request objects by a media access request executor in accordance with an embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     While certain embodiments of the inventions will be described, these embodiments are presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions. 
     Overview 
     Embodiments of the invention described herein are directed to pipelined or multi-staged systems and methods for controlling and managing access to physical memory media in solid-state storage subsystems. In an embodiment, the system separates media access request management tasks such as address translation, preparation for execution at the physical memory media, and completion status reporting into discrete stages and processing components to improve performance. In an embodiment, the components in the various stages of processing media access requests cooperate with one another by using a common data structure termed a media access request object to share information on a pending request that is being processed and report any result data and/or completion status information to an entity that originated the media access request. In an embodiment, the components in the various stages of processing media access requests are executed on different processors and/or processor threads. In an embodiment, the system manages the execution timing of requests on memory units (e.g., dies) of a solid-state storage subsystem to improve concurrent use of the memory units. 
     An overview of a solid state storage subsystem  120  implementing a multi-stage media access control system embodiment is presented in  FIG. 1 . As shown in  FIG. 1 , a plurality of media access requesters  101  within the storage subsystem  120  may have requests for memory access to memory arrays  105  in the storage subsystem  120 . In an embodiment, a host device  100  is in communication with at least one of the media access requesters  101 . The host device  100  may have requests to read or write data stored in memory arrays  105 . In an embodiment, the plurality of media access requesters  101  include a media access requester responsible for handling host commands received from the host device  100 , a media access requester responsible for issuing garbage collection commands, and a media access requester responsible for issuing wear leveling commands. 
     As shown in this embodiment, the plurality of media requesters  101  send media access requests to a media access request manager  102  in the storage subsystem  120 . In an embodiment, the media access request manager  102  performs any necessary processing including address translation and ensures the media requests can be executed on the memory arrays  105 . The address translation performed by the media access request manager  102  can include translation from a logical to a physical address. In an embodiment, once translation is complete, the media access request manager  102  then sends the media access requests to a media access request executor  103  in the storage subsystem  120 . The media access request executor  103  is in communication with a media controller  104 , and in an embodiment the media access request executor  103  determines when to send media requests to the media controller  104  for execution in the memory arrays  105 . In an embodiment, since the media access requests are embedded in media access request objects, the media access request executor, as part of the sending process, may also perform any required translation of data within the media access request objects to data in a format that is understandable by the media controller  104 . The media controller  104  performs media requests on the memory arrays  105  and returns a status and/or an error message to the media access request executor  103 . In an embodiment, the media access request executor  103  in turn returns the status/error message of media access requests to the plurality of media access requesters  101 . 
     Storage Subsystem Overview with Media Request Objects and Pointers 
       FIG. 2  provides an additional overview of the storage subsystem illustrating the use of media access request objects and pointers to process media access requests according to an embodiment. As shown in  FIG. 2 , a plurality of media access requesters  201  may comprise several requesters, such a host interface requester  201   a , a garbage collection requester  201   b , and a wear leveling requester  201   c . In this embodiment, a host  200  communicates with the host interface requester  201   a , but in other embodiments the host may communicate with a plurality of requesters. When the media access requesters  201  want to perform a memory access command on and interface with memory arrays  208 , the media access requesters first obtain a media access request pointer to a media access request object from a free pointer pool  202   a  associated with a shared pointer memory  202 . In this embodiment, the shared pointer memory  202  is stored in Static Random Access Memory (SRAM), though storage in other types of memory is possible in other embodiments. The media access requesters  201  may each have a dedicated portion of the shared pointer memory  202  associated with each media access requester, or the media access requesters may each request a pointer from a common pool of pointers. 
     In an embodiment, the shared pointer memory  202  includes a memory structure  202   b  with memory allocated to memory access request objects. The shared pointer memory  202  can also include a plurality of pointers to the memory access request objects. In an embodiment, each media access requester is associated with a dedicated pool of memory and may maintain its own free pointer pool. 
     As described below, in an embodiment, each component may access and update information in a memory access request object located in shared pointer memory  202 . Each component&#39;s access is denoted by dotted lines in  FIG. 2 . In an embodiment, each component may be executed on a separate thread on a processor. In an embodiment, one or more components may be located on a different processor. Though in these embodiments the components are executed on different processors or separately executing threads, in some embodiments the components nonetheless all access shared pointer memory  202 . Contention between processing units for access to shared pointer memory  202  is controlled by logically programming each component to access only memory associated with memory access request pointers currently being handled by that component. For example, once a media access requester  201  completes its processing of a media access request object, it passes the associated pointer to the media access request manager  203  and thus relinquishes to the media access request manager  203  any access right to that object. Likewise, when the media access request manager  203  completes processing of the same object and passes the associated pointer to the media access request executor  206 , the media access request manager  203  no longer has access to the object. In summary, in various embodiments, the processing unit that has the pointer is the sole unit with access to the referenced object. Data integrity in the shared memory objects is preserved even as they are accessed by different processing units executed on different processors and/or threads, without the use of a synchronization and/or locking mechanism that may require substantial implementation costs and/or run-time system resources. 
     Returning to the top of  FIG. 2 , after obtaining a media access request pointer, the media access requester can populate the referenced media access request object with information relating to the media access request to be issued by the media access requester. For example, the media access requester may populate the media access request object with request data such as a logical address, data indicating the type of memory operation to be performed (e.g., read, write, or erase), and data associated with the operation (e.g., data to be written to a memory array for a write operation). Additional details on request data will be provided below in conjunction with the description of  FIG. 4 . 
     Once the media access request object is populated with request data, the media access requesters  201  can then pass the pointer for the media access request object to the media access request manager  203 . The passing of the pointer, rather than the object itself, reduces the memory footprint necessary to implement this shared message architecture and improves system performance. In an embodiment, the media access request manager  203  maintains an invalid page table  205  and a mapping table  204 . The mapping table  204  maintains a mapping between logical and physical addresses, while the invalid page table  205  maintains a list of physical addresses that contain invalid data. Typically, addresses (e.g., pages) referenced by the invalid page table can be freed through a garbage collection operation and made ready for further use. In an embodiment, the media access request manager  203  processes the incoming requests and resolves any contention between requests submitted by the plurality of media access requesters  201 . For example, if the execution of a request requires an alteration to the mapping table  204  and/or invalid page table  205 , the media access request manager  203  may update the tables to reflect the alteration. In certain circumstances, a media request may specify a media operation on data that has been altered since the request was first issued by the originating media access requester. In these circumstances, the media access request manager  203  may alter the request to conform to the alterations, or the media access request manager  203  may reject the request due to the conflict and return the associated media access request pointer to the originating media access requester. 
     In an embodiment, the media access request manager  203  may also populate the physical address field of the media access request object with a physical address that corresponds to a logical address indicated in the media access request object. For a read operation, determining a physical address that corresponds to a logical address may involve a simple lookup in the mapping table  204 . For write operations, the media access request manager  203  may maintain a list of upcoming physical write locations associated with the media access requester that submitted the request. In an embodiment, the upcoming physical write locations may be associated with an open group of blocks, and the media access request manager  203  can choose a write location from the open group of blocks that is currently assigned to the requester. In an embodiment, the media access request manager  203  is configured to assign one or more open groups of blocks to individual requesters  201 . In further embodiments, other ways of assigning a write location to a logical address may be implemented in any way understood to one skilled in the art. 
     In an embodiment, after processing the request as described above, the media access request manager  203  passes the media access request pointer to a media access request executor  206 . The media access request executor  206  may be in communication with a media controller  207  to execute commands on memory arrays  208 . The media access executor  206  schedules execution of requests on the memory arrays  208 . In an embodiment, each memory array  208  has an associated execution queue, which stores commands issued by the media controller  207  in response to requests dispatched by the media access executor  206 . In an embodiment, the media access request executor  206  maintains a record of the outstanding commands in each execution queue, and passes requests to the media controller according to the outstanding commands in each execution queue. In some embodiments, the media access request executor  206  passes the media access request pointer to the media controller  207 , and the media controller  207  executes a command specified by the referenced media access request object on the memory arrays  208 . In other embodiments, the media access request executor  206  translates data within the media access request object into a format executable by the media controller  207  on the memory arrays  208 . 
     After execution of the media request on the memory arrays  208 , the memory arrays  208  and/or media controller  207  in an embodiment return result data such as a status (e.g., indicating success or failure) and any other information associated with the execution to the media access request executor  206 . For example, associated information for a read operation may include the data read from the memory arrays. In certain embodiments, the media access request executor  206  populates a status field in the media access request object with the returned result data and other returned information. In other embodiments, the media controller  207  populates the media request object data fields with result data. 
     Once result data is inserted into the media request object, in an embodiment the associated media access request pointer can then be passed from the media access request executor  206  to a request status pool/status reporter  209 . In an embodiment, the media access requesters  201  will poll the request status reporter/pool  209  to retrieve any updated status for requests referenced by media access request pointers associated with that requester. In another embodiment, the request status pool/reporter can push status information to the associated requester(s). 
     Media Access Request Processing Cycle 
       FIG. 3  is a flowchart depicting the processing cycle of a media access request pointer/object according to an embodiment. As shown in this embodiment, at block  300 , a media access request pointer is maintained in a pool of available request pointers. When a media access request is needed by a media access requester, a media access request pointer can be allocated to the requester at block  301 . Next, as shown at block  302 , the media access requester can populate the media access request object referenced by the allocated pointer with command data, such as the command type, the logical address associated with the command, and any other associated data (such as write data for a write command). The media access request pointer is then passed to the media access request manager at block  303 , where the media access request data may be updated for translation. If the media access request is determined to have a conflict with other pending requests or otherwise cannot be completed according to the data populated in the referenced media access request object, the media access request may be updated with a status indicating it was not executed and the media access request pointer may be passed to a status pool  306 . After updating the media access request object with a physical address location by translation of the logical address, the media access request manager can pass the media access request pointer to the media access request executor at block  304 . The media access request is then executed on the media at block  305  and the request will be updated with a result and status. As described above, the media access request pointer may or may not require translation for a media controller for execution at block  305 . After execution, the media access request pointer is passed to the status pool  306 . As described above, the status pool  306  holds media access request pointers until they are reviewed by the media access requesters. Once reviewed, the media access requester will then free the pointers by placing them back into the pool of available request pointers, where they can be allocated to the next requesters. 
     Media Access Request Object Structure 
       FIG. 4  depicts data fields allocated for the media access request object according to an embodiment. As shown in the embodiment in  FIG. 4 , the media access request object includes a variety of data fields to hold information about the request. The media access request object embodiment shown in  FIG. 4  includes data fields for a macro pointer  401 , a single/dual plane designation  403 , plane selection  405  indicating which plane on a die is being accessed, operation data  408  indicating the associated memory operation type, a logical chunk number  402 , a physical chunk number  404 , a requester ID  406 , a result status  407 , and a count  409  for the number of buffers in the request. The media access request object may also contain a field for other data  410 . In certain embodiments, the logical chunk number data field  402  may also be used for metadata about the request. Certain embodiments of the media access request object may include data for specifying single or dual plane operations which can be designated by the single/dual plane designation  403 . The media access request object may also include buffers  411  to store read and write data relating to the request. 
     As the media access request object is passed among the components of the media request control system within the storage subsystem, various portions of the media access request object are populated. For example, the media access requester may populate fields such as the requester ID data field  406  (indicating the originating requester), the operation data field  408 , and the logical chunk number data field  402 . For a write request, the media access requester may populate data to be written into buffers  411 . In an embodiment, when the media access request object is passed to the media access request manager, the physical chunk number field  404 , and the plane field  405  are populated. In an embodiment, if the request cannot be executed, the media access request manager can update the result data field  407 . After the execution of the media request in the physical memory media, the media access executor and/or the media controller can update the result data field  408 , and for some operations, e.g., a read operation, the buffers  411  with data read from the target physical address. 
     Media Access Request Manager 
     A detailed view of an embodiment of the media access request manager is shown in  FIG. 5 . In this embodiment, three media access requesters are shown, a garbage collector  503 , a user data manager  502 , and a requester for other internal operations  501 . Other internal operations  501  may include, for example, wear leveling operations. The user data manager  502  is in communication with a host interface  500  and is responsible for responding to host communications (e.g., accepting incoming memory commands from the host). In this embodiment, each requester may have a dedicated request queue  504   a - c . In other embodiments, certain requesters may share one or more request queues  504 . For example, the user data manager  502  and other internal operations  501  may share a single request queue. Each media access requester may obtain a media access request pointer and populate the media access request object as described above. In an embodiment, the media access requester can then place the media access request pointer in the associated request queue. 
     In an embodiment, the media access request manager  505  receives media access request pointers from the request queues  504   a - c . After receiving the media access request pointers, the media access request manager  505  in an embodiment updates the media access request object with reference to a mapping table  506  and an invalid page table  507 . In an embodiment, the media access request manager  505  maintains a designated write location for each requester or request queue. In an embodiment, the media access request manager  505  determines a physical address for the write location corresponding to an open block in a group of blocks for each requester or request queue. In an embodiment, after receiving a write command request at the media access request manager  505 , the mapping table  506  is updated with the new physical address corresponding to the logical address of the request. The invalid page table  507  is then updated to indicate that the physical address formerly mapped to the logical address is invalid. In an embodiment, the media access request manager  505  also updates an invalid page counter for the block or group of blocks corresponding to the old physical address. In an embodiment, the media access request manager  505  is the only entity with access rights to write to the mapping table  506  and the invalid page table  507 . Restricting write accesses to the media access request manager  505  ensures a single point of control for these tables, and prevents possible contentions and data coherency errors that may result if the requesters were granted direct access to the tables. 
     In an embodiment, when the media access request manager  505  receives media access request pointers from the request queues  504   a - c , the media access request manager  505  may determine a ratio at which to process media access request pointers from each queue. While requests from each requester must generally be executed in-order relative to other requests from that requester, requests from each requester may generally be executed in a different order than received relative to requests from another requester. For example, if a read command from the user data manager  502  is directed to the same logical address as a garbage collection command from the garbage collector  503 , the media access request manager  505  can ensure that the physical address of each command is updated to appropriately compensate for the order of execution. If the garbage collection command is processed first, the read command can be directed to the new physical address corresponding to the same logical address. If the read command is processed first, the read command is executed prior to any data movement associated with the garbage collection command and revisions to the physical addresses of the data. As such, the media access request manager ensures coherency with the mapping and invalid page tables, thereby preventing downstream request execution conflicts. 
     In an embodiment, the option of processing requests from the queues in different ratios allows the media access request manager  505  to throttle, or provide dynamic ratios, of requests received from each request queue  504  and thereby prioritize a specific requester over other requesters. For example, the media access request manager  505  can receive and process two or three or more requests from the user data manager  502  for every request from garbage collector  503  (or vice versa). In an embodiment, the media access request manager  505  temporarily halts receiving media access request pointers from a request queue when certain conditions are met. For example, if the invalid page table shows a high need for garbage collection, the requests for user data manager  502  may be throttled down or halted to allow for more garbage collection requests to be processed. Alternatively, if the user data manager  502  has an unusually high number of requests, or if the request queue  504   a  for the user data manager  502  is nearing a threshold, the requests associated with the garbage collector  503  may be throttled down or halted. After preparing the media access request for execution, the media access request manager  505  passes the media access request pointer to the media access request executor. 
     Media Access Request Executor 
       FIG. 6  depicts a media access request executor according to an embodiment. In an embodiment, the media access request executor  602  receives media access request pointers from a media access request manager  601 . The media access request executor improves throughput and concurrency on memory arrays  607  by determining when to pass media access requests down to a media controller  605  for execution. In an embodiment, the media access request executor  602  maintains a request list  603  and execution counters  604 . In this embodiment, received media access requests from the media access request manager  601  are placed on the request list  603 . In certain embodiments, the request list identifies a specific memory unit associated with the media access request. The memory unit associated with a media access request may be a die of solid-state memory. In other embodiments, the request list identifies another granularity of the memory arrays  607  (e.g., plane(s) or block(s) with a die or a device including multiple dies), which is generally the lowest level of memory access on which the memory array can accept commands while other commands can be executed on a portion of the memory array addressed at the same level. In this embodiment, the execution counters  604  maintain a record of the pending requests associated with each memory unit. As requests are directed to each memory unit and requests are completed at each memory unit, the execution counter of each memory unit increments and decrements accordingly. 
     In an embodiment, when the media access request executor  602  has determined that a media access request should be executed, the media access request executor  602  will pass the media access request to media controller  605 . As described above, in certain embodiments the media access request executor  602  can pass a media access request pointer to media controller  605 , while in other embodiments the media access request executor  602  performs a translation to a format acceptable to the media controller  605 . Media controller  605  directs each media access request to an execution queue  606  associated with each memory arrays  607 . Each memory array  607  may be accessed via an associated data channel. The execution queue  606  for each memory array  607  may be maintained on the memory arrays  607 , or may be maintained as a part of media controller  605 . As media access requests are completed on memory arrays  607 , the media controller  605  will return a status and any associated information to the media access request executor  602 . As necessary, the media access request executor  602  incorporates the status and associated information into the media access request object and passes the media access request pointer to status pool  608 . In an embodiment, the media controller  605  populates the status and associated information fields of the media access request object instead of the media access request executor  602 . As described above, the media access requesters receive media access request pointers from the status pool  608 . 
     One example method of determining when to execute media access requests is shown in  FIGS. 7A-7D . As shown in  FIG. 7A , in an embodiment the request list includes a reference to the memory unit associated with each memory access request (each box represents a memory access request and the number within is the reference). In this example, starting with  FIG. 7A , the execution counters show that all of the memory units ( 0 - 7 ) do not have any pending media access requests. In an embodiment, the media access request executor also maintains a current position on the request list, which is shown in this example by the arrow pointing to the first request directed at memory unit  1 . As the media access request executor sends requests to the media controller for execution, the media access request executor updates the execution counter corresponding to the memory unit associated with the media access request.  FIG. 7B  shows the status of the request list and execution counters after the first three requests have been processed (now shaded in grey) and sent to the media controller for execution. As shown, two requests for memory unit  1  and one request for memory unit  2  have been sent to the media controller for execution. Accordingly, the media execution counters have been updated, and the request list indicates that these requests have been sent. The current position on the request list as shown is from a time immediately after the last-sent request. 
     In various embodiments, each memory unit is capable of having a certain threshold number of outstanding requests at once. As such, the media access request executor cannot send additional requests past this outstanding request threshold. The outstanding request threshold is typically dictated by the depth of the execution queue  606  depicted in  FIG. 6 . In the example shown in  FIGS. 7A-7D , for ease of explanation, the threshold number of outstanding requests is two. In actual systems, the number of possible outstanding requests may be higher. In an embodiment, when the current position in the request list points to a request directed to a memory unit for which the outstanding request threshold has been reached, the media access request executor skips that request on the request list and continues to process the next requests on the list. As shown in  FIG. 7C , a media access request directed to memory unit  1  has been skipped because that memory unit had already reached the threshold number of outstanding requests (as indicated by the counter). After the skip, the media access request executor has gone on to process requests directed to memory units  3 ,  4 ,  2 , and  5 , and the associated media execution counters have been incremented. 
     In an embodiment, as the request list is traversed, the current position in the request list will reach a reset point and return to the first skipped request to determine if that request can now be executed. The reset point can be determined in a variety of ways. The current position can return to the first skipped request upon the triggering of a condition. As examples, the condition can be based at least partially on any of: the number of requests that have been skipped (in total or for a particular memory unit), receiving an indication notifying of a request completion from a memory unit that has been skipped, and reaching the end of the request list. In various embodiments, as the request list is continually changing to reflect processed requests and new incoming requests from the media access request manager, periodic snapshots of the request list are taken and the aforementioned condition for reset may also be based on reaching the end of a snapshot of the request list.  FIG. 7D  shows an embodiment of the request list and execution counters if a reset point has been reached immediately after the system state in  FIG. 7C . When the reset point is reached, the media controller is polled to determine if any requests have completed, since these completed requests can result in the media access request executor decrementing the execution counters. In  FIG. 7D , requests corresponding to memory units  1  and  2  have completed and the corresponding execution counters are decremented. The current position in the request list now returns to the first skipped request, which here corresponds to memory unit  1 . Returning to the first skipped request ensures that memory requests for a particular memory unit are executed in-order relative to that memory unit. Because certain requests are skipped, the requests for a memory unit may be executed out-of-order with respect to the time when the requests for other memory units were received. However, because these requests correspond to different, independently accessible memory units, data coherency is maintained. In the embodiment shown in  FIG. 7D , at the reset point the media access request executor also clears the request list of requests that have already been sent to the media controller and/or added to the appropriate execution queues for the respective memory units. Other embodiments may remove a request from the request list only when the request is completed on the memory unit. Completed requests may be passed to a status pool as described above. 
     In alternate embodiments, other methods for maintaining a list of requests to be executed may be used. For example, at the media access request executor level, the received requests may be placed in individual queues associated with each memory unit. The media access request executor may then send media access requests from the queues to the media controller by checking whether the associated media unit has reached a threshold in the media execution counter. In an embodiment, the media access request executor may check only those queues which have not reached the threshold in the execution counter. In an embodiment, the media access request executor checks the queues associated with each memory unit by a round robin algorithm. A flag can be used to designate which memory units have an execution counter that has reached the threshold. According to another embodiment, the media access request executor reorders the media access requests in the request list at least partly based on the status of the associated memory units. The status of the associated memory units may be determined with reference to the execution counters. Other data management means can be used to control the flow of media requests to the media controller in accordance with this disclosure. 
     By temporarily skipping certain requests, the media access request executor increases the concurrent use of the memory units (by sending requests to those units before other units have finished requests). Further, by using a reset point, the media access request executor can return to process skipped requests to preserve data coherency. 
     CONCLUSION 
     The features and attributes of the specific embodiments disclosed above may be combined in different ways to form additional embodiments, all of which fall within the scope of the present disclosure. Although the present disclosure provides certain preferred embodiments and applications, other embodiments that are apparent to those of ordinary skill in the art, including embodiments which do not provide all of the features and advantages set forth herein, are also within the scope of this disclosure. Accordingly, the scope of the present disclosure is intended to be defined only by reference to the appended claims.