Storage device read-disturb-based block read temperature utilization system

A storage device read-disturb-based block read temperature utilization system includes a storage device chassis housing a storage subsystem. A local read temperature utilization subsystem in the storage device chassis determines that data in a first block in the storage subsystem should be moved and, in response determines read disturb information for the first block and uses it to identify relative read temperatures for a plurality of rows in the first block in the storage subsystem. The local read temperature utilization system then moves the data from the first block in the storage subsystem to at least one second block in the storage subsystem based on the relative read temperatures identified for the plurality of rows in the first block in the storage subsystem.

BACKGROUND

The present disclosure relates generally to information handling systems, and more particularly to a storage device in an information handling system utilizing read-disturb-based read temperatures identified for an individual block in that storage device.

Information handling systems such as, for example, server devices and/or storage systems, and/or other computing devices known in the art, includes storage systems having one or more storage devices (e.g., Solid State Drive (SSD) storage devices) for storing data generated by the computing device. In some situations, it may be desirable to identify how often logical storage locations associated with any storage device are read. For example, different storage devices with different capabilities are associated with different costs (e.g., storage devices with relatively “higher” capabilities are more expensive than storage devices with relative “lower” capabilities), while different data stored in a storage system may have different characteristics, with some data being read relatively often (also referred to as data having a relatively “hot” read temperature) and other data being read relatively less often (also referred to as data having a relatively “cold” read temperature). As will be appreciated by one of skill in the art in possession of the present disclosure, financial margins of storage providers (e.g., entities that provide storage for customers) may be improved by offering particular storage Service Level Agreements (SLAs) while using relatively cheaper storage devices, and value can be passed on to customers by providing improved storage SLAs for data with relatively “hot” read temperatures without incurring higher costs for all storage devices in the storage system (e.g., by storing data with relatively “hot” read temperatures on relatively higher capability/cost storage devices, and storing data with relatively “cold” read temperatures on relatively lower capability/cost storage devices).

Conventional read temperature identification systems typically utilize a host processor (or a storage processor) and a host memory in a server device and/or storage system to identify read temperatures of logical storage locations in SSD storage device(s) included in, connected to, and/or otherwise coupled to that server device and/or storage system. For example, a Logical Block Address (LBA) range may be divided into smaller address ranges or logical “chunks” (e.g., 128 KB chunks). A counter (e.g., a Dynamic Random Access Memory (DRAM) counter) in the host memory may then be assigned to track read access to each logical chunk, and when the host processor performs read operations to read data from each of the SSD storage device(s) in the server device and/or storage system, the host processor will map the LBA range of that read operation to the corresponding logical chunk(s) being read, and increment the counter(s) for those physical storage element chunk(s) in the host memory. However, such conventional read temperature identification systems suffer from a number of issues.

For example, the conventional read temperature identification systems discussed above require dedicated host memory (e.g., for a 16 TB SSD storage device with 128 KB logical chunks, 32 MB of dedicated host memory is required if 8 bit counters are utilized), and the read temperature information identified will not be power-fail safe without a persistent power implementation (e.g., a battery backup, the use of Storage Class Memory (SCM) devices, etc.), each of which increases costs. In another example, the conventional read temperature identification systems discussed above increase complexity, as for a High Availability (HA) system each of multiple host processors included in a server device and/or storage system must generate its own read temperature map that tracks read temperatures of its storage devices in that server device and/or storage system, and then those host processors must synchronize their respective read temperature maps. Further complexity may be introduced when more Input/Output (I/O) initiators are utilized (e.g., when additional host processors are utilized in Non-Volatile Memory express over Fabrics (NVMe-oF) Just a Bunch Of Drives (JBOD) systems, disaggregated storage systems, and/or other systems that would be apparent to one of skill in the art in possession of the present disclosure).

In yet another example, the conventional read temperature identification systems discussed above may be inaccurate in some situations, as read temperature identification operations may be performed “in the background” with a “best effort” approach, and when host processors in a server device and/or storage system are busy performing other operations, those read temperature identification operations may not be performed in order to prevent I/O latency and/or other performance issues. While the host processors in a server device and/or storage system may sometimes only delay the read temperature identification operations in those situations, in some cases the read temperature identification operations may simply not be performed. In yet another example, the conventional read temperature identification systems discussed above can introduce a performance impact to data path(s) in a server device and/or storage system due to the use of the host processor and the host memory bus in performing the read temperature identification (e.g., via Read Modify Write (RMW) operations to provide these relatively small read temperature data writes via 64 byte cache line host memory entries, resulting in increased cache thrashing operations).

One conventional read temperature identification solution to the issues discussed above is to assume or characterize (a priori) the read temperatures of a storage device based on the type of data being read (e.g., metadata vs customer data), the type of application instructing the read operation (e.g., Relational Database Management System (RDBMS) applications vs. social media post applications (e.g., applications provided “tweets” via the TWITTER® social networking service available from TWITTER® of San Francisco, California, United States) vs. video streaming applications), the type of workload being performed (e.g., 4K vs. 8K video streaming workloads, sequential access vs. random access workloads, etc.). However, such conventional read temperature identification solutions suffer from a number of issues as well.

For example, the conventional read temperature identification solutions discussed above require pre-qualification or classification of data attributes, and cannot provide granularity beyond the particular classification that is used. In another example, conventional read temperature identification solutions do not allow for sub-classifications of data (e.g., a video type of the read data) that may be useful, will not allow data (e.g., video data such as that providing a movie) that is read often to be provided on a faster storage device or replicated at additional storage locations, and present issues with tracking effective “hits” per storage device and load balancing (as conventional read temperature identification solutions are typically limited to tracking data requests (e.g., video data requests) at the application level). In yet another example, conventional read temperature identification solutions require modification of software when new types of data, applications, and/or workloads are introduced and, as such, are less resilient with regard to optimizing read performance for use cases that emerge over time, and present additional costs associated with research and development to qualify new workloads or applications, develop software, test that software, perform software patch updates on server devices and/or storage systems that will use that software, and/or introduce other added cost factors that would be apparent to one of skill in the art in possession of the present disclosure.

Accordingly, it would be desirable to provide read temperature identification system that addressees the issues discussed above.

SUMMARY

According to one embodiment, a storage device includes a processing system; and a memory system that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide a local read temperature utilization engine that is configured to: determine that data in a first block in a storage subsystem should be moved; determine, for the first block in the storage subsystem in response to determining that the data in the first block in the storage subsystem should be moved, read disturb information; identify, using the read disturb information, relative read temperatures for a plurality of rows in the first block in the storage subsystem; and move the data from the first block in the storage subsystem to at least one second block in the storage subsystem based on the relative read temperatures identified for the plurality of rows in the first block in the storage subsystem.

DETAILED DESCRIPTION

Referring now toFIG.2, an embodiment of a computing device200is illustrated that may include the read-disturb-based read temperature identification system of the present disclosure. In an embodiment, the computing device200may be provided by the IHS100discussed above with reference toFIG.1and/or may include some or all of the components of the IHS100, and in the specific examples below is illustrated and described as being provided by a server device and/or a storage system. However, while illustrated and discussed as being provided by particular computing devices, one of skill in the art in possession of the present disclosure will recognize that the functionality of the computing device200discussed below may be provided by other devices that are configured to operate similarly as the computing device200discussed below. In the illustrated embodiment, the computing device200includes a chassis202that houses the components of the computing device200, only some of which are illustrated and discussed below. For example, the chassis202may house a processing system (not illustrated, but which may include the processor102discussed above with reference toFIG.1that may be provided by a Central Processing Unit (CPU) and/or other processing systems that one of skill in the art in possession of the present disclosure would recognize as providing a computing device host processor) and a memory system (not illustrated, but which may include the memory114discussed above with reference toFIG.1) that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide a global read temperature identification engine204that is configured to perform the functionality of the global read temperature identification engines and/or computing devices discussed below.

The memory system housed in the chassis202may also include a global read temperature identification database206that is configured to store any of the information utilized by the global read temperature identification engine204discussed below. The chassis202may also house a storage system208that, in the illustrated embodiment, includes a plurality of storage devices210a,210b, and up to210c. In the specific examples below, each of the storage devices210a-210cin the storage system208are described as being provided by particular Solid State Drive (SSD) storage devices, but one of skill in the art in possession of the present disclosure will appreciate how the teachings of the present disclosure may benefit other storage device technologies, and thus storage devices utilizing those other types of storage device technologies are envisioned as falling within the scope of the present disclosure as well. However, while a specific computing device200has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that computing devices (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the computing device200) may include a variety of components and/or component configurations for providing conventional computing device functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now toFIG.3A, an embodiment of a storage device300is illustrated that may provide any or each of the storage devices210a-210cin the storage system208discussed above with reference toFIG.2. In an embodiment, the storage device300may be provided by the IHS100discussed above with reference toFIG.1and/or may include some or all of the components of the IHS100, and in the specific examples below is illustrated and described as being provide by an SSD storage device (e.g., a Non-Volatile Memory express (NVMe) SSD storage device). However, while illustrated and discussed as being provided by a particular storage device, one of skill in the art in possession of the present disclosure will appreciate that the teachings of the present disclosure may be implemented in other storage devices that are configured to operate similarly as the storage device300discussed below. In the illustrated embodiment, the storage device300includes a chassis302that houses the components of the storage device300, only some of which are illustrated and discussed below.

For example, the chassis302may house a storage device processing system304(which may include the processor102discussed above with reference toFIG.1such as a Central Processing Unit (CPU), storage device controller, and/or other processing systems that one of skill in the art in possession of the present disclosure would recognize as being provided in an SSD storage device) that is coupled via a first memory interface306(e.g., a Dual Data Rate (DDR) interface) to a first memory system308(which may include the memory114discussed above with reference toFIG.1such as Dynamic Random Access Memory (DRAM) devices and/or other memory systems that would be apparent to one of skill in the art in possession of the present disclosure). As illustrated in the specific examples provided herein, the first memory system308may include instructions that, when executed by the storage processing system304, cause the storage device processing system304to provide a local read temperature identification engine310that is configured to perform the functionality of the local read temperature identification engines and/or storage devices discussed below.

As also illustrated in the specific examples provided herein, the first memory system308may include a local read temperature identification database312that is configured to store any of the information utilized by the local read temperature identification engine310discussed below. However, one of skill in the art in possession of the present disclosure will recognize that other embodiments of the present disclosure may provide the local read temperature identification database312in other locations while remaining within the scope of the present disclosure as well. For example, as illustrated, the storage device processing system304may also be coupled via a second memory interface314(e.g., a Storage Class Memory (SCM) interface) to a second memory system316(which may include the memory114discussed above with reference toFIG.1such as SCM devices and/or other memory systems that would be apparent to one of skill in the art in possession of the present disclosure), and the local read temperature identification database312may be provided by the second memory system316while remaining within the scope of the present disclosure as well.

The storage device processing system304may also be coupled via a storage interface318to a storage subsystem320. With reference toFIG.3B, in some embodiments, the storage subsystem320may include a storage subsystem chassis320athat supports a plurality of NAND die322. With reference toFIG.3C, each NAND die322may include a chassis322athat supports a plurality of NAND blocks324, with each NAND block324including a chassis324athat supports a plurality of NAND wordlines326. Furthermore, each NAND wordline326may include a plurality of cells that provide a plurality of data portions326a, and a respective error check portion326b(e.g., a Cyclic Redundancy Check (CRC)portion and/or other error check data known in the art) may be associated with each of those data portions326a. However, one of skill in the art in possession of the present disclosure will appreciate how in some embodiments the data written to a NAND block324may include “padding” data or other data which conventionally does require the writing of associated error check portions.

To provide a specific example, the storage subsystem320may include 128, 256, or 512 NAND die, with each NAND die including approximately 2000 NAND blocks, and with each NAND block including NAND wordlines grouped into 100-200 NAND layers (although forecasts predict that NAND wordlines will be grouped into up to 800 layers by the year 2030). As will be appreciated by one of skill in the art in possession of the present disclosure, conventional Triple Level Cell (TLC) technology typically allows on the order of tens to hundreds of K of data (e.g., 96 KiB on a NAND wordline, 48 KiB on a NAND wordline with two NAND wordlines activated at any particular time, up to hundreds of KiB when more planes are utilized, etc.) to be stored per NAND wordline (i.e., in the data portions of those NAND wordlines), resulting in NAND wordlines with −250K cells.

With reference toFIG.3D, a simplified representation of how data may be stored in a cell328is provided, and one of skill in the art in possession of the present disclosure will appreciate how data may be stored in any of the plurality of cells in any of the plurality of NAND wordlines discussed above in the manner described below. The data storage representation of the cell328inFIG.3Dincludes a graph330with voltage330aon the X-axis, and illustrates how different voltages of the cell328may be associated with different values for that cell328, which in specific example illustrated inFIG.4Dincludes values “A”, “B”, “C”, “D”, “E”, “F”, “G”, and “H”. Furthermore, the data storage representation of the cell328also illustrated how reference voltages may be defined to distinguish whether a voltage in the cell provide a particular value, with a B reference (“B REF”) distinguishing between a value “A” or a value “B” for the cell328, a C reference (“C REF”) distinguishing between a value “B” or a value “C” for the cell328, a D reference (“D REF”) distinguishing between a value “C” or a value “D” for the cell328, an E reference (“E REF”) distinguishing between a value “D” or a value “E” for the cell328, an F reference (“F REF”) distinguishing between a value “E” or a value “F” for the cell328, a G reference (“G REF”) distinguishing between a value “F” or a value “G” for the cell328, an H reference (“H REF”) distinguishing between a value “G” or a value “H” for the cell328.

As such, when the cell328includes a voltage below “B REF” it will provide a value “A”, when the cell328includes a voltage between “B REF” and “C REF” it will provide a value “B”, when the cell328includes a voltage between “C REF” and “D REF” it will provide a value “C”, when the cell328includes a voltage between “D REF” and “E REF” it will provide a value “D”, when the cell328includes a voltage between “E REF” and “F REF” it will provide a value “E”, when the cell328includes a voltage between “F REF” and “G REF” it will provide a value “F”, when the cell328includes a voltage between “G REF” and “H REF” it will provide a value “G”, when the cell328includes a voltage over “H REF” it will provide a value “H”. While not illustrated or described in detail herein, one of skill in the art in possession of the present disclosure will appreciate that each value A-H illustrated inFIG.3Dmay be configured to store more than one bit depending on the amount of voltage that is provided to indicate that value (e.g., a first voltage level between “B REF” and “C REF” will provide a first set of bits for the value “B”, a second voltage level between “B REF” and “C REF” will provide a second set of bits for the value “B”, and so on).

As will be appreciated by one of skill in the art in possession of the present disclosure, different storage device manufacturers/providers may configure the NAND wordlines/NAND layers in NAND blocks differently, with some storage devices including NAND blocks with separate NAND wordlines, some storage devices including NAND blocks with NAND layers that each include a plurality of NAND wordlines, and some storage devices including NAND blocks with groups of NAND layers that each include a plurality of NAND wordlines. As such, with reference toFIG.3E, the present disclosure abstracts the physical implementation of NAND wordlines and NAND layers into “NAND rows”, with each NAND block324discussed in the examples below including a plurality of NAND rows332. In other words, any one of the NAND rows332may include NAND wordline(s), NAND layer(s) each including a plurality of NAND wordlines, or group(s) of NAND layers that each include a plurality of NAND wordlines. As will be appreciated by one of skill in the art in possession of the present disclosure, the read disturb signatures discussed below may vary based on the design of the storage subsystem/storage device, as it may effect a NAND wordline or group of NAND wordlines, and thus the abstraction of the physical implementation of NAND wordlines into NAND rows is provided to simplify the discussion below while encompassing such different storage subsystem/storage device designs.

However, while the specific examples discussed above describes the storage device300as including the storage interface318that may be provided by a flash device interface and the storage subsystem320that is described as being provided by NAND devices (e.g., NAND flash devices), one of skill in the art in possession of the present disclosure will appreciate how the teachings of the present disclosure may benefit other storage technologies, and thus storage devices utilizing those other types of storage technologies are envisioned as falling within the scope of the present disclosure as well. Furthermore, while a specific example of cells that may store 8 values (“A”-“H” in the examples above) are provided, one of skill in the art in possession of the present disclosure will appreciate how the cells may store 2 values (e.g., “A”/“0” and “B”/“1”), 4 values (e.g., “A”/“00”, “B”/“01”, “C”/“10”, and “D”/“11”), or more than 8 values while remaining within the scope of the present disclosure as well). Furthermore, one of skill in the art in possession of the present disclosure will appreciate how different NAND rows332in any particular NAND block324of the storage subsystem302may use different value encoding techniques (e.g., “A” and “B”, “A”-“D”, “A”-“H” in the examples above), and such mixed encoding NAND rows332will fall within the scope of the present disclosure.

In the illustrated embodiment, a power loss prevention system334is housed in the chassis302and coupled to the storage subsystem320, and in specific examples may be provided by a Power Loss Prevention (PLP) capacitor and/or other power storage/provisioning subsystems that would be apparent to one of skill in the art in possession of the present disclosure. In the illustrated embodiments, the storage device processing system304is also coupled to an expansion bus336such as, for example, a Peripheral Component Interconnect express (PCIe) expansion bus that may provide the connection to the global read temperature identification engine204, as well as to one or more storage device processing system peripherals338. Furthermore, the expansion bus336may provide one or more connections for performing operations associated with the storage device300(e.g., connection(s) for reading/writing, connections for managing any of the data/information discussed below, etc.), and may also provide out-of-band interface(s), side channel interface(s), and/or other interfaces that provide access to the storage device processing system304for other systems. However, while a specific storage device300has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that storage devices (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the storage device300) may include a variety of components and/or component configurations for providing conventional storage device functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now toFIG.4, an embodiment of a method400for providing read-disturb-based read temperature identification is illustrated. As discussed below, the systems and methods of the present disclosure utilize the read disturb effect that produces noise in adjacent NAND rows when any particular NAND row is read in order to identify NAND rows that are read more often than other NAND rows and thus have higher “read temperatures” than those other NAND rows. For example, the read-disturb-based read temperature identification system of the present disclosure may include storage device(s) that each determine read disturb information for each block in that storage device, use that read disturb information to identify a subset of rows in at least one block in that storage device that have a higher read temperature than the other rows in the at least one block in that storage device and, based on that identification, generate and store a local logical storage element read temperature map that identifies a subset of logical storage elements associated with that storage device that have a higher read temperature than the other logical storage elements associated with that storage device. A global read temperature identification subsystem coupled to the storage device(s) may then retrieve the local logical storage element read temperature map generated by each of the storage device(s) and use them to generate a global logical storage element read temperature map.

As such, the read disturb effect, which occurs automatically in response to conventional read operations and persists across power cycles, may be leveraged to generate read temperature maps for storage devices and storage systems, thus addressing many of the issues with conventional read temperature identification systems discussed above. As will be appreciated by one of skill in the art in possession of the present disclosure, the local logical storage element read temperature maps of the present disclosure are generated by the storage devices themselves (rather than the host processor/storage processor in the server device and/or storage system in which they are located), limiting read temperature identification operations performed by that host processor/storage processor to the utilization of those local logical storage element read temperature maps to generate a global logical storage element read temperature map in embodiments of the present disclosure. Furthermore, the local logical storage element read temperature maps of the present disclosure may be generated without any knowledge of the type of data being read, the application performing the read operation, or the workload being performed that resulted in the read operation.

The method400begins at block402where storage device(s) determine read disturb information for each of a plurality of blocks in that storage device. During or prior to the method400, the computing device200may be utilized to write data to the storage devices210a,210b, and up to210cin the storage system208, and then read that data from those storage device(s). As will be appreciated by one of skill in the art in possession of the present disclosure, a simplified example of the writing of data to a NAND block in a storage device using the specific examples provided above includes a processing system in the computing device200(e.g., the processing system that provides the global read temperature identification engine204) erasing all of the NAND rows in that NAND block to set each of their cells to the “A” value, and then selectively applying voltages across “vertical” bitlines in the NAND block and one or more “horizontal” NAND wordline(s) in NAND row(s) in that NAND block in order to cause the cells in those one or more NAND wordlines(s) to switch from the “A” value to a value indicated by a higher voltage in that cell (e.g., one of the values “B”, “C”, “D”, “E”, “F”, “G”, and “H” in the example above), resulting in each of those cells in the one or more NAND row(s) in that NAND block storing some number of electrons to provide one of the values “A” “B”, “C”, “D”, “E”, “F”, “G”, or “H” discussed above. As will be appreciated by one of skill in the art in possession of the present disclosure, the selective application of voltages discussed above may include no application of a voltage for a cell that is desired to have an “A” value.

Subsequently, data may be read from a NAND block by determining what values the cells in its NAND rows store. As will be appreciated by one of skill in the art in possession of the present disclosure, a simplified example of the reading of data from a first NAND row in a NAND block in a storage device includes a processing system in the computing device200(e.g., the processing system that provides the global read temperature identification engine204) “selecting” the first NAND row by providing a voltage across the “vertical” bitlines in the NAND block, with the electrons stored in the cells in the first NAND row (i.e., to provide the values discussed above) operating to reduce the current that is sensed at the bottom of the “vertical” bitlines in the NAND block and that is produced in response to the applied voltage (with the sensing of that reduced current operating to identify particular values in the cells in the first NAND row). However, in order to prevent other second NAND rows in that NAND block that are not being read from effecting the current resulting from the voltage provided across the “vertical” bitlines in that NAND block (i.e., in order to ensure the effect on that current by the electrons stored in the cells of the first NAND row may be sensed as discussed above), those second NAND rows are “deselected” by providing a “bypass” voltage across each of those “horizontal” second NAND rows that forces its cell(s) to conduct current on the bitline.

As will be appreciated by one of skill in the art in possession of the present disclosure, the provisioning of that “bypass” voltage across each of the “horizontal” second NAND rows results in the “micro-programming” of those second NAND rows caused by electrons accumulating in those second NAND rows each time the first NAND row is read (i.e., due to the “bypass” voltage used to “deselect” them attracting electrons out of the bitline), which is referred to as the “read disturb effect” herein. Furthermore, one of skill in the art in possession of the present disclosure will appreciate how the read disturb effect/microprogramming/electron accumulation discussed above is higher in second NAND rows that are closer to the first NAND row being read, and the amount of read disturb effect/microprogramming/electron accumulation in NAND rows will depend on the type of SSD technology used in the SSD storage device.

As will be appreciated by one of skill in the art in possession of the present disclosure, while each NAND row includes many cells that may each identify multiple different values, any read of a NAND row operates to read all the cells in that NAND row and, as such, it is the read temperature of the NAND row that is of concern. Furthermore, while it is possible to read a portion of a NAND row (some subset of the NAND wordlines in that NAND row), that read operation will still apply a voltage to that entire NAND row in order to accomplish the read, thus introducing the same read disturb effect in that NAND row that would have occurred if the entire NAND row had been read.

Conventionally, the read disturb effect discussed above is considered a problem that must be corrected, as the microprogramming/electron accumulation in second NAND row(s) adjacent a first NAND row that is read often can cause a desired value in one or more of the cells in those second NAND row(s) to be mistakenly read as a different value, which one of skill in the art in possession of the present disclosure will appreciate results in a number of incorrect or “flipped” bits (i.e., bits that do not match their originally written value) that provide a “fail bit count” and must be corrected (e.g., using the error check portion326bassociated with the data portion326aprovided by the cell in the NAND wordline) to reverse the change in the value read for that cell in that NAND row. However, as discussed in further detail above, the inventors of the present disclosure have discovered that the read disturb effect may be leveraged in order to identify the read temperature of NAND rows in NAND blocks in a manner that eliminates many of the issues present in conventional read temperature identification systems.

As such, subsequent to the writing of data to the storage device(s)210a-210cand the reading of that data from those storage device(s)210a-210c, any or all of those storage device(s)210a-210c/300may operate at block402to determine read disturb information for each of the NAND blocks324included in the NAND die322in the storage subsystem320in that storage device. As will be appreciated by one of skill in the art in possession of the present disclosure, the discussion below of the determination of read disturb information by the storage device300may apply to any of the storage devices210a-210c, and may be performed upon startup, reset, or other initialization of the storage device300, periodically by the storage device during runtime, at the request of a user of the computing device200, and/or on any other schedule or at any other time that would be apparent to one of skill in the art in possession of the present disclosure.

With reference toFIG.5A, in an embodiment of block402, the storage device processing system304in the storage device300may perform read disturb information retrieval operations500that may include the storage device processing system304retrieving, via the storage interface318, read disturb information associated with each of the NAND blocks324included in the NAND die322in the storage subsystem320. The inventors of the present disclosure have developed techniques for retrieving read disturb information that are described in U.S. patent application Ser. No. 17/578,694, filed Jan. 19, 2022, the disclosure of which is incorporated by reference herein in its entirety. However, while the retrieval of read disturb information from each of the NAND rows332in each of the NAND blocks324in the storage subsystem320is described, one of skill in the art in possession of the present disclosure will appreciate how the retrieval of read disturb information for a subset of NAND rows332in a subset of NAND blocks324in the storage subsystem320will fall within the scope of the present disclosure as well (e.g., when a previously “hot” subset of NAND rows in NAND block(s) are being checked to determine whether they are still “hot”).

For example, the read disturb information retrieval operations500performed at block402may include the storage device processing system304in the storage device300accessing each NAND block324to identify a read disturb “signature” for each of a plurality of NAND rows332a-332iin that NAND block324that may be provided by fail bit counts in one or more adjacent NAND rows. With reference toFIG.5B, a specific example of the read disturb information for the NAND row332ein a NAND block324is provided, and illustrates a read disturb signature502provided by fail bit counts for some of the NAND rows332a-332dand332f-332i.

In particular, the read disturb signature502illustrated inFIG.5Bincludes a fail bit count portion502aassociated with the NAND rows332a-332eon a “first side” of the NAND row332e, and a fail bit count portion502bassociated with the NAND rows332f-332ion a “second side” of the NAND row332e. As will be appreciated by one of skill in the art in possession of the present disclosure, the distribution of the read disturb signature502provides a histogram across all the cells in NAND wordlines of the NAND rows (e.g., with some cells in the NAND wordline(s) in the NAND row332frelatively more effected by the read disturb effect than other cells in that NAND wordlines in that NAND row3320, with all of the NAND wordlines in the NAND rows impacted by the read disturb effect to some extent (i.e., due to electron accumulation prior to the attenuation effects discussed above). However, while a particular example is provided, one of skill in the art in possession of the present disclosure will appreciate that other storage subsystem technologies (e.g., SCM storage devices, Dual Data Rate (DDR) storage devices, etc.) provide similar effects (e.g., DDR storage devices experience a “row hammer” effect) that will fall within the scope of the present disclosure as well. As will be appreciated by one of skill in the art in possession of the present disclosure, the fail bit count portion502aillustrates how the NAND row332dexperiences a higher fail bit count than the NAND row332c, which experiences a higher fail bit count than the NAND row332b, and so on due to their relative proximity to the NAND row332e. Similarly, the fail bit count portion502billustrates how the NAND row332fexperiences a higher fail bit count than the NAND row332g, which experiences a higher fail bit count than the NAND row332h, and so on due to their relative proximity to the NAND row332e.

Furthermore, the fail bit count portions502aand502balso illustrate how the NAND row332dexperiences a higher fail bit count than the NAND row332fdue to the NAND row332dseeing the “full” current resulting from the voltage applied to the “vertical” bitlines in the NAND block when performing a read of the NAND row332erelative to the current that is attenuated by the charge in the cell of the NAND row332e(e.g., if the charge in the cell(s) of a NAND row provides a relatively low voltage value (e.g., the “A” value discussed below), the “downstream” NAND rows will see relatively more electrons than the “upstream” NAND rows, while if the charge in the cell(s) of a NAND row provides a relatively high voltage value (e.g., the “H” value discussed below), the “downstream” NAND rows will see relatively less electrons than the “upstream” NAND rows).

With reference toFIG.5C, the simplified representation fromFIG.3Dof how data may be stored in a cell328is reproduced, but with the data storage representation of the cell328inFIG.5Cincluding the graph330with voltage330aon the X-axis and probability504on the Y-axis. The graph330inFIG.5Calso includes voltage/value probabilities504a(provided in solid lines) for each value “A”-“H” available in the cell328that, as can be seen, is highest in the middle of the voltage range for each value “A”-“H”, and reduces to near-zero near the bounds of the voltage range for each value “A”-“H” (e.g., the voltage/value probability for the value “A” is highest midway between “0” and “B REF” and reduces to near-zero at both “0” and “B REF”, the voltage/value probability for the value “B” is highest midway between “B REF” and “C REF” and reduces to near-zero at both “B REF” and “C REF”, and so on).

As will be appreciated by one of skill in the art in possession of the present disclosure, the cell328associated with the graph330inFIG.5Chas experienced the read disturb effect (e.g., it is a cell in one of the NAND rows332b-d,332f, or332ginFIG.5B), and the graph330illustrates a read disturb effect skew504b(provided in dashed lines) that illustrates how the read disturb effect skews the voltage/value probabilities504afor each value “A”-“H” available in the cell328. As discussed above, after a plurality of reads to an adjacent NAND row (e.g., the NAND row332e) that causes the accumulation of charge in a particular NAND row (e.g., the NAND row332d), a desired value in some cells may be mistakenly read as a different value due to the voltage in those cells crossing the reference voltage that defines that different value.

For example,FIG.5Cillustrates how the accumulation of charge in the NAND row332dmay introduce the read disturb effect skew504bfor one or more of the values “A”-“H” that can cause at least a portion of the voltage/value probabilities504afor those values to shift across the reference voltage for an adjacent value. As can be seen inFIG.5C, the read disturb effect skew504bto the voltage/value probability504afor the value “A” causes that voltage/value probability504ato skew past the B REF, and thus some reads of voltages in the cell328that are desired to provide the value “A” will instead mistakenly provide the value “B” (i.e., due to the actual voltage read being between the B REF and the C REF because it was “pushed” in that “direction” due to the read disturb effect). Furthermore, while a single example is provided, one of skill in the art in possession of the present disclosure will appreciate how the read disturb effect skew504bto the voltage/value probability504afor any of the values “B”-“H” can result in the identification of a mistaken value in a similarly manner.

One of skill in the art in possession of the present disclosure will recognize how conventional systems (e.g., SSD storage device firmware and controllers) may utilize software to shift the reference voltages for one or more values in a cell to compensate for this read disturb effect. However, at block402, the storage device processing system304may instead identify this read disturb information for each NAND row in each of a plurality of NAND blocks324in its storage device300, and one of skill in the art in possession of the present disclosure will appreciate how the read disturb signature for each of those NAND rows will differ depending on whether that NAND row has been read a relatively higher number of times (in which case its read disturb signature will include relatively high fail bit counts for its adjacent NAND rows), whether that NAND row has been read a relatively lower number of times (in which case its read disturb signature will include relatively lower fail bit counts for its adjacent NAND rows), whether that NAND row has been read a relatively intermediate number of times (in which case its read disturb signature will include relatively intermediate fail bit counts for its adjacent NAND rows), etc.

Techniques for using fail bit counts that provide read disturb signatures in order to determine read disturb information are described by the inventors of the present disclosure in more detail in U.S. patent application Ser. No. 17/581,882, filed Jan. 22, 2022; and U.S. patent application Ser. No. 17/581,896, filed Jan. 22, 2022; the disclosures of which are incorporated by reference herein in their entirety. However, one of skill in the art in possession of the present disclosure will appreciate that other techniques may be utilized to determine read disturb information while remaining within the scope of the present disclosure. For example, the inventors of the present disclosure have developed techniques for determining read disturb information without the need to explicitly identify failed bit counts, which are described in U.S. patent application Ser. No. 17/581,879, filed Jan. 22, 2022, the disclosure of which is incorporated by reference herein in its entirety.

With reference toFIG.5D, the storage device processing system304may then perform read disturb information storage operations506that include accessing the first memory system308vis the first memory interface306and storing the read disturb information in the local read temperature identification database312. As such, following block402, each of the storage devices210a-210c/300in the computing device200may have determined and stored read disturb information for each NAND row332a-332iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320.

In some embodiments, at block402and prior to or subsequent to storing the read disturb information in the local read temperature identification database312, the storage device processing system304in the storage device300may perform read disturb information isolation operations in order to isolate data in the read disturb information determined for each NAND row332a-332iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320. The inventors of the present disclosure have developed several techniques for isolating read disturb information that are described in U.S. patent application Ser. No. 17/578,694, filed Jan. 19, 2022, the disclosure of which is incorporated by reference herein in its entirety. As described in those patent documents, the read disturb information determined at block402by the storage device processing system304for each NAND row332a-332iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320may include “noise” and/or other information artifacts that are not indicative of the read disturb effect, and thus different isolation techniques may be performed on the read disturb information in order to allow the storage device processing system304to more accurately characterized the read disturb effect for each NAND row332a-332iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320. As such, in some embodiments, the read disturb information stored in the local read temperature identification database312in each storage device210a-210c/300may be isolated read disturb information.

The method400then proceeds to block404where the storage device(s) use the read disturb information to identify a subset of rows in block(s) in that storage device that have higher read temperatures than other rows in the block(s) in that storage device. With reference toFIG.6, in an embodiment of block404, the storage device processing system304may perform read temperature identification operations600that may include accessing the read disturb information stored in the local read temperature identification database312(e.g., via the first memory interface306and the first memory system308), and identifying relative read temperatures of each NAND row332a-332iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320. However, while the read temperature information is described as being stored in the local read temperature identification database312prior to being accessed and used to identify relative read temperatures of NAND rows, one of skill in the art in possession of the present disclosure will recognize that the read temperature information may be used to identify relative read temperatures of NAND rows upon its collection and without storing it in the local read temperature identification database312(e.g., relative read temperatures of NAND rows may be identified “on the fly” as read disturb information is collected at block402) while remaining within the scope of the present disclosure as well.

As discussed above, the read disturb signature determined for each NAND row will differ depending on whether that NAND row has been read a relatively higher number of times, whether that NAND row has been read a relatively lower number of times, whether that NAND row has been read a relatively intermediate number of times, etc. Furthermore, one of skill in the art in possession of the present disclosure will appreciate how NAND rows that have been read a relatively higher number of times may be identified as having a relatively high read temperature, NAND rows that have been read a relatively lower number of times have be identified as having a relatively low read temperature, NAND rows that have been read a relatively intermediate number of times may be identified as having a relatively intermediate temperature, and so on.

As such, in some embodiments of block404, the storage device processing system304may analyze each read disturb signature determined for each NAND row328a-328iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320, and determine whether that read disturb signature identifies a relatively high read temperature (e.g., because the NAND rows adjacent the NAND row for which that read disturb signature was generated experienced relatively high fail bit counts), whether that read disturb signature identifies a relatively low read temperature (e.g., because the NAND rows adjacent the NAND row for which that read disturb signature was generated experienced relatively low fail bit counts), whether that read disturb signature identifies a relatively intermediate read temperature (e.g., because the NAND rows adjacent the NAND row for which that read disturb signature was generated experiences relatively intermediate fail bit counts), and/or whether that read disturb signature identifies other read temperature granularities that are distinguishable from the relatively high read temperature, the relatively low read temperature, and the relatively intermediate read temperature discussed above.

Furthermore, in some embodiments of block404, the storage device300may operate to process current and previously determined read disturb information for a NAND row in order to generate a read temperature for that NAND row, and the inventors of the present disclosure describe techniques for storage devices to generate read temperature in U.S. patent application Ser. No. 17/580,359, filed Jan. 20, 2022; U.S. patent application Ser. No. 17/580,756, filed Jan. 21, 2022; and U.S. patent application Ser. No. 17/580,888, filed Jan. 21, 2022; the disclosures of which are incorporated by reference herein in their entirety.

As will be appreciated by one of skill in the art in possession of the present disclosure, any relative read temperature metrics may be assigned to the relatively high read temperature, relatively low read temperature, relatively intermediate read temperature, and so on, in order to indicate the different read temperatures for each of the NAND rows. For example, the NAND row read temperatures identified as discussed above using NAND row read disturb signatures may not provide exact numerical read temperatures (e.g., as may be provided in conventional read temperature identification systems may operate to record the exact number of reads of a NAND row), but the inventors of the present disclosure have found that the use of the NAND row read disturb signatures to provide NAND row read temperatures as discussed above provide for the accurate identification of relative temperatures of the different NAND rows in a storage subsystem. Furthermore, as read disturb signature analysis becomes more accurate in the future, the inventors of the present disclosure expect that any particular read disturb signature may then be associated with a number of reads of a NAND row, and thus envision doing so while remaining within the scope of the present disclosure as well.

Many of the embodiments discussed below describe the use of the read temperatures determined by any storage device as described above to generate local logical storage element read temperature map for that storage device. However, as also described in further detail below, storage devices may identify and use read-disturb-based read temperatures for a single block to move data without the need to generate a multi-block local logical storage element read temperature map, which one of skill in the art in possession of the present disclosure will appreciate requires less complex storage device/storage subsystem processing/functionality while still allowing several benefits of the read-disturb-based read temperatures identified as described herein to be realized.

The method400then proceeds to block406where each of the storage device(s) generate a local logical storage element read temperature map identifying a subset of logical storage elements that are associated with that storage device and that have higher read temperatures than other logical storage elements associated with that storage device, as well as to block408where the storage device(s) store the local storage element read temperature map generated by that storage device. The inventors of the present disclosure have developed several techniques for generating and/or maintaining a local logical storage element read temperature map that are described in U.S. patent application Ser. No. 17/581,874, filed Jan. 22, 2022; U.S. patent application Ser. No. 17/581,785, filed Jan. 21, 2022; U.S. patent application Ser. No. 17/581,677, filed Jan. 21, 2022; U.S. patent application Ser. No. 17/579,988, filed Jan. 20, 2022; the disclosures of which are incorporated by reference herein in their entirety. As described in those patent documents, local logical storage element read temperature maps may be generated by mapping read temperatures identified for physical storage to a logical-to-physical mapping (e.g., via a “reverse lookup”), and may be maintained by identifying any historical read temperature(s) associated with a logical storage element when its data is moved to a new storage element (i.e., by writing that data to a physical storage location mapped to that new storage element), and then mapping those historical read temperatures to that new storage element in the logical-to-physical mapping as well (and in some cases, persistently storing those historical read temperatures in the NAND block that stores the corresponding data, in a metadata NAND block in the storage subsystem320, and/or in other storage locations that would be apparent to one of skill in the art in possession of the present disclosure).

In an embodiment, at block406, the storage device processing system304in the storage device300may generate a local logical storage element read temperature map using the relative read temperatures identified for each NAND row332a-332iincluded in each of its NAND blocks324provided by each of its NAND dies322in its storage subsystem320at block404. In an embodiment, the storage device300may utilize Logical Block Addressing (LBA), which one of skill in the art in possession of the present disclosure will recognize logically divides the storage subsystem320into logical storage elements (e.g., 512 byte to 4096 byte sectors), with the first logical storage element identified as logical block 0, the second logical storage element identified as logical block 1, and so on, and with each logical storage element mapped to a respective NAND row332a-332iincluded in the NAND blocks324provided by the NAND dies322in the storage subsystem320.

As such, block406may include the storage device processing system304mapping, in a local logical storage element read temperature map, the read temperature determined for each NAND row at block404to the logical storage element that is mapped to that NAND row. Thus, continuing with the example provided above, a read temperature identified for a first NAND row may be mapped, in the local logical storage element read temperature map, to an LBA block 0 that is mapped to the first NAND row; a read temperature identified for a second NAND row may be mapped, in the local logical storage element read temperature map, to an LBA block 1 that is mapped to the second NAND row; a read temperature identified for a third NAND row may be mapped, in the local logical storage element read temperature map, to an LBA block 2 that is mapped to the third NAND row; and so on until a read temperature is mapped to each of the LBA blocks.

In a specific example, any NAND row may be mapped to one or more logical storage elements, and in the event a NAND row has a particular read temperature, each logical storage element mapped to that NAND row will have that particular read temperature. As such, the present disclosure may provide read temperature granularity at the NAND row level. Furthermore, while most storage device implementations today map logical blocks to NAND rows such that each logical block is fully contained within that NAND row, one of skill in the art in possession of the present disclosure will appreciate that a logical block may “straddle” multiple NAND rows, and in such cases read temperatures of that logical block may be computed by combining the read temperatures determined for those multiple NAND rows using any of variety of techniques that would be apparent to one of skill in the art in possession of the present disclosure. Thus, following block406, each of the storage devices210a-210c/300may have generated a respective local logical storage element read temperature map, and at block408each of the storage devices210a-210c/300may have stored that local logical storage element read temperature map in its local read temperature identification database312.

With reference toFIG.7A, an example of a local logical storage element read temperature map700is illustrated that may have been generated and stored by a first of the storage devices210a-210c/300in its local read temperature identification database312. In the illustrated example, the local logical storage element read temperature map700includes relatively high read temperature logical storage elements700a(illustrated as black boxes in the local logical storage element read temperature map700), relatively low read temperature logical storage elements700b(illustrated as white boxes in the local logical storage element read temperature map700), relatively high-intermediate read temperature logical storage elements700c(illustrated as dark grey boxes in the local logical storage element read temperature map700), and relatively low-intermediate read temperature logical storage elements700d(illustrated as light grey boxes in the local logical storage element read temperature map700). However, one of skill in the art in possession of the present disclosure will recognize that the inclusion of other levels of read temperature granularity in local logical storage element read temperature maps will fall within the scope of the present disclosure as well. As will be appreciated by one of skill in the art in possession of the present disclosure, the dashed line inFIG.7Ais provided to indicate that the local logical storage element read temperature map700is one of a plurality of local logical storage element read temperature maps that provide the total logical storage space for the computing device200.

With reference toFIG.7B, an example of a local logical storage element read temperature map702is illustrated that may have been generated and stored by a second of the storage devices210a-210c/300in its local read temperature identification database312. In the illustrated example, the local logical storage element read temperature map702includes relatively high read temperature logical storage elements702a(illustrated as black boxes in the local logical storage element read temperature map702), relatively low read temperature logical storage elements702b(illustrated as white boxes in the local logical storage element read temperature map702), relatively high-intermediate read temperature logical storage elements702c(illustrated as dark grey boxes in the local logical storage element read temperature map70), and relatively low-intermediate read temperature logical storage elements702d(illustrated as light grey boxes in the local logical storage element read temperature map702). However, one of skill in the art in possession of the present disclosure will recognize that the inclusion of other levels of read temperature granularity in local logical storage element read temperature maps will fall within the scope of the present disclosure as well. As will be appreciated by one of skill in the art in possession of the present disclosure, the dashed line inFIG.7Bis provided to indicate that the local logical storage element read temperature map702is one of a plurality of local logical storage element read temperature maps that provide the total logical storage space for the computing device200. Furthermore, while examples of only two local logical storage element read temperature maps700and702generated by two storage devices are provided, one of skill in the art in possession of the present disclosure will appreciate that storage systems may include many more storage devices, and each of those storage devices may generate a local logical storage element read temperature map while remaining within the scope of the present disclosure as well.

While the embodiments discussed below describe the use of local logical storage element read temperature maps from different storage devices to generate a global logical storage element read temperature map, the inventors of the present disclosure have developed techniques for storage devices to use their local logical storage element read temperature map to move data that are described in U.S. patent application Ser. No. 17/579,689, filed Jan. 20, 2022, the disclosure of which is incorporated by reference herein in its entirety. Furthermore, the global read temperature identification engine204(or other host subsystem in the computing device200) may operate to adjust read temperatures included in the local logical storage element read temperature map based on data characteristics of the data stored in corresponding logical storage elements, and the inventors of the present disclosure describe techniques for host subsystem read temperature adjustments in U.S. patent application Ser. No. 17/580,756, filed Jan. 21, 2022; and U.S. patent application Ser. No. 17/580,888, filed Jan. 21, 2022; the disclosures of which are incorporated by reference herein in their entirety.

The method400then proceeds to block410where a global read temperature identification subsystem retrieves the local logical storage element read temperature map(s) generated by the storage device(s). With reference toFIG.8A, in an embodiment of block410, the global read temperature identification engine204in the computing device200may perform local logical storage element read temperature map retrieval operations800in order to retrieve the local logical storage element read temperature maps generated and stored by the storage devices210a,210b, and up to210cat blocks406and408. The inventors of the present disclosure have developed several techniques for accessing and utilizing local logical storage element read temperature maps, information provided therein, and/or associated information, which are described in U.S. patent application Ser. No. 17/579,282, filed Jan. 19, 2022; and U.S. patent application Ser. No. 17/579,020, filed Jan. 19, 2022; the disclosures of which are incorporated by reference herein in their entirety.

As will be appreciated by one of skill in the art in possession of the present disclosure, in some examples the global read temperature identification engine204in the computing device200may access the local read temperature identification databases312in the first memory system308in each of the storage devices210a-210c/300in order to retrieve the local logical storage element read temperature maps stored therein, while in other embodiments the global read temperature identification engine204in the computing device200may provide requests for those local logical storage element read temperature maps such that each of the storage devices210a-210c/300transmit them to the global read temperature identification engine204.

The method400then proceeds to block412where the global read temperature identification subsystem uses the local logical storage element read temperature map(s) to generate a global logical storage element read temperature map, as well as to optional block414where the global read temperature identification subsystem stores the global logical storage element read temperature map. In some embodiment, at block412, the global read temperature identification engine204in the computing device200may concatenate the local logical storage element read temperature maps retrieved from the storage devices210a-210c/300to generate a global logical storage element read temperature map that it then stores in the global read temperature identification database206. For example,FIG.8Billustrates an example of a global logical storage element read temperature map802that may have been generated by the global read temperature identification engine204via concatenation of the local logical storage element read temperature maps700and702discussed above with reference toFIGS.7A and7B, and then stored by the global read temperature identification engine204in its global read temperature identification database312.

However, one of skill in the art in possession of the present disclosure will appreciate how the generation of a global logical storage element read temperature map via concatenation of local logical storage element read temperature maps provides a simplified example of the use of local logical storage element read temperature maps to generate a global logical storage element read temperature map, and that the local logical storage element read temperature maps discussed above may be utilized to generate the global logical storage element read temperature map in other manners that will fall within the scope of the present disclosure as well. For example, the generation of the global logical storage element read temperature map using the local logical storage element read temperature maps may depend on how the global read temperature identification engine204is configured to track read temperatures, how the global read temperature identification engine204is configured to organize data (e.g., data may be “striped” across the storage devices210a-210c), and/or based on other factors that would be apparent to one of skill in the art in possession of the present disclosure.

For example, in some embodiments the computing device200may store data in the storage devices210a-210bindependently of each other (e.g., as the logical blocks discussed above), in which case the generation of the global logical storage element read temperature map via concatenation of local logical storage element read temperature maps may be appropriate. However, in other embodiments, the computing device200may utilize more complex software that organizes the storage of the data in the storage devices210a-210cin “groups” of logical blocks. For example, for performance considerations a data group of data A, B, and C may be provided by three respective logical blocks, and may be written to each of three respective storage devices, but the computing device200may view that data group as “atomic” such that the read temperature that matters is the read temperature of that data group. In such an embodiment, a “higher level” “group global logical storage element read temperature map” may be generated in order to allow the tracking of data group read temperatures, and the inventors of the present disclosure are developing techniques for doing so. One example of such an embodiment is a Redundant Array of Independent Drives (RAID) storage system, but one of skill in the art in possession of the present disclosure will appreciate that other storage systems may introduce similar considerations as well.

Furthermore, the inventors of the present disclosure have developed techniques for generating global logical storage element read temperature maps using local logical storage element read temperature maps which are described in U.S. patent application Ser. No. 17/579,020, filed Jan. 19, 2022, the disclosure of which is incorporated by reference herein in its entirety. As described in that patent document, data in local logical storage element read temperature maps generated by different storage devices may be scaled relative to each other so that the read temperatures of storage locations in different storage devices with different utilizations (e.g., a first storage device with one million reads and a second storage device with ten million reads) may be accurately compared relative to each other (i.e., “hot” storage locations in the first storage device with one million reads may not be “hot” relative to “hot” storage locations in the second storage device with ten million reads).

In some embodiments, a subset of the local logical storage element read temperature maps generated by the storage devices210a-210cmay be retrieved and used by the global read temperature identification engine204at block412. For instance, if the memory system in the computing device300does not have sufficient space to store the global logical storage element read temperature map (or for other reasons that memory space is allocated for the storage of other data), a global logical storage element read temperature map may be generated that only identifies logical storage elements with relatively “hot” red temperatures. As such, in some embodiments, the local logical storage element read temperature maps retrieved from the storage devices210a-210cmay only identify logical storage elements having a particular read temperature (e.g., those with relatively “hot” read temperatures), allowing for the generation of the global logical storage element read temperature map identifying logical storage elements with that particular temperature as well. However, in other embodiments, the global read temperature identification engine204may be configured to retrieve the local logical storage element read temperature map(s) from the storage devices210a-210c, and then utilize a filter to generate a filtered global logical storage element read temperature map that identifies particular read temperatures from the local logical storage element read temperature map(s).

In an embodiment, following block412, the computing device200(e.g., the processing system in the computing device200) may utilize the global logical storage element read temperature map (which may have been stored in the global read temperature identification database206) in order to provide for the storage of data, movement of data, and/or other data operations that would be apparent to one of skill in the art in possession of the present disclosure. For example, data stored in NAND rows with relatively high read temperatures may be moved to relatively high capability/cost storage devices, data stored in NAND rows with relatively low read temperatures may be moved to relatively low capability/cost storage devices, data stored in NAND rows with relatively intermediate read temperatures may be moved to relatively intermediate capability/cost storage devices, etc. In another example, the computing device200(e.g., the processing system in the computing device200) may utilize the global logical storage element read temperature map in order to perform load balancing (e.g., when the storage devices210a-210care the same type of capability/cost storage device, load balancing reads to those storage devices can result in a higher performing storage system (relative to the performance of that storage system without the load balancing). However, while a few specific examples of operations based on identified read temperatures have been described, one of skill in the art in possession of the present disclosure will appreciate how the read temperatures identified in the global logical storage element read temperature map may be utilized to perform any of a variety of read-temperature-based operations while remaining within the scope of the present disclosure as well.

As will be appreciated by one of skill in the art in possession of the present disclosure, global read temperature identification engine204that created the global logical storage element read temperature map802may perform any of the read-temperature-based operations discussed above. However, one of skill in the art in possession of the present disclosure will also recognize that other subsystems in the computing device200(i.e., other than the global read temperature identification engine204) may perform the read-temperature-based operations discussed above while remaining within the scope of the present disclosure. Furthermore, subsystems outside the computing device200may perform the read-temperature-based operations discussed above while remaining within the scope of the present disclosure as well. As such, access to the global logical storage element read temperature map802and/or the local logical storage element read temperature maps700and702may be provided to subsystems other than the storage devices and global read temperature identification engine204discussed above, allowing the information stored therein (as well as information used to generated those maps) to be utilized by those other subsystems in any of a variety of manners that will be apparent to one of skill in the art in possession of the present disclosure.

Thus, systems and methods have been described that utilize the read disturb effect that produces noise in adjacent NAND rows when any particular NAND row is read to identify NAND rows that are read more often than other NAND rows and thus have higher “read temperatures” than those other NAND rows. For example, the read-disturb-based read temperature identification system of the present disclosure may include storage device(s) that each determine read disturb information for each block in that storage device, use that read disturb information to identify a subset of rows in at least one block in that storage device that have a higher read temperature than the other rows in the at least one block in that storage device and, based on that identification, generate and store a local logical storage element read temperature map that identifies a subset of logical storage elements associated with that storage device that have a higher read temperature than the other logical storage elements associated with that storage device. A global read temperature identification subsystem coupled to the storage device(s) may then retrieve the local logical storage element read temperature map generated by each of the storage device(s) and use them to generate a global logical storage element read temperature map.

As such, the read disturb effect that happens automatically in response to conventional read operations and that persists across power cycles may be leveraged to generate read temperature maps for storage devices and storage systems, thus addressing many of the issues with conventional read temperature identification systems discussed above. As will be appreciated by one of skill in the art in possession of the present disclosure, systems and methods of the present disclosure allow a determination of the relative read temperatures of data within storage devices by the storage device themselves (i.e., without requiring processing cycles of a host processor in the server device and/or storage system in which they are located), and with the advent of Storage Class Memory (SCM) devices and low-cost NAND devices that is causing the storage landscape to fracture further than it already has, the opportunity and value associated with placing particular data in the most efficient storage media has increased, and may be realized with the novel read-disturb-based read temperature identification techniques described herein.

Referring now toFIG.9, an embodiment of the storage device300discussed above with reference toFIGS.3A-3Eis illustrated. In the embodiments illustrated and discussed below, the first memory system308in the storage device300that is coupled to the storage device processing system304may include instructions that, when executed by the storage device processing system304, cause the storage device processing system304to provide a local read temperature utilization engine900that is configured to perform the functionality of the local read temperature utilization engines and/or storage devices discussed below. As discussed in further detail below, the local read temperature utilization engine900may be configured to move data within the storage subsystem320using the read temperatures for a single NAND block324that stores that data and that are generated by the storage device300based on the read disturb effect as discussed above with reference to the method400. In some examples, the local read temperature identification engine204discussed above and the local read temperature utilization engine900described below may be integrated as part of the same engine, although systems with separate local read temperature identification engines and local read temperature utilization engines are envisioned as falling within the scope of the present disclosure as well.

As also illustrated in the specific examples provided herein, the first memory system308in the storage device300may also include a local read temperature utilization database902that is configured to store any of the information utilized by the local read temperature utilization engine900discussed below. In some examples, the local read temperature identification database312discussed above and the local read temperature utilization database902described below may be integrated as part of the same database, although systems with separate local read temperature identification databases and local read temperature utilization databases are envisioned as falling within the scope of the present disclosure as well. However, while a specific storage device300has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that storage devices (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the storage device300) may include a variety of components and/or component configurations for providing conventional storage device functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now toFIG.10, an embodiment of a method1000for storage device utilization of read-disturb-based block read temperatures is illustrated. As discussed below, the systems and methods of the present disclosure provide for the utilization of a read-disturb-based block read temperatures for a block in a storage device in order to move data from that block and within that storage device. For example, the storage device read-disturb-based block read temperature utilization system of the present disclosure may include a storage device chassis housing a storage subsystem. A local read temperature utilization subsystem in the storage device chassis determines that data in a first block in the storage subsystem should be moved and, in response, determines read disturb information for the first block, uses it to identify relative read temperatures for a plurality of rows in the first block in the storage subsystem, and then moves the data from the first block in the storage subsystem to at least one second block in the storage subsystem based on the relative read temperatures identified for the plurality of rows in the first block in the storage subsystem. As such, a storage device may move data internally (i.e., between blocks in its storage subsystem) based on read temperatures it generated for that data in a block according to the teachings of the present disclosure, which allows several of the benefits of the read-disturb-based read temperatures discussed above to be realized without the associated processing overhead/functionality requirements associated with generating the local logical storage element read temperature maps discussed above, allowing for more efficient placement of that data within the storage device during garbage collection operations, storage endurance optimization operations, read performance optimization operations, and/or other storage device operations that would be apparent to one of skill in the art in possession of the present disclosure.

The method1000begins at decision block1002where it is determined whether data should be moved in a storage device. In an embodiment, at block1002, the local read temperature utilization engine900in the storage device300may determine whether data in the storage subsystem320should be moved. While specific examples of determining that data should be moved due to garbage collection operations, storage endurance optimization operations, and read performance optimization operations are described below, one of skill in the art in possession of the present disclosure will appreciate how other scenarios that require data movement may be determined at decision block1002while remaining within the scope of the present disclosure as well. Furthermore, one of skill in the art in possession of the present disclosure will appreciate how another subsystem may initiate the operations that require data movement (e.g., garbage collection operations, storage endurance optimization operations, and read performance optimization operations discussed below may be initiated by a “host” subsystem in the computing device200, a Flash Translation Layer (FTL) in the storage device or in the “host” subsystem, etc.), and the local read temperature utilization engine900in the storage device300may determine how data must be moved at block1002as part of those operations. If, at decision block1002, it is determined that data should not be moved in the storage subsystem302, the method1000returns to decision block1002. As such, the method1000may loop at decision block1002to monitor whether data should be moved in the storage subsystem320(e.g., the read-disturb-based block read temperature determination and utilization operations discussed below may be performed any time data is moved within the storage device300).

If, at decision block1002, it is determined that data should be moved in the storage subsystem320, the method1000then proceeds to block1004where the storage device determines read disturb information for a first block in the storage device that stores the data that will be moved. As discussed above, at decision block1002, the local read temperature utilization engine900in the storage device300may determine that data should be moved in the storage subsystem320due to garbage collection operations. As will be appreciated by one of skill in the art in possession of the present disclosure, garbage collection operations for storage devices such as the Solid State Drive (SSD) storage device discussed above operate to optimize space, improve write performance for the storage device by pro-actively eliminating the need for whole NAND block erasure prior to every write operation, and improve efficiency of the storage device.

For example, as “valid” data stored in a first NAND block is updated (e.g., a newer version of that data is written to a second NAND block) or deleted, that “valid” data becomes “obsolete” data that remains in the first NAND block, and garbage collection operations may operate to identify the first NAND block when it has a threshold amount of “obsolete” data, move any “valid” data remaining in the first NAND block to a second NAND block, and then erase the first NAND block so that data may later be relatively quickly written to that first NAND block (without the need to perform the erasure of the NAND block at the time of the write operation). As such, garbage collection operations may involve moving any “valid” data in a NAND block324upon which garbage collection operations are being performed. In an embodiment, the local read temperature utilization engine900in the storage device300may perform garbage collection operations as a background process in the storage device300, and may determine at decision block1002that data should be moved as part of those garbage collection operations.

As also discussed above, at decision block1002, the local read temperature utilization engine900in the storage device300may determine that data should be moved in the storage subsystem320due to storage endurance optimization operations. As will be appreciated by one of skill in the art in possession of the present disclosure, storage endurance optimization operations for storage devices such as the SSD storage device discussed above operate to place data within the storage subsystem320of the storage device300in a manner that increases the “endurance” or life of the storage device300by, for example, reducing write amplification in the storage device300. For example, each time data is provided by a host or other subsystem for writing to the storage device300, the storage device300may have to perform multiple write operations to move “valid” data, erase “obsolete” data, and/or perform other data writes that allow the data provided by the host to be written to the storage device300. As will be appreciated by one of skill in the art in possession of the present disclosure, the number of times data may be written to and erased from the storage subsystem320in the storage device300is limited, and by reducing the write amplification discussed above the “endurance” or life of the storage device300may be extended. As such, data movement may be performed in a manner that reduces the write amplification discussed above by placing data in the storage subsystem320in a manner that attempts to minimize the number of write operations that will be performed by the storage device300in response to any single write request from a host.

Furthermore, as the number of reads from a NAND block324increase, the ability to perform error correction for that data reduces (e.g., as reads increase, errors increase, and because Error Correction Code (ECC) provides a limited amount of data to correct errors, the capabilities of ECC to correct errors may be exceeded after a threshold number of reads). As such, storage endurance may be optimized by ensuring that any movement/placement of data within the storage subsystem320will provide that data in a location from which it will not be moved for as long as possible (e.g., due to read disturb issues, error correction limits being reached/exceeded, etc.) As will be appreciated by one of skill in the art in possession of the present disclosure, each time data is read from a NAND block324, the error correction information associated with that data may indicate how reliable that data is and may require movement of that data in the event its reliability drops below a reliability threshold. As such, the local read temperature utilization engine900may operate to determine that the data in a first NAND block in the storage subsystem320has reached a reliability threshold and, in response, determined at decision block1002that the data should be moved from the first NAND block in the storage subsystem320to at least one second NAND block in the storage subsystem320. In an embodiment, the local read temperature utilization engine900in the storage device300may perform storage endurance optimization operations as a background process in the storage device300, and may determine at decision block1002that data should be moved as part of those storage endurance optimization operations.

As also discussed above, at decision block1002, the local read temperature utilization engine900in the storage device300may determine that data should be moved in the storage subsystem320due to read performance optimization operations. As will be appreciated by one of skill in the art in possession of the present disclosure, read performance optimization operations for storage devices such as the SSD storage device discussed above operate to place data within the storage subsystem320of the storage device300in order to reduce data retrieval delays due to storage subsystem bandwidth issues. For example, the storage subsystem320may introduce limits to concurrent accesses of NAND blocks324due to the bandwidth available via the physical channel provided to the NAND die322that includes those NAND blocks324. As such, if too many read requests are received for data stored in a NAND block324in the same NAND die322, a queue may be created so that reads of any particular NAND row332in that NAND block324may be made sequentially via the physical channel provided to the NAND die322, thus delaying read operations placed further down in the queue. In an embodiment, the local read temperature utilization engine900in the storage device300may perform read performance optimization operations as a background process in the storage device300, and may determine at decision block1002that data should be moved as part of those read performance optimization operations.

However, while several specific examples of scenarios where data must be moved within the storage subsystem320have been described, one of skill in the art in possession of the present disclosure will appreciate how other data movement scenarios will fall within the scope of the present disclosure as well. For example, one of skill in the art in possession of the present disclosure will appreciate how the block-level read-disturb-based read temperature identification and utilization discussed below may benefit from the provisioning of a respective counter with each NAND block324in the storage subsystem320that provides for the identification of a number of times any particular NAND block324is read, and allows the block read temperatures of any NAND block324to be determined when that NAND block324reaches a read threshold. As such, the local read temperature utilization engine900in the storage device300may operate to maintain counters (e.g., in the first memory system308) for each of its NAND blocks324, and then determine at block1002that data should be moved from a NAND block324based on that counter for that NAND block exceeding a read threshold (e.g., exceeding a total read threshold, exceeding a mean number of reads identified by the NAND block counters by a standard deviation, etc.)

In an embodiment, at block1004and in response to determining that data should be moved within the storage subsystem320, the local read temperature utilization engine900in the storage device300may operate similarly as described above with reference to block402of the method400in order to determine read disturb information for the NAND block324in the storage subsystem320that stores that data. As will be appreciated by one of skill in the art in possession of the present disclosure, the method1000may be performed by any of the storage devices210a-210cin the storage system208of the computing device200in response to determining data should be moved within the storage subsystem320in that storage device300at decision block1002. As such, block1004may be performed in response to determining that data should be moved within the storage subsystem320in a storage device300as part of garbage collection operations, storage endurance optimization operations, read performance optimization operations, other storage device operations, and/or in other scenarios that would be apparent to one of skill in the art in possession of the present disclosure. Thus, following block1004, the storage device300may have determined read disturb information for a plurality of the NAND rows332(and in some cases, all of the NAND rows332) in the NAND block324in its storage subsystem320that stores the data that will be moved (as determined at decision block1002).

The method1000then proceeds to block1006where the storage device uses the read disturb information to identify relative read temperatures of rows in the first block of the storage device. In an embodiment, at block1006, the local read temperature utilization engine900in the storage device300may operate similarly as described above with reference to block404of the method400in order to identify a subset of NAND rows332in the NAND block324in the storage subsystem320of the storage device300(which stores the data being moved) that have higher read temperatures than other NAND rows332in that NAND block324in the storage subsystem320of the storage device300. As discussed in detail above, the read disturb information determined at block1004allows the local read temperature utilization engine900in the storage device300to determine the relative read temperatures of the NAND rows332in the NAND block324in the storage subsystem320of the storage device300that stores the data that will be moved (as determined at decision block1002), which in the examples herein may include NAND rows332with relatively “hot” read temperatures, NAND rows332with relatively “high intermediate” read temperatures, NAND rows332with relatively “low intermediate” read temperatures, and NAND rows332with relatively “low” read temperatures. However, while four particular relative read temperatures are described in the examples herein, different read temperature granularities (e.g., two read temperatures: “hot” and “cold”, three read temperatures, more than four read temperatures, etc.) are envisioned as falling within the scope of the present disclosure as well. As such, following block1004, a plurality of NAND rows332in the NAND block324in the storage subsystem320of the storage device300that stores the data being moved may be associated with a read temperature.

With reference toFIG.11, a NAND block1100is illustrated with a plurality of NAND rows1100a,1100b,1100c,1100d,1100e,1100f,1100g,1100h, and up to1100iin that NAND block1100, and with each NAND row1100a-1100ihaving been associated with a read temperature at block1006. In the specific example illustrated inFIG.11, the NAND rows1100dand1100ghave been associated with a relatively “high” read temperatures (illustrated as black boxes in the block read temperature map inFIG.11), the NAND rows1100aand1100fhave been associated with a relatively “high-intermediate” read temperatures (illustrated as dark grey boxes in the block read temperature map inFIG.11), the NAND rows1100eand1100ihave been associated with a relatively “low-intermediate” read temperatures (illustrated as light grey boxes in the block read temperature map inFIG.11), and the NAND rows1100b,1100c, and1100hhave been associated with a relatively “low” read temperatures (illustrated as white boxes in the block read temperature map inFIG.11). However, while a specific example of NAND rows in a NAND block associated with read temperatures has been illustrated and described, one of skill in the art in possession of the present disclosure will appreciate how read temperatures may be associated with the NAND rows included in a NAND block in a variety of manners that will fall within the scope of the present disclosure as well (e.g., via an association with the logical storage elements that point to those NAND rows similarly as discussed above). As such, following block1008, the read temperatures of data stored in NAND rows in the NAND block324in the storage subsystem320of the storage device300(which stores the data that will be moved as determined at decision block1002) may be stored in the local read temperature utilization database902.

The method1000then proceeds to block1008where the storage device moves data from the first block in the storage device to second block(s) in the storage device based on relative read temperatures identified for the rows in the first block in the storage device. In some embodiments, data may be moved at block1008from the NAND block324into different NAND subsystem groups within the storage subsystem320as part of the storage endurance optimization operations described above, and one of skill in the art in possession of the present disclosure will appreciate that by separating the relatively “high” read temperature data and the relatively “low” read temperature data from that NAND block324into different NAND subsystems will improve the “endurance” or lifespan of the storage device300by, for example, placing relatively “high” read temperature data in different NAND blocks than relatively “low” read temperature data (e.g., with later movement of the relatively “high” read temperature data to a different NAND block not requiring movement of the relatively “low” read temperature data that is stored on different NAND block(s), thus reducing the number of writes utilized with the relatively “low” read temperature data).

In some embodiments, data may be moved at block1008from the NAND block324into different NAND subsystem groups within the storage subsystem320as part of the read performance optimization operations described above, and one of skill in the art in possession of the present disclosure will appreciate that by storing the relatively “high” read temperature data from the NAND block324across different NAND subsystems will improve the read performance of the storage device300by, for example, placing relatively “high” read temperature data from the NAND block324on different NAND die322(e.g., splitting up the relatively “high” read temperature data in the storage subsystem320across different NAND die322such that the bandwidth of the physical channels to those NAND die322will not be exceeded when concurrent accesses of that relatively “high” read temperature data are performed), or placing relatively “high” read temperature data from the NAND block324on different NAND blocks324(e.g., splitting up the relatively “high” read temperature data in the storage subsystem320across different NAND blocks324to prevent “read contention”/degraded Quality of Service (QoS) that would otherwise result if that relatively “high” read temperature data were all stored on the same NAND block324).

As such, the local read temperature utilization engine900may determine that the read performance of the storage subsystem320is below a read performance threshold and, in response, may move data from the “first” NAND block324in the storage subsystem320to second NAND block(s) in the storage subsystem320such that the read performance of the storage subsystem320is improved. As will be appreciated by one of skill in the art in possession of the present disclosure, such data movement operations performed as part of read performance optimization operations may be configured to distribute the data across the different NAND die to optimize the average read response time, QoS, and/or other performance characteristics of the storage device300that would be apparent to one of skill in the art in possession of the present disclosure. In some embodiments, different data that is accessed often (e.g., relatively “high” read temperature data) may be correlated and then placed in NAND blocks in different NAND die322as part of the read performance optimization operations. Furthermore, while the discussion above focuses on the optimization of read response, one of skill in the art in possession of the present disclosure will appreciate how other storage device characteristics (e.g., wear) may be optimized (e.g., to reduce wear on the storage device) in a similar manner while remaining within the scope of the present disclosure as well.

With reference toFIG.12, a specific embodiment of data movement operations at block1008is illustrated using the read temperatures determine for the NAND rows1100a-1100iin the NAND block1100. As can be seen, data stored the NAND rows1100a-1100iin the NAND block1100may be moved into a different NAND block1200in the storage subsystem320of the storage device300in order to group similar read temperature data adjacent each other in the NAND block1200. For example,FIG.12illustrates how the relatively “high” read temperature data in the NAND rows1100dand1100gin the NAND block1100may be moved to adjacent NAND rows1200aand1200bin the NAND block1200; the relatively “high-intermediate” read temperature data in the NAND rows1100aand1100fin the NAND block1100may be moved to adjacent NAND rows1200cand1200din the NAND block1200; the relatively “low-intermediate” read temperature data in the NAND rows1100eand1100iin the NAND block1100may be moved to adjacent NAND rows1200eand1200fin the NAND block1200; and the relatively “low” read temperature data in the NAND rows1100b,1100c, and1100hin the NAND block1100may be moved to adjacent NAND rows1200g,1200h, and1200iin the NAND block1200. Furthermore, whileFIG.12illustrates movement of data from a first NAND block to a second NAND block, as described herein, data from a first NAND block may be moved into a plurality of different second NAND blocks while remaining within the scope of the present disclosure as well.

In some embodiments, the movement of different read temperature data into different adjacent NAND row groups in a NAND block may be performed as part of the storage endurance optimization operations described above, and one of skill in the art in possession of the present disclosure will appreciate that by placing the relatively “high” read temperature data in the adjacent NAND rows1200aand1200b, the relatively “high-intermediate” read temperature data in the adjacent NAND rows1200cand1200d, the relatively “low-intermediate” read temperature data in the adjacent NAND rows1200eand1200f, and the relatively “low” read temperature data in the adjacent NAND rows1200gand1200h, will allow the movement of the relatively “high” read temperature data (e.g., when associated error correction limits are reached) without the need to move some or all of the relatively lower read temperature data stored in the NAND block1200.

With reference toFIG.13, a specific embodiment of data movement operations at block1008is illustrated using read temperatures determined for the NAND rows1100a-1100iin the NAND block1100. As can be seen, data stored in subsets of the NAND rows1100a-1100iin the NAND block1100may be moved into a different NAND block1300in the storage subsystem320of the storage device300in order to group similar read temperature data in the NAND block1300. For example,FIG.13illustrates how the relatively “high” read temperature data in the NAND rows1100dand1100gand the relatively “high-intermediate” read temperature data in the NAND rows1100aand1100fin the NAND block1100may be moved to NAND rows1300a,1300b,1300c, and1300d, respectively, in the NAND block1300. Furthermore, one of skill in the art in possession of the present disclosure will appreciate how the relatively “low-intermediate” read temperature data in the NAND rows1100eand1100iand the relatively “low” read temperature data in the NAND rows1100aand1100fin the NAND block1100may be moved to NAND rows in a different NAND block in the storage subsystem320. As discussed above, the grouping of data with the same read temperatures (or similar read temperatures in the case of the grouping of the “high” and “high-intermediate” read temperature data, and the “low” and “low-intermediate” read temperature data in the example above) in different NAND blocks can operate to reduce write amplification issues and optimize endurance/life span of the storage device300.

In some embodiments, the movement of different read temperature data into different NAND blocks may be performed as part of the garbage collection operations described above, and one of skill in the art in possession of the present disclosure will appreciate that by placing the relatively “high” read temperature data from the NAND block324in a high performance NAND block will improve the read performance of the storage device300(e.g., reads of more frequently read data will be quicker because that data is now stored in a NAND block that provide for faster reads). As such, data movements performed as part of garbage collection operations may leverage the read temperatures of the NAND rows in the NAND block that stores that data in order to place data within the storage device300in a manner that improves the efficiency, performance, life, and/or other characteristics of the storage device300.

Thus, systems and methods have been described that provide for a simplified implementation of the teachings of the present disclosure in which read-disturb-based read temperature identification is performed for an individual block in a storage subsystem of a storage device when data stored in that individual block must be moved. As will be appreciated by one of skill in the art in possession of the present disclosure, the read temperature determinations at the storage subsystem block level discussed above may prevent the ability to move data based on relative read temperatures across different blocks in the storage subsystem, but still offer several benefits with regard to the movement of data from any particular block in the storage subsystem to one or more other blocks of the storage subsystem.