Patent Description:
Persistent storage devices employing flash memory (e.g., not-AND (NAND) flash memory) may perform read and write operations with page granularity and erase operations with block granularity, which may be significantly coarser. This difference in granularity may lead to write amplification, for example, in a garbage collection operation in which valid data remaining in a block are moved to another block before the block is erased so that it may be reused. The cells of flash memory may have limited endurance, e.g., they may degrade gradually with repeated erase and write cycles.

It is with respect to this general technical environment that aspects of the present disclosure are related.

<CIT> discloses tiered storage using storage class memory.

<CIT> discloses an SSD with heterogeneous NVM types.

<CIT> discloses memory virtualization for accessing heterogeneous memory components.

According to an embodiment of the present disclosure, there is provided a system including: a persistent storage device, the persistent storage device including: a controller circuit; persistent storage media, connected to the controller circuit; nonvolatile memory, connected to the controller circuit; and volatile memory, connected to the controller circuit.

In some embodiments, the persistent storage media may be flash memory.

In some embodiments, the volatile memory may be dynamic random access memory.

In some embodiments, the nonvolatile memory may be storage class memory.

In some embodiments, the controller circuit may be configured to receive a first write request, for a first page of the persistent storage media; store the first write request in the nonvolatile memory; receive a second write request, for the first page of the persistent storage media; store the second write request in the nonvolatile memory; and flush the first write request and the second write request to the first page of the persistent storage media.

In some embodiments, the controller circuit may be configured to store logging information in the nonvolatile memory.

In some embodiments, the controller circuit may be configured to store metadata in the nonvolatile memory.

In some embodiments, the controller circuit may be configured to store journaling information in the nonvolatile memory.

In some embodiments, the system further may include an accelerator circuit, connected to the volatile memory. In some embodiments, the accelerator circuit may be configured to identify a frequently accessed page in the persistent storage media.

In some embodiments, the accelerator circuit may be configured to cause the frequently accessed page to be moved to the nonvolatile memory.

According to an embodiment of the present disclosure, there may be provide an operating method of persistent storage device, including: receiving, by a persistent storage device, a first write request, for a first page of persistent storage media of the persistent storage device; storing the first write request in a nonvolatile memory of the persistent storage device; receiving a second write request, for the first page of the persistent storage media; storing the second write request in the nonvolatile memory; and flushing the first write request and the second write request to the first page of the persistent storage media.

In some embodiments, the operating method of persistent storage device further may include storing logging information in the nonvolatile memory.

In some embodiments, the operating method of persistent storage device further may include storing metadata in the nonvolatile memory.

In some embodiments, the operating method of persistent storage device further may include storing journaling information in the nonvolatile memory.

In some embodiments, the persistent storage device may include an accelerator circuit, and a volatile memory connected to the accelerator circuit.

In some embodiments, the accelerator circuit may be configured to identify a frequently accessed page in the persistent storage media.

According to an embodiment of the present disclosure, there may be provided a system including: a persistent storage device, the persistent storage device including:
means for processing; means for persistent storage, connected to the means for processing; nonvolatile memory, connected to the means for processing; and volatile memory, connected to the means for processing.

These and other features, aspects and advantages of the embodiments of the present disclosure will be more fully understood when considered with respect to the following detailed description, appended claims, and accompanying drawings. Of course, the actual scope of the invention is defined by the appended claims.

The detailed description set forth below in connection with the appended drawings is intended as a description of exemplary embodiments of a high endurance persistent storage device provided in accordance with the present disclosure and is not intended to represent the only forms in which the present disclosure may be constructed or utilized. The description sets forth the features of the present disclosure in connection with the illustrated embodiments. As denoted elsewhere herein, like element numbers are intended to indicate like elements or features.

Some embodiments address write amplification and wear acceleration in persistent storage devices, e.g., in Nonvolatile Memory Express (NVMe) solid-state drives (SSDs). Features of such embodiments may have the effect of improving the SSD lifetime (long-term data endurance (LDE)) and reducing the total cost of ownership. Features of such embodiments may also improve the performance of computational storage (e.g., SmartSSDs) by leveraging machine learning (ML)-based hot block and hot page caching on a high-speed non-volatile memory (e.g., phase change memory (PCM)).

The write amplification factor (WAF) and wear acceleration index (WAI) of a persistent storage device may be relevant for the endurance and lifetime of the persistent storage device. Reducing the write amplification factor and the wear acceleration index may have a positive impact on the lifespan of the persistent storage device, which may be inversely correlated with the storage deployment cost of the persistent storage device.

Increasing the write amplification factor and wear acceleration index may be negative for the lifetime of the persistent storage device. For example, in some solid-state drives, updates to the same pages and blocks may be done out-of-place (e.g., because of the coarse erase granularity of some flash memory media) which may cause a significant increase in the write amplification factor. Moreover, each write to the flash memory of a solid-state drive may include additional journaling and wear leveling metadata overhead, which may also contribute to an increase in the wear acceleration index.

A solid-state drive may include a volatile memory (e.g., dynamic random access memory (DRAM)) buffer, which may be used to store, e.g., data to be written to the flash memory, and which may in some circumstances (e.g., if a data value to be written is overwritten by a new value while it is still in the buffer) make it possible to avoid some write operations to the flash memory. The buffer may however be flushed to the flash memory relatively frequently to avoid the risk of unacceptable data loss in the event of a power failure; this may limit the effectiveness of the volatile memory buffer for avoiding write operations to the flash memory.

As such, in some embodiments, a high-speed non-volatile memory (NVM) (e.g., storage class memory (SCM), which may be phase change memory (PCM) or non-volatile random access memory (RAM), for example) may be employed as an additional write buffer located between the volatile memory buffer and the flash memory, to reduce the wear acceleration index and the write amplification factor. As used herein, "storage class memory" means memory that is inherently nonvolatile (e.g., that does not require a power-supply connection or a battery-backup to retain data), and that has a read latency of less than <NUM> microsecond, and a write latency of less than <NUM> microsecond.

<FIG> shows an architecture of a persistent storage device <NUM>, in some embodiments. The persistent storage device <NUM> includes a high-speed non-volatile memory buffer <NUM> as an additional write buffer between the volatile memory (e.g., DRAM) write buffer <NUM> and the flash memory <NUM>, which is an example of a persistent storage media. In addition to user data, the persistent storage device <NUM> may store logging blocks and metadata blocks, which are information for use in executing the garbage collection and wear levelling operations. These logging blocks and metadata blocks may not only cause additional writes, but also may be updated more frequently and hence such logging blocks and metadata blocks (if stored in the flash memory <NUM>) may wear out faster than user data blocks. Logging information may include, for example self-monitoring, analysis and reporting technology (SMART) information (which may, for example, be recorded pursuant to the NVMe standard) that monitors errors, device health, and endurance. Metadata may include, for example, an erase count, a free block count, or file data itself (e.g., a file name, a date, a time, a description of the file's contents, or superblock information). A controller (e.g. controller <NUM> of <FIG>) (or control circuit, which may be or include a processing circuit) may control various aspects of the persistent storage device <NUM> including interactions with the host, and data transfers between the volatile memory buffer <NUM>, the high-speed non-volatile memory buffer <NUM>, and the flash memory <NUM>.

The inclusion of the high-speed non-volatile memory buffer <NUM> as an additional write buffer may make it possible to batch multiple updates (to the same block or pages) and to write them to the flash memory <NUM> in one out-of-place update. Further, the metadata and log information may be decoupled from the flash memory <NUM> and stored entirely in the high-speed non-volatile memory buffer <NUM>. A mapping between the metadata and log information and the corresponding data blocks may then be maintained. Thus the more frequently updated metadata and log information writes may be absorbed in the high endurance high-speed non-volatile memory buffer <NUM>, saving space in the flash memory <NUM> and avoiding wear in the flash memory <NUM>.

The high-speed non-volatile memory buffer <NUM>, in additional to being non-volatile (i.e., once data is stored in the high-speed non-volatile memory buffer <NUM> it can persist across power cycles), also may have high (e.g., nearly infinite) write endurance compared to the flash memory <NUM>. Therefore if a larger number of writes is batched in the high-speed non-volatile memory buffer <NUM>, then the number of writes to flash may be significantly reduced, which may directly reduce the write amplification factor and wear acceleration index. Moreover, if there is a crash (e.g., a power outage or a crash of the controller of the persistent storage device <NUM>) before the flash memory <NUM> is updated, the data in the high-speed non-volatile memory buffer <NUM> may still be retrieved. Similarly, storing the wear leveling (WL) metadata and log information in the high-speed non-volatile memory buffer <NUM> may make it possible to retrieve this data even in the event of a power loss or crash. The mapping table (which maps logical to physical blocks), which in some circumstances may be stored in the volatile memory buffer <NUM>, may, in some embodiments, also be stored instead in the high-speed non-volatile memory buffer <NUM>; this may significantly reduce the bootstrapping time, as the mapping table, in such an embodiment, does not need to be reconstructed at every startup. Further, the high-speed non-volatile memory buffer <NUM> may be constructed to have a large capacity (e.g., larger than the volatile memory buffer <NUM>) and it may therefore be capable of hosting the data for a longer time, thereby increasing the likelihood that it will be possible to batch write operations.

In some embodiments, updates of the metadata, of the log information, and of the journaling information during garbage collection and wear levelling operations may be performed on the high-speed non-volatile memory buffer <NUM>, which may have up to <NUM> times lower write latency than the flash memory <NUM>. This may improve the latency and performance of the garbage collection and wear leveling processes, potentially resulting in significant performance improvements if during garbage collection the persistent storage device <NUM> cannot serve user requests (e.g., requests from a host connected to (or containing) the persistent storage device <NUM>).

The architecture of <FIG> may be extended to a computational storage device (e.g., a SmartSSD) to provide intelligent machine learning based caching solutions, and to improve the performance and throughput of the computational storage device. In such an embodiment, the on-device accelerator may be used to gather data access statistics and to identify access patterns, so as to classify hot (frequently accessed) pages, hot blocks, and hot data. Further, the identified hot pages, hot blocks, and hot data may then be moved to the on-device high-speed non-volatile memory buffer <NUM> and any subsequent request to the hot data may be served from the high-speed non-volatile memory buffer <NUM>, thereby eliminating the need to fetch such data (e. g hot data) from the flash memory <NUM>. This may improve both the throughput and latency of the computational storage device.

Write amplification factor in a solid-state drive(SSD) is defined as the ratio of data written to the flash memory to the data written in response to write requests from the host. That is, any additional data (over the user data) written to the flash memory <NUM> may increase the write amplification factor.

<FIG> is a diagram illustrating a method of reducing write amplification factor, according to an embodiment of the present disclosure. <FIG> below presents an architecture for reducing the write amplification factor by using (as in the embodiment of <FIG>) an additional high-speed non-volatile memory buffer <NUM> between the volatile memory buffer <NUM> and the flash memory <NUM>. Contents that overlap with those described in <FIG> will be omitted.

The use of the high-speed non-volatile memory buffer <NUM> as an intermediate tier may make it possible to batch a larger number of write operations (each corresponding to a write request (e.g., a write request received from the host)) before writing to the flash memory, and to then flush the corresponding write requests to the flash memory <NUM> in a batch. In an embodiment without the high-speed non-volatile memory buffer <NUM>, the flash memory <NUM> may be updated whenever the volatile memory buffer <NUM> becomes full or when the application (running on the host) issues a flush operation (e.g., a call to msync() or fsync()).

Even if the write buffer (e.g. the volatile memory buffer <NUM>) is not full it may be necessary to update the flash memory <NUM> to ensure crash consistency for the user data. In the <FIG>, however, the write operations may simply be written to the high-speed non-volatile memory buffer <NUM>, and, because the high-speed non-volatile memory buffer <NUM> is nonvolatile, be permitted to reside there until the high-speed non-volatile memory buffer <NUM> is full, this may increase the likelihood that opportunities for write batching will be present. As the high-speed non-volatile memory buffer <NUM> becomes full, all updates to a block or page may be batched and written to the flash memory <NUM> as a single update.

The legend of <FIG> shows the symbols (i) for write operations (triangles) and (ii) for commands from the host (e.g., calls to msync()) to flush the volatile memory buffer <NUM>. It may be seen that by avoiding the need to flush data to the flash memory <NUM> with every flush command, it is possible to accumulate a significantly larger number of write commands between write operations to the flash memory <NUM>, providing significantly greater opportunities to perform batching when writing to the flash memory <NUM>.

Such batching may be advantageous because each write to the flash memory <NUM> may use an out-of-place update, i.e., a new page or block may be used to make updates to the existing page. This not only reduces the write latency but also reduces the garbage collection and wear leveling overhead, as writing to a new block may involve erase and rewrite operations. As such, the greater the extent to which batching is employed, the fewer flash memory <NUM> updates may be used, reducing garbage collection and wear leveling overhead. This may increase the lifespan of the persistent storage device <NUM> and reduce the total cost of ownership of the persistent storage device <NUM>.

In a persistent storage device <NUM>, the write amplification factor may be related not only to the writing of user data but also to some of the internal operations of the persistent storage device <NUM>, such as wear levelling, journaling, and garbage collection. The metadata and logging (for wear levelling and normal write operations to the flash memory <NUM>) may cause more significant wear of the flash memory <NUM> than writing to the user data blocks because such metadata and log blocks may be more frequently updated. As such, it may be possible to further reduce wear of the flash memory <NUM> by (in addition to performing batching, as described above) leveraging the high-speed non-volatile memory buffer <NUM> to decouple such metadata from the flash memory <NUM>. <FIG> shows such an embodiment.

<FIG> is a block diagram of a portion of a persistent storage device, according to an embodiment of the present disclosure. Referring to <FIG>, the persistent storage device <NUM> may include the high-speed non-volatile memory buffer <NUM>, the volatile memory buffer <NUM>, and the flash memory <NUM>. As shown in <FIG>, in some embodiments, a portion of the high-speed non-volatile memory buffer <NUM> is reserved for storing the logging data (for each block <NUM>) and the wear-levelling metadata, which may be used to store metadata for free blocks <NUM>. To associate the metadata and logging data in the high-speed non-volatile memory buffer <NUM> with the corresponding block in the flash memory <NUM>, a two-way mapping may be maintained. Thus, the metadata may be updated in the high-speed non-volatile memory buffer <NUM> rather than incurring write operations to the flash memory <NUM>. Because the high-speed non-volatile memory buffer <NUM> is non-volatile, these metadata and log data may be consistently retrieved across power cycles. Further, the high-speed non-volatile memory buffer <NUM> may allow fine-grained updates (e.g., <NUM> byte updates, <NUM> byte updates, or <NUM> byte updates) so this may further reduce any potential write amplification.

<FIG> is a block diagram of a portion of a computational storage device, according to an embodiment of the present disclosure. As illustrated in <FIG>, the features of embodiments discussed above may be employed in an analogous manner in a computational storage device <NUM>, with the potential additional benefit that to the extent that the replacement cost of a computational storage device <NUM> exceeds the replacement cost of a persistent storage device(e.g. persistent storage device <NUM> of <FIG>), the benefits of prolonging the life of the device (by reducing wear on the flash memory) may be greater for a computational storage device than for a persistent storage device <NUM>. In addition to improving the lifespan, a high-speed non-volatile memory buffer 110a may be used in a computational storage device <NUM> to provide a rich caching solution. The high-speed non-volatile memory buffer 110a and the volatile memory 115a of <FIG> may correspond to the high-speed non-volatile memory buffer <NUM> and the volatile memory <NUM> of <FIG>.

<FIG> shows an embodiment in which a high-speed non-volatile memory buffer 110a is used as a caching layer and a write-staging buffer. The computational storage device <NUM> is connected to a host <NUM>. In addition to the high-speed non-volatile memory buffer 110a, the computational storage device <NUM> includes a controller (or controller circuit) <NUM> (which may be or include a processing circuit), a volatile memory buffer 115a (which may be used for computations or computational storage functions (CSF)), a dedicated on-device accelerator (or accelerator circuit) <NUM> (which may be or include a processing circuit), and one or more solid-state drives <NUM>, which are examples of a persistent storage media. The computational storage device <NUM> is an example of a persistent storage device.

The on-device accelerator <NUM> may be or include a processing circuit such as a tensor processing unit (TPU) or a neural processing unit (NPU), and it may be used for gathering data access statistics and identifying hot blocks or pages. The on-device accelerator <NUM> may be used to run machine learning (ML) algorithms to detect the host applications' data access patterns and predict hot blocks, pages, or data by leveraging its training data. The identified hot data may be moved from the solid-state drives to the high-speed non-volatile memory buffer 110a by the controller <NUM>, and subsequent requests to access the hot data may be served from the high-speed non-volatile memory buffer 110a without a need to access the solid-state drives <NUM>. In some embodiments the on-device accelerator <NUM> may cause frequently accessed data to be moved, by the controller <NUM>, to the high-speed non-volatile memory buffer 110a (e.g., the accelerator <NUM> may send, to the controller <NUM> a report identifying such frequently accessed data, or the accelerator <NUM> may send, to the controller <NUM> a request that such frequently accessed data be moved to the high-speed non-volatile memory buffer 110a). This may significantly improve the latency and performance of the computational storage device <NUM>.

A caching algorithm (e.g., least recently used (LRU), least frequently used (LFU)) may be used to handle the eviction and migration of blocks, data, or pages from the solid-state drives <NUM> to the high-speed non-volatile memory buffer 110a and vice versa. Further, the high-speed non-volatile memory buffer 110a may also be used as a staging buffer for write operations. For example, all of the write requests received from an application may initially be written to the high-speed non-volatile memory buffer 110a, and the computational storage device <NUM> may then batch the updates where possible and issue a single write request to flush the batch of write requests to the target solid-state drive <NUM>. Because the high-speed non-volatile memory buffer 110a has lower write latency than the solid-state drives <NUM> this may significantly improve the write performance while also supporting write batching external to the solid-state drives <NUM>. If an application performs a skewed access, the cache hit rate may be increased because of the use of the high-speed non-volatile memory buffer 110a as a staging buffer for write operations. Because the high-speed non-volatile memory buffer 110a is non-volatile, writes may be consistently retrieved in the event of an unexpected crash.

<FIG> illustrates a method, in some embodiments. The persistent storage device may receive, at <NUM>, by a persistent storage device, a first write request, for a first page of persistent storage media of the persistent storage device. The persistent storage device may store, at <NUM>, the first write request in a nonvolatile memory of the persistent storage device. The persistent storage device may receive, at <NUM>, a second write request, for the first page of the persistent storage media. The persistent storage device may store, at <NUM>, the second write request in the nonvolatile memory. The persistent storage device may flush, at <NUM>, the first write request and the second write request to the first page of the persistent storage media.

The persistent storage devices <NUM> and computational storage device <NUM> discussed herein may have any of various suitable form factors, including U. <NUM>, and AIC. As used herein, "storing" a write request means storing at least the data associated with the write request.

As used herein, "a portion of" something means "at least some of" the thing, and as such may mean less than all of, or all of, the thing. As such, "a portion of" a thing includes the entire thing as a special case, i.e., the entire thing is an example of a portion of the thing. As used herein, the term "or" should be interpreted as "and/or", such that, for example, "A or B" means any one of "A" or "B" or "A and B".

The background provided in the Background section of the present disclosure section is included only to set context, and the content of this section is not admitted to be prior art. Any of the components or any combination of the components described (e.g., in any system diagrams included herein) may be used to perform one or more of the operations of any flow chart included herein. Further, (i) the operations are example operations, and may involve various additional steps not explicitly covered, and (ii) the temporal order of the operations may be varied.

Each of the terms "processing circuit" and "means for processing" may be used herein to mean any combination of hardware, firmware, and software, employed to process data or digital signals. Processing circuit hardware may include, for example, application specific integrated circuits (ASICs), general purpose or special purpose central processing units (CPUs), digital signal processors (DSPs), graphics processing units (GPUs), and programmable logic devices such as field programmable gate arrays (FPGAs). In a processing circuit, as used herein, each function is performed either by hardware configured, i.e., hard-wired, to perform that function, or by more general-purpose hardware, such as a CPU, configured to execute instructions stored in a non-transitory storage medium. A processing circuit may be fabricated on a single printed circuit board (PCB) or distributed over several interconnected PCBs. A processing circuit may contain other processing circuits. For example, a processing circuit may include two processing circuits, an FPGA and a CPU, interconnected on a PCB.

As used herein, when a method (e.g., an adjustment) or a first quantity (e.g., a first variable) is referred to as being "based on" a second quantity (e.g., a second variable) it means that the second quantity is an input to the method or influences the first quantity, e.g., the second quantity may be an input (e.g., the only input, or one of several inputs) to a function that calculates the first quantity, or the first quantity may be equal to the second quantity, or the first quantity may be the same as (e.g., stored at the same location or locations in memory as) the second quantity.

As used herein, "connected" means connected by a signal path (e.g., a conductor or a waveguide) that may contain arbitrary intervening elements, including intervening elements the presence of which qualitatively changes the behavior of the circuit. Any pair of elements shown in the drawings with a line connecting the elements may mean (i) connected, in some embodiments, (ii) directly connected, in some embodiments, or (iii) immediately connected, in some embodiments. As used herein, "directly connected" may mean (i) "immediately connected" or (ii) connected with intervening elements, the intervening elements being ones (e.g., low-value resistors or inductors, short sections of transmission line, or short sections of waveguide) that do not qualitatively affect the behavior of the circuit.

Claim 1:
A system comprising:
a persistent storage device (<NUM>; <NUM>), the persistent storage device (<NUM>; <NUM>) comprising:
a controller circuit (<NUM>);
persistent storage media (<NUM>; <NUM>), connected to the controller circuit (<NUM>);
nonvolatile memory (<NUM>; 110a), connected to the controller circuit (<NUM>); and
volatile memory (<NUM>; 115a), connected to the controller circuit (<NUM>), wherein the controller circuit (<NUM>) is configured to
receive a first write request, for a first page of the persistent storage media (<NUM>; <NUM>);
store the first write request in the nonvolatile memory (<NUM>);
receive a second write request, for the first page of the persistent storage media (<NUM>; <NUM>);
store the second write request in the nonvolatile memory (<NUM>); and
flush the first write request and the second write request to the first page of the persistent storage media (<NUM>; <NUM>).