Media Controller and Method for Management of CPU-Attached Non-Volatile Memory

A media controller and method for management of CPU-attached non-volatile memory are provided. In one embodiment, a storage system is provided comprising a plurality of non-volatile memory devices and a controller in communication with the plurality of non-volatile memory devices. The controller is configured to receive a read command from a host; in response to receiving the read command from the host, read data from the plurality of non-volatile memory devices; perform an operation having an undetermined duration from the host's perspective; send a ready signal to the host after the operation has been performed; receive a send command from the host; and in response to receiving the send command from the host, send the data to the host.

BACKGROUND

Many computer systems use one or more dual in-line memory modules (DIMMs) attached to a central processing unit (CPU) to store data. Some DIMMs contain dynamic random-access memory (DRAM) chips. However, DRAM is relatively expensive, requires a relatively-large amount of power, and is failing to scale capacity at a rate matching processor power, which can be undesirable when used in servers, such as enterprise and hyperscale systems in data centers where vast amounts of data are stored. To address these issues, non-volatile DIMMs (NV-DIMMs) have been developed, which replaces volatile DRAM chips with non-volatile memory devices. As compared to DRAM-based DIMMs, NV-DIMMs can provide lower cost per gigabyte, lower power consumption, and longer data retention, especially in the event of a power outage or system crash. Like some DRAM-based DIMMs, some NV-DIMMs are designed to communicate over a clock-data parallel interface, such as a double-data rate (DDR) interface.

DETAILED DESCRIPTION

Overview

By way of introduction, the below embodiments relate to a media controller and method for management of CPU-attached non-volatile memory. In one embodiment, a storage system is provided comprising a plurality of non-volatile memory devices and a controller in communication with the plurality of non-volatile memory devices. The controller is configured to receive a read command from a host; in response to receiving the read command from the host, read data from the plurality of non-volatile memory devices; perform an operation having an undetermined duration from the host's perspective; send a ready signal to the host after the operation has been performed; receive a send command from the host; and in response to receiving the send command from the host, send the data to the host.

In some embodiments, the controller is configured to communicate with the host using a clock-data parallel interface.

In some embodiments, the clock-data parallel interface comprises a double data rate (DDR) interface.

In some embodiments, the ready signal is sent to the host after an undetermined amount of time after the read command is received.

In some embodiments, the operation comprises one or more of the following: a data protection operation and a decryption operation.

In some embodiments, the read command is associated with an identifier, and wherein the identifier is sent back to the host with the data.

In some embodiments, the controller is distributed among the plurality of non-volatile memory devices.

In some embodiments, the storage system comprises a non-volatile dual in-line memory module (NV-DIMM).

In some embodiments, at least one of the plurality of non-volatile memory devices comprises a three-dimensional memory.

In another embodiment, a storage system is provided comprising a plurality of non-volatile memory devices and a controller in communication with the plurality of non-volatile memory devices. The controller is configured to receive a write command from a host, wherein the host is allowed only a certain number of outstanding write commands as tracked by a write counter in the host; perform an operation having an undetermined duration from the host's perspective; write data to the plurality of non-volatile memory devices; and after the data has been written, send a write counter increase signal to the host.

In some embodiments, the controller is configured to communicate with the host using a clock-data parallel interface.

In some embodiments, the clock-data parallel interface comprises a double data rate (DDR) interface.

In some embodiments, the operation comprises one or more of the following: a data protection operation, an encryption operation, and a wear-leveling operation.

In some embodiments, the controller is distributed among the plurality of non-volatile memory devices.

In some embodiments, the storage system comprises a non-volatile dual in-line memory module (NV-DIMM).

In some embodiments, at least one of the plurality of non-volatile memory devices comprises a three-dimensional memory.

Other embodiments are possible, and each of the embodiments can be used alone or together in combination.

General Introduction to One Implementation of One Embodiment

As explained in the background section above, dual in-line memory modules (DIMMs) can be attached to a central processing unit (CPU) of a host to store data. Non-volatile dual in-line memory modules (NV-DIMMs) have been developed to replace volatile DRAM chips on standard DIMMs with non-volatile memory devices, such as NAND. As compared to DRAM-based DIMMs, NV-DIMMs can provide lower cost per gigabyte, lower power consumption, and longer data retention, especially in the event of a power outage or system crash. Like some DRAM-based DIMMs, some NV-DIMMs are designed to communicate over a clock-data parallel interface, such as a double-data rate (DDR) interface.

However, existing standards that are appropriate for DRAM-based DIMMs may not be appropriate for NV-DIMMs. For example, some existing standards require read and write operations to be completed within a specified (“deterministic”) amount of time. While completing read and write operations in the specified amount of time is typically not a problem for DRAM memory, the mechanics of reading and writing to non-volatile memory can cause delays that exceed the specified amount of time. That is, DRAM-based DIMM protocols expect consistent, predictable, and fast responses, which non-volatile memory may not be able to provide. To account for this, some emerging standards (e.g., JEDEC's NVDIMM-P standard) allow for “non-deterministic” read and write operations to put “slack” in the communication between the storage system and the host. Under such standards, read and write operations to the NV-DIMM are not required to be completed by a certain amount of time. Instead, in the case of a read operation, the NV-DIMM informs the host when the requested data is ready, so the host can then retrieve it. In the case of a write operation, the host can be restricted from having more than a certain number of write commands outstanding to ensure that the non-volatile memory device does not receive more write commands than it can handle.

The approach of allowing non-deterministically timed operations at a protocol level is just one possible approach for dealing with the unpredictable nature of non-volatile memories. Other approaches do not take advantage of non-deterministic modifications to the DDR standard. Instead, they rely on software approaches to construct compound read and write procedures out of conventional DDR primitives. Each DDR primitive may correspond either to a direct access to the non-volatile memory itself, or it may correspond to indirect operations performed via the use of intermediate circuit elements, such as control registers or buffers. Though the read or write algorithms themselves may require an unspecified number of iterations or DRR commands to complete and thus may not complete within a specific timeframe each individual primitive DDR operation completes within the well-defined time limits set by the usual (deterministically-timed) DDR standards.

Some of the following embodiments take advantage of the non-deterministic aspect of the emerging standard to allow the NV-DIMM to perform time-consuming actions that it may not have the time to do under conventional, DRAM-based DIMM standards. These actions will sometimes be referred to herein as operations having an undetermined duration from the host's perspective and may include memory and data management operations. These memory and data management operations may be important to the operation of the NV-DIMM. For example, as compared to DRAM, a non-volatile memory device can have lower endurance (i.e., number of writes before failure) and less reliably store data (e.g., because of internal memory errors that cause bits to be stored incorrectly). These issues may be even more pronounced with emerging non-volatile memory technologies that would likely be used as a DRAM replacement in an NV-DIMM. As such, in one embodiment, the NV-DIMM takes advantage of not being “under the gun” to perform operations having an undetermined duration from the host's perspective, such as memory and data management operations (e.g., wear leveling and error correction operations) that it may not be able to perform in the allotted time under conventional, DRAM-based DIMM standards.

It should be noted that this introduction merely discusses one particular implementation of an embodiment and that other implementations and embodiments can be used, as discussed in the following paragraphs. Further, while some of these embodiments will be discussed in terms of an NV-DIMM attached to a CPU of a host, it should be understood that any type of storage system can be used in any suitable type of environment. Accordingly, specific architectures and protocols discussed herein should not be read into the claims unless expressly recited therein.

General Discussion of Clock-Data Parallel Interfaces and New Protocols

Clock-data parallel interfaces are a simple way of transferring digitized data and commands between any two devices. Any transmission line carrying data or commands from one device to the other are accompanied by a separate “clock” transmission-line, which provides a time-reference for sampling changes in the data and command buses. In some embodiments, the clock may be deactivated when the interface is inactive, transmitting no data or commands. This provides a convenient way of reducing power dissipation when inactive. In some embodiments of clock-data parallel interfaces, the clock is a single-ended transmission-line, meaning that the clock consists of one additional transmission line, whose voltage is compared to a common voltage reference shared by many transmission lines travelling between the CPU and memory devices. In other embodiments, the timing reference might be a differential clock, with both a positive clock reference and a clock complement, which switches to a low voltage simultaneously with every low-to-high-voltage switch of the positive clock—an event known as the “rising-edge” of the clock—and conversely the clock complement switches to high-voltage state with every high-to-low-voltage transition of the positive clock reference—and event known as the “falling-edge” of the clock. Clock-data parallel interfaces are often classified by how many beats of data are sent along with the clock. In “single-data rate” or SDR interfaces, the command or data buses transition once per clock cycle, often with the rising edge of the reference clock. In “double-data rate” or DDR interfaces, the command and data buses send twice as much data per clock period, by allowing the command and data buses to switch twice per period, once on the rising edge of the clock, and once on the falling edge of the clock. Furthermore, there are quad-data rate (QDR) protocols, which allow for four data or command transitions per clock. Typically, clock-data parallel interfaces are, by their simplicity, efficient and low latency, and the receiver circuitry may be as simple as a single bank of logic flip-flops. However, there may be additional complexity induced by the need to synchronize the newly-latched data with the internal clock of the devices themselves, one of the many jobs handled by a collection of signal conditioning circuits known as the “physical communication layer” or simply “Phy Layer.”

Serial interfaces, by contrast, typically rely on clock-data recovery processes to extract the time-reference from a single electrical transmission line, which switches voltage at regular time intervals, but in such a pattern that also communicates commands and/or data (in some embodiments, many different lines are run in parallel for increased bandwidth, and thus each line may encode data for an entire command, and entire sequence of data, or just a portion of a command or data sequence). Encoding the clock and the data in the same physical transmission line reduces timing uncertainties caused by mismatched delays between clock and data or command lines and thus allows for clock frequencies of 25 GHz or higher, for very-high bandwidth communication. However, such interfaces also have some disadvantages. Due to the nature of clock-data recovery, the transmission line must remain active continuously in order to maintain synchronization of the inferred clock reference between the communication partners. Power-saving modes are possible, but re-entering the active mode requires significant retraining delays. Moreover, the very nature of clock-data recovery requires slightly more time to decode each message, and one-way communication delays are common for even a well-trained serial link. This adds extra latency to any data request.

The interface between computer CPUs and their corresponding memory devices is one example of an interface where optimization of both power and latency are desired. So, though there exists high bandwidth serial CPU-memory interfaces, such as Hybrid Memory Cube, the bulk of contemporary interfaces between CPUs and memory devices still use clock-data parallel interfaces. For instance, synchronous dynamic random access memory (SDRAM) uses a single clock to synchronize commands on a command bus consisting of a plurality of transmission lines, each encoding one-bit of command-sequence information. Depending on the embodiment, commands in a SDRAM command sequence may include, but are not limited to, the following: activate a row of cells in a two-dimensional data array for future reading or writing; read some columns in a currently-active row; write some columns in a currently-active row; select a different bank of cells for reading or writing; write some bits to the memory mode registers to change aspects of the memory device's behavior; and read back values from the mode registers to identify the status of the memory device.

Data associated with these commands is sent or received along a separate data bus consisting of a separate and parallel plurality of data transmission lines, referred to as the DQ bus. In some embodiments, the DQ bus may be half-duplex and bi-directional, meaning that the same lines are used for receipt and transmission of data, and data cannot be simultaneously sent from the memory device to the CPU while data is flowing in the opposite direction, nor vice-versa. In other embodiments, the DQ bus may be full-duplex with separate lines for receipt or transmission of data. The data on the DQ bus may be safely assumed to be synchronous with the device command clock. However, for longer transmission lines or faster operational frequencies, this may lead to poor synchronization. Thus, other embodiments exist where the overall DQ bus is subdivided into a plurality of smaller DQ groups, each with its own “DQ strobe” signal, DQS, which serves as a separate timing reference for the wires in that DQ group. For instance, in one embodiment, a 64-bit DQ bus may be divided into 8 groups (or “byte-lanes”) of 8 DQ-lines in each, each synchronized by its own DQS strobe. The DQS strobes may be differential or single-ended, depending on the embodiment. In some embodiments, some DQ lines may provide encode for not just data stored by the host, but also additional parity or other signal data for the purpose of recording additional error correcting codes. Depending on the embodiment, many DDR protocols have a range of other control signal transmission lines driven by CPU to the memory device, which for example may, in some embodiments, command the functions include but are not limited to: Command Suppression lines (CS_N), Clock Enable (CKE), or enablement of on-die termination (ODT).

An electronic system may consist of one or a plurality of data processing elements—where the act of processing may include computation, analysis, storage of data or transmission of the data over a network or peripheral bus—attached to a plurality of memory devices. Examples of data processing elements include, but are not limited to, CPUs, CPU caches, application-specific integrated circuits, peripheral buses, Direct Memory Access (DMA) engines, or network interface devices. In the many DRAM configurations, a plurality of memory circuits are bundled together into modules; for example, in modules described by the dual-inline memory module (DIMM) standard. Within a module, some devices may transmit data in parallel along separate DQ groups, while others may be all be connected in parallel to the same transmission lines within a DQ group. Again, in many typical DRAM configurations, a plurality of modules then may be connected in parallel to form a channel. In addition to the memory modules, each channel is connected to exactly one data processing element, hereafter referred to as the host. Each memory device may be connected to the host via a portion of a half-duplex DQ bus (as opposed to a full-duplex DQ bus) or may furthermore be attached to the same DQ transmission lines as several other memory devices—either on the same module or on other adjacent modules in the same channel. Therefore, there is the risk that a memory device could choose to assert data on the DQ bus or at the same time as other memory devices on the same bus, and thus there is need for arbitration on the bus. Therefore, SDRAM protocols rely on a centralized, time-windowed, bus allocation scheme: the host by default is the only device permitted to transmit data on the DQ bus, and by default all memory devices leave their DQ lines high-impedance most of the time. When a command requiring a response is sent to a particular memory device, that device is permitted to transmit data on the DQ bus but only within a certain window of time following the first pulse of the command. The window starts a fixed number of clock cycles after the command and has a typical duration of just one or two clock-cycles longer than the time required to transmit the data. Memory devices transmitting data outside this window will either fail to get their data to the host successfully, or will corrupt data coming back from adjacent memory devices.

The DQ bus arbitration scheme used by these clock-data parallel SDRAM protocols works well for DRAM. The technology behind DRAM devices has advanced to the point where their data access times are extremely consistent and predictable. DRAM however is a relatively power-hungry technology, as it requires frequent refresh thousands of times a second.

Non-volatile memories such as phase-change random access memory (PCM), oxidative resistive random access memory (OxRAM or ReRAM), conductive-bridge random access memory (CBRAM), NAND Flash (NAND), magnetic tunnel junction-based magnetic random access memory (MRAM), memristor, NOR Flash (NOR), spin torque-transfer magnetic memory (STT-MRAM), and ferroelectric random-access memory (FeRAM), all promise low-latency data access for data, can be optimized for lower power-consumption for many data heavy workloads, and may soon offer random-access storage at higher density than DRAM. However, they require slightly more relaxed data-access protocols than DRAM. All of these non-volatile memories exhibit non-deterministic read and write latencies. It is impossible to accurately know at the time a read or write command is written how long it would take to access or commit the data to or from a cell of non-volatile memory for all NVM choices and for all NVM device architectures. However, it is possible to mimic deterministic latencies. Deterministic latencies may be mimicked by assuming worst case timing conditions or giving up on a read that may be taking too long. Modifications of the DDR SDRAM protocols could be specified based on pessimistic read or write latency specifications. For example, a memory that commits most writes within 100 ns, but occasionally takes 10 us to commit data for unpredictable reasons, could use a DDR protocol that does not allow writes for a whole 10 us after the previous write, and does not allow reads in this period also (since for some memory technologies writes mean that reads must also be delayed). This however would present a dramatic limit to the maximum bandwidth achievable by such a device, and furthermore, could limit the performance of other devices on the same channel. Conversely, one can imagine a modification of the standard DDR or SDR or QDR SDRAM protocols that allow flexibility for non-deterministic read latencies and non-deterministic write latencies. In one embodiment, this protocol is referred to as a synchronous non-volatile RAM (hereafter SNVRAM) protocol.

For example, in some embodiments of SNVRAM protocols, the read command may be split into three smaller commands. Where before a read command-sequence consisted of two-parts: an activate command, followed by a read to specify the row and column of the data requested, the command would now consist of an activate command, a read command, and finally—after some undetermined delay—a send command. The activate/read combination would specific the two part request to read a specific region. However, no response would be sent following the read command; instead, the memory device would assert a signal, called for example “READ READY” (sometimes referred to herein as “R_RDY”), back to the host at some non-determined time after the read command. This assertion would then prompt the host to issue the SEND command as other SDRAM activity is allowed to transfer the completely extracted data from the memory device back to the host. The response from the SEND command would go out over the shared DQ bus within predetermined window following the SEND command. In this way, the typical read command would support non-deterministic read latencies; however, performance characteristics such as the average minimum latency or overall bandwidth of the system is not limited by the slowest possible read. The average performance of the protocol matches the typical performance of the device while still allowing some flexibility for outliers which are clearly expected as a physical consequence of the choice of media.

In one embodiment, the SNVRAM includes the following characteristics:Much like existing SDRAM or DDR protocols, it supports communication between a single host and a plurality of memory devices on the same memory channel. Hosts may be attached to separate memory channels, though each channel operates independently, and thus the protocol does not specify the behavior of devices in other channels. Transmission lines for the operation of one channel can be used exclusively by that channel. In other embodiments, the host may attach to a single memory device, and that memory device may relay the commands and data on to a second device in a chained style of deployment.As in existing SDRAM or DDR protocols, each signal or bus from the host to the channel can be synchronous to a clock signal following a parallel transmission line.As in existing SDRAM or DDR protocols, there exist logical commands such as “activate address block,” “read element within active address block,” or “write to element within active address block” which can be sent along a command bus.As in existing SDRAM or DDR protocols, the command bus can be synchronized to a master clock or master command strobe for the channel.As in existing SDRAM or DDR protocols, data returning from the memory device can be sent along a separate data bus, which consists of a plurality of transmission lines referred to as the DQ bus.As in existing SDRAM or DDR protocols, each line in the DQ bus may be synchronous to the master clock in some embodiments. In other embodiments, the DQ bus is synchronous to a separated DQ strobe signal (generated either by the host or by the memory device), here after labelled DQS. There may be multiple DQS lines in some embodiments, each corresponding to a subset of the DQ bus lines.As in existing SDRAM or DDR protocols some embodiments exist in which the DQ bus may be bidirectional, and may accommodate storable data from the host to the memory device. Other embodiments may include a separate write DQ bus.As in existing SDRAM or DDR protocols, data from the host to the memory device on a DQ bus can be transmitted synchronous with either the master clock or the appropriate DQS lines, depending on the embodiment under consideration.As in existing SDRAM or DDR protocols, the DQ buses may be attached to multiple memory devices in addition to the single host. Arbitration on this bus is done on the basis of time-windows. When a memory device receives from the host a command requiring a response, it has a narrow window of time in which it owns the DQ-bus and may assert data.As in existing SDRAM or DDR protocols, within a channel, memory devices may be grouped together as a plurality to form coordinated modules.SNVRAM protocols are typically unique from SDRAM protocols in that there are additional control lines sending signals from the storage system to the host. (Typical SDRAM interfaces only include control signals sent from the host to the storage system). These additional control lines are hereafter referred to as the “response bus” (or RSP). The response bus may be synchronous to the master clock in some embodiments, or in other embodiments may have its own strobe signal generated by the memory module. The response bus includes, but is not limited to, signals, which for our purposes are here identified as “READ READY” (R_RDY) and “WRITE CREDIT INCREMENT.” (WC_INC). However, it should be noted that different embodiments of SNVRAM protocols may have electrical signals with similar functions, though the protocol may refer to them by a different name. Accordingly, it should be understood that specific signal names used herein are merely examples.In some embodiments of NVRAM protocols, the response bus may be shared by all modules in a channel and arbitrated by the host, or in other embodiments the response bus may consist of distinct transmission lines—not shared between any modules—passing only from each module to the host, not making electrical contact with any other modules.Just as different embodiments of the SDRAM or DDR protocols transmit data at protocol-specified rates, data on any command bus may be specified for transmission at SDR, DDR, or QDR rates by the particular protocol embodimentData on any command bus, clocks or strobes may be sent single-ended or differentially, depending on the specifications included by the embodiment of the SNVRAM protocolSNVRAM protocols provide a simple way of accommodating the irregular behavior of nondeterministic non-volatile media without unnecessarily restricting their bandwidth. However, there are many other opportunities that can be realized by such protocols. In addition to compensating for non-deterministic behavior of the memory, these protocols also can be used to provide time for various maintenance tasks and data quality enhancements, such as error correction, I/O scheduling, memory wear-leveling, in-situ media characterization, and logging of controller-specific events and functions. Once the hardware implementing these functions becomes more complex, contention for hardware resources performing these functions become another potential source of delays. All such delays can cause significant performance or reliability issues when using a standard SDRAM communication protocol. However, the use of non-deterministically timed SNVRAM protocol allows for flexible operation and freedom of hardware complexity. Furthermore, non-deterministic read-timings allow for the possibility of occasional faster read response through caching.

Discussion of the Drawings

Turning now to the drawings,FIG. 1is a block diagram of a host100in communication with storage systems of an embodiment. As used here, the phrase “in communication with” could mean directly in communication with or indirectly in communication with through one or more components, which may or may not be shown or described herein. In this illustration, there are two storage systems shown (storage system A and storage system B); however, it should be understood that more than two storage systems can be used or only one storage system can be used. In this embodiment, the host100comprises one or more central processing units (CPUs)110and a memory controller120. In this illustration, there are two CPUs (CPU A and CPU B); however, it should be understood that more than two CPUs can be used or only a single CPU can be used. The memory controller may also be connected to devices other than just CPUs and may be configured to relay memory requests on behalf of other devices, such as, but not limited to, network cards or other storage systems (e.g., a hard drive or a solid-state drive (SSD)). Furthermore, the memory controller may relay memory requests on behalf of one or more software applications running on the CPU, which sends requests to the memory controller120for access to the attached storage systems.

In this embodiment, the host100also comprises a memory controller120in communication with the CPUs110(although, in other embodiments, a memory controller is not used), which communicates with the storage systems using a communication interface, such as a clock-data parallel interface (e.g., DDR) and operates under a certain protocol (e.g., one set forth by the Joint Electron Device Engineering Council (JEDEC)). In one embodiment, the memory controller120correlates access requests to the storage systems from the CPUs110and sorts out replies from the storage systems and delivers them to the appropriate CPUs110.

As also shown inFIG. 1, storage system A comprises a media (non-volatile memory) controller130in communication with a plurality of non-volatile memory devices140. In this embodiment, storage systems A and B contain the same components, so storage system A also comprises a media (non-volatile memory) controller150in communication with a plurality of non-volatile memory devices160. It should be noted that, in other embodiments, the storage systems can contain different components.

The media controller130(which is sometimes referred to as a “non-volatile memory (NVM) controller” or just “controller”) can take the form of processing circuitry, a microprocessor or processor, and a computer-readable medium that stores computer-readable program code (e.g., firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, for example. The controller130can be configured with hardware and/or firmware to perform the various functions described below and shown in the flow diagrams.

In general, the controller130receives requests to access the storage system from the memory controller120in the host100, processes and sends the requests to the non-volatile memories140, and provides responses back to the memory controller120. In one embodiment, the controller130can take the form of a non-volatile (e.g., flash) memory controller that can format the non-volatile memory to ensure the memory is operating properly, map out bad non-volatile memory cells, and allocate spare cells to be substituted for future failed cells. Some part of the spare cells can be used to hold firmware to operate the non-volatile memory controller and implement other features. In operation, when the host100needs to read data from or write data to the non-volatile memory, it will communicate with the non-volatile memory controller. If the host100provides a logical address to which data is to be read/written, the flash memory controller can convert the logical address received from the host100to a physical address in the non-volatile memory. (Alternatively, the host100can provide the physical address.) The non-volatile memory controller can also perform various operations having an undetermined duration from the host's perspective, such as, but not limited to, wear leveling (distributing writes to avoid wearing out specific blocks of memory that would otherwise be repeatedly written to) and garbage collection (after a block is full, moving only the valid pages of data to a new block, so the full block can be erased and reused). More information about one particular embodiment of the controller130is set forth below in conjunction withFIG. 6.

A non-volatile memory device140can also take any suitable form. For example, a non-volatile memory device140can contain a single memory die or multiple memory dies, and can be equipped with or without an internal controller. As used herein, the term “die” refers to the collection of non-volatile memory cells, and associated circuitry for managing the physical operation of those non-volatile memory cells, that are formed on a single semiconductor substrate. A non-volatile memory die104may include any suitable non-volatile storage medium, including NAND flash memory cells, NOR flash memory cells, PCM, RRAM, OxRAM, CBRAM, MRAM, STT-RAM, FeRAM, or any other non-volatile technology. Also, volatile storage that mimics non-volatility can be used, such as a volatile memory that is battery-backed up or otherwise protected by an auxiliary power source. The memory cells can take the form of solid-state (e.g., flash) memory cells and can be one-time programmable, few-time programmable, or many-time programmable. The memory cells can also be single-level cells (SLC), multiple-level cells (MLC), triple-level cells (TLC), or use other memory cell level technologies, now known or later developed. Also, the memory cells can be fabricated in a two-dimensional or three-dimensional fashion. Some other memory technologies were discussed above, and additional discussion of possible memory technologies that can be used is provided below as well. Also, different memory technologies may have different algorithms (e.g., program in place and wear leveling) applicable to that technology.

For simplicity,FIG. 1shows a single line connecting the controller130and non-volatile memory device140, it should be understood that that connection can contain a single channel or multiple channels. For example, in some architectures, 2, 4, 8, or more channels may exist between the controller130and a memory device140. Accordingly, in any of the embodiments described herein, more than a single channel may exist between the controller130and the memory device140, even if a single channel is shown in the drawings.

The host100and storage systems can take any suitable form. For example, in one embodiment (shown inFIG. 2A), the storage module takes the form of a non-volatile dual in-line memory module (NV-DIMM)200, and the host100takes the form of a computer with a motherboard that accepts one or more DIMMs. In the NV-DIMM200shown inFIG. 2A, there are nine non-volatile memory devices40, and the NV-DIMM200has an interface210that includes 9 data input/output DQ groups (DQ0-DQ8), a command bus, and a response bus. Of course, these are merely examples, and other implementations can be used. For example,FIG. 2Bshows an alternate embodiment, in which the storage system has a distributed controller31and a master controller212(which, although not shown, connects to all the distributed controllers31). As compared to the storage system inFIG. 2A, each NVM device41communicates with its own NVM controller31, instead of all NVM devices communicating with a single NVM controller. In one embodiment, the master controller212does any synchronizing activity needed, including determining when all the distributed controllers31are read to send the RD_RDY signal, which will be discussed in more detail below.

As mentioned above, multiple storage systems can be used, in which signals can be passed through one storage system to reach another. This is shown inFIG. 3. InFIG. 3, storage system A is closer in line to the host100than storage system B. Arrow300represents shared memory input signals that are sent from the host100to the command pin in both the first and second storage systems. Examples of shared memory input signals that can be used include, but are not limited to, an address signal, a read chip select signal, a bank group signal, a command signal, an activate signal, a clock enable signal, a termination control signal, and a command identifier (ID) signal. Arrow310represents a memory channel clock, which can also be sent on the command pin. Arrow320represents shared memory output signals, which can be sent on the DQ0-DQ8groups. Examples of shared memory output signals include, but are not limited to, data signals, parity signals, and data strobe signals. Arrow330represents dedicated memory input signals to storage system B, and arrow350represents dedicated memory input signals to storage system A. Examples of dedicated memory input signals, which can be sent on the command pin, include, but are not limited to, clock enable signals, data strobe, chip select signals, and termination control signals. Arrow340represents a device-dedicated response line to storage system B, and arrow360represents a device-dedicated response line to storage system A. Examples of signals send on the device-dedicated response lines, which can be sent on the command pin, include, but are not limited to, read data ready (R_RDY) signals, a read identifier (ID) signal, and a write flow control signal. These signals will be discussed in more detail below.

One aspect of these embodiments is how the NVM controller130in the storage system handles read and write commands. Before turning to that aspect of these embodiments, the flow chart400inFIG. 4will be discussed to illustrate how a conventional host reads data from a convention DDR-based DRAM DIMM. This flow chart400will be discussed in conjunction with the timing diagram500inFIG. 5. As shown inFIG. 4, when the host required data from the DIMM (referred to as the “device” inFIG. 4) (act410), the memory controller in the host sends an activate command with the upper address (act420). The memory controller in the host then sends a read command with the lower address (act430). This is shown as the “Act” and “Rd” boxes on the command/address line inFIG. 5. The memory controller in the host then waits a predetermined amount of time (sometimes referred to as the “preamble time”) (act440). This is shown as “predefined delay” inFIG. 5. After the predetermined (“deterministic”) amount of time has expired, the memory controller in the host accepts the data (with data strobes for fine grained timing synchronization) (act450) (boxes D1-DN on the data line inFIG. 5), and the data is provided to the host (act460).

As mentioned above, while this interaction between a host and the storage system is adequate with the storage system is a DRAM DIMM, complications can arise when using a deterministic protocol with an NV-DIMM because of the mechanics behind reading and writing to non-volatile memory can cause delays that exceed the amount of time specified for a read or write operation under the protocol. To account for this, some emerging standards allow for “non-deterministic” read and write operations. Under such standards, read and write operations to the NV-DIMM are not required to be completed by a certain amount of time.

In the case of a read operation, the NV-DIMM informs the host100when the requested data is ready, so the host can then retrieve it. This is shown in the flow charts600,700inFIGS. 6 and 7and timing diagram800inFIG. 8A. As shown inFIG. 6, when the host100requires data from the storage system (act610), the host100generates a double data rate identifier (DDR ID) for the request (act620). The host100then associates the DDR ID with a host request ID (e.g., an ID of the CPU or other entity in the host100that requested the data) (act630). Next, the host100sends the activation command and the upper address (act640) and then sends the read command, lower address, and DDR ID (act650). This is shown by the “Act” and “Rd+ID” boxes on the command/address line inFIG. 8A. (FIG. 8Bis another timing diagram810for the read process discussed above, but, here, there are two read commands, and the later-received read (read command B) command completes before the first-received read command (read command B). As such, data B is returned to the host100before data A.)

In response to receiving the read command, the controller130takes an undetermined amount of time to read the data from the non-volatile memory140. After the data has been read, the controller130tells the host100the data is ready by sending a R_RDY signal on the response bus (act710inFIG. 7). In response, the host100sends a “send” command on the command/address line (act720), and, after a pre-defined delay, the controller130returns the data to the host100(act730) (as shown by the “D1”-“DN” boxes on the data line and the “ID” box on the ID line inFIG. 8B). The memory controller120in the host100then accepts the data and the DDR ID (act740). Next, the memory controller120determines if the DDR ID is associated with a specific host ID of one of the CPUs110in the host100(act750). If there is, the memory controller120returns the data to the correct CPU110(act760); otherwise, the memory controller120ignores the data or issues an exception (act770).

In the case of a write operation, the host100can be restricted from having more than a certain number of write commands outstanding to ensure that the non-volatile memory device does not receive more write commands than it can handle. This is shown in the write timing diagram820inFIG. 8C. As shown inFIG. 8C, every time the host100issues a write command, it decreases its write flow control credits (labeled “WC” in the drawing). When a write operation is complete, the media controller130sends a response to the host100for it to increase its write flow control credits.

The protocol discussed above is one embodiment of a NVRAM protocol which supports reads and write operations of unpredictable duration. As discussed previously, in some embodiments, the controller130can take advantage of the non-deterministic aspect in read and write operations to perform time-consuming actions (which may be referred to herein as operations having an undetermined duration from the host's perspective) that it may not have the time to do under conventional, DRAM-based DIMM standards. These operations having an undetermined duration from the host's perspective, such as memory and data management operations, may be important to the operation of the NV-DIMM. For example, as compared to DRAM, a non-volatile memory device140can have lower endurance (i.e., number of writes before failure) and less reliably store data (e.g., because of internal memory errors that cause bits to be stored incorrectly). These issues may be even more pronounced with emerging non-volatile memory technologies that would likely be used as a DRAM replacement in an NV-DIMM. As such, in one embodiment, the NV-DIMM takes advantage of not being “under the gun” to perform operations having an undetermined duration from the host's perspective (e.g., wear leveling and error correction operations) that it may not be able to perform in the allotted time under conventional, DRAM-based DIMM standards.

In general, an operation that has an undetermined duration from the host's perspective refers to an operation that (1) by its nature, does not have a predetermined duration (e.g., because the operation's duration depends on one or more variables) or (2) has a predetermined duration but that duration is not known to the host (e.g., a decryption operation may have a predetermined duration, but that duration is undetermined from the host's perspective because the host does not know whether or not the storage system will be performing a decryption operation). An “operation that has an undetermined duration from the host's perspective” can take any suitable form. For example, such an operation can be a “memory and data management function,” which is an action taken by the controller130to manage the health and integrity of the NVM device. Examples of memory and data management function include, but are not limited to, wear leveling, data movement, metadata writing/reading (e.g., logging, controller status and state tracking, wear leveling tracking updates), data decode variations (ECC engine variations (syndromes, BCH vs LDPC, soft bit decodes), soft reads or re-reads, layered ECC requiring increased transfers and reads, RAID or parity reads with their compounded decoding and component latencies), resource contention (ECC engine, channels, NVM property (die, block, plane, JO circuitry, buffers), DRAM access, scrambler, other hardware engines, other RAM contention), controller exceptions (bugs, peripherals (temperature, NOR), media characterization activities (determining the effective age of memory cells, determining the bit error rate (BER), or probing for memory defects). Furthermore, the media controller may introduce elements, such as caches, that have the inverse effect (fast programs, temporary writes with reduced retention or other characteristics), and serve to accelerate read or write operations in ways that would be difficult to predict deterministically.

Further, operations of undetermined duration from the host perspective can include, but are not limited to, program refreshes, steps for verification (e.g., skip verify, regular settings, tight settings), data movement from one media/state to another location or another state (e.g., SLC to TLC, ReRam to NAND, STT-MRAM to ReRam, burst settings to hardened settings, low ECC to high ECC), and longer media settings (e.g., easier voltage transients). Such operations can be performed, for example, for endurance stretching, retention improvement or mitigation, and performance acceleration (e.g., writing this burst of data quickly or programming this data more strongly in the preferred direction such that future reading settle more quickly).

The media/NVM controller130can be equipped with various hardware and/or software modules to perform these memory and data management operations. As used herein, a “module” may take the form of a packaged functional hardware unit designed for use with other components, a portion of a program code (e.g., software or firmware) executable by a (micro)processor or processing circuitry that usually performs a particular function of related functions, or a self-contained hardware or software component that interfaces with a larger system, for example.

FIG. 9is a block diagram of an NVM controller130of one embodiment showing various modules that can be used to perform memory and data management functions. In this particular embodiment, the controller130is configured to perform encryption, error correction, wear leveling, command scheduling, and data aggregation. However, it should be noted that the controller130can be configured to perform other types and numbers of memory and data management functions.

As shown inFIG. 9, this NVM controller900comprises a physical layer900and a non-volatile RAM (“SNVRAM”) protocol logical interface (which included command and location decoding)905that is used to communicate with the host100(via the memory controller120). The physical layer900is responsible for latching in the data and commands, and the interface905separates out the commands and locations and handles additional signaling pins between the host100and the controller130. The controller130also includes N number of memory finite state machines (MemFSMs)910and NVM physical layer (Phy)910that communicate with M number of non-volatile memory devices140.

In between these input and output portions, the controller130has a write path on the right, a command path in the middle, and a read path on the left. Although not shown, the controller130can have a processor (e.g., a CPU running firmware) that can control and interface with the various elements shown inFIG. 9. Turning first to a write operation, after a command and location have been decoded by the interface905, the address is sent to a wear-leveling address translation module955. In this embodiment, the host100sends a logical address with a command to write data, and the wear-leveling address translation module955translates the logical address to a physical address in memory140. In this translation, the wear-leveling address translation module955shuffles the data to be placed at a physical address that has not been well worn. The wear-leveling data movement module960is responsible for rearranging the data if a sufficiently unworn memory area cannot be found within the address translation scheme. The resulting physical address, along with the associated command and address where the data can be found in local buffers inside the controller130, are inputted to the NVM I/O scheduling module940, which schedules read and write operations to the memory140. The NVM I/O scheduling module940can include other functions to schedule, such as, but not limited to, erases, setting changes, and defect management.

In this embodiment, in parallel to the address translation, for a write operation, the data is first encrypted by the encryption engine925. Next, the media error correction code (ECC) encoder930generates ECC protection for the data while it is at rest in the NVM memory140. Protecting data while at rest may be preferred since non-volatile memories are much more prone to errors than DRAM when retrieving previously-stored data. However, decoding data with error correction is not always a constant time operation, so it would be difficult to perform such operations under deterministic protocols. While ECC is used in this example, it should be understood that any suitable data protection scheme can be used, such as, but not limited to, cyclic redundancy check (CRC), redundant array of independent disks (RAID), scrambling, data weighting/modulation, or other alteration to protect from degradation from physical events such as temperature, time, and voltage exposure (DRAM is also prone to error, but NVM is prone to different errors. Thus, each NVM likely requires a different protection scheme while at rest. Often, it is a tradeoff latency to cost). Also, while not shown to simplify the drawing, it should be noted that other data protection systems can be used by the controller130to protected data when “in flight” between the host100and the controller130and when moving around in the controller130(e.g., using CRC, ECC, or RAID).

As mentioned above, data protection schemes other than ECC can be used. The following paragraphs provide some additional information on various data protection schemes.

Regarding ECC, some embodiments of error-checking codes, such as BCH or other Hamming codes, allow for the decoding engine, which can use a nearly-instantaneous syndrome, to check to validate the correctness of the data. However, a syndrome-check failure may entail the solution of complex algebraic equations which can add to significant delay. Moreover, if multiple syndrome-check failures occur at the same time, there may be hardware-resource-generated backlogs due to the unavailability of hardware resources for decoding. However, these occasional delays can be handled by delaying the read-ready notification to the host. Other coding schemes, such as LDPC or additional CRC checks, may also be included for more efficient use of space or higher reliability, and though these others schemes are likely to have additional variations in time to process the data coming out of the storage media, these variations can also be handled by a simple delay of the read-ready signal.

Another form of data protection may take the form of soft-bit decoding, whereby the binary value of the data stored in the medium is measured with higher confidence by measuring the analog values of the data stored in the physical memory medium several times, relative to several threshold values. Such techniques will take longer to perform, and may add additional variability to the combined data read and decoding process. However these additional delays if needed can be handled gracefully by postponing the READ READY signal back to the host.

Further, reliability still can be added using nested or layered error correcting schemes. For instance, the data in the medium may be encoded such that the data that can survive N errors out of every A bytes read, and can survive M (where M>N) errors out of every B where (B>A) bytes read. A small read of size A may thus be optimal for fast operation, but sub-optimal for data-reliability in the face of a very bad data-block with greater than N errors. Occasional problems in this scheme can be corrected by first reading and validating A bytes. If errors persist, the controller has the option to read the much larger block, at the penalty of a delay, but with successful decoding of the data. This is another emergency decoding option made possible by the non-deterministic read-timings afforded by the SNVRAM-supported media controller.

Also, gross failures of a particular memory device could be encoded via RAID techniques. Data could be distributed across a plurality of memory devices to accommodate the complete failure of some number of memory devices within this set. Spare memory devices could be included in a memory module as fail-in-place spares to receive redundancy data once a bad memory devices is encountered.

Returning toFIG. 9, after the media error correction code (ECC) encoder930generates ECC protection for the data, the data is sent to the write cache management module935, which determines whether or not there is space in the write data cache buffers945and where to put the data in those buffers945. The data is stored in the write data cache buffers945where it is stored until read. So, if there is a delay in scheduling the write command, the data can be stored in the write data cache buffers945indefinitely until the memory140is ready to receive the data.

Once the write command associated with that write-data-cache-buffer entry comes to the front of the queue, the data entry is passed to the NVM write I/O queue950. When indicated by the NVM I/O scheduler940, the command is passed from the NVM I/O scheduler940to the NVM data routing, command routing, and data aggregation module920, and the data is passed from the NVM write I/O queue950to the NVM data routing, command routing, and data aggregation module920. The command and data are then passed to the appropriate channel. The memory finite state machine (MemFSM)910, which is responsible for parsing the commands into more fine-grain, NVM-specific commands and controlling the timing of when those commands are dispersed to the NVM devices140. The NVM Phy915controls timing to an even finer level, making sure that the data and command pulses are placed at well-synchronized intervals with respect to the NVM clock.

Turning now to the read path, as data from read commands come back from the NVM devices140, the NVM data routing, command routing, and data aggregation module920places the read data in the NVM read I/O queue965. In this embodiment, the read data can take one of three forms: data that is requested by a user, NVM register data (for internal use by the controller130), and write-validation data. In other embodiments, one or more of these data classes can be held in different queues. If the data was read for internal purposes, it is processed by the internal read processing module960(e.g., to check that previously-written data was correctly written before sending an acknowledgement back to the host100or sending a rewrite request to the scheduler940). If the data was requested by the user, metadata indicating the command ID associated with the read data is attached to the data. This command ID metadata is associated with the read data as it is transmitted through the read pipeline (as indicated by the double arrow). The data is then sent to the media ECC decoder975, which decodes the data, and then to the decryption module980, which decrypts the data before sending it to the read data cache955. The data stays in the read data cache955until the host100requests it by identifying the command ID block. At that time, the data is sent to the interface905and physical layer900for transmission to the host100.

FIG. 10is a flow chart1000of a method for reading data using the controller130ofFIG. 6. As shown inFIG. 10, first the host100sends a read request to the storage system (act1050). The NVM controller130in this embodiment then extracts the following elements from the request: address, read request ID, and length of the request (act1010). The NVM controller130then converts the logical address from the request to a physical address for wear leveling (act1015).

The NVM controller130then determines if the physical address corresponds to a portion of the memory array that is busy or unavailable for reads (act1020). If the memory portion is busy or unavailable, the NVM controller130schedules the read of the non-volatile memory devices140for a later time (act1022) At that later time, if the physical address becomes available (act1024), the NVM controller130determines if there are other higher priority operations pending that prevent the read (act1026). If there are, the NVM controller130waits (act1028).

If/when the memory portion becomes available, the NVM controller130sends read commands to the NVM devices140to read the requested data (act1030). The NVM devices140then returns the requested data (act1035). Depending on the type of devices used, the NVM devices140can return the data after a fixed, pre-determined time period. The NVM controller130then can process the returned data. For example, after aggregating the data returned from the various NVM devices140(act1040), the NVM controller130can determine if the data passes an error correction code (ECC) check (act1045). If the data does not pass the ECC check, the NVM controller130can initiate an error recovery process (act1046). After the error recovery process is completed (act1048) or if the aggregated data passed the ECC check, the NVM controller130determines if the data is encrypted (act1050). If the data is encrypted, the NVM controller130initiates a decryption process (act1052).

After the decryption process is completed (act1054) or if the data was not encrypted, the NVM controller130optionally determines whether the host100previously agreed to use non-deterministic reads (act1055). (Act1055allows the NVM controller130to be used for both deterministic and non-deterministic reads but may not be used on certain embodiments.) If the host100previously agreed, the NVM controller130holds (or puts aside) the read data for a future send command (as discussed below) (act1060). The NVM controller130also sends a signal on the “READ READY” line to the host100(act1065). When it is ready, the memory controller120in the host100sends a send command (act1070). In response to receiving the send command from the host100, the NVM controller130transmits the processed, read data, along with the command ID, to the host100(e.g., after a pre-defined delay (there can be global timeouts from the memory controller in the host)) (act1075).

If the host100did not previously agree to use non-deterministic reads (act1055), the NVM controller130will handle the read, as in the conventional system discussed above. That is, the NVM controller130will determine if the elapsed time exceeds the pre-agreed transmission time (act1080). If the elapsed time has not exceeded the pre-agreed transmission time, the NVM controller130transmits the data to the host100(act1075). However, if the elapsed time has exceeded the pre-agreed transmission time, the read has failed (act1085).

Turning now to a write operation,FIG. 11is a flow chart1100that starts when the host100has data to write (act1105). Next, the host1110checks to see if there is an available flow control credit for the write operation (acts1110and1115). If there is a flow control credit available, the host100issues the write request (act1130), and the media controller130receives the write request from the host10(act1125). The controller130then extracts the destination address and user data from the request (act1130). Since a non-deterministic protocol is used in this embodiment, the controller130can now spend time performing memory and data management operations. For example, if the data requires encryption (act1135), the controller130encrypts the data (act1140). Otherwise, the controller130encodes the data for error correction (act1145). As noted above, any suitable error correction scheme can be used, such as, but not limited to, ECC, cyclic redundancy check (CRC), redundant array of independent disks (RAID), scrambling, or data weighting/modulation. Next, the controller130uses wear-leveling hardware (or software) to convert the logical address to a physical (NVM) address (act1150). The controller130then determines if the write cache is full (act1155). If it is, the controller130signals a failure (act1160). A failure can be signaled in any suitable way, including, but not limited to, using a series of voltages on a dedicated pin or pins on the response bus, writing the error in log (e.g., in the NVM controller), or incrementing or annotating the error in the serial presence detect (SPD) data. If it isn't, the controller130associates a write cache entry with the current request (act1165) and writes the data to the write cache (act1170).

The controller130then determines if the physical media is busy at the required physical address (act1175). If it is, the controller130schedules the write operation for future processing (act1180). If it isn't, the controller130waits for the current operation to complete (act1182) and then determines if there is a higher-priority request still pending (act1184). If there isn't, the controller130distributes the data to the NVM devices140via write commands (act1186). The controller130then waits, as there are typical delays in writing to NVM devices (act1188). Next, optionally, the controller140ensures that the write commit was successful (act1190) by determining if the write was successful (act1192). If the write was not successful, the controller130determines if further attempts are warranted (act1193). If they are not, the controller130optionally can apply error correction techniques (act1194). If and when the write is successful, the controller130releases the write cache entry (act1195) and notifies the host100of additional write buffer space (act1196), and the write operation than concludes (act1197).

The flow charts inFIGS. 10 and 11both describe the process for performing a single read operation or a single write operation. However, in many media controller embodiments, multiple read or write operations may proceed in parallel, thus creating a continuous pipeline of read or write processes. Many of these steps in turn will support out-of-order processing. The flow charts serve as an example of the steps that may be required to process a single read or write request.

In summary, some of the above embodiments provide a media controller that interfaces to a host via a particular embodiment of the SNVRAM protocol and also interfaces to a plurality of memory devices. In addition to using non-deterministic read- and write-timing features of the SNVRAM protocol, the media controller is specifically designed to enhance the life of the media (NVM), optimally correct errors in the media, and schedule requests through the media to optimize throughput, all while presenting a low-latency, high-bandwidth memory interface to the host. In this way, the media controller can manage the health and integrity of the storage medium by “massaging” memory idiosyncrasies. Also, the media controller can collect and aggregate data from NVM chips for more efficient data processing and error-handling.

There are many alternatives that can be used with these embodiments. For example, while a clock-data parallel interface was in the examples above, other types of interfaces can be used in different embodiments, such as, but not limited to, SATA (serial advanced technology attachment), PCIe (peripheral component interface express), NVMe (non-volatile memory express), RapidIO, ISA (Industry Standard Architecture), Lightning, Infiniband, or FCoE (fiber channel over Ethernet). Accordingly, while a parallel, DDR interface was used in the above example, other interfaces, including serial interfaces, can be used in alternate embodiments. However, current serial interfaces may encounter long latencies and I/O delays (whereas a DDR interface provides fast access times). Also, as noted above, while the storage system took the form of an NV-DIMM in the above examples, other types of storage systems can be used, including, but not limited to embedded and removable devices, such as a solid-state drive (SSD) or memory card (e.g., secure digital (SD), micro secure digital (micro-SD) card, or universal serial bus (USB) drives.

As another alternative, NVM chips can be built that can speak either standard DDR or newer SNVRAM protocols without the use of a media controller. However, use of a media controller is presently preferred as currently-existing NVM devices have much larger features than more-developed DRAM devices; thus, NVM chips cannot be depended on to speak at current DDR frequencies. The memory controller can slow down DDR signals to communicate with the NVM chips. Also, the functions that the media controller performs can be relatively complex and expensive to integrate into the memory chips themselves. Further, media controller technology is likely to evolve, and it may be desired to allow for upgrading the media controller separately to better handle a particular type of memory chip. That is, sufficiently isolating the NVM and NVM controller enables incubation of new memories while also providing a DRAM speed flow through for mature NVMs. Additionally, the media controller allows error checking codes and wear levelling schemes that distribute data across all chips and handle defects, and there is a benefit from aggregating data together through one device.

As discussed above, in some embodiments, the controller130can take advantage of the non-deterministic aspect in read and write operations to perform time-consuming actions that have an undetermined duration from the host's perspective. While memory and data management operations were mentioned above as examples of such actions, it should be understood that there are many other examples of such actions, such as monitoring the health of the individual non-volatile media cells, protecting them from wear, identifying failures in the circuitry used to access the cells, ensuring that user data is transferred to, or removed from the cells in a timely matter that is consistent with the operational requirements of the NVM device, and ensuring that user data is reliably stored and not lost or corrupted due to bad cells or media circuit failures. Furthermore, in cases where sensitive data may be stored on such device, operations that have an undetermined duration from the host's perspective can include encryption as a management service to prevent the theft of non-volatile data by malicious entities.

More generally, an operation that has an undetermined duration from the host's perspective can include, but is not limited to, one or more of the following: (1) NVM activity, (2) protection of data stored in the NVM, and (3) data movement efficiencies in the controller.

Examples of NVM activity include, but are not limited to, user data handling, non-user media activity, and scheduling decisions. Examples of user data handling include, but are not limited to, improving or mitigating endurance of NVM (e.g., wear leveling data movement where wear leveling is dispersing localized user activity over a larger physical space to extend the device's endurance, and writing or reading the NVM in a manner to impact the endurance characteristics of that location), improving or mitigating retention of the NVM (e.g., program refreshes, data movement, and retention verifications), varied media latency handling to better manage the wear impact on the media during media activity (writes, reads, erases, verifications, or other interactions) (e.g., using longer or shorter latency methods as needed for NVM handling to improve a desired property (endurance, retention, future read latency, BER, etc.)), and folding of data from temporary storage (SLC or STT-MRAM) to more permanent storage (TLC or ReRam). Examples of non-user media activity include, but are not limited to, device logs (e.g., errors, debug information, host usage information, warranty support information, settings, activity trace information, and device history information), controller status and state tracking (e.g., algorithm and state tracking updates for improved or continuous behavior on power loss or power on handling, and intermediate verification status conditions for media write confirmations, defect identifications, and data protection updates to ECC (updating parity or layered ECC values), media characterization activities (e.g., characterizations of NVM age or BER, and examination of NVM for defects), and remapping of defect areas.

Examples of protection of data stored in the NVM include, but are not limited to, various ECC engine implementations (e.g., BCH or Hamming (hardware implementation choices of size, parallelization of implementation, syndromes, and encoding Implementation choices such as which generator polynomial, level of protection, or special case arrangements), LDPC (e.g., hardware implementation choices of size, parallelization of implementation, array size, and clock rate; and encoding implementation choices such as level of protection and polynomial selection to benefit media BER characteristics), parity (e.g., user data CRC placed before the ECC, and RAID), layered protection of any of the above in any order (e.g., CRC on the user data, ECC over the user data and CRC, two ECC blocks together get another ECC, calculate the RAID over several ECC'ed blocks for a full stripe of RAID), decode retry paths (e.g., choices on initiating and utilizing the other layers of protection (e.g., speculatively soft reading, wait until failure before reading the entire RAID stripe, low power vs high power ECC engine modes)), ECC Retries with or without any of the following: speculative bit flips, soft bit decodes, soft reads, new reads (e.g., re-reads and soft reads (re-reading the same data with different settings), and decode failure), and data shaping for improved storage behavior (e.g., reduced intercell interference (e.g., using a scrambler or weighted scrambler for improved sense circuitry performance).

Examples of data movement efficiencies in the controller include, but are not limited to, scheduling architecture and scheduling decisions. Scheduling architecture can relate to the availability of single vs multiple paths for each of the following: prioritization, speculative early starts, parallelization, component acceleration, resource arbitration, and implementation choices specific to that component. The quantity, throughput, latencies, and connections of every device resource will implicitly impact the scheduling. Scheduling architecture can also include internal bus conflicts during transfers (e.g., AXI bus conflicts), ECC engines, NVM communication channels (e.g., bandwidth, speeds, latencies, idle times, congestion of traffic to other NVM, ordering or prioritization choices, and efficiencies of usage for command, data, status, and other NVM interactions), NVM access conflicts often due to the arrangement and internal circuitry access of each specific NVM (e.g., die, block, plane, IO circuitry, buffers, bays, arrays, word lines, strings, cells, combs, layers, and bit lines), memory access (e.g., external DRAM, SRAM, eDRAM, internal NVMs, and ECC on those memories), scrambler, internal data transfers, interrupt delays, polling delays, processors and firmware delays (e.g., processor code execution speed, code efficiency, and function, thread or interrupt exchanges), and cache engines (e.g., efficiency of cache searches, cache insertion costs, cache filling strategies, cache hits successfully and efficiently canceling parallel NVM and controller activity, and cache ejection strategies). Scheduling decisions can include, but are not limited to, command overlap detections and ordering, location decoding and storage schemes (e.g., cached look-up tables, hardware driven tables, and layered tables), controller exceptions (e.g., firmware hangs, component timeouts, and unexpected component states), peripheral handling (e.g., alternative NVM handling such as NOR or EEPROM, temperature, SPD (Serial Presence Detect) interactions on the NVDIMM-P, and alternative device access paths (e.g., low power modes and out of band commands), power circuitry status), and reduced power modes (e.g., off, reduced power states, idle, idle active, and higher power states that may serve for accelerations or bursts).

One of skill in the art will recognize that this invention is not limited to the two dimensional and three dimensional exemplary structures described but cover all relevant memory structures within the spirit and scope of the invention as described herein and as understood by one of skill in the art.