Reactive power management for non-volatile memory controllers

Systems, methods, and apparatus are provided that can reduce power consumption of memory controllers in response to memory command backlog in various situations. A data storage device includes a plurality of sets of non-volatile memory (NVM) devices, a central controller, and a plurality of channel controllers. Each channel controller is coupled to a distinct set of the plurality of sets of NVM devices. Each channel controller includes a command queue configured to store pending memory commands and provide backlog information. The central controller is configured to receive the backlog information of the command queues of the plurality of channel controllers, and adjust a clock frequency of the central controller and one or more clock frequencies of the plurality of channel controllers based on the backlog information such that the pending memory commands in each of the command queues are below a predetermined threshold level.

BACKGROUND

Solid state data storage systems are increasingly used for storing and managing data for electronic devices. A solid state data storage uses non-volatile memory for storing data. A typical non-volatile data storage device stores data as an electrical value (e.g., voltage) in non-volatile memory cells, and utilizes one or more memory controllers to manage data transactions across multiple non-volatile memory devices of the storage system.

Data transactions in a solid state data storage system are generally carried out by executions of memory commands. Some exemplary memory commands are reading, writing, and erasing non-volatile memory chips, for example, NAND flash memory chips. To facilitate this process, memory controllers are often constructed with command queues that facilitates command executions across multiple memory cells. In some data storage systems, multiple commands may be executed in parallel across multiple channels of the data storage system. However, the memory controllers may not be able to operate at full speed in some scenarios due to power limitation and/or processing delay in executing the memory commands.

SUMMARY

Various embodiments of systems, methods, and devices within the scope of the appended claims each have several aspects, no single one of which is solely responsible for the attributes described herein. Without limiting the scope of the appended claims, after considering this disclosure, and particularly after considering the section entitled “Detailed Description” one will understand how the aspects of various embodiments are used to manage power consumption in storage devices (e.g., solid-state drives, sometimes called SSDs).

Aspects of the present disclosure provides systems, methods, and/or apparatus to reduce power consumption of memory controllers in response to memory command backlog in various situations.

One embodiment of the present disclosure provides a data storage device. The data storage device includes a plurality of sets of non-volatile memory (NVM) devices, a central controller, and a plurality of channel controllers. Each channel controller is coupled to a distinct set of the plurality of sets of NVM devices. Each channel controller includes a command queue configured to store pending memory commands and provide backlog information. The central controller is configured to receive the backlog information of the command queues of the plurality of channel controllers, and adjust a clock frequency of the central controller and one or more clock frequencies of the plurality of channel controllers based on the backlog information such that the pending memory commands in each of the command queues are below a predetermined threshold level.

Another embodiment of the present disclosure provides a method of operating a data storage device including a central controller and a plurality of channel controllers. The central controller issues memory commands to the plurality of channel controllers. Each of the channel controllers includes a command queue for storing the corresponding memory commands for a distinct set of a plurality of sets of non-volatile memory (NVM) devices and providing backlog information. The central controller receives the backlog information of the pending memory commands in the command queues, and adjusts a clock frequency of the central controller and one or more clock frequencies of the plurality of channel controllers based on the backlog information such that the pending memory commands in each of the command queues are below a predetermined threshold level.

Another embodiment of the present disclosure provides a data storage device. The data storage device includes a plurality of sets of non-volatile memory (NVM) devices and a plurality of first means. Each first means stores memory commands for a distinct set of the plurality of sets of NVM devices and provides backlog information on the pending memory commands. The data storage device further includes a second means for issuing the memory commands to the plurality of first means, and for receiving the backlog information. The data storage device further includes a third means for adjusting a memory command processing throughput of the second means and the plurality of first means, based on the backlog information such that the pending memory commands in each first means are below a predetermined threshold level.

DETAILED DESCRIPTION

Numerous details are described herein in order to provide a thorough understanding of the example embodiments illustrated in the accompanying drawings. However, some embodiments may be practiced without many of the specific details, and the scope of the claims is only limited by those features and aspects specifically recited in the claims. Furthermore, well-known methods, components, and circuits have not been described in exhaustive detail so as not to unnecessarily obscure pertinent aspects of the embodiments described herein.

A computing device (host) can use a data storage hierarchy that puts faster data storage closer to the host and slower storage farther away from the host. The faster storage may be volatile storage and generally is referred to as “memory”, and the slower storage is typically persistent memory and often referred to as “data storage.” Some examples of persistent memory are solid state storage device like flash memory (e.g., NAND flash). Some data storage systems may have a certain power budget or limitation that cannot be exceeded during system operation. In some implementations, a data storage system may use dynamic voltage and frequency scaling (DVFS) to reduce power as needed. However, the complexity and overhead associated with DVFS may not be justified for some low power implementations and practical for high performance solid state data storage device. Moreover, DVFS solutions may not be suitable in high-load and power throttling situations.

In a distributed data storage architecture, the non-volatile data storage may be organized into different channels or groups. Each channel includes non-volatile memory devices (e.g., NAND flash chips) that are controlled by a channel controller. The data storage system can use a central controller to control memory operations among the channels through the channel controllers. The central controller can provide a host interface to a host (e.g., a computer) to access the data in the data storage system. In such distributed data storage system, the loading of the multiple downstream channels needs to be carefully considered when adjusting the clock frequency of the central controller, for example, to reduce power consumption.

Referring now to the drawings, embodiments of systems and methods are provided for managing power consumption in memory systems for storing data. Some embodiments include systems, methods, and/or devices to reduce power consumption of memory controllers in response to memory command backlog in various situations.

FIG. 1is a block diagram illustrating an implementation of a data storage system100in accordance with some embodiments. While some example features are illustrated, various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the exemplary implementations disclosed herein.

The data storage system100can provide persistent data storage to a host102or computer. The data storage system includes a central controller110and channel controllers112. The central controller110includes various components, for example, a clock generator114, a dynamic frequency scaling (DFS) control block116, a host interface118, a memory command processor120, and a power credit allocation block124. The host interface118provides the central controller110with an interface to a host102for data and control signal communication. The host102may be a computer or central processing unit that can store data in the data storage system. The clock generator114can be configured to generate one or more clock signals for driving various components of the central controller110. For example, the clock generator114can output a clock signal for driving the memory command processor120. The faster the clock signal (i.e., higher frequency), the faster the memory command processor120can process memory commands. However, faster clock speed can increase power consumption of the controller. Some examples of memory commands are used to write, read, and erase data at the non-volatile memory (NVM)130of the data storage system100.

The DFS control block116can generate a frequency scale factor signal that dynamically controls the clock frequency of the clock generator114. In one example, the clock generator114may have a base frequency F. When the scale factor is X as indicated by the frequency scale factor signal, the clock frequency may be determined as F multiple by X. Therefore, a scale factor greater than 1 increases the output clock frequency, while a scale factor less than 1 decreases the output clock frequency. The central controller110can dynamically change the clock scale factor to change the clock frequency as needed.

The memory command processor120receives memory commands from the host102via the host interface118. In some examples, the host interface118may be a part or functional block of the memory command processor120. Based on the commands or instructions received from the host102, the memory command processor120generates and sends corresponding memory commands to the channel controllers112for writing, reading, and erasing data at the non-volatile memory (NVM)130. In some embodiments, the NVM130may be NAND flash memory or the like.

Each channel controller112includes a clock generator132and a command queue134. The clock generator132generates one or more clock signals for driving various components of the channel controller112based on a frequency scale factor signal received from the DFS control block116. Therefore, the central controller110, using the DFS control block116, can dynamically adjust the clock frequencies outputted by the clock generator132of each channel controller112. In some examples, the command queue134may be a first-in-first-out (FIFO) buffer configured to receive memory commands from the central processor110(e.g., memory command processor120). Each channel controller112is configured to execute the memory commands stored in its command queue134to manage the data stored in the NVM130. The faster the clock signal generated by the clock generator132, the faster the channel controller112can perform the memory commands in its command queue134. For example, the memory commands can cause the channel controller112to write, read, or erase data at the corresponding NVM130.

The data storage system100may have a power monitor block140that is configured to monitor the power consumption of the system. For example, the power monitor block140may include sensors and circuitry (e.g., one or more current sensors and/or voltage sensors) configured to measure power consumption of different components of the data storage system100. In some aspects of the disclosure, when the power consumption of the system is above a predetermined threshold, limit, or power budget, the data storage system100may perform power management operations to reduce or limit power consumption. In some power management scenarios, the performance of some components (e.g., the NVM130) may be reduced, limited, or throttled. The data storage system100may include components and circuitry configured to perform power management functions. The power credit allocation block124allocates power credits to the channel controllers112and the associated NVM130, and each channel controller112has a wait control block146and NVM control block148for controlling power throttling based on the allocated power credits. Power throttling refers to operations, for example, frequency and/or voltage reduction, that can reduce the power consumption of the throttled circuitry or components. A power credit may correspond to a certain amount of power that may be used by the circuitry. When a channel controller122and its associated NVM130are allocated certain amount of power credits, the channel controller122and NVM130can use an amount of power corresponding to the allocated power credits, for example, to execute memory commands.

FIG. 2is a flow chart illustrating a power management process200in accordance with some embodiments. At the central controller side, the power credit allocation block124allocates power credits to each channel controller (202). At the channel controller side, the wait control block146may determine (204) if the allocated power credits are sufficient to perform the memory commands pending in its command queue134. The wait control block146can output a WAIT signal to the NVM control block148to control power throttling of the NVM130, if needed. If sufficient power credits are available to perform the memory commands in the command queue134, the WAIT signal may indicate no throttling (206). Otherwise, if the allocated power credits are not sufficient, the WAIT signal may indicate throttling (208).

During power throttling, the NVM control block148can slow down the speed of sending memory commands to the NVM130and/or the operations at the NVM130such that power consumption can be reduced. The backlog of the command queue134may increase when the NVM control block148slows down memory command execution. In that case, the command queue134has backpressure or increasing backpressure. The wait control block146can provide the power credit allocation block124with feedback on the backlog or backpressure of the command queue134. When the NVM130are throttled to reduce power consumption, the central controller110and/or channel controller112may not need to be running at full speed (i.e., at the rated clock rate or frequency). In that case, the DFS control block116may scale down (210) the clock frequencies of the central controller110and/or channel controller112. The clocks of the channel controllers112may be scaled down to different degrees depending on the backpressure at their respective command queues.

FIG. 3is a flow chart illustrating a reactive power control process300in accordance with some embodiments. The data storage system100can utilize this reactive power control process to dynamically change clock frequencies of the controllers in response to the imbalance in memory command processing throughputs between the central controller110and channel controllers112. The power saved in the controllers can be reclaimed as power credits, and thus more power credits may be made available for the channel controllers/NVM or reduce overall power consumption of the system.

At block302, the central controller110issues memory commands304to a plurality of channel controllers112. For example, each channel controller112includes a command queue134for storing the corresponding memory commands for a distinct set of a plurality of sets of NVM devices (e.g., NVM130). The command queue may be a FIFO buffer that can hold a certain number of pending memory commands.

At block306, each channel controller112provides the central controller110with backlog information308of the memory commands pending in the command queue134. Each of the command queue134may have an output that indicates its backlog or backpressure. For example, the signal may indicate the number of memory commands pending in the command queue or a usage level of the command queue. Backpressure increases when the channel controller112receives new memory commands faster than it can complete the pending memory commands.

At block310, the central controller110dynamically adjusts the clock frequency of the central controller110and/or respective clock frequencies of the channel controllers112based on the backlog information such that the memory commands pending in the respective command queues can be kept below a predetermined threshold level or in a certain range. For example, the DFS control block116can output suitable frequency scale factor signals312to the clock generator114in the central controller110and clock generators132in the channel controllers112to set the respective clock frequencies such that the data storage system100can meet a desired or minimum memory processing throughput or performance level. In general, when the clock frequency is reduced, the controller has lower memory command processing throughput. Therefore, when the central controller110has a lower throughput, it sends fewer memory commands to the command queue134. In this close-loop reactive power control process, the central controller110can reduce the upstream processing speed so that the controllers stay just slightly ahead of the NVM in terms of memory command processing throughput. Therefore, the NVM interface150(e.g., a memory interface) may be kept fully occupied as much as possible while the controllers can reduce power consumption when running at lower clock frequencies.

When the clocks of the central controller110and/or channel controller112are dynamically reduced in response to the backlog of the command queues134of the channel controllers112, the power saved at the controllers may be reclaimed as power credits that can be made available for the NVM130. When more power credits can be allocated to the NVM130, power throttling may not be needed or may be reduced. This close-loop power control process may be called reactive power management because the power is adjusted in response to the imbalance in command processing throughputs between the central controller and channel controllers.

In general, when the command queue134has backpressure, the channel controller112operates as fast as possible to keep up with the command traffic upstream from the central controller110. In that case, the central controller110may slow down its clock so that fewer memory commands are processed and sent to the command queues134at the channel controllers112. During throttling as described above, memory operations (e.g., read, write, erase) at the NVM130are gated, limited, or modulated. For example, the NVM control block148may gate or limit the issuing of memory commands to the NVM130through the NVM interface150. Therefore, during throttling, the channel controller112may operate slower and still keep up with the NVM130. That is, during throttling, backpressure may be caused by the gating of memory commands, not the channel controller112being too slow to keep up with the upstream central controller110. In some cases, therefore, the central controller110and/or channel controllers112may slow down to reduce power consumption without increasing backpressure at the command queues. The central controller110or channel controller112may maintain the clock frequency of the NVM interface150when the clock frequency of the central controller110and/or clock frequency of any of the channel controllers112is adjusted (e.g., reduced) so that memory commands and/or data can move across the memory interface150without slowing down.

FIG. 4is a flow chart illustrating a process400for controlling the clock frequency of the central controller110in accordance with some embodiments. At block402, the central controller110determines the command queue levels of all the command queues in the channel controllers112. For example, the DFS control block116may receive backlog information from the command queue134of each channel controller. The backlog information indicates the level or quantity of memory commands pending in the command queue134. At decision block404, the central controller determines whether all of the queue levels (e.g., combined queue level) are within a threshold dead zone. For example, when all of the queue levels are within the threshold dead zone (i.e., the “Yes” path), the central controller110does not change the clock frequency of the central controller110. The threshold dead zone may be a predetermined range of queue levels in which the central controller can maintain its current clock frequency and achieve a predetermined (e.g., minimum) memory command processing throughput among the channel controllers. In some examples, different channel controllers may have the same or different threshold dead zones for their respective command queues.

At decision block406, when one or more queue levels are not within the threshold dead zone, the central controller110determines whether all of the queue levels are above a predetermined threshold. A same threshold or different respective thresholds may be used for the channel controllers112. When all of the queue levels are above the predetermined threshold, it may indicate that all of the command queues have backpressure. If there is backpressure from any of the channel controllers, whether throttling or not, it means that the channel controller(s) cannot keep up and therefore the central controller110can slow down and still keep up with the channel controllers112. At block408, the central controller110determines whether its clock frequency is above a minimum frequency. If the central controller's clock frequency is above the minimum frequency, at block410, the central controller110may utilize the DFS control block116to reduce the clock frequency of the central controller. For example, the DFS control block116may output a frequency scale factor signal to the clock generator114to reduce its clock frequency.

At block412, when not all of the queue levels are above the threshold, the central controller110determines whether its clock frequency is below a maximum frequency. If the central controller's clock frequency is below the maximum frequency, at block414, the central controller110may utilize the DFS control block116to increase the clock frequency of the central controller.

FIGS. 5 and 6are flow charts illustrating control processes500and600for controlling the clock frequency of the channel controllers112in accordance with some embodiments. For example, the central controller110may perform the process500to control the clock frequency of N channel controllers112after adjusting the clock frequency of the central controller110using the process400ofFIG. 4. Referring toFIG. 5, at block416, the central controller110may control the clock frequency of a first channel controller112according to an algorithm illustrated inFIG. 6. Referring toFIG. 6, at decision block602, the central controller110determines whether the command queue level is within a threshold dead zone. For example, when the queue level is within the threshold dead zone (i.e., follow the “Yes” path), the central controller110does not change the clock frequency of this particular channel controller112. The threshold dead zone may be a predetermined range of queue levels in which the channel controller can maintain its current clock frequency while providing the desired memory command processing throughput. In some examples, different channel controllers may have the same or different threshold dead zones for their respective command queues.

At decision block604, when the queue level is not within the threshold dead zone, the central controller110determines whether the queue level is above a predetermined threshold. A same threshold or different respective thresholds may be used for different channel controllers112. When the queue level is above the predetermined threshold, it may indicate that the command queue/channel controller has a high backpressure. In that case (i.e., follow the “yes” path), at decision block606, the central controller110determines whether the channel controller's clock frequency is below a maximum frequency. If the channel controller's clock frequency is below the maximum frequency, at block608, the central controller110may utilize the DFS control block116to increase the clock frequency of the channel controller. Increasing the clock frequency can increase memory command processing throughput to reduce the backpressure of the command queue.

At decision block610, if the queue level is not above the predetermined threshold, the central controller110determines whether the channel controller's clock frequency is above a minimum frequency. If the channel controller's clock frequency is above the minimum frequency, at block612, the central controller110may utilize the DFS control block116to reduce the clock frequency of the channel controller. For example, the DFS control block116may output a frequency scale factor signal to the clock generator132to adjust (e.g., reduce or increase) the clock frequency of the channel controller112. Reducing the clock frequency can reduce power consumption of the channel controller.

The central controller110may repeat the above-described algorithm illustrated inFIG. 6to control the clock frequency of each channel controller112. Referring back toFIG. 5, at block418, the central controller110may utilize the same algorithm to control the clock frequency of a second channel controller112. Subsequently, at block420, the central controller110may utilize the algorithm illustrated inFIG. 6to control the clock frequency of the N-th channel controller112.

As described above, the clock frequencies of the central controller110and channel controllers112are dynamically controlled or adjusted in response to backpressure of the command queues. If there is backpressure from any of the channel controllers, whether throttling or not, it means that the channel controller(s)112cannot keep up and therefore the central controller110can slow down and still keep up with the channel controller(s)112. If throttling is in use, the power saved in the central controller can be reclaimed by, for example, the channel controllers112and/or NVM130, resulting in reduced throttling and increased performance for the same power draw. If no throttling is in use, the overall system power can be reduced, resulting in lower overall system power consumption for the same performance.

In some embodiments, each channel controller112may be implemented in a number of pipeline stages including the NVM130. The clock frequencies of the pipeline stages can be adjusted individually in order to keep the pipeline stages balanced in throughput. For example, if there is a certain amount of power (e.g., power credits) available for the pipeline stages, but not enough to operate all pipeline stages concurrently at the rated frequency, the central controller110can adjust the clock frequency of each pipeline stage or component to dynamically share the available power while ensuring that each stage can maintain a minimum performance level. For example, when current NVM operations (e.g., read, write, erase) are completed, the amount of power available may increase and one or more pipeline stages can be sped up to a higher clock frequency. In another example, when “excess” performance is within the pipeline stages and NVM operations are still pending, it means the pipeline stages' throughput are higher than the NVM. In that case, one or more pipeline stages can be slowed down to balance the NVM-to-pipeline performance levels.

In some embodiments, a data storage device may include various means for performing the various functions and procedures described above in relation toFIGS. 1-6. For example, the data storage device may include a plurality of first means. Each first means may be the command queue134that can store memory commands for a set of NVM devices130and provide backlog information on the pending memory commands. The data storage device may include a second means for issuing the memory commands to the plurality of first means, and for receiving the backlog information. For example, the second means may be the memory command processor120. The data storage device may include a third means for adjusting a memory command processing throughput of the second means and the plurality of first means, based on the backlog information such that the pending memory commands in each first means are below a predetermined threshold level. For example, the third means may be the dynamical frequency scaling (DFS) control block116.

In one embodiment, the process can perform the sequence of actions in a different order. In another embodiment, the process can skip one or more of the actions. In other embodiments, one or more of the actions are performed simultaneously. In some embodiments, additional actions can be performed.

While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as examples of specific embodiments thereof. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.