Patent Description:
Modern server designs often incorporate persistent memory (PMEM), such as Data Center Persistent Memory Modules (DCPMMs) or non-volatile dual in-line memory modules (NVDIMMS), into the memory architecture. Persistent memory provides several advantages over block-based persistent media including low latency random access times and the ability to directly perform remote direct memory access (RDMA) operations into persistent memory.

Committing data directly to persistent memory devices is expensive, and servers with persistent memory typically support treating some volatile on-chip state as persistent in order to limit the number of explicit commit operations software is required to perform. If the system can guarantee that the state of a volatile buffer will be flushed to persistent memory upon all resets or power transitions that would otherwise destroy the content stored within the volatile buffer, then programs may treat any data committed to the volatile buffer as persistent. One such approach for flushing volatile buffers is referred to as asynchronous dynamic random access memory refresh (ADR) whereby volatile buffers in the memory controller are included in the persistent domain. According to this approach, the system reserves a small amount of energy necessary to keep the system powered following the loss of power for long enough to flush the volatile memory controller buffers out to persistent memory devices.

Another technology, referred to as enhanced ADR (eADR) or persistent cache flushing (PCF), expands the volatile state which can be handled as persistent to include all processor caches and on-chip buffers. Typically, processor caches are orders of magnitude larger than the volatile memory buffers in the memory controller. Thus, the system requires significantly more energy to complete the flush process. Servers which support persistent cache flushing must include some form of auxiliary energy storage to power the system during the persistent cache flush operation. Some servers include a battery backup unit (BBU) to provide sufficient energy to complete flushing data out of the processor caches into persistent memory after power has been lost. BBUs may store a significant amount of energy; however, they suffer from numerous challenges including a large footprint, limited ability to supply the high currents required by server systems, thermal constraints, and additional costs.

Asynchronous hardware reset events further complicate the implementation of persistent cache flushing mechanisms. Asynchronous hardware resets are typically implemented by directly asserting a reset request pin and may not be detectable by the processor or chipset's power sequencing logic. If the system allows externally initiated reset events to trigger a hardware reset without invoking a persistent flush handler prior to the reset, then the persistent memory state may not be properly flushed. If applications rely on persistent cache flushing when not fully supported by the platform hardware, then application data may become lost or corrupted during power disruption events.

<CIT> discloses a system for flush-on-fail (FoF) operations. A FoF operation may be initiated by, for example, a system failure, low power, or a system reset. The FoF operation may enable flushing of results stored in, for example, dirty cache lines, registers, or non-persistent portions of a memory management unit to persistent memory upon failure or expected failure of a system or portions of the system. FoF operations may be used in a FoF mode that may be enabled by default, or may be selectively enabled or disabled. The system includes a FoF interrupt handler, as well as suitable flags, registers, or other storage denoted as initiate FoF, FoF enabled, FoF power threshold, and FoF successful.

<CIT> discloses an apparatus and method for handling caching of persistent data. The apparatus comprises cache storage having a plurality of entries to cache data items associated with memory addresses in a non-volatile memory. The data items may comprise persistent data items and non-persistent data items. Write back control circuitry is used to control write back of the data items from the cache storage to the non-volatile memory. In addition, cache usage determination circuitry is used to determine, in dependence on information indicative of capacity of a backup energy source, a subset of the plurality of entries to be used to store persistent data items. In response to an event causing the backup energy source to be used, the write back control circuitry is then arranged to initiate write back to the non-volatile memory of the persistent data items cached in the subset of the plurality of entries. By constraining the extent to which the cache storage is allowed to store persistent data items, taking into account the capacity of the backup energy source, the persistence of those data items can then be guaranteed in the event of the backup energy source being triggered, for example due to removal of the primary energy source for the apparatus.

Due account shall be taken of any element which is equivalent to an element specified in the claims.

Techniques are described herein for utilizing system power supply units (PSUs) to provide auxiliary energy for flushing volatile system memory to persistent memory after the loss of alternating current (AC) power. In some embodiments, the techniques include implementing an extended hold-up window long enough to complete a full flush of processor caches and memory controller buffers using energy available in PSU bulk capacitors after a power outage event. The techniques may enable flushing the volatile system caches without requiring a BBU even though the amount of energy available in a PSU is relatively small compared to most BBUs.

Many PSUs include a bulk capacitor that allows the system to handle a temporary <NUM> milliseconds (ms) loss of AC power. An example PSU implementation is to assume the worst-case output load and provide a <NUM> timer which turns off the supplies outputs when the timer expires. This implementation limits the maximum hold-up window that may be implemented by the PSU to <NUM> regardless of the system power consumption, which may not be enough time to flush all system caches.

In some embodiments, PSUs are implemented to extend the hold-up window for an indefinite window of time determined by system power consumption rather than a fixed window of time. The voltage on the bulk capacitors within one or more PSUs may be monitored, and a notification may be triggered when a programmable, power-failure warning threshold voltage is detected on the bulk capacitors. The system may configure the voltage threshold to indicate that a certain minimum amount of energy required to successfully complete a cache flush operation is available in the PSU. The PSU may further implement a second voltage threshold associated with the minimum amount of energy required to safely sequence down the system's power rails. Since both notifications are based upon the amount of energy available in the PSU's bulk capacitors, the system may implement a configurable hold-up window whose duration is determined by the system's power consumption rather than a fixed duration. As a result, the system may define an operating point to minimize energy consumption rather than being constrained by fixed duration timers.

In some embodiments, system logic may implement an energy counter that estimates the total amount of energy available across all the installed PSUs and generates an interrupt signal to invoke the persistent flush handler when the estimated total system energy has reached a threshold associated with the minimum energy required to successfully complete a cache flush operation. The system logic may implement an energy counter for each PSU installed in the system. After a PSU has generated a power failure warning signal to the system logic, the system logic may start decrementing the energy counter associated with that PSU at a rate proportional to the number of active power supplies in the system and the system's mode of operation. The system logic may estimate the total energy available by summing each of the per PSU counters. When the total estimated energy drops below a critical threshold value, the system logic may generate an interrupt signal to invoke the persistent cache flush handler.

In some embodiments, the system is configured to reduce power consumption during the flush process to minimize the amount of energy required to complete a flush. Processors, persistent memory devices, and supporting circuitry may remain powered. Other system components that are not involved in the flush process, such as fans, input/output (I/O) devices, and hard-disk drives, may have power disabled. The power control loops in the system may further contain hooks to decrease central processing unit (CPU) power consumption such as by reducing processor frequency.

To ensure volatile system resources containing state considered to be persistent are properly flushed to persistent media prior to a system reset or power transition, in some embodiments, all resets or power transitions are preceded by the execution of a persistent flush handler responsible for pushing all volatile state out to persistent media prior to the reset or power transition. The system may trap accesses to registers that are used to initiate resets or power-state transitions and initiate a persistent cache flush prior to allowing the trapped write to complete. Trapping accesses to the registers allows the system to run a cache flush handler prior to performing the requested reset or power transition action. A similar mechanism may be implemented to handle resets and power transitions requested by platform entities external to a host subsystem.

System resets and power transitions may occur not only in response to power outages but also in response to events initiated by external agents. For example, certain system errors may trigger a hardware (HW) initiated system reset. As another example, a user may initiate a warm reset or a forced power down by holding a button or flipping a switch. If the system allows externally initiated reset or power transition events to trigger a hardware reset without invoking a flush handler prior to the reset, then the data residing in volatile processor caches or memory buffers may be lost. To ensure that externally initiated system resets or power transitions properly invoke the persistent flush handler, the system may proxy these asynchronous events through system logic, which generates an interrupt to invoke a special persistent flush interrupt handler that performs a persistent cache flush prior to invoking the HW operation requested. Additionally or alternatively, the system may include a HW backup mechanism to ensure all resets and power-transitions requested in HW reliably complete within a bounded window of time independent of whether the persistent cache flush handler succeeds.

Techniques described herein further provide a handshake mechanism and protocol for notifying an operating system whether system hardware supports persistent cache flushing. The system may determine whether the hardware is capable of supporting a full flush of processor caches and volatile memory buffers in the event of a power outage or asynchronous reset. If the hardware is capable, then persistent cache flushing may be selectively enabled and advertised to the operating system. Once persistent cache flushing is enabled, the operating system may treat data committed to volatile processor caches as persistent. If disabled or not supported by system hardware, then such data may be subject to loss through a power failure or reset event and the platform may not advertise support for persistent cache flushing to the operating system.

In some embodiments, the techniques described herein are implemented on one or more computing devices, such as a server appliance or other network host, that includes persistent memory in the memory layout. While example computing architectures are provided herein, the techniques are applicable to a variety of different computing architectures, which may vary depending on the particular implementation. The techniques may be used to (a) determine whether a particular combination of system components are capable of supporting persistent cache flushing, (b) configure system components to enable persistent cache flushing if supported, and/or (c) execute a persistent cache flushing handler prior to power transitions or resets if persistent cache flushing is enabled.

<FIG> illustrates a system for performing persistent cache flush operations in accordance with some embodiments. As illustrated, <FIG> includes PSU 102a, PSU 102b, power management subsystem <NUM>, persistent cache flush handler <NUM>, memory subsystem <NUM>, CPU <NUM>, system management module <NUM>, system firmware <NUM>, peripheral components <NUM>, and operating system <NUM>. In other embodiments, system <NUM> may include more or fewer components than the components illustrated in <FIG>. In some cases, components illustrated in <FIG> may be local to or remote from each other.

PSU 102a and PSU 102b convert electrical power into a form that allows proper operation of components of system <NUM>. In some embodiments, PSU 102a and PSU 102b convert AC power into direct current (DC) energy used to power components of system <NUM>. Additionally or alternatively, PSU 102a and PSU 102b may comprise a DC-to-DC power convert, such as a converter that steps up or steps down an input voltage. PSU 102a and PSU 102b may be electrically coupled to other components of system <NUM> via one or more power rails, such as a +<NUM> Volt (V), +5V, and/or +12V rail. Although two PSUs are illustrated, system may have only a single PSU or additional PSUs, depending on the particular implementation.

Power management subsystem <NUM> controls the delivery of power to components of system <NUM> by the system PSUs. In some embodiments, power management subsystem <NUM> selectively powers down components during reset or power outage events to gracefully shutdown system <NUM>. Additionally or alternatively, power management subsystem <NUM> may monitor voltage levels across the bulk capacitors in PSU 102a and PSU 102b. If the voltage level falls below a programmable threshold, then power management subsystem <NUM> may assert or de-assert a signal to notify other components of system <NUM>.

Memory subsystem <NUM> includes volatile and non-volatile storage areas. In some embodiments, the volatile storage areas include processor caches <NUM> and memory buffers <NUM>. Processor caches <NUM> may include caches within CPU <NUM>, such as a level <NUM> (L3) and level <NUM> (L4) cache, which may be used by CPU <NUM> to reduce data access times to main memory. Memory buffers <NUM> may include registers in CPU <NUM> and/or a memory controller that provides intermediate storage for data being transferred between different areas. For example, a memory controller buffer may provide temporary storage to data that is being transferred between processor caches <NUM> and main memory.

Persistent memory <NUM> includes one or more non-volatile memory device, such as Data Center Persistent Memory Modules (DCPMMs) and non-volatile dual in-line memory module (NVDIMM). In some embodiments, persistent memory <NUM> is byte-addressable and resides on the memory bus, providing similar speeds and latency as volatile DRAM, which is typically much faster than peripheral non-volatile storage devices that do not reside on the memory bus, such as hard disks and flash drives. Further, persistent memory <NUM> may be paged and mapped by operating system <NUM> in the same manner as volatile DRAM, which is generally not the case with other forms of persistent storage. Persistent memory <NUM> may serve as main memory within system <NUM>. In other cases, main memory may include one or more volatile memory modules, such as DRAM.

When a persistent cache flush handler is installed and platform signaling mechanisms are enabled, data stored within volatile memory areas, including processor caches <NUM> and memory buffers <NUM>, may be treated as part of the persistent memory state even in the event of power outages or other power transition events. To maintain the persistent state, cache flush handler <NUM> executes and manages cache flush operations responsive to detecting triggering events. If persistent cache flush operations are not enabled, then a full cache flush may not be performed during power transition events, and some or all of the data may be subject to loss within the volatile memory areas. Without a persistent cache flush handler, data from memory buffers <NUM> may be flushed but not processor caches <NUM>, which may reduce the amount of time required to perform a flush operation.

System management module <NUM> comprises software and/or hardware for managing system-level operations. In some embodiments, system management module <NUM> includes a service processor (SP) and a CPU chipset. System management module <NUM> may interface with one or more sensors to monitor hardware components. Additionally or alternatively, system management module <NUM> may perform other functions including trapping writes to system registers, generating system management interrupts (SMIs), and monitoring system boot status.

System firmware <NUM> comprise software providing low-level control of system hardware. In some embodiments, system firmware <NUM> includes software, such as basic input/output system (BIOS) firmware, that manages a booting process when the system is powered on or reset. System firmware <NUM> may further provide runtime services for operating system <NUM>, such as managing persistent cache flush operations and peripheral components <NUM>.

Operating system <NUM> includes software that supports operations including scheduling instruction execution on CPU <NUM>, providing services to software applications, and controlling access to peripheral components <NUM>. In some embodiments, system firmware <NUM> may advertise the ability to include cache contents in the persistence domain if supported by system <NUM>. Operating system <NUM> may then selectively enable or disable persistent cache flushing. When enabled, operating system <NUM> may treat data committed to volatile memory, including processor caches <NUM> and memory buffers <NUM>, as persistent.

Peripheral components <NUM> include auxiliary hardware devices such as hard disks, input devices, display devices, and/or other output devices that may be electrically coupled with other components of system <NUM>. The power consumption of system <NUM> may vary based in part on which peripheral components <NUM> are connected and active. A worst-case scenario maximum power load may be computed by assuming that all hardware components, including peripheral components <NUM>, are operating at full capacity.

When AC power is disrupted, it may not be desirable to immediately trigger a cache flush operation since power may be quickly restored. However, there is a risk that hold-up energy within PSU 102a and PSU 102b will be insufficient to perform a full cache flush if too much time passes without the power being restored. If persistent cache flushing is enabled, then the persistent memory state may become corrupted. To maintain the persistent state, power management subsystem <NUM> may generate a warning signal when remaining energy within the bulk capacitors of PSU 102a and PSU 102b drops below a threshold level.

<FIG> illustrates an example set of operations for executing cache flush operations to maintain a persistent memory state in accordance with some embodiments. One or more operations illustrated in <FIG> may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in <FIG> should not be construed as limiting the scope of one or more embodiments.

Referring to <FIG>, process <NUM> includes estimating a total amount of ride-through time and hold-up time based on system load (operation <NUM>). The ride-through time corresponds to an estimated amount of time that system <NUM> may operate without AC power while leaving sufficient energy to perform a cached flush and sequence down the power rails. The hold-up time corresponds to the amount of time to perform a full cache flush and sequence down the power rails given the system load. The estimates may be computed based on a full system load or a reduced system load bounded to a maximum as described further herein.

In some embodiments, process <NUM> programs one or more energy thresholds based on the estimated ride-through time and hold-up time (operation <NUM>). For example, process <NUM> may estimate a voltage level in a PSU's bulk capacitors that would guarantee system <NUM> the estimated amount of hold-up time under a bounded system load to complete the cache flush and sequential power rail shutdown operations. The voltage level may then be programmed as a threshold. In other implementations, a time may be set based on the estimated ride-through time rather than a voltage/energy-based threshold.

In some embodiments, operations <NUM> and <NUM> are implemented as a separate process from the remaining operations described herein. For example, operations <NUM> and <NUM> may be performed during a boot sequence for system <NUM>, which may calculate the amount of energy required and associated voltage threshold for each operating point. The calculations may be performed based in part on which system components the boots sequence detected and the estimated power requirements to run the components during normal operations and/or a reduced power operating mode. The boot sequence may then set the programmable voltage thresholds for system <NUM>. In other embodiments, the programmable thresholds may be set or modified through user input. For instance, a system administrator may set the programmable voltage thresholds for each operating point, allowing the system administrator to inject domain knowledge about the system power requirements.

Referring again to <FIG>, process <NUM> includes monitoring for the loss of AC power (operation <NUM>). In some embodiments, system <NUM> includes sensors and/or circuitry, which may be embedded in PSU 102a and/or PSU 102b, that detects when input AC power is disrupted. In other embodiments, external circuitry and/or sensors may signal system <NUM> when AC power is lost.

Based on the monitoring circuitry, process <NUM> may detect the loss of AC power (operation <NUM>). In response, process <NUM> triggers a notification (operation <NUM>). In some embodiments, the notification is triggered by de-asserting an acok signal. De-asserting the signal provides warning that power is no longer stable and the energy reserves within the PSU's bulk capacitor have dropped to a critical point where initiating a system shutdown may be necessary to preserve the persistent state of the data, marking the beginning of the estimated hold-up time. Stated another way, the notification serves to alert power management subsystem <NUM> to reserve sufficient energy in the system PSUs to hold-up power rails for long enough to perform a full cache flush of processor caches <NUM> and memory buffers <NUM>.

In some implementations, the early warning mechanism is associated with a fixed time interval prior to shutdown, in which case power management subsystem <NUM> may assume system <NUM> is operating at a maximum load and guarantees a minimum amount of time under maximum load to complete the cache flush and to sequence down the power rails. However, this approach may lead to a conservative implementation where a system shutdown may be initiated earlier than desired, especially where the maximum PSU load is significantly higher than the actual system load during the persistent cache flush. In comparison, a programmable early warning threshold allows the system to trade-off energy consumption prior to the assertion of the warning signal for ride-through against energy consumption after the assertion of the warning signal for hold-up.

After the notification has been triggered, process <NUM> continues powering the system components in a first operating mode (operation <NUM>). While running in the first operating mode, system components may be powered using the energy in the PSU bulk capacitors. In some embodiments, power may be provided as if AC had not been disrupted. In other embodiments, power saving adjustments may be made within system <NUM>. For example, processor frequency may be throttled, display brightness may be dimmed, and/or other power conserving actions may be taken. Additionally or alternatively, data may continue to be written to and updated in processor caches <NUM> and memory buffers <NUM>.

Process <NUM> further monitors the energy level within one or more system PSUs based on the programmed thresholds (operation <NUM>). In some embodiments, system <NUM> includes sensors to monitor voltage across bulk capacitors in the PSUs. Since the capacitance values of the bulk capacitors are fixed, voltage in the capacitors may be used as a proxy for the PSU energy levels when AC power is disrupted. In other embodiments, the energy level may be computed as a function of the capacitance value of the bulk capacitor and the measured voltage.

Process <NUM> further determines whether the energy level of the one or more PSUs satisfies a threshold (operation <NUM>). For example, process <NUM> may determine that the threshold is satisfied if the measured voltage across the one or more bulk capacitors crosses below the voltage threshold programmed at operation <NUM>. If the threshold is not satisfied, then process <NUM> may continue monitoring the PSU energy levels until power is restored or the voltage in the PSU bulk capacitors reaches or falls below the programmable threshold. Once the threshold is satisfied, a warning signal may be asserted to trigger the cache flush and power down sequence.

In some embodiments, process <NUM> enters a second operating mode by reducing the system load to minimize power consumption (operation <NUM>). During this phase, power management subsystem <NUM> may power down components that are not involved in the cache flush operation. For example, power management subsystem <NUM> may power down peripheral components <NUM>, which may include hard disk drives, fans, displays, peripheral component interconnect express (PCIe) devices, and/or other peripheral hardware. Additionally or alternatively, power management subsystem <NUM> may throttle clock speed and the frequency of CPU <NUM> to minimize power consumption.

Process <NUM> further performs a cache flush (operation <NUM>). During the cache flush operation, CPU <NUM> may write data stored in processor caches <NUM> and memory buffers <NUM> to persistent memory <NUM>, to maintain the persistent state of the data. In some embodiments, process <NUM> may continue to monitor the PSU energy levels during this operation. If the PSU energy levels fall below a second voltage threshold, then process <NUM> may trigger the power down sequence even if the cache flush is not complete to prevent all the power rails from dropping off at the same time. The second voltage threshold may be programmed at a much lower level than the first threshold, leaving enough energy to sequentially bring down the power rails.

Once the cache flush is complete, process <NUM> powers down the remaining system components (operation <NUM>). Process <NUM> may sequence down the power rails to gracefully shutdown system <NUM>. The sequence in which the power rails are brought down may vary from system to system.

The process depicted in <FIG> may maintain a persistent memory state without having to install or otherwise rely on energy from BBUs. Instead, the energy within bulk capacitors of one or more PSUs may be managed by power management subsystem <NUM> to guarantee persistence. Further, power management subsystem <NUM> accounts for runtime power load, which allows for variable ride-through and hold-up times to more efficiently and effectively use the stored energy.

When there are multiple PSUs in a system and AC power is lost to one or more of the PSUs, the amount of energy in a single PSU may not be enough to complete a cache flush operation. However, the aggregate energy across multiple PSUs may be sufficient to complete a cache flush to maintain the persistent state of the data. If there are multiple PSUs, then power management subsystem <NUM> may monitor the total energy available across all power supplies. Power management system <NUM> may signal a power failure warning when the aggregate voltage level crosses a threshold to trigger a cache flush operation.

In some embodiments, power management subsystem <NUM> detects the following events with respect to each PSU being managed:.

In some embodiments, power management subsystem <NUM> maintains a set of per-PSU counters to track estimated energy levels in each PSU in the event of an AC power loss. The initial value of the per-PSU counters may be hard-coded or programmable to correspond to the amount of energy available in the PSU when vwarn asserts. When power management subsystem <NUM> detects that a PSU has asserted vwarn, it may start decrementing the associated PSU's energy counter at a rate proportional to the number of active supplies in the system and the maximum load per supply. For example, if there is a single active PSU and the maximum load is <NUM> Watts (W), then the counter may be decremented at a rate of <NUM> Joules (J) per millisecond. If there are two active supplies, the load is 600W per supply, and the energy counter may be decremented by 600mJ/ms. With four active supplies, the energy counter may be decremented at 300mJ/ms. As another example, if the worst case system load is reduced to 1000W, then the counter decrement rate may be modified to 1J/ms for a single supply, 500mJ/ms for two supplies, and 250mJ/ms for four supplies. The counters may be tuned to provide maximum ride-through time while maintaining sufficient energy to maintain the persistent cache flush for prolonged outages.

<FIG> illustrates an example set of operations for managing persistent cache flush operations in systems with multiple power supplies accordance with some embodiments. One or more operations illustrated in <FIG> may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in <FIG> should not be construed as limiting the scope of one or more embodiments.

Referring to <FIG>, process <NUM> detects the assertion of one or more vwarn signals from one or more PSUs (operation <NUM>). As previously noted, each PSU may be configured to assert the signal when AC power is lost and the energy in the PSUs bulk capacitor(s) falls below a threshold, which may be programmable.

Responsive to detecting the vwarn signal(s), process <NUM> initiates one or more associated countdown timers (operation <NUM>). In some embodiments, the countdown timers track the estimated energy levels for each PSU that has asserted the vwarn signal. Process <NUM> may decrement the counter at a rate proportional to the number of PSUs in the system and the maximum load per supply. In other embodiments, other mechanisms may be used to track the energy levels within the PSUs. For instance, process <NUM> may increment a counter rather than decrement a counter until a threshold is reached or use other tracking logic.

Additionally or alternatively, process <NUM> may cause system <NUM> to enter a reduced power mode responsive to detecting one or more vwarn signals. The reduced power operating mode may be triggered by a single signal or a threshold number of signals, depending on the particular implementation. In other embodiments, process <NUM> may gradually reduce power with each new detected signal. For example, process <NUM> may increasingly throttle the CPU frequency with each new vwarn signal and/or initiate or increase other power conserving actions as previously described.

Process <NUM> further monitors (a) the aggregate energy level of the combined PSUs based on the countdown timers (or other tracking logic), (b) the assertion of additional vwarn signals from other PSUs, and (c) the assertion of pwrok signals from the PSUs (operation <NUM>). If additional vwarn signals are detected, then process <NUM> initiates the associated countdown timers for the PSU(s) that asserted the signals (operation <NUM>).

If the aggregate energy level satisfies a first threshold, then process <NUM> performs a cache flush operation (operation <NUM>). For example, process <NUM> may determine that the aggregate energy level across all PSUs equals or falls below a minimum threshold. During the cache flush operation, CPU <NUM> may write data stored in processor caches <NUM> and memory buffers <NUM> to persistent memory <NUM>, to maintain the persistent state of the data. In some embodiments, process <NUM> may continue to monitor the PSU energy levels during this operation.

If the cache flush operation completes, or the PSUs fall below a second voltage threshold number triggering one or more pwrok signals, then process <NUM> sequences down the power rails (operation <NUM>). When a pwrok signal is detected, process <NUM> may initiate the power down sequence even if the cache flush is not complete to prevent all the power rails from dropping off at the same time. The second voltage threshold may be programmed at a much lower level than the first threshold, leaving enough energy to sequentially bring down the power rails.

<FIG> illustrates example system <NUM> for managing multiple power supplies in accordance with some embodiments. System <NUM> includes PSUs <NUM> and <NUM>. However, the number of PSUs may vary depending on the particular implementation. Each PSU may include one or more bulk capacitors, such as capacitor <NUM>, which stores electrostatic energy obtained from a connected AC power network. The capacitor-based storage allows PSUs to be implemented with a smaller footprint than BBUs and provides faster charge and discharge rates. PSUs <NUM> and <NUM> may be connected to the same AC power network or different AC power networks depending on the particular implementation. If connected to different AC power networks, then one PSU may lose AC power while another PSU continues to be supplied with power by a different AC network. In this scenario, each PSUs may provide separate acok signals (not shown) to power management subsystem <NUM>. These signals may be independently de-asserted by the individual PSUs when AC power is lost to signal which PSU lost power. In other cases, the de-assertion of an acok signal may signal that a group or all of the PSUs have lost AC power.

In some embodiments, each PSU asserts a vwarn signal when the energy in the bulk capacitor (e.g., bulk capacitor <NUM>) reaches a threshold. Thus, a vwarn signal notifies the power management subsystem <NUM> that the available energy of the associated PSU is at a first threshold level. Power management subsystem <NUM> maintains separate energy counters for each PSU, which are triggered when the associated PSU asserts the vwarn signal. For example, when PSU <NUM> asserts a vwarn signal, then power management subsystem <NUM> may decrement energy counter <NUM> at a rate proportional to the number of PSUs in the system and the maximum load per supply. Energy counter <NUM> is managed independently of energy counter <NUM> (the vwarn signals from PSUs do not trigger the count on unassociated counters for other PSUs) and is decremented responsive to PSU <NUM> asserting a vwarn signal. Power management subsystem <NUM> includes adder <NUM>, which sums together the estimated energy counts for the PSUs to compute aggregate energy counter <NUM>.

In some embodiments, power management subsystem monitors aggregate energy counter <NUM> to determine whether the aggregate energy across all PSUs has reached or fallen below a system threshold, which may be programmable and vary depending on the particular implementation. If the threshold is reached, then power management subsystem <NUM> asserts an SMI signal to halt the current task being executed by CPU/chipset <NUM> in preparation for a persistent cache flush and reset. Responsive to the SMI, persistent cache flush handler <NUM> may initiate the persistent cache flush operations described previously.

<FIG> illustrates example timing diagram <NUM> with staggered warning signals from different power supply units in accordance with some embodiments. The top half of diagram <NUM> depicts the timing of the power failures while the bottom half of diagram <NUM> depicts the potential power reductions system <NUM> may implement in response to the impending power failure. Diagram <NUM> assumes worst case behavior, where the system operates at maximum load until the power-fail cache flush is triggered. When the power fail cache flush is triggered, system power consumption is reduced to minimize load when completing the flush operation.

The variables of diagram <NUM> may be defined as follows:.

Referring to <FIG>, when the first PSU asserts v1warn, at tpsu0_v1warn, that PSU has (Ev1warn + Epwrok) energy available before the supply shuts off the associated power rail and stops providing power to the system. There are N active PSUs and system load is split amongst all the active PSUs in the system. Some point later in time the second PSU asserts v1warn at tpsu1_v1warn. The amount of energy consumed during the period of time between tpsu0_v1warn and tpsu1_v1warn is represented as Ev1warn_delay.

During the period of time after the first supply has asserted v1warn but prior to the point where the second supply has asserted v1warn, system <NUM> is drawing energy from all the N active PSUs. The maximum energy consumed from the first PSU after it asserts v1warn is ((Ev1warn + Epwrok). The energy remaining in the first PSU when the second CPU asserts v1warn is represented as Epsu0reserve.

If Tv1warn_delay is small, then the second PSU de-asserts pwrok before the system has consumed all the energy from the fist supply. In the worst case, when both supplies each assert vlwarn simultaneously, both supplies also de-assert pwrok simultaneously. Under these circumstances, it may not be possible to use any of the Epwrok energy in the first supply if the system is shutdown when all supplies de-assert pwrok. To allow for this possibility, system <NUM> may be configured with the assumption that the Epwrok energy is not available in the first supply.

To utilize all the energy in both supplies, system <NUM> may be configured such that the power down flush is not immediately started when the second PSU asserts v1warn. System <NUM> may instead delay the flush trigger until the amount of energy reserved in all the active PSUs is equal to the amount required to complete the flush. System <NUM> may further be configured to guarantee that Ereserve ≥ Eflush to reserve sufficient energy to complete the cache flush operation.

If both PSUs assert v1warn simultaneously, then Tv1warn_delay = <NUM>, and Tv1warn_debounce = Tv1warn. If the PSUs assert vlwarn far apart in time, then Tv1warn_debounce = <NUM>, and the power fail flush may be triggered as soon as the second PSU asserts v1warn. Power management subsystem <NUM> may program the energy/voltage thresholds accordingly.

Power disruption events are not the only cause of system shutdowns or resets. In some cases, system errors or user actions may trigger a system shutdown or reset. For these externally-initiated asynchronous events, monitoring for power loss may not be sufficient to maintain the persistent memory state since A/C power may be relatively constant. Asynchronous hardware resets are typically implemented by directly asserting a reset request pin, which initiates a reset in HW and may not provide any ability to invoke a software cache flush handler prior to the reset. In some embodiments, to prevent data loss, board logic is configured to generate an SMI signal to initiate a cache flush when an externally initiated reset request is detected.

<FIG> illustrates an example set of operations for processing externally initiated asynchronous reset events in accordance with some embodiments. One or more operations illustrated in <FIG> may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in <FIG> should not be construed as limiting the scope of one or more embodiments.

Referring to <FIG>, process <NUM> intercepts the assertion of a HW reset request signal (operation <NUM>). In some embodiments, platform-initiated reset requests, including those initiated by system management module <NUM>, are proxied through power management subsystem <NUM>. This allows system <NUM> to run persistent cache flush handler <NUM> prior to performing the requested reset or power-transition action.

In some embodiments, process <NUM> determines whether persistent cache flushing is enabled (operation <NUM>). As described further below, system firmware (or other system logic) may selectively enable or disable persistent cache flushing to configure whether data in processor caches <NUM> are included in the persistent memory state.

If persistent cache flushing is not enabled, then process <NUM> routes the request to the reset pin (operation <NUM>). In some embodiments, power management subsystem <NUM> routes the request to a system chipset. The chipset may initiate a HW reset sequence.

If persistent cache flushing is enabled, then process <NUM> routes the request to system management module <NUM> (operation <NUM>). In this case, the reset pin is not immediately asserted responsive to the platform or user-initiated reset to allow time to invoke the software-based cache flush handler.

In some embodiments, process <NUM> generates an SMI signal to place system <NUM> into a system management mode (operation <NUM>). The SMI signal may be asserted by system management module <NUM>, which may use a special signaling line directly tied to CPU <NUM>. The signal may cause system firmware <NUM> (e.g., BIOS) to halt the current task being executed by CPU <NUM> in preparation for the cache flush and reset.

In some embodiments, if persistent cache flushing is enabled, then system firmware <NUM> (e.g., BIOS) configures a general-purpose input/output (GPIO) pin within system management module <NUM> as a trigger for an SMI. The GPIO pin may be used to signal system firmware <NUM> when a cache flush followed by a warm reset is to be performed. This GPIO may be different from the GPIO used to signal an impending power-failure to the chipset to communicate that the persistent cache flush handler should terminate with the request warmreset rather than a power-off.

Process <NUM> next performs a cache flush operation (operation <NUM>). Responsive to the SMI signal, system firmware <NUM> may invoke cache flush handler <NUM> to manage a cache flush operation as previously described. Thus, data is transferred from volatile memory, such as processor caches <NUM> and memory buffers <NUM>, to persistent memory <NUM>, thereby maintaining the persistent state.

Process <NUM> further determines whether the flush is complete (operation <NUM>). Persistent cache flush handler <NUM> may assert a signal or otherwise provide a notification when the data in processor caches <NUM> and memory buffers <NUM> have been written to persistent memory <NUM>.

Once the cache flush completes, process <NUM> generates a reset request (operation <NUM>). For example, persistent cache flush handler <NUM> may initiate a system reset by writing a particular value to a specific IO port/register of a PCH (e.g., 0x06 to port CF9) or by requesting that system logic assert the HW reset request signal to the chipset.

If the flush is not complete, process <NUM> may determine whether a timeout has been reached (operation <NUM>). For example, process <NUM> may allow one second or another threshold period of time, which may be configurable by system <NUM>, for the flush operation to complete. In some cases, a system state associated with the reset event may prevent a flush from completing. Implementing a timeout may prevent system <NUM> from entering a state in which a warm reset cannot be executed.

If the timeout is reached, process <NUM> generates a reset request signal directly to the chipset (operation <NUM>). The reset request in operation <NUM> may also be a direct reset request to the chipset or may be a software-based request. Thus, the mechanisms for resetting the system may differ based on whether the flush successfully completes or not.

Responsive to the reset signal, system <NUM> is then reset (operation <NUM>). A reset in this context may cause system <NUM> to shutdown or restart.

<FIG> illustrates example system <NUM> for intercepting and processing externally initiated asynchronous reset events in accordance with some embodiments. System <NUM> includes system management module <NUM>, which may be implemented in programmable hardware, such as a field programmable gate array (FPGA), or through other hardware components (or combination or hardware and software) as described further below. System management module <NUM> acts a proxy, intercepting the assertion of hardware request signals, which may be triggered by a user pressing a reset button or a reset request asserted by a board management controller (BMC) or a debug header.

System management module <NUM> includes logic gate <NUM>, which routes the asserted reset request signal to demultiplexer <NUM>. The select wire coupled to demultiplexer <NUM> is set based on whether persistent cache flushing is enabled or disabled. A "<NUM>" or low voltage state represents a memory operation mode where persistent cache flushing is disabled and data in processor caches <NUM> and memory buffers <NUM> are not managed as part of the persistence domain. A "<NUM>" or high voltage state represents a persistent cache operation mode where persistent cache flushing is enabled and data in processor caches <NUM> and memory buffers <NUM> are part of the persistent domain. However, the values on the select wire may be swapped, depending on the particular implementation.

When persistent cache flushing is disabled, then system management module <NUM> asserts a request reset interrupt signal to a pin that is electrically coupled on CPU/chipset <NUM>. In response, reset control logic <NUM> on CPU/chipset <NUM> halts the current task being executed and initiates a hardware reset, which may comprise sending a signal to reset finite state machine (FSM) <NUM>. Reset FSM <NUM> may sequence down the power rails in a particular order to avoid damaging hardware components. As previously noted, the sequence in which power rails are brought down may vary depending on the system architecture.

When persistent cache flushing is enabled, system management module <NUM> asserts an SMI using a special signaling line directly tied to another pin on CPU/chipset <NUM>. The signaling line is distinct from the line used to perform the HW reset previously described when persistent cache flushing is not enabled. In response to detecting the SMI, CPU/chipset <NUM> sends a software-based request to power fail flush handler <NUM> to initiate a persistent cache flush.

Responsive to the request, persistent cache flush handler <NUM> initiates the persistent cache flush operations to transfer data from processor caches and memory buffers to persistent storage media. If the cache flush successfully completes, then power fail flush handler <NUM> sends a software reset request to reset control logic <NUM>, which may trigger the power down sequence as previously described.

When persistent cache flushing is enabled, system management module <NUM> further initializes timer <NUM>. Timer <NUM> may decrement or increment until canceled or a timeout value is reached. The count may be canceled responsive to detecting the assertion of a signal on the input pin to reset FSM <NUM>. The signal indicates that power fail flush handler <NUM> successfully flushed the processor caches and memory buffers to persistent storage media, and the reset sequence has been initiated. If the timeout value is reached before the timer is canceled, then system management module <NUM> may directly assert the rst_req_in pin on CPU/chipset <NUM> to trigger a HW reset.

System boot firmware may expose persistent cache flushing support via a user configurable option. However, the boot firmware may be deployed across a wide variety of hardware platforms and presenting the option may not imply that a specific platform hardware is capable of supporting persistent cache flushing. Whether or not the platform is capable of supporting persistent cache flushing may depend on the hardware configuration, the presence and/or health of energy storage modules and capabilities of the underlying hardware components. In some embodiments, components of system <NUM> engage in a handshake to (a) determine whether the hardware has sufficient capability to support persistent cache flushing; (b) selectively enable/disable persistent cache flushing; (c) configure system components to support persistent cache flushing when persistent cache flushing is enabled; and (d) communicate to the operating system whether persistent cache flushing has been successfully enabled.

<FIG> illustrates an example set of operations for coordinating persistent memory operating modes in accordance with some embodiments. One or more operations illustrated in <FIG> may be modified, rearranged, or omitted all together. Accordingly, the particular sequence of operations illustrated in <FIG> should not be construed as limiting the scope of one or more embodiments.

Referring to <FIG>, process <NUM> initiates a boot sequence (operation <NUM>). In some embodiments, the boot sequence loads system firmware <NUM>, which may include BIOS firmware. The firmware may be configured to expose a user-configurable option for persistent cache flushing. For example, the firmware may present a prompt a user whether the user would like to enable persistent cache flushing, or the user may navigate user interface, such as a BIOS setup utility screen.

In some embodiments, the user interface exposes multiple settings for a "Durability Domain" setup option to configure whether a platform will operate in ADR mode or persistent cache flush mode. For example, the user interface may expose an option to select a "Memory Controller" setting or a "CPU Cache Hierarchy" setting. In the "Memory Controller" setting, ADR is enabled but persistent cache flushing is disabled. When this setting is selected, memory buffers <NUM> are flushed during power outage events, but the flush operation is not applied to processor caches <NUM>. In some embodiments, system hardware may be configured in this setting by default.

In the "CPU Cache Hierarchy" setting, persistent cache flushing is enabled. Thus, if this option is selected, then data in memory buffers <NUM> and processor caches <NUM> are flushed upon power outage events if the platform-hardware supports persistent cache flush operation.

Additionally or alternatively, other settings may be supported. For example, a "Standard Domain" setting may be selected where cached data is not flushed upon power failure events. Users may select the preferred setting via a user interface, as previously described. If a user has not selected a setting, then system firmware <NUM> may select a default setting, which may vary depending on the particular implementation.

In some embodiments, system firmware <NUM> checks to determine whether persistent cache mode has been selected by the user or default (operation <NUM>). Even if the option is selected, the platform hardware may not support persistent cache flush operations in some cases. Further, system hardware may evolve over time as components are added, removed, age, and/or fail.

If persistent cache mode has not been selected, then system firmware <NUM> continues the boot sequence without advertising support for persistent cache mode (operation <NUM>). The boot sequence may include initializing hardware components, loading the operating system, and/or processing system boot files that have not already been processed. The boot sequence may continue without performing the hardware capability checks described further below.

If persistent cache mode has been selected, then system firmware <NUM> sends a request to system management module <NUM> to determine whether system <NUM> is capable of supporting persistent cache flushing operations (operation <NUM>).

Responsive to the request, system management module <NUM> evaluates the hardware capabilities of system <NUM> (operation <NUM>). In some embodiments, system management module <NUM> may engage in a handshake with one or more hardware components to determine settings, configurations, and/or other information indicative of whether persistent cache flushing is supported. For example, during the boot sequence, the connected hardware components may include firmware that provides a list of features supported by the component to the system firmware. System management module <NUM> may scan the feature list and/or other information provided to determine if the features are compatible with persistent cache flushing.

In some embodiments, evaluating the hardware capabilities of system <NUM> comprises determining whether PSU 102a and/or PSU 102b support generating the pre-warning signal and configuring a programmable vwarn threshold. For example, system management module <NUM> may determine whether PSU <NUM> includes a pin for asserting the vwarn signal. If the PSUs do not have these capabilities, then system management module <NUM> may determine that the platform-hardware does not support persistent cache flush operations.

Additionally or alternatively, system management module <NUM> may determine whether power management subsystem <NUM> includes logic for detecting the vwarn signals, monitoring aggregate energy levels across multiple PSUs, and/or triggering interrupts when a systemwide energy level is below a threshold. If power management subsystem <NUM> does not have these capabilities, then system management module <NUM> may determine that the platform-hardware does not support persistent cache flush operations.

Additionally or alternatively, system management module <NUM> may evaluate other hardware capabilities. For example, system management module <NUM> may evaluate system <NUM> to determine whether the system supports intercepting reset signals and configuring a GPIO pin to process asynchronous reset events. As another example, system management module <NUM> may evaluate CPU <NUM> to determine whether it includes a special signaling line for invoking a persistent cache flush handler.

Additionally or alternatively, system management module <NUM> may determine whether any BBUs have been installed that support persistent cache flush operations. If BBUs have been installed, then system management module <NUM> may determine that persistent cache flushing is supported even if the PSU architecture does not provide support. On the other hand, system management module <NUM> may determine that persistent cache flushing is not supported if BBUs are not installed and the PSUs and/or power management subsystem do not support persistent cache flush operations.

Additional or alternatively, system management module <NUM> may evaluate other hardware capabilities. For example, system management module <NUM> may evaluate the capacity of the auxiliary energy storage devices installed in the platform, such as BBUs, and determine whether the devices provide sufficient energy to power the system components that are active during the flush process. Additionally or alternatively, system management module <NUM> may evaluate the health of a battery, such as by measuring the battery impedance, to determine whether the platform hardware supports persistent cache flushing.

Based on the evaluation, system management module <NUM> returns a response to system firmware <NUM> indicating whether the platform is capable of supporting persistent cache flushing or not (operation <NUM>). The response may grant system firmware <NUM> permission to enable persistent cache flushing if supported. Otherwise, system management module <NUM> denies system firmware <NUM> the ability to enable persistent cache flushing.

Upon receiving the response, system firmware <NUM> determines whether system <NUM> supports persistent cache flushing (operation <NUM>).

If the platform hardware does not support persistent cache flushing, then system firmware <NUM> continues the boot sequence without advertising support for persistent cache flushing to operating system <NUM> (operation <NUM>). When persistent cache flushing is not advertised and enabled, operating system <NUM> may prevent applications from attempting to treat processor caches as persistent in system <NUM>.

If persistent cache flushing is supported, system firmware <NUM> and/or system management module <NUM> then configure system components to support persistent cache flushing operations (operation <NUM>). For example, system firmware <NUM> may establish the GPIO pins, initialize the per-PSU timers, configure the PSUs, and otherwise configure system hardware/software to perform cache flush operations as previously described.

System firmware <NUM> and/or system management module <NUM> then advertises support for persistent cache flushing to the operating system <NUM> (operation <NUM>). In some embodiments, system firmware <NUM> may provide a list of supported features and/or configuration settings to operating system <NUM>. The list may include an entry indicating that persistent cache flushing is supported and enabled. However, the manner in which support is advertised may vary depending on the particular implementation.

Based on the advertisement, operating system <NUM> detects persistent cache mode is supported (operation <NUM>). For example, operating system <NUM> may scan a list of supported features during the boot sequence to determine whether system firmware or system management module <NUM> is advertising support for persistent cache flushing.

If persistent cache mode is enabled and supported by the platform hardware, then operating system <NUM> advertises the persistent cache mode to one or more applications (operation <NUM>). In some embodiments, the applications may query operating system <NUM> to determine whether persistent cache mode is available and supported. Operating system <NUM> may provide a response to indicating whether or not the applications may rely on persistent caching. An application may implement different logic depending on whether persistent caching is enabled and supported or not. For instance, if enabled, a database application may treat reads and writes as committed without implementing complicated software-based checks, which may simplify the application code and provide more efficient execution of reads and writes.

As system components evolve, process <NUM> may be repeated to determine whether the support for persistent cache mode has changed. A change in hardware, such as the installation of a BBU or PSU upgrade, may lead system <NUM> to advertise support for persistent cache flushing when it was previously not supported. In other cases, the advertisement may be removed if components, such as a BBU, are removed or fail.

The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the techniques.

For example, <FIG> is a block diagram that illustrates computer system <NUM> upon which an embodiment of the invention may be implemented. Computer system <NUM> includes bus <NUM> or other communication mechanism for communicating information, and a hardware processor <NUM> coupled with bus <NUM> for processing information. Hardware processor <NUM> may be, for example, a general-purpose microprocessor.

Computer system <NUM> also includes main memory <NUM>, such as a random-access memory (RAM) or other dynamic storage device, coupled to bus <NUM> for storing information and instructions to be executed by processor <NUM>.

Computer system <NUM> further includes read only memory (ROM) <NUM> or other static storage device coupled to bus <NUM> for storing static information and instructions for processor <NUM>. Storage device <NUM>, such as a magnetic disk or optical disk, is provided and coupled to bus <NUM> for storing information and instructions.

Computer system <NUM> may be coupled via bus <NUM> to display <NUM>, such as a cathode ray tube (CRT) or light emitting diode (LED) monitor, for displaying information to a computer user. Input device <NUM>, which may include alphanumeric and other keys, is coupled to bus <NUM> for communicating information and command selections to processor <NUM>. Another type of user input device is cursor control <NUM>, such as a mouse, a trackball, touchscreen, or cursor direction keys for communicating direction information and command selections to processor <NUM> and for controlling cursor movement on display <NUM>. Input device <NUM> typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, content-addressable memory (CAM), and ternary content-addressable memory (TCAM).

The remote computer can load the instructions into its dynamic memory and send the instructions over a network line, such as a telephone line, a fiber optic cable, or a coaxial cable, using a modem. A modem local to computer system <NUM> can receive the data on the network line and use an infra-red transmitter to convert the data to an infra-red signal.

ISP <NUM> in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" <NUM>.

Embodiments are directed to a system with one or more devices that include a hardware processor and that are configured to perform any of the operations described herein and/or recited in any of the claims below.

In an embodiment, a non-transitory computer readable storage medium comprises instructions which, when executed by one or more hardware processors, causes performance of any of the operations described herein and/or recited in any of the claims.

Claim 1:
A method comprising:
determining that a configuration option for a computing system (<NUM>, <NUM>) has been selected to enable a first operating mode where data stored in volatile processor caches (<NUM>) are part of a persistent memory state;
determining, by the computing system (<NUM>, <NUM>), whether hardware components of the computing system (<NUM>, <NUM>) are capable of supporting persistent cache flushing, wherein firmware (<NUM>) executing on the computing system (<NUM>, <NUM>) is configured to not advertise support for the first operating mode to an operating system (<NUM>) if the hardware components are not capable of supporting persistent cache flushing even if the configuration option to enable the first operating mode has been selected; and
responsive to determining that the first operating mode is enabled and the hardware components are capable of supporting persistent cache flushing configuring one or more components of the computing system (<NUM>, <NUM>) to perform persistent cache flush operations when power disruption events or power transition requests are detected.