Patent ID: 12210414

DETAILED DESCRIPTION

For purposes of this disclosure, an information handling system may include any instrumentality or aggregate of instrumentalities operable to compute, calculate, determine, classify, process, transmit, receive, retrieve, originate, switch, store, display, communicate, manifest, detect, record, reproduce, handle, or utilize any form of information, intelligence, or data for business, scientific, control, or other purposes. For example, an information handling system may be a personal computer (e.g., desktop or laptop), tablet computer, mobile device (e.g., personal digital assistant (PDA) or smart phone), server (e.g., blade server or rack server), a network storage device, or any other suitable device and may vary in size, shape, performance, functionality, and price. The information handling system may include random access memory (RAM), one or more processing resources such as a central processing unit (CPU) or hardware or software control logic, ROM, and/or other types of nonvolatile memory. Additional components of the information handling system may include one or more disk drives, one or more network ports for communicating with external devices as well as various input and output (I/O) devices, such as a keyboard, a mouse, touchscreen and/or a video display. The information handling system may also include one or more buses operable to transmit communications between the various hardware components.

In one embodiment, IHS100,FIG.1, includes a processor102, which is connected to a bus104. Bus104serves as a connection between processor102and other components of IHS100. An input device106is coupled to processor102to provide input to processor102. Examples of input devices may include keyboards, touchscreens, pointing devices such as mouses, trackballs, and trackpads, and/or a variety of other input devices known in the art. Programs and data are stored on a mass storage device108, which is coupled to processor102. Examples of mass storage devices may include hard discs, optical disks, magneto-optical discs, solid-state storage devices, and/or a variety of other mass storage devices known in the art. IHS100further includes a display110, which is coupled to processor102by a video controller112. A system memory114is coupled to processor102to provide the processor with fast storage to facilitate execution of computer programs by processor102. Examples of system memory may include random access memory (RAM) devices such as dynamic RAM (DRAM), synchronous DRAM (SDRAM), solid state memory devices, and/or a variety of other memory devices known in the art. In an embodiment, a chassis116houses some or all of the components of IHS100. It should be understood that other buses and intermediate circuits can be deployed between the components described above and processor102to facilitate interconnection between the components and the processor102.

Referring now toFIG.2, an embodiment of a networked system200is illustrated that may include the tiered memory management system of the present disclosure. In the illustrated embodiment, the networked system200includes a plurality of computing devices202a,202b, and up to202c. In an embodiment, any or each of the computing devices202a-202cmay be provided by the IHS100discussed above with reference toFIG.1, and/or may include some or all of the components of the IHS100, and in specific examples may be provided by server devices, desktop computing devices, laptop/notebook computing devices, tablet computing devices, or mobile phones. However, while illustrated and discussed as being provided by particular computing devices, one of skill in the art in possession of the present disclosure will recognize that computing devices provided in the networked system200may include any devices that may be configured to operate similarly as the computing devices202a-202cdiscussed below.

In the illustrated embodiment, the computing devices202a-202care coupled to a network204that may be provided by a Local Area Network (LAN), the Internet, combinations thereof, and/or any other networks that would be apparent to one of skill in the art in possession of the present disclosure. In the illustrated embodiment, a network-attached memory system206is coupled to the computing devices202a-202cvia the network204. In an embodiment, the network-attached memory system206may be provided by the IHS100discussed above with reference toFIG.1, and/or may include some or all of the components of the IHS100, and in specific examples may be provided by a storage system such as a Just a Bunch Of Flash (JBOF) storage system. However, while illustrated and discussed as being provided by a particular storage system, one of skill in the art in possession of the present disclosure will recognize that network-attached memory systems provided in the networked system200may include any devices that may be configured to operate similarly as the network-attached memory system206discussed below. However, while a specific networked system200has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that the tiered memory management system of the present disclosure may be provided using a variety of components and component configurations while remaining within the scope of the present disclosure as well.

Referring now toFIG.3, an embodiment of a computing device300is illustrated that may provide any of the computing devices202a-202cdiscussed above with reference toFIG.2. As such, the computing device300may be provided by the IHS100discussed above with reference toFIG.1and/or may include some or all of the components of the IHS100, and in specific examples may be provided by server devices, desktop computing devices, laptop/notebook computing devices, tablet computing devices, or mobile phones. Furthermore, while illustrated and discussed as being provided by particular computing devices, one of skill in the art in possession of the present disclosure will recognize that the functionality of the computing device300discussed below may be provided by other devices that are configured to operate similarly as the computing device300discussed below. In the illustrated embodiment, the computing device300includes a chassis302that houses the components of the computing device300, only some of which are illustrated and discussed below. For example, the chassis302may house a processing system (not illustrated, but which may include the processor102discussed above with reference toFIG.1) and a memory subsystem (not illustrated, but which may include the memory114discussed above with reference toFIG.1) that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide a computing engine304that is configured to perform the functionality of the computing engines and/or computing devices discussed below.

The chassis302may also house a storage system (not illustrated, but which may include the storage108discussed above with reference toFIG.1) that is coupled to the computing engine304(e.g., via a coupling between the storage system and the processing system) and that includes a computing database306that is configured to store any of the information utilized by the computing engine304discussed below. The chassis302may also house a memory system308that is coupled to the computing engine304(e.g., via a coupling between the memory system308and the processing system). As will be appreciated by one of skill in the art in possession of the present disclosure, the memory system308may include the memory subsystem discussed above that are used by the processing system to provide the computing engine304, as well as any of a variety of other memory subsystems that would be apparent one of skill in the art in possession of the present disclosure. For example, the memory system308may include On-Package Memory (OPM) subsystems, direct Dynamic Random Access Memory (DRAM) subsystems, local Compute eXpress Link (CXL) DRAM subsystems, local CXL persistent DRAM subsystems, local Non-Volatile Memory express (NVMe) memory subsystems discussed in the specific embodiments described below, as well as other “local” memory subsystem that would be apparent to one of skill in the art in possession of the present disclosure.

The chassis302may also house a communication system310that is coupled to the computing engine304(e.g., via a coupling between the communication system310and the processing system) and that may be provided by a Network Interface Controller (NIC), wireless communication systems (e.g., BLUETOOTH®, Near Field Communication (NFC) components, WiFi components, etc., and/or any other communication components that would be apparent to one of skill in the art in possession of the present disclosure. In the illustrated embodiment, the chassis302also houses a tiered memory management system312, discussed in further detail below with reference toFIG.4, that is coupled to the memory system308, the computing engine304(e.g., via a coupling between the tiered memory management system312and the processing system), and the communication system310. In an embodiment, the tiered memory management system312may be provided by the IHS100discussed above with reference toFIG.1, and/or may include some or all of the components of the IHS100.

In a specific example, the tiered memory management system312may be provided using software in a memory management system for the memory system308, a hypervisor provided by the computing engine304, an operating system provided by the computing engine304, and/or other software enabled systems in the computing device300. As such, the tiered memory management system312may be provided using hardware such as a co-processor, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), and/or other hardware subsystem that has access (e.g., control plane access) to Direct Memory Access (DMA) engines, memory devices, and/or other hardware/software that one of skill in the art in possession of the present disclosure would recognize as enabling the functionality described below.

However, while the tiered memory management system312is illustrated and described as being included in the computing device300, one of skill in the art in possession of the present disclosure will appreciate how the tiered memory management system of the present disclosure may be coupled to any of the computing devices202a-202c/300via the network204(e.g., as a stand-alone device, as part of the network-attached memory system206, included in a different computing device202a-202c/300, etc.) while remaining within the scope of the present disclosure as well. As such, while a specific computing device300has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that computing devices (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the computing device300) may include a variety of components and/or component configurations for providing conventional computing device functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now toFIG.4, an embodiment of a tiered memory management system400is illustrated that may provide the tiered memory management system312discussed above with reference toFIG.3. As such, the tiered memory management system400may be provided by the IHS100discussed above with reference toFIG.1and/or may include some or all of the components of the IHS100. In the illustrated embodiment, the tiered memory management system400includes a chassis402that houses the components of the tiered memory management system400, only some of which are illustrated and discussed below. In embodiments in which the tiered memory management system400is included in the computing device300ofFIG.3, the chassis402may be provided by a circuit board that supports the components of the tiered memory management system400(e.g., a motherboard in the computing device300, a circuit board that is configured to couple to a motherboard in the computing device300, etc., while in embodiments in which the tiered memory management system400is provided by a stand-alone device or otherwise outside of the computing device300ofFIG.3, the chassis402may house the components of the tiered memory management system400.

As such, the chassis402may support/house a processing system (not illustrated, but which may include the processor102discussed above with reference toFIG.1) and a memory system (not illustrated, but which may include the memory114discussed above with reference toFIG.1) that is coupled to the processing system and that includes instructions that, when executed by the processing system, cause the processing system to provide a tiered memory management engine404that is configured to perform the functionality of the tiered memory management engines and/or tiered memory management systems discussed below. Similarly as discussed above, the tiered memory management engine404may be provided using software in a memory management system for a memory system, a hypervisor, an operating system, and/or other software enabled systems that would be apparent to one of skill in the art in possession of the present disclosure. As such, the tiered memory management engine404may be provided using hardware such as a co-processor, an ASIC, an FPGA, a CPU, and/or other hardware subsystem that has access (e.g., control plane access) to DMA engines, memory devices, and/or other hardware/software that one of skill in the art in possession of the present disclosure would recognize as enabling the functionality described below

The chassis402may also support/house a storage system (not illustrated, but which may include the storage108discussed above with reference toFIG.1) that is coupled to the tiered memory management engine404(e.g., via a coupling between the storage system and the processing system) and that includes one or more tiered memory management databases406that are configured to store any of the information utilized by the tiered memory management engine404discussed below. The chassis402may also support/house a communication system408that is coupled to the tiered memory management engine404(e.g., via a coupling between the communication system308and the processing system) and that may be provided by a Network Interface Controller (NIC), wireless communication systems (e.g., BLUETOOTH®, Near Field Communication (NFC) components, WiFi components, etc.), and/or any other communication components that would be apparent to one of skill in the art in possession of the present disclosure.

The chassis402may also support/house one or more data mover devices408that are coupled to the tiered memory management engine404(e.g., via a coupling between the data mover device(s)408and the processing system) and to the communication system410. However, while the data mover device(s)408are described as being supported/housed by the chassis402of the tiered memory management system400, one of skill in the art in possession of the present disclosure will appreciate how the data mover device(s)408may be included in the computing device300and coupled to the tiered memory management system400, coupled to the tiered memory system400via the network204, and/or other accessible to the tiered memory management engine404in a variety of other manners that will fall within the scope of the present disclosure. As such, while a specific tiered memory management system400has been illustrated and described, one of skill in the art in possession of the present disclosure will recognize that tiered memory management systems (or other devices operating according to the teachings of the present disclosure in a manner similar to that described below for the tiered memory management system400) may include a variety of components and/or component configurations for providing conventional functionality, as well as the functionality discussed below, while remaining within the scope of the present disclosure as well.

Referring now toFIG.5, an embodiment of a method500for managing tiered memory is illustrated. As discussed below, the systems and methods of the present disclosure provide for the management of memory subsystems in a tiered memory system in a manner that allows the preemptive removal of failing memory subsystems without interruption of the computing subsystems using them, as well as the alleviation of uneven wear patterns memory subsystems that are part of a memory subsystems tier. For example, the tiered memory management system of the present disclosure may be coupled to a first memory subsystem associated with a first memory subsystem tier, and a second memory subsystem associated with a second memory subsystem tier that is different than the first memory subsystem tier. The tiered memory management system monitors a health of the first memory subsystem associated with the first memory subsystem tier and the second memory subsystem associated with the second memory subsystem tier. When the tiered memory management system identifies a health issue with the first memory subsystem associated with the first memory subsystem tier, it moves data stored in the first memory subsystem associated with the first memory subsystem tier to the second memory subsystem associated with the second memory subsystem tier. As such, failure issues present in conventional tiered memory systems are eliminated.

The method500begins at block502where a tiered memory management system defines a tiered memory system associating memory subsystem types with different memory subsystem tiers. In an embodiment, at block502, the tiered memory management engine404in the tiered memory management system312/400may define a tiered memory system for the computing device300that associates memory subsystem types with different memory subsystem tiers. With reference toFIG.6, a tiered memory system600is illustrated that includes a plurality of memory subsystem tiers for respective memory subsystem types of memory subsystems that may be accessible to the computing device202a/300(e.g., accessible to the computing engine304in the computing device202a/300for the storage and utilization of data, as well as other memory subsystems functionality that would be apparent to one of skill in the art in possession of the present disclosure). As will be appreciated by one of skill in the art in possession of the present disclosure, while the tiered memory system600is described as being defined for a single computing device202a/300, similar tiered memory systems may be defined for the other computing devices202band up to202cin the networked system200while remaining within the scope of the present disclosure.

As illustrated, the tiered memory system600includes a first memory subsystem tier602that is associated with on-package memory subsystems that may be included in the memory system308of the computing device202a/300and that may be provided by, for example, on-package High Bandwidth Memory (HBM) devices defined by Joint Electron Device Engineering Council (JEDEC) standards and including Dynamic Random Access Memory (DRAM) memory technology using Through-Silicon Vias (TSVs) to interconnect stacked DRAM die, as well as other on-package memory subsystems that would be apparent to one of skill in the art in possession of the present disclosure. The tiered memory system600also includes a second memory subsystem tier604that is associated with direct DRAM subsystems that may be included in the memory system308of the computing device202a/300and that may be provided by, for example, Double Data Rate (DDR) memory devices and/or other direct DRAM subsystems that would be apparent to one of skill in the art in possession of the present disclosure. The tiered memory system600also includes a third memory subsystem tier606that is associated with local Compute express Link (CXL) DRAM subsystems that may be included in the memory system308of the computing device202a/300.

The tiered memory system600also includes a fourth memory subsystem tier608that is associated with local CXL persistent DRAM subsystems that may be included in the memory system308of the computing device202a/300. The tiered memory system600also includes a fifth memory subsystem tier610that is associated with local CXL Storage Class Memory (SCM) subsystems that may be included in the memory system308of the computing device202a/300and that may be provided by, for example, OPTANE® memory devices available from INTEL® corporation of Redmond, Washington, United States, as well as other local CXL SCM subsystems that would be apparent to one of skill in the art in possession of the present disclosure. The tiered memory system600also includes a sixth memory subsystem tier612that is associated with external CSL memory subsystems that may be included in the computing devices202b-202cand accessible to the computing device202a/300, and that may be provided by, for example, OPTANE® memory devices available from INTEL® corporation of Redmond, Washington, United States; DRAM devices; SCM devices; and/or other external CXL memory subsystems that would be apparent to one of skill in the art in possession of the present disclosure.

The tiered memory system600also includes a seventh memory subsystem tier614that is associated with local Non-Volatile Memory express (NVMe) memory subsystems that may be included in the computing device202a, and that may be provided by, for example, OPTANE® memory devices available from INTEL® corporation of Redmond, Washington, United States; NAND memory devices in NVMe Solid State Drive (SSD) storage device; and/or other local NVMe memory subsystems that would be apparent to one of skill in the art in possession of the present disclosure. The tiered memory system600also includes an eighth memory subsystem tier616that is associated with (NVMe) over Fabric (NVMe-oF) memory subsystems that may be included in the network-attached memory system206and accessible to the computing device202avia the network204, and that may be provided by, for example, OPTANE® memory devices available from INTEL® corporation of Redmond, Washington, United States; NAND memory devices in NVMe Solid State Drive (SSD) storage devices; and/or other NVMe-OF memory subsystems that would be apparent to one of skill in the art in possession of the present disclosure.

As discussed above, the different memory subsystem tiers in a tiered memory system may be ranked relative to each other based on cost factors, availability factors, performance factors, capability factors, and/or other tiered memory subsystem factors that would be apparent to one of skill in the art in possession of the present disclosure. For example, performance factors used to rank different memory subsystem tiers in a tiered memory system may include memory subsystem latency, memory subsystem bandwidth, memory subsystem power consumption, memory subsystem write endurance, and/or other performance factors that would be apparent to one of skill in the art in possession of the present disclosure. In another example, capability factors used to rank different memory subsystem tiers in a tiered memory system may include data recovery capabilities of memory subsystems, Reliability/Availability/Serviceability (RAS) capabilities of memory subsystems, data persistence capabilities of memory subsystems, metadata support capabilities of memory subsystems, and/or other capabilities factors that would be apparent to one of skill in the art in possession of the present disclosure.

As such, one of skill in the art in possession of the present disclosure will appreciate how tiered memory system600illustrated inFIG.6ranks its memory subsystem tiers602,604,606,608,610,612,614, and616with the first memory subsystem tier602as the “highest” tier, the second memory subsystem tier604as the second “highest” tier, the third memory subsystem tier606as the third “highest” tier, the fourth memory subsystem tier608as the fourth “highest” tier, the fifth memory subsystem tier610as the fifth “highest” tier, the sixth memory subsystem tier612as the sixth “highest” tier, the seventh memory subsystem tier614as the seventh “highest” tier, and the eighth memory subsystem tier616as the “lowest” tier. However, while a specific tiered memory system with specific memory subsystem types ranked in specific memory subsystem tiers is illustrated and described herein, one of skill in the art in possession of the present disclosure will appreciate how tiered memory systems, memory subsystems tiers, and memory subsystem types may differ depending on the tiered memory system configuration, priorities, and/or other factors while remaining within the scope of the present disclosure as well.

The method500then proceeds to block504where the tiered memory management system configures one or more memory subsystems in the tiered memory system to store data for one or more computing subsystems. In an embodiment, at block504, the tiered memory management engine404in the tiered memory management system312/400may configure memory subsystem(s) in the memory system308of the computing devices202a/300, in the computing devices202b-202c, and/or in the network-attached memory system206to store data for one or more computing subsystems provided by the computing engine304. With reference toFIG.7A, in the examples illustrated and described below, the computing engine304has access to DDR memory subsystems DDR0and DDR1that, as discussed above, are provided by the memory system308and included in the second memory subsystem tier604of the tiered memory system600that is associated with direct DRAM subsystems; Local CXL SCM memory subsystems LCS0, LCS1, LCS2, and LCS3that, as discussed above, are provided by the memory system308and included in the fifth memory subsystem tier610of the tiered memory system600; and External CXL memory subsystems ECM0, ECM1, ECM2, and ECM3that, as discussed above, are provided in the computing devices202b-202cand included in the sixth memory subsystem tier612of the tiered memory system600.

To provide a specific example, the computing subsystems provided by the computing engine202a/300may include an operating system and/or one or more applications, although other computing subsystems that utilize memory subsystems will fall within the scope of the present disclosure as well. In some embodiments, at block504, the tiered memory management engine404in the tiered memory management system312/400may define a plurality of logical memory address spaces such as, for example, “bins” that represent the granularity at which the tiered memory management engine404and/or the data mover device(s)408operate, and/or any other logical memory addresses spaces that would be apparent to one of skill in the art in possession of the present disclosure. The tiered memory management engine404may then assign subsets of the logical memory address spaces to one or more computing subsystems provided by the computing engine304(e.g., assign a first subset of the logical memory address spaces to an operating system, assign a second subset of the logical memory address spaces to a first application, assign a third subset of the logical memory address spaces to a second application, and so on).

The tiered memory management engine404in the tiered memory management system312/400may then associate each logical memory address space with one or more of the plurality of memory subsystems, and one of skill in the art in possession of the present disclosure will appreciate how the association of any logical memory address space with memory subsystem(s) may be based, at least in part, on the computing subsystem that was assigned that logical memory address space, the data provided for storage by the computing subsystem that was assigned that logical memory address space, the memory subsystem tier in which that memory subsystem is included, as well as any other memory subsystem/logical address space association factors that would be apparent to one of skill in the art in possession of the present disclosure. For example, any logical memory address space may be associated with a memory subsystem in a memory subsystem tier that includes characteristics desired for the data that will be stored in that logical memory address space by the computing subsystem to which it is assigned.

With reference toFIG.7B, a logical memory address space/memory subsystem association table700is illustrated that may be stored in the tiered memory management database(s)406and that includes a logical memory address space column702and a physical memory subsystem column704, and one of skill in the art in possession of the present disclosure will appreciate how the logical memory address space/memory subsystem association table700may be used by the tiered memory management engine404to associate logical memory address spaces with memory subsystems as discussed above.

For example, as illustrated inFIG.7B, at block504the tiered memory management engine404in the tiered memory management system312/400may have associated each of: a logical memory address space 0 with the DDR0memory subsystem that is included in the second memory subsystem tier604, a logical memory address space 1 with the LCS2memory subsystem that is included in the fifth memory subsystem tier610, a logical memory address space 2 with the DDR1memory subsystem that is included in the second memory subsystem tier604, a logical memory address space 3 with the LCS0memory subsystem that is included in the fifth memory subsystem tier610, a logical memory address space 4 with the ECM2memory subsystem that is included in the sixth memory subsystem tier612, a logical memory address space 5 with the DDR1memory subsystem that is included in the second memory subsystem tier604, a logical memory address space 6 with the DDR1memory subsystem that is included in the second memory subsystem tier604, a logical memory address space 7 with the LCS0memory subsystem that is included in the fifth memory subsystem tier610, a logical memory address space 8 with the ECM3memory subsystem that is included in the sixth memory subsystem tier612, a logical memory address space 9 with the LCS2memory subsystem that is included in the fifth memory subsystem tier610, and a logical memory address space 10 with the LCS3memory subsystem that is included in the fifth memory subsystem tier610.

As such, one of skill in the art in possession of the present disclosure will appreciate how the assignment of logical memory address space(s) to a computing subsystem and the association of any of those logical memory address space(s) with a memory subsystem will configure that memory subsystem to store data for that computing subsystem. However, while a specific example of the configuration of memory subsystems to store data for computing subsystems has been illustrated and described, one of skill in the art in possession of the present disclosure will appreciate how other techniques for configuring the memory subsystems to store data for computing subsystems will fall within the scope of the present disclosure as well. As will be appreciated by one of skill in the art in possession of the present disclosure, following block504, the computing subsystems in the computing device202a/300may operate to store and utilize data in the memory subsystems associated with their assigned logical memory address spaces.

The method500then proceeds to decision block506where it is determined whether a health issue exists in a first memory subsystem. In an embodiment, at decision block506, the tiered memory management engine404in the tiered memory management system312/400may monitor the plurality of memory subsystems utilized in the tiered memory system600in order to determine whether a health issue exists in any of those memory subsystems. For example, the tiered memory management engine604may be configured to receive write telemetry data, error telemetry data, and/or other health telemetry data that would be apparent to one of skill in the art in possession of the present disclosure, from each of the memory subsystems utilized in the tiered memory system600. At decision block506, the tiered memory management engine604may analyze any health telemetry data received from any of the memory subsystems in the tiered memory system600in order to determine whether a health issue exists in that memory subsystem by, for example, determining whether that health telemetry data reaches or exceeds a threshold. However, while the use of thresholds to determine a health issue is described below, one of skill in the art in possession of the present disclosure will appreciate how the health issues discussed below may be determined using other techniques that will fall within the scope of the present disclosure as well.

For example, at decision block506, the tiered memory management engine404in the tiered memory management system312/400may use the health telemetry data to determine whether a threshold number of errors have occurred in a memory subsystem, whether a threshold number of warnings have been received from a memory subsystem, whether a threshold number of write retries have been attempted to write data to a memory subsystem, to determine whether a memory subsystem includes a threshold number of “bad” pages, to determine whether a memory subsystem includes a threshold number of “bad” blocks (e.g., in an SCM memory subsystem, a flash memory subsystem, and/or other storage-device-like memory subsystem), to determine whether a memory subsystem had been written to a threshold number of times, to determine whether a memory subsystem has experienced a threshold number of correctable errors in a particular subset of the memory subsystem (e.g., a rank, page, or other subset of the memory subsystem), and/or to determine whether a variety of other thresholds have been reached that may be indicated of health issues in a memory subsystem. However, while the monitoring for several specific examples of health issues in memory subsystems have been described, one of skill in the art in possession of the present disclosure will appreciate how any of a variety of memory subsystem health issues may be monitored for at decision block506while remaining within the scope of the present disclosure.

If, at decision block506, it is determined that a health issue does not exist in the first memory subsystem, the method500returns to decision block506. As such, the method500may loop such that the tiered memory management engine404in the tiered memory management system312/400monitors the health of the memory subsystems while the computing subsystems in the computing device202a/300operate to store and utilize data in the memory subsystems associated with their assigned logical memory address spaces.

If, at decision block506, it is determined that a health issue exists in the first memory subsystem, the method500proceeds to block508where the tiered memory management system moves data stored in the first memory subsystem associated with a first memory subsystem tier to a second memory subsystem associated with a second memory subsystem tier that is different than the first memory subsystem tier. With reference toFIG.8A, in an embodiment of decision block506, the tiered memory management engine404in the tiered memory management system312/400may determine that a health issue exists in the DDR1memory subsystem that is included in the second memory subsystem tier604(e.g., as indicated by the bolded box inFIG.8A). In this example of decision block506, the tiered memory management engine404may determine that the DDR1memory subsystem has reached a memory subsystem failure threshold in response to determining that a threshold number of errors have occurred in the DDR1memory subsystem that is included in the second memory subsystem tier604.

With reference toFIG.8B, the tiered memory management engine404in the tiered memory management system312/400may use the logical memory address space/memory subsystem association table700to determine that the DDR1memory subsystem in the second memory subsystem tier604that was determined to have reached the memory subsystem failure threshold at decision block506(e.g., as indicated by the bolded boxes inFIG.8B) is associated with the logical memory address spaces 2 and 5. In an embodiment, at block508and in response to determining that the DDR memory subsystem in the second memory subsystem tier604has reached the memory subsystem failure threshold, the tiered memory management engine404in the tiered memory management system312/400may mark the data stored in the DDR1memory subsystem in the second memory subsystem tier604for memory subsystem tier movement. Furthermore, in response to the data stored in the DDR1memory subsystem being marked for memory subsystem tier movement, the data mover device(s)408may move the data stored in the DDR1memory subsystem in the second memory subsystem tier604to the LCS2memory subsystem in the fifth memory subsystem tier610and to the LCS3memory subsystem in the fifth memory subsystem tier610, as illustrated inFIG.8C.

In different embodiments, the movement of the data from the DDR1memory subsystem to the LCS2and LCS3memory subsystems in response to being marked for memory subsystem tier movement may be performed using a variety of tiered memory system data movement techniques that would be apparent to one of skill in the art in possession of the present disclosure. For example, one of skill in the art in possession of the present disclosure will appreciate how the marking of the data in the DDR1memory subsystem for memory subsystem tier movement may include “demoting” the logical memory address spaces 2 and 5 associated with the DDR1memory subsystem that has reached the memory subsystem failure threshold to a lower memory subsystem tier, and thus may result in the movement of the data stored in the logical memory address spaces 2 and 5 to memory subsystem(s) in lower memory subsystem tiers like the LCS2and LCS3memory subsystems in the fifth memory subsystem tier610in the illustrated examples. However, one of skill in the art in possession of the present disclosure will recognize how tiered memory system data movement techniques may be utilized to move the data stored in the logical memory address spaces 2 and 5 from the DDR1memory subsystem to memory subsystem(s) in lower memory subsystem tiers, the same memory subsystem tier, and/or higher memory subsystem tiers (e.g., when such memory subsystems have available space) while remaining within the scope of the present disclosure as well.

As such, with reference toFIG.8C, in an embodiment of block508and in response to the marking of the data stored in the logical memory address spaces 2 and 5 associated with the DDR1memory subsystem that has reached the memory subsystem failure threshold, the data mover device(s)408may move that data from the DDR1memory subsystem to the LCS2and LCS3memory subsystems. Thus, the marking of the data by the tiered memory management engine404may result in the disassociation of the logical memory address space 2 from the DDR1memory subsystem in the second memory subsystem tier604and the association of that logical memory address space 2 with the LCS2memory subsystem in the fifth memory subsystem tier610, and may result in the disassociation of the logical memory address space 5 from the DDR1memory subsystem in the second memory subsystem tier604and the association of that logical memory address space 5 with the LCS3memory subsystem in the fifth memory subsystem tier610.

The method500then proceeds to block510where the tiered memory management system causes the computing subsystem that was storing data in the first memory subsystem to store data in the second memory subsystem. With reference toFIG.8D, in an embodiment of block510and following the moving of the data stored in the logical memory address spaces 2 and 5 from the DDR1memory subsystem to the LCS2and LCS3memory subsystems, the tiered memory management engine404in the tiered memory management system312/400may identify the DDR1memory subsystem as unavailable (as indicated by element800inFIG.8D) to the computing subsystem(s) that were utilizing the DDR1memory subsystem. As such, following block510, those computing subsystems may store and utilize data in the LCS2and LCS3memory subsystems.

For example, when the computing device300is booted, initialized, or otherwise configured, the memory subsystems illustratedFIG.8Amay be identified as its physical address space (e.g., that includes the DDR0memory subsystem, the DDR1memory subsystem, the LCS0, LCS1memory subsystem, the LCS2memory subsystem, the LCS3memory subsystem, the ECM0memory subsystem, the ECM1memory subsystem, the ECM2memory subsystem, and the ECM3memory subsystem). A logical address space may then be mapped to the physical address space as illustrated inFIG.8B. As such, when the DDR1memory subsystem fails or otherwise becomes unavailable, the tiered memory management system312/400may identify memory subsystems that are included in a lower tier and that include memory capacity that is not currently mapped to the logical address space. In this specific example, the LCS2and LCS3memory subsystems include memory capacity that is not currently mapped to the logical address space and thus may be mapped to the logical memory address spaces 2 and 5. As will be appreciated by one of skill in the art in possession of the present disclosure, in the event all the LCS devices are full, the tiered memory management system312/400may evict some logical memory addresses from the LCS2and LCS3memory subsystems to a subset of ECM memory subsystems to make spaces for the logical memory address spaces 2 and 5 in the LCS2and LCS3memory subsystems. In the event the ECM memory subsystems are also full, logical memory address space mapped to the ECM memory subsystems may be “paged out” in order to allow memory space to be swapped on a storage device.

As will be appreciated by one of skill in the art in possession of the present disclosure, following the movement of the data to the LCS2and LCS3memory subsystems, that data may be later moved to another memory subsystem tier as per conventional tiered memory system operations. For example, in the event the access frequency of that data reaches or exceeds a threshold, that data may be moved from the LCS2and LCS3memory subsystems in the fifth memory subsystem tier610to at least one memory subsystem in a higher memory subsystem tier in the tiered memory system600, and one of skill in the art in possession of the present disclosure will appreciate how that data may be moved to at least one memory subsystem in a lower memory subsystem tier in a similar manner.

With reference toFIGS.9A,9B, and9C, another embodiment of decision blocks506,508, and510is illustrated and described in which a memory subsystem in a memory subsystem tier has experienced a health issue that includes that memory subsystem being written to a threshold amount more than the other memory subsystems in the same memory subsystem tier and, in response, the number of logical memory address spaces associated with that memory subsystem is reduced to reduce the write frequency to that memory subsystem relative to the other memory subsystems in its memory subsystem tier. With reference toFIG.9A, in this example of decision block506, the tiered memory management engine404may identify a health issue in the LCS2memory subsystem (e.g., as indicated by the bolded box inFIG.9A) based on the LCS2memory subsystem reaching a relative memory subsystem write threshold in response to a number of writes performed in the LCS2memory subsystem that is included in the fifth memory subsystem tier604exceeding a number of writes performed in the LCS0, LCS1, and/or LCS2memory subsystems in the fifth memory subsystem tier610by some threshold amount.

With reference toFIG.9B, the tiered memory management engine404in the tiered memory management system312/400may use the logical memory address space/memory subsystem association table700to determine that the LCS2memory subsystem in the fifth memory subsystem tier610that was determined to have reached the relative memory subsystem write threshold at decision block506(e.g., as indicated by the bolded boxes inFIG.9B) is associated with the logical memory address spaces 1 and 9. In an embodiment, at block508and in response to determining that the LCS2memory subsystem has reached the relative memory subsystem write threshold, the tiered memory management engine404may mark the data stored in the logical memory address space 1 associated with the LCS2memory subsystem in the fifth memory subsystem tier610for memory subsystem tier movement. Furthermore, in response to the data stored in the logical memory address space 1 associated with the LCS2memory subsystem being marked for memory subsystem tier movement, the data mover device(s)408may move that data to the ECM0memory subsystem in the sixth memory subsystem tier612, as illustrated inFIG.9C.

In different embodiments, the movement of the data stored in the logical memory address space 1 associated with the LCS2memory subsystem to the ECM0memory subsystem in response to being marked for memory subsystem tier movement may be performed using a variety of tiered memory system data movement techniques that would be apparent to one of skill in the art in possession of the present disclosure. For example, one of skill in the art in possession of the present disclosure will appreciate how the marking of the data logical memory address space 1 associated with the LCS2memory subsystem for memory subsystem tier movement may include “demoting” the logical memory address space 1 associated with the LCS2memory subsystem that has reached the relative memory subsystem write threshold to a lower memory subsystem tier, and thus may result in the movement of the data stored in the logical memory address space 1 to memory subsystem(s) in lower memory subsystem tiers like the ECM0memory subsystem in the sixth memory subsystem tier612in the illustrated examples. However, one of skill in the art in possession of the present disclosure will recognize how tiered memory system data movement techniques may be utilized to move the data stored in the logical memory address space 1 from the LCS2memory subsystem in the fifth memory subsystem tier610to memory subsystem(s) in lower memory subsystem tiers, the same memory subsystem tier, and/or higher memory subsystem tiers (e.g., when such memory subsystems have available space) while remaining within the scope of the present disclosure as well.

As such, with reference toFIG.9C, in an embodiment of block508and in response to the marking of the data stored in the logical memory address space 1 associated with the LCS2memory subsystem that has reached the relative memory subsystem write threshold, the data mover device(s)408may move that data from the LCS2memory subsystem to the ECM0memory subsystems. Thus, the marking of the data by the tiered memory management engine404may result in the disassociation of the logical memory address space 1 from the LCS2memory subsystem in the fifth memory subsystem tier610and the association of that logical memory address space 1 with the ECM0memory subsystem in the sixth memory subsystem tier612. The method500then proceeds to block510where the tiered memory management system causes the computing subsystem that was storing data in the first memory subsystem to store data in the second memory subsystem similarly as described above. Following block510, the computing subsystem(s) assigned the logical memory address space 1 may store and utilize data in the ECM0memory subsystem.

As will be appreciated by one of skill in the art in possession of the present disclosure, following the movement of the data to the ECM0memory subsystem, the LCS2memory subsystem should experience fewer writes (e.g., due the data associated with logical memory address space 1 now being located to the ECM0memory subsystem), but at the cost of reducing the amount of memory subsystem space in the fifth memory subsystem tier610. However, the data moved to the ECM0memory subsystem may be later moved to another memory subsystem tier as per conventional tiered memory system operations. For example, in the event the access frequency of that data reaches or exceeds a threshold, that data may be moved from the ECM0memory subsystem in the sixth memory subsystem tier612to at least one memory subsystem in a higher memory subsystem tier in the tiered memory system600, and one of skill in the art in possession of the present disclosure will appreciate how that data may be moved to at least one memory subsystem in a lower memory subsystem tier in a similar manner. In a specific example, following block510the disparity in writes between the LCS2memory subsystem and the other memory subsystems in the fifth memory subsystem tier610(e.g., the LCS0, LCS1, and LCS3memory subsystems) may reduce below the relative memory system write threshold, and the logical memory address space 1 may again be associated with the LCS2memory subsystem.

Thus, systems and methods have been described that provide for the management of memory subsystems in a tiered memory system in a manner that allows the preemptive removal of failing memory subsystems without interruption of the computing subsystems (e.g., the operating system and/or applications) using them, as well as the alleviation of uneven wear patterns in memory subsystems tiers by adjusting the rate of writes to memory subsystem that are reaching their write endurance limits faster than other memory subsystems in a memory subsystem tier. For example, the tiered memory management system of the present disclosure may be coupled to a first memory subsystem associated with a first memory subsystem tier, and a second memory subsystem associated with a second memory subsystem tier that is different than the first memory subsystem tier. The tiered memory management system monitors a health of the first memory subsystem associated with the first memory subsystem tier and the second memory subsystem associated with the second memory subsystem tier. When the tiered memory management system identifies a health issue with the first memory subsystem associated with the first memory subsystem tier, it moves data stored in the first memory subsystem associated with the first memory subsystem tier to the second memory subsystem associated with the second memory subsystem tier. As such, failure issues present in conventional tiered memory systems are eliminated.

Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. Accordingly, it is appropriate that the appended claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.